TUMexam iOS Usage Analytics Final talk for the Bachelors Thesis by - PowerPoint PPT Presentation

Chair of Network Architectures and Services Department of Informatics Technical University of Munich TUMexam iOS Usage Analytics Final talk for the Bachelor’s Thesis by Henri Allgöwer advised by Stephan Günther, Maurice Leclaire, Benedikt Jaeger, Georg Wechslberger, Johannes Naab Monday 16 th December, 2019 Chair of Network Architectures and Services Department of Informatics Technical University of Munich

General Topic / Motivation Background (a) Home screen (b) Correction screen Figure 1: Screenshots of the TUMexam correction application H. Allgöwer — TUMexam Usage Analytics 2

General Topic / Motivation Goal of the thesis Improving the existing iOS application 1. Analyze user behaviour • Does the user interact with the app as expected? • Is it intuitive, easy to use? 2. Determine weaknesses in the app • UI / UX • Missing features 3. Figuring out a way to optimize • Improve existing features • Add desired functionalities H. Allgöwer — TUMexam Usage Analytics 3

Measurements 1. Time • Timestamps • Amount of time invested by correctors • Comparison to correction by hand Figure 2: Stopwatch 2. Pencil gestures • Amount and length of lines drawn, erased, undone, redone • UI elements clicked • Change in corrector behaviour Figure 3: Toolbar 3. User feedback • Questionnaire in the app • Rating system similar to App Store • Multiple flags to be set for each subproblem Figure 4: Flagging in the app H. Allgöwer — TUMexam Usage Analytics 4

Privacy Agreement • Privacy of users is of utmost importance • Inform the users on what is being tracked and why Figure 5: Prototype privacy agreement screen in the app H. Allgöwer — TUMexam Usage Analytics 5

Privacy Agreement • Privacy of users is of utmost importance • Inform the users on what is being tracked and why Figure 5: Prototype privacy agreement screen in the app Necessity vs risk of jeopardizing progress H. Allgöwer — TUMexam Usage Analytics 5

Statistical Analyses Overview • In-depth analysis of the correction of the GRNVS endterm and retake exams • Python script that evaluates user data gathered of any exam and provides the user with a quick overview. This includes: – Overall correction time – Correction time split into correction group (i.e. first and second correctors), problems, individual users – Performance overview for each corrector – Raw user data csv split into subgroups for easier analysis H. Allgöwer — TUMexam Usage Analytics 6

Statistical Analyses Time stamp problematic Problem: Improper dismissal of the application resulting in inaccurate time stamps 2 , 500 Amount of Database Entries 2 , 000 1 , 500 1 , 000 500 0 ( 0 – 25 ) ( 25 – 50 ) ( 50 – 75 ) ( 75 – 100 ) ( 100 – 125 ) ( 125 – 150 ) ( 150 – 175 ) ( 175 – 200 ) ( 200 – 225 ) ( 225 – 250 ) ( 250 – 275 ) ( 275 – 300 ) ( 300 – 325 ) ( 325 – 350 ) Time Intervalls in Seconds Figure 6: Time distribution of the correction of a single problem H. Allgöwer — TUMexam Usage Analytics 7

Statistical Analyses Time stamp problematic Problem: Improper dismissal of the application resulting in inaccurate time stamps 2 , 500 Amount of Database Entries 2 , 000 1 , 500 1 , 000 500 0 ( 0 – 25 ) ( 25 – 50 ) ( 50 – 75 ) ( 75 – 100 ) ( 100 – 125 ) ( 125 – 150 ) ( 150 – 175 ) ( 175 – 200 ) ( 200 – 225 ) ( 225 – 250 ) ( 250 – 275 ) ( 275 – 300 ) ( 300 – 325 ) ( 325 – 350 ) Time Intervalls in Seconds Figure 6: Time distribution of the correction of a single problem ⇒ Upper limit of 800s and need for more precise time stamps H. Allgöwer — TUMexam Usage Analytics 7

Statistical Analyses GRNVS Endterm > 1000 enrolled students 711 written exams 22 correctors/users 231 hours spent correcting H. Allgöwer — TUMexam Usage Analytics 8

Statistical Analyses First vs. second correctors Problem First group Second group 1 2 min 07 s 1 min 35 s 2 1 min 16 s 1 min 03 s 3 1 min 48 s 1 min 42 s 4 1 min 10 s 0 min 36 s 5 1 min 09 s 0 min 49 s Table 1: Average time per problem H. Allgöwer — TUMexam Usage Analytics 9

Statistical Analyses First vs. second correctors Overall time Group Problem First group Second group 30 h 49 min 2 22 h 31 min 2 1 2 min 07 s 1 min 35 s 18 h 48 min 2 2 1 min 16 s 1 min 03 s 18 h 00 min 1 3 1 min 48 s 1 min 42 s 12 h 47 min 1 4 1 min 10 s 0 min 36 s 12 h 23 min 1 5 1 min 09 s 0 min 49 s 12 h 15 min 1 . . . Table 1: Average time per problem Table 2: Overall correction time H. Allgöwer — TUMexam Usage Analytics 9

Statistical Analyses Correction timeline 800 700 600 Time in Seconds 500 400 300 200 100 0 0 26 100 200 243 300 Index Figure 7: Typical timeline of the correction process H. Allgöwer — TUMexam Usage Analytics 10

Statistical Analyses Correction on subproblem basis 800 700 600 Time in Seconds 500 400 300 200 100 0 0 250 500 750 1 , 000 1 , 250 1 , 500 1 , 750 Index Figure 8: Timeline of a subproblem by subproblem correction H. Allgöwer — TUMexam Usage Analytics 11

Statistical Analyses Correction on subproblem basis 800 700 600 Time in Seconds 500 400 300 200 100 0 0 250 500 750 1 , 000 1 , 250 1 , 500 1 , 750 Index Figure 8: Timeline of a subproblem by subproblem correction ⇒ Need for a selection of correction type H. Allgöwer — TUMexam Usage Analytics 11

Statistical Analyses GRNVS Retake 343 registered students 243 written exams 16 correctors/users 84 hours spent correcting H. Allgöwer — TUMexam Usage Analytics 12

Statistical Analyses First vs. second corrector Total Time t avg / prob Total Exams Correction group 8 h 02 min 3 min 24 s 142 1 7 h 31 min 1 min 51 s 243 2 7 h 21 min 1 min 49 s 243 2 6 h 38 min 2 min 37 s 152 1 6 h 32 min 1 min 26 s 176 1 5 h 49 min 1 min 26 s 243 2 5 h 48 min 1 min 25 s 243 2 5 h 35 min 3 min 20 s 101 1 5 h 19 min 2 min 06 s 151 1 4 h 54 min 1 min 32 s 191 1 4 h 39 min 1 min 09 s 243 2 . . . Table 3: Overview of the corrector’s individual performance during the retake exam H. Allgöwer — TUMexam Usage Analytics 13

Statistical Analyses Problem distribution Figure 9: Distribution of time and pencil gestures for the five problems of the retake exam H. Allgöwer — TUMexam Usage Analytics 14

Statistical Analyses Pencil pressure 2 . 5 First corrector Second corrector 2 Pencil Pressure 1 . 5 1 0 . 5 0 0 25 50 75 100 Index Figure 10: Pencil pressure timeline for an exemplary first and second corrector H. Allgöwer — TUMexam Usage Analytics 15

Outlook Future improvements 1. Statistics view in the application 2. Smart timestamps 3. Selectable correction mode 4. Shortcut to additional pages 5. TUMexam for examinees H. Allgöwer — TUMexam Usage Analytics 16

References Marie Davidian und Thomas A. Louis. “Why Statistics?” In: Science 336.6077 (2012), S. 12–12. Cem Kaner, Senior Member und Walter P . Bond. “Software Engineering Metrics: What Do They Measure and How Do We Know?” In: In METRICS 2004. IEEE CS. Press, 2004. Bruce D. Weinberg u. a. “Internet of Things: Convenience vs. privacy and secrecy”. In: Business Horizons 58.6 (2015). SPECIAL ISSUE: THE MAGIC OF SECRETS, S. 615 –624. Tobias Röhm. “The MALTASE Framework For Usage-Aware Software Evolution”. Diss. München: Technische Universität München, 2015. Stephan Günther. TUMexam Flyer. Available online at https://tumexam.de/static/tumexam-booklet-20191021.pdf ; last accessed on 2019/10/31. H. Allgöwer — TUMexam Usage Analytics 17

References Images • Figure 2 Icons made by https://www.flaticon.com/authors/freepik • Classroom icon on slide 8 made by https://www.flaticon.com/authors/freepik • Test icons on slides 8 and 13 made by https://www.flaticon.com/authors/surang • Hourglass icons on slides 8 and 13 made by https://www.flaticon.com/authors/smash icons • Domain icon on slide 13 made by https://www.flaticon.com/authors/eucalyp H. Allgöwer — TUMexam Usage Analytics 18

Backup Distribution endterm Figure 11: Distribution of time and pencil gestures for the endterm’s five problems H. Allgöwer — TUMexam Usage Analytics 19

Backup Timeline - First vs. second corrector 800 800 700 700 600 600 Time in Seconds Time in Seconds 500 500 400 400 300 300 200 200 100 100 0 0 0 26 100 200 243 300 0 100 200 300 400 500 600 700 800 900 Index Index (a) First corrector (b) Second corrector Figure 12: Timeline of correction time per problem for the two correction groups H. Allgöwer — TUMexam Usage Analytics 20

Backup Timeline - First vs. second corrector 800 800 700 700 600 600 Time in Seconds Time in Seconds 500 500 400 400 300 300 200 200 100 100 0 0 0 26 100 200 243 300 0 100 200 300 400 500 600 700 800 900 Index Index (a) First corrector (b) Second corrector Figure 13: Timeline of correction time per problem for the two correction groups H. Allgöwer — TUMexam Usage Analytics 21

Backup First vs. second correctors 80% 83% 17% 20% Figure 14: Lines drawn Figure 15: Lines erased 79% 21% Figure 16: Lines undone H. Allgöwer — TUMexam Usage Analytics 22