Tests for Algorithmic Complexity in Programming Exercises - PowerPoint PPT Presentation

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Tests for Algorithmic Complexity in Programming Exercises Intermediate talk for the Guided Research by Stefan Wölfert advised by Johannes Naab and Henning Stubbe Friday 17 th January, 2020 Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Tests for Algorithmic Complexity in Programming Exercises Programming exercises in GRNVS: • Automatic testing of student submissions for functional requirements • Non-functional assessment by hand S. Wölfert — ethstats runtime test case 2

Tests for Algorithmic Complexity in Programming Exercises Programming exercises in GRNVS: • Automatic testing of student submissions for functional requirements • Non-functional assessment by hand Problem statement: • Finding a method for estimating the used data structure • Creating a new test case for ethstats programming exercise S. Wölfert — ethstats runtime test case 2

Programming exercise: ethstats Problem statement 5 0 1 2 3 4 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 0 B Destination Address 4 B Source Address 8 B 12 B Ethertype Figure 1: IEEE 802.3 Ethernet Header 1 1 GRNVS Cheatsheet S. Wölfert — ethstats runtime test case 3

Programming exercise: ethstats Problem statement 5 0 1 2 3 4 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 0 B Destination Address 4 B Source Address 8 B 12 B Ethertype Figure 1: IEEE 802.3 Ethernet Header 1 Tasks: Sample output: I am ready! • Reading data from a raw socket 0x0800: 3 frames, 192 bytes 0x0806: 3 frames, 209 bytes • Evaluating the destination address 0x86dd: 4 frames, 256 bytes • Determining the EtherType 3 of them were for me 4 of them were multicast • Counting amount of frames and data IPv4 accounted for 29.2% and IPv6 for 39.0% of traffic per EtherType 1 GRNVS Cheatsheet S. Wölfert — ethstats runtime test case 3

Programming exercise: ethstats Test cases Current test cases: • Handling EtherTypes • Output format • Correct counting • Correct destination evaluation • Appropriate runtime S. Wölfert — ethstats runtime test case 4

Programming exercise: ethstats Test cases Current test cases: • Handling EtherTypes • Output format • Correct counting • Correct destination evaluation • Appropriate runtime Additional requirements: choosing an appropriate data structure S. Wölfert — ethstats runtime test case 4

Scaling of data structures Runtime behaviour Time-determining operations: • Data structure independent input and output behaviour • Main influencing operations: insert , update (s. Table 1 and Table 2) Table 1: Best-case runtime behaviour for a single operation. Table 2: Worst-case runtime behaviour for a single operation. Map Linked-List Map Linked-List Insert Insert O (1) O (1) O (1) O ( n ) Update O (1) O (1) Update O (1) O ( n ) S. Wölfert — ethstats runtime test case 5

Scaling of data structures Scaling behaviour O ( n 2 ) 10 8 scaling 6 4 2 O ( n ) 0 0 2 4 6 8 10 n Figure 2: Sample scaling behaviour. S. Wölfert — ethstats runtime test case 6

Runtime test case Goal: estimating the used data structure by runtime measurements • Reference implementations for ethstats in c and java with array and linked-list data struc- ture • Creating a testbed for runtime measurements • Using linear scaling behaviour as main criterion for estimation S. Wölfert — ethstats runtime test case 7

Runtime test case Testbed eth1 eth1 eth0 eth0 Figure 3: Network topology for tests on ethstats S. Wölfert — ethstats runtime test case 8

Runtime test case Testbed eth1 eth1 eth0 eth0 Figure 3: Network topology for tests on ethstats Workflow: 1. Providing clean environment on testee 2. Building ethstats on testee 3. Sending ready-marker on control channel 4. Sending frames on data channel in ascending and descending order, while observing user space runtime via /usr/bin/time 5. Evaluating output and measured time values S. Wölfert — ethstats runtime test case 8

Runtime test case Parameters Influences on the measurement: • Frame size • Sending rate and time-out value • Amount of frames S. Wölfert — ethstats runtime test case 9

Runtime test case Parameters Influences on the measurement: • Frame size • Sending rate and time-out value • Amount of frames 1 c map c linked-list avg. user space time [s] 0 . 8 java map java linked-list 0 . 6 0 . 4 0 . 2 0 0 2 4 6 8 10 12 amount of frames [2 n ] Figure 4: Runtime behaviour of reference implementations. S. Wölfert — ethstats runtime test case 9

Runtime test case Simple linear regression • For a single point: (1) t i = β · x i + t 0 + ǫ i O ( n 2 ) 10 8 scaling 6 4 2 O ( n ) 0 0 2 4 6 8 10 n S. Wölfert — ethstats runtime test case 10

Runtime test case Simple linear regression • For a single point: (1) t i = β · x i + t 0 + ǫ i • Estimating β and t 0 by method of least squares: � n � t � t i − ¯ i =1 t 0 = ¯ β = t − β · ¯ x (2) � n x ) 2 O ( n 2 ) i =1 ( x i − ¯ 10 8 scaling 6 4 2 O ( n ) 0 0 2 4 6 8 10 n S. Wölfert — ethstats runtime test case 10

Runtime test case Simple linear regression • For a single point: (1) t i = β · x i + t 0 + ǫ i • Estimating β and t 0 by method of least squares: � n � t � t i − ¯ i =1 t 0 = ¯ β = t − β · ¯ x (2) � n x ) 2 O ( n 2 ) i =1 ( x i − ¯ 10 8 • Spreading due to regression SQR : scaling 6 n t � 2 � � t i − ¯ ˜ ˜ SQR = t i = t i − ǫ i (3) 4 i =1 2 O ( n ) 0 0 2 4 6 8 10 n S. Wölfert — ethstats runtime test case 10

Runtime test case Simple linear regression • For a single point: (1) t i = β · x i + t 0 + ǫ i • Estimating β and t 0 by method of least squares: � n � t � t i − ¯ i =1 t 0 = ¯ β = t − β · ¯ x (2) � n x ) 2 O ( n 2 ) i =1 ( x i − ¯ 10 8 • Spreading due to regression SQR : scaling 6 n t � 2 � � t i − ¯ ˜ ˜ SQR = t i = t i − ǫ i (3) 4 i =1 2 O ( n ) • Total spreading SQT : 0 0 2 4 6 8 10 n n t � 2 � � t i − ¯ SQT = (4) i =1 S. Wölfert — ethstats runtime test case 10

Runtime test case Simple linear regression • For a single point: (1) t i = β · x i + t 0 + ǫ i • Estimating β and t 0 by method of least squares: � n � t � t i − ¯ i =1 t 0 = ¯ β = t − β · ¯ x (2) � n x ) 2 O ( n 2 ) i =1 ( x i − ¯ 10 8 • Spreading due to regression SQR : scaling 6 n t � 2 � � t i − ¯ ˜ ˜ SQR = t i = t i − ǫ i (3) 4 i =1 2 O ( n ) • Total spreading SQT : 0 0 2 4 6 8 10 n n t � 2 � � t i − ¯ SQT = (4) i =1 • Quality of fitting η : η = 1 − SQR (5) SQT S. Wölfert — ethstats runtime test case 10

Runtime test case Testing reference implementations User space time measurement: • Tested by sending 512, 1024, 2048, 3072, 4096 and 5120 frames • 5 Repetitions Table 3: Fit quality of reference implementations. language data rate(asc) rate(dsc) t max structure [%] [%] [s] c map 41.6 25.6 0.05 c list 95.7 32.9 0.19 java map 90.1 89.6 0.83 java list time-out time-out time-out S. Wölfert — ethstats runtime test case 11

Runtime test case Testing reference implementations User space time measurement: • Tested by sending 512, 1024, 2048, 3072, 4096 and 5120 frames • 5 Repetitions Table 3: Fit quality of reference implementations. language data rate(asc) rate(dsc) t max structure [%] [%] [s] c map 41.6 25.6 0.05 c list 95.7 32.9 0.19 java map 90.1 89.6 0.83 java list time-out time-out time-out Test criteria: • Small difference in quality rate for appropriate implementations • Small maximum runtime as second alternative criterion S. Wölfert — ethstats runtime test case 11

Runtime test case Testing 2019SS students submissions User space time measurement: • Tested by sending 512, 1024, 2048, 3072, 4096 and 5120 frames • 5 Repetitions Table 4: Identifying students submissions based on simple linear regression and total time in user space. language data false correct structure identification identification c map 5 0 c list 8 1 java map 2 43 java list 9 2 S. Wölfert — ethstats runtime test case 12

Runtime test case Testing 2019SS students submissions User space time measurement: • Tested by sending 512, 1024, 2048, 3072, 4096 and 5120 frames • 5 Repetitions Table 4: Identifying students submissions based on simple linear regression and total time in user space. language data false correct structure identification identification c map 5 0 c list 8 1 java map 2 43 java list 9 2 • Evaluation of c and java implementations only • Multiple crashes of test framework causing missing data S. Wölfert — ethstats runtime test case 12