Measuring Performance on OpenBSD Alexander Bluhm bluhm@openbsd.org - PowerPoint PPT Presentation

What did exist before? How does it work? What are the findings? What is the Conclusion? Measuring Performance on OpenBSD Alexander Bluhm bluhm@openbsd.org BSDCan, May 2019

What did exist before? How does it work? What are the findings? What is the Conclusion? Detect Changes in Performance over Commits and Releases What did exist before? 1 How does it work? 2 What are the findings? 3 What is the Conclusion? 4

What did exist before? How does it work? What are the findings? What is the Conclusion? Agenda What did exist before? 1 How does it work? 2 What are the findings? 3 What is the Conclusion? 4

What did exist before? How does it work? What are the findings? What is the Conclusion? genua Firewall Testbed HPF Numbers for Customers Marketing Developers Hardware . . .

What did exist before? How does it work? What are the findings? What is the Conclusion? HPF Result

What did exist before? How does it work? What are the findings? What is the Conclusion? Multi User, Multi Purpose Hardware Setup 1 Gbit 1 Gbit Source 1 Switch Drain 1 Source 2 Drain 2 . . . . . . Target 1 Target 2 Source n Drain n . . . 10 Gbit 10 Gbit Target n

What did exist before? How does it work? What are the findings? What is the Conclusion? Unsuitable HPF too many requirements too much complexity not enough flexibility

What did exist before? How does it work? What are the findings? What is the Conclusion? Performance Hardware Design 10 Gbit Target Target

What did exist before? How does it work? What are the findings? What is the Conclusion? Existing Regression Tests daily test runs using /usr/src/regress multi architecture history of pass and fail useful information web access http://bluhm.genua.de/

What did exist before? How does it work? What are the findings? What is the Conclusion? All Regression Tests

What did exist before? How does it work? What are the findings? What is the Conclusion? Regression Tests with Links setup logs tested obj download version run history cvs web colored status test output

What did exist before? How does it work? What are the findings? What is the Conclusion? Regression History for i386 history severity

What did exist before? How does it work? What are the findings? What is the Conclusion? Regression Test Hardware server remote router local

What did exist before? How does it work? What are the findings? What is the Conclusion? Regression Master web publish configure master control server remote router local console install

What did exist before? How does it work? What are the findings? What is the Conclusion? Agenda What did exist before? 1 How does it work? 2 What are the findings? 3 What is the Conclusion? 4

What did exist before? How does it work? What are the findings? What is the Conclusion? Performance Goals history reproducable details available drill down automatic

What did exist before? How does it work? What are the findings? What is the Conclusion? Performance History advance by step install collect release results run checkout compile tests at date kernel

What did exist before? How does it work? What are the findings? What is the Conclusion? Performance Tests Overview cvs run install checkout date version steps run history run detail test command

What did exist before? How does it work? What are the findings? What is the Conclusion? Performance Run at Date checkout build kernel date quirks commits run and install log average numbers unstable results

What did exist before? How does it work? What are the findings? What is the Conclusion? Performance Repeat at CVS Checkout repeat standard outlier count deviation reboot log single result

What did exist before? How does it work? What are the findings? What is the Conclusion? Weekly from 6.2 to 6.3 TCP Performance A C B D E F G H I J 4 × 10 9 3 . 5 × 10 9 3 × 10 9 2 . 5 × 10 9 bits/sec 2 × 10 9 1 . 5 × 10 9 1 × 10 9 5 × 10 8 0 2017-11-01 2017-12-01 2018-01-01 2018-02-01 2018-03-01 Checkout (date)

What did exist before? How does it work? What are the findings? What is the Conclusion? Quirks from 6.2 to 6.3 A OpenBSD/amd64 6.2 release B fix cvs vendor branch checkout C clang update LLVM to 5.0.0 D pfctl pf packet rate matching E move kernel source file dwiic.c F pfctl pf divert type G sysctl struct vfsconf H clang update LLVM to 5.0.1 I pfctl pf syncookies J OpenBSD/amd64 6.3 release

What did exist before? How does it work? What are the findings? What is the Conclusion? Build Quirks advance build checkout by step toolchain userland run checkout compile tests at date kernel

What did exist before? How does it work? What are the findings? What is the Conclusion? Performance Hardware Future Target 1 Target 1 Target 2 Target 2 Linux

What did exist before? How does it work? What are the findings? What is the Conclusion? Performance Master web publish configure and compile master control remote local target target console install release

What did exist before? How does it work? What are the findings? What is the Conclusion? Agenda What did exist before? 1 How does it work? 2 What are the findings? 3 What is the Conclusion? 4

What did exist before? How does it work? What are the findings? What is the Conclusion? Drilldown from Week to Days TCP Performance 4 × 10 9 3 . 5 × 10 9 3 × 10 9 2 . 5 × 10 9 bits/sec 2 × 10 9 1 . 5 × 10 9 1 × 10 9 5 × 10 8 0 2017-12-28 2017-12-30 2018-01-01 2018-01-03 2018-01-05 2018-01-07 2018-01-09 Checkout (date)

What did exist before? How does it work? What are the findings? What is the Conclusion? Reproduce and Reboot run compile tests kernel run reboot relink again machine kernel

What did exist before? How does it work? What are the findings? What is the Conclusion? 6.4, 5 Days, 10 Tests, Keep Machine Running TCP Performance Y 4 × 10 9 3 . 5 × 10 9 3 × 10 9 2 . 5 × 10 9 bits/sec 2 × 10 9 1 . 5 × 10 9 1 × 10 9 5 × 10 8 0 2018-10-11 2018-10-11 2018-10-12 2018-10-12 2018-10-13 2018-10-13 2018-10-14 2018-10-14 2018-10-15 Checkout (date)

What did exist before? How does it work? What are the findings? What is the Conclusion? 6.4, 5 Days, 10 Tests, Reboot Machine TCP Performance Y 4 × 10 9 3 . 5 × 10 9 3 × 10 9 2 . 5 × 10 9 bits/sec 2 × 10 9 1 . 5 × 10 9 1 × 10 9 5 × 10 8 0 2018-10-11 2018-10-11 2018-10-12 2018-10-12 2018-10-13 2018-10-13 2018-10-14 2018-10-14 2018-10-15 Checkout (date)

What did exist before? How does it work? What are the findings? What is the Conclusion? 6.4, 5 Days, 10 Tests, Link and Reorder Kernel TCP Performance Y 4 × 10 9 3 . 5 × 10 9 3 × 10 9 2 . 5 × 10 9 bits/sec 2 × 10 9 1 . 5 × 10 9 1 × 10 9 5 × 10 8 0 2018-10-11 2018-10-11 2018-10-12 2018-10-12 2018-10-13 2018-10-13 2018-10-14 2018-10-14 2018-10-15 Checkout (date)

What did exist before? How does it work? What are the findings? What is the Conclusion? 6.4, 15 Days, 5 Tests, 2 CPU Sockets, Keep running TCP Performance Y Z ab c 4 × 10 9 3 . 5 × 10 9 3 × 10 9 2 . 5 × 10 9 bits/sec 2 × 10 9 1 . 5 × 10 9 1 × 10 9 5 × 10 8 0 2018-10-12 2018-10-14 2018-10-16 2018-10-18 2018-10-20 2018-10-22 2018-10-24 Checkout (date)

What did exist before? How does it work? What are the findings? What is the Conclusion? 2 CPU Sockets, Repeat, Keep running second cycle

What did exist before? How does it work? What are the findings? What is the Conclusion? from 6.3 to 6.4, 202 Days TCP Performance K M L N O PQ RS TU V W X Y 4 × 10 9 3 . 5 × 10 9 3 × 10 9 2 . 5 × 10 9 bits/sec 2 × 10 9 1 . 5 × 10 9 1 × 10 9 5 × 10 8 0 2018-04-01 2018-05-01 2018-06-01 2018-07-01 2018-08-01 2018-09-01 2018-10-01 Checkout (date)

What did exist before? How does it work? What are the findings? What is the Conclusion? from 6.4 to 6.5, 185 Days TCP Performance Y Z bc a d e f g h i j k l 4 × 10 9 3 . 5 × 10 9 3 × 10 9 2 . 5 × 10 9 bits/sec 2 × 10 9 1 . 5 × 10 9 1 × 10 9 5 × 10 8 0 2018-11-01 2018-12-01 2019-01-01 2019-02-01 2019-03-01 2019-04-01 Checkout (date)

What did exist before? How does it work? What are the findings? What is the Conclusion? from 6.5, 19 Days TCP Performance m 4 × 10 9 3 . 5 × 10 9 3 × 10 9 2 . 5 × 10 9 bits/sec 2 × 10 9 1 . 5 × 10 9 1 × 10 9 5 × 10 8 0 2019-04-14 2019-04-16 2019-04-18 2019-04-20 2019-04-22 2019-04-24 2019-04-26 2019-04-28 2019-04-30 Checkout (date)