What Exactly do we Mean by JIT Warmup ? Edd Barrett, Carl Friedrich - PowerPoint PPT Presentation

What Exactly do we Mean by JIT Warmup ? Edd Barrett, Carl Friedrich Bolz, Rebecca Killick (Lancaster), Vincent Knight (Cardiff), Sarah Mount, Laurence Tratt Software Development Team April 20, 2016 1 / 40 http://soft-dev.org/

Agenda Agenda 1 JIT Warmup Background 2 The Back-Story 3 The Warmup Experiment v2.0 4 Results 5 Automated Analyses 6 Conclusion and Future Work 2 / 40 http://soft-dev.org/

JIT Warmup Background 3 / 40 http://soft-dev.org/

JIT Warmup Background JIT Warmup Background Informally: Time taken for a JITted VM to reach peak performance 4 / 40 http://soft-dev.org/

JIT Warmup Background JIT Warmup Background 5 / 40 http://soft-dev.org/

JIT Warmup Background JIT Warmup Background 6 / 40 http://soft-dev.org/

Why is Warmup Important? Why is Warmup Important? Warmup contributes to overall performance. Long warmup is bad for user-facing and short-lived programs. VM authors report peak performance. 7 / 40 http://soft-dev.org/

The Back-Story 8 / 40 http://soft-dev.org/

The Back-Story The Back-Story We have a hunch that warmup is longer than people expect. We have some preliminary ideas to improve warmup. 9 / 40 http://soft-dev.org/

The Back-Story The Back-Story Goal: Measure how long modern JITs take to warm up. 10 / 40 http://soft-dev.org/

The Warmup Experiment v1.0 The Warmup Experiment v1.0 Microbenchmarks Reasonable number of repetitions. 10 process executions. 50 in-process iterations. Run on various VMs. Plot and report warmup time. 11 / 40 http://soft-dev.org/

The Back-Story The Back-Story Weird Results Many benchmarks don’t warmup under the classic model. 12 / 40 http://soft-dev.org/

The Back-Story The Back-Story New goal: Try to understand why we see “weird ” results. 13 / 40 http://soft-dev.org/

The Warmup Experiment v2.0 14 / 40 http://soft-dev.org/

Microbenchmarks Revisited Microbenchmarks Revisited CFG determinism. Each run takes same path through CFG. Checksums. Ensures di ff erent languages do the same work. Harder for VMs to optimise away whole benchmark. Code for microbenchmarks: https://github.com/softdevteam/warmup_experiment 15 / 40 http://soft-dev.org/

The Benchmark Runner Revisited The Benchmark Runner Revisited Krun Benchmark runner that aims to control sources of variation. WRT: memory limits, I /O, system state, . . . https://github.com/softdevteam/krun 16 / 40 http://soft-dev.org/

VMs VMs Graal-0.13 HHVM-3.12.0 JRuby/Tru ffl e (recent git version) Hotspot-8u72b15 LuaJit-2.0.4 PyPy-4.0.1 V8-4.9.385.21 GCC-4.9.3 (not really a VM) Same GCC across the board, minor VM patching. 17 / 40 http://soft-dev.org/

Machines Machines Linux-Debian8/i4790K, 24GiB RAM Linux-Debian8/i4790, 32GiB RAM OpenBSD-5.8/i4790, 32GiB RAM “ Turbo boost ” disabled. SSH blocked from non-local machines. Daemons disabled (e.g. cron, smtpd). 18 / 40 http://soft-dev.org/

Run for Longer Run for Longer Run many more in-process iterations (2000). Plot results and see if we see classic warmup now. 19 / 40 http://soft-dev.org/

Results 20 / 40 http://soft-dev.org/

Classical Warmup Classical Warmup Richards, Graal, Linux1/i7-4790K, Process execution #3 0.884 0.884 0.558 0.775 0.232 0 1 2 3 4 5 6 7 8 9 0.666 Time(s) 0.558 0.449 0.341 0.232 0 200 400 600 800 1000 1200 1400 1600 1800 2000 In-process iteration 21 / 40 http://soft-dev.org/

Classical Warmup Classical Warmup Fasta, V8, Linux2/i7-4790, Process execution #1 1.176 1.169 1.163 Time(s) 1.156 1.150 1.143 1.137 0 200 400 600 800 1000 1200 1400 1600 1800 2000 In-process iteration 21 / 40 http://soft-dev.org/

Classical Warmup Classical Warmup Spectral Norm, PyPy, Linux1/i7-4790K, Process execution #7 0.480 0.478 0.476 Time(s) 0.473 0.471 0.469 0.466 0 200 400 600 800 1000 1200 1400 1600 1800 2000 In-process iteration 21 / 40 http://soft-dev.org/

Classical Warmup Classical Warmup Fasta, V8, Linux1/i7-4790K, Process execution #1 Fasta, V8, Linux2/i7-4790, Process execution #1 1.055 1.176 1.050 1.169 1.044 1.163 Time(s) Time(s) 1.038 1.156 1.032 1.150 1.027 1.143 1.021 1.137 0 200 400 600 800 1000 1200 1400 1600 1800 2000 0 200 400 600 800 1000 1200 1400 1600 1800 2000 In-process iteration In-process iteration (Di ff erent machines) 22 / 40 http://soft-dev.org/

Slowdown Slowdown Fannkuch Redux, LuaJIT, OpenBSD/i7-4790, Process execution #10 0.567 0.566 0.566 Time(s) 0.565 0.564 0.563 0.562 0 200 400 600 800 1000 1200 1400 1600 1800 2000 In-process iteration 23 / 40 http://soft-dev.org/

Slowdown Slowdown Richards, Hotspot, Linux2/i7-4790, Process execution #2 0.298 0.293 0.287 Time(s) 0.282 0.276 0.271 0.266 0 200 400 600 800 1000 1200 1400 1600 1800 2000 In-process iteration 23 / 40 http://soft-dev.org/

Cycles Cycles Fannkuch Redux, Hotspot, Linux1/i7-4790K, Process execution #1 0.347 0.340 0.332 Time(s) 0.324 0.316 0.309 0.301 0 200 400 600 800 1000 1200 1400 1600 1800 2000 In-process iteration 24 / 40 http://soft-dev.org/

Cycles Cycles Fannkuch Redux, Hotspot, OpenBSD/i7-4790, Process execution #4 0.405 0.386 0.372 0.397 0.359 250 300 350 400 450 500 550 600 0.389 Time(s) 0.382 0.374 0.366 0.358 0 200 400 600 800 1000 1200 1400 1600 1800 2000 In-process iteration 24 / 40 http://soft-dev.org/

Cycles Cycles Binary Trees, PyPy, Linux2/i7-4790, Process execution #1 0.556 0.515 0.510 0.547 0.506 200 205 210 215 220 225 230 235 240 0.539 Time(s) 0.530 0.522 0.513 0.504 0 200 400 600 800 1000 1200 1400 1600 1800 2000 In-process iteration 24 / 40 http://soft-dev.org/

Changing Phases Changing Phases Fasta, LuaJIT, OpenBSD/i7-4790, Process execution #5 0.355 0.354 0.354 Time(s) 0.353 0.352 0.351 0.350 0 200 400 600 800 1000 1200 1400 1600 1800 2000 In-process iteration 25 / 40 http://soft-dev.org/

Vastly I nconsistent Process-executions Vastly I nconsistent Process-executions Fasta, PyPy, Linux2/i7-4790, Process execution #3 Fasta, PyPy, Linux2/i7-4790, Process execution #4 3.681 3.681 3.668 3.668 3.655 3.655 Time(s) Time(s) 3.643 3.643 3.630 3.630 3.618 3.618 3.605 3.605 0 200 400 600 800 1000 1200 1400 1600 1800 2000 0 200 400 600 800 1000 1200 1400 1600 1800 2000 In-process iteration In-process iteration (same machine) 26 / 40 http://soft-dev.org/

Vastly I nconsistent Process-executions Vastly I nconsistent Process-executions Binary Trees, C, Linux2/i7-4790, Process execution #1 Binary Trees, C, OpenBSD/i7-4790, Process execution #1 1.039 3.380 1.026 3.357 1.014 3.334 Time(s) Time(s) 1.001 3.311 0.989 3.288 0.976 3.265 0.963 3.242 0 200 400 600 800 1000 1200 1400 1600 1800 2000 0 200 400 600 800 1000 1200 1400 1600 1800 2000 In-process iteration In-process iteration (Di ff erent machines. Bouncing ball pattern Linux-speci fi c) 26 / 40 http://soft-dev.org/

Full Results Full Results https://archive.org/download/softdev_warmup_ experiment_artefacts/v0.2/ all_graphs.pdf All plots in one huge PDF. warmup_results*.json.bz2 Raw results. 27 / 40 http://soft-dev.org/

Automated Analyses 28 / 40 http://soft-dev.org/

Automated Analyses: Outlier Detection Automated Analyses: Outlier Detection Measurement Outliers Spectral Norm, PyPy, Linux1/i7-4790K, Process execution #1 Spectral Norm, PyPy, Linux1/i7-4790K, Process execution #2 0.480 0.480 0.478 0.478 0.476 0.476 Time(s) Time(s) 0.473 0.473 0.471 0.471 0.469 0.469 0.466 0.466 0 200 400 600 800 1000 1200 1400 1600 1800 2000 0 200 400 600 800 1000 1200 1400 1600 1800 2000 In-process iteration In-process iteration 29 / 40 http://soft-dev.org/

Automated Analyses: Outlier Detection Automated Analyses: Outlier Detection 5 ¾ Measurement Outliers Spectral Norm, PyPy, Linux1/i7-4790K, Process execution #1 Spectral Norm, PyPy, Linux1/i7-4790K, Process execution #2 0.480 0.480 0.478 0.478 0.476 0.476 Time(s) Time(s) 0.473 0.473 0.471 0.471 0.469 0.469 0.466 0.466 0 200 400 600 800 1000 1200 1400 1600 1800 2000 0 200 400 600 800 1000 1200 1400 1600 1800 2000 In-process iteration In-process iteration outliers outside 5 σ of rolling average 29 / 40 http://soft-dev.org/

What Exactly do we Mean by JIT Warmup ? Edd Barrett, Carl Friedrich - PowerPoint PPT Presentation

What Exactly do we Mean by JIT Warmup ? Edd Barrett, Carl Friedrich Bolz, Rebecca Killick (Lancaster), Vincent Knight (Cardiff), Sarah Mount, Laurence Tratt Software Development Team April 20, 2016 1 / 40 http://soft-dev.org/ Agenda Agenda

Assisted warmup with the Zing JVM Ivn Kr lov @JohnWings Assisted warmup with the Zing JVM

CSE 373: Asymptotic Analysis Michael Lee Wednesday, Jan 10, 2018 1 Warmup Warmup: construct a

CSE 373: More on graphs; DFS and BFS Michael Lee Wednesday, Feb 14, 2018 1 Warmup Warmup:

CSE 373: Graph traversal Michael Lee Friday, Feb 16, 2018 1 Warmup Warmup Given a graph,

CSE 373: Hash functions and hash tables Michael Lee Monday, Jan 22, 2018 1 Warmup Warmup:

CSE 373: Tradeofgs and Abstractions Michael Lee Friday Jan 5, 2017 1 Warmup Warmup questions:

Games, etc Warmup: argmin Recursion types Games ReasonML representa6on of games Warmup

CS 141, Lecture 3 Warmup: You're working at a fast

Foundations of Computer Science Lecture 1 Warmup: A Taste for Discrete Math and Computing

Foundations of Computer Science Lecture 1 Warmup: A Taste for Discrete Math and Computing

(Today) Warmup: A Taste for Discrete Math and Computing Foundations of Computer Science Lecture 1

WARMUP (10, -4) AND (6, 0) FIND THE DISTANCE BETWEEN THE 2 POINTS. FIND THE SLOPE

Warmup Approximately what is the largest number that can be represented with a signed 32-bit

Warmup What is the current i 3 and voltage V 3 ? 1V -100mA 5V 20mA i 3 V 3 Go to menti.com and

Purpose: How do we make invisible heat visible? WARMUP : 50 Mm = ____________ mm #1 Video

Warmup Find the voltage across and current through each resistor, using KVL, KCL and Ohm's law:

Warmup Draw the schematic for this circuit: Battery B 1 L 1 ON S 1 Light bulb OFF B 2 L 2

Warmup Use a k-map to fi nd a minimal implementation of this truth table: A B C D | Y 0 0 0 0 0

1. SNAP!: WARMUP ACTIVITY to elicit ideas on Open Data (13:45 to 14:00) Snap! is a

Maps, and Iterators Algorithms CSE 373 SU 18 BEN JONES 1 Warmup Draw a tree for this

Analysis + Recurrences Algorithms CSE 373 SU 18 BEN JONES 1 Warmup Write a model and find

Warmup : On paper, write a C++ function that takes a single int argument (n) and returns the

Warmup You are asked organize the Boston Marathons, which this year will be fi ve separate races

How to Count like a Mathematician Counting Warmup Round 1 squares Lara Pudwell dominoes