Toward Standard Formats and Benchmark Suites for Floating Point - PowerPoint PPT Presentation

Toward Standard Formats and Benchmark Suites for Floating Point Tools Zach Tatlock

Collaborators Nasrine Damouche Pavel Panchekha Matthieu Martel Alex Sanchez-Stern Eva Darulova Chen Qiu Heiko Becker Sorin Lerner Debasmita Lohar Bill Zorn Dan Grossman

Yardsticks and Assembly Lines

Diverse Yardsticks Across Numerics Accuracy absolute, relative, ulp, bound, average Performance space, runtime, analysis time Expressiveness (domain) HPC, embedded, comp geom, BLAS, libm

Diverse Yardsticks Across Numerics Accuracy absolute, relative, ulp, average “It is impossible to escape the impression that people commonly use false standards of measurement -- Performance that they seek power, success and wealth for themselves and admire them in others, and that they space, runtime, analysis time underestimate what is of true value in life.” Domain Sigmund Freud Civilization and Its Discontents HPC, embedded, BLAS, libm

Diverse Yardsticks Across Numerics Accuracy absolute, relative, ulp, average “It is impossible to escape the impression that people commonly use false standards of measurement -- Performance that they seek power, success and wealth for themselves and admire them in others, and that they space, runtime, analysis time underestimate what is of true value in numerics cs .” Domain Si Sigm gmoid F-ro round HPC, embedded, BLAS, libm Si Simu mulation a and It Its Di Discontinuities

Diverse Tools Across Numerics Accuracy / Performance / Domain

Diverse Tools Across Numerics Problem: - disjoint benchmarks / paper - completely different reprs - difficult to compare & combine Accuracy / Performance / Domain

Measures & Fmts Across CS Compilation (SPEC INT, EEMBC) compile time, run time, code size SAT/SMT (DIMACS, SMT-LIB) solver time, model size, theory support Synthesis (SyGuS) invariant synth, programming by example, etc. . . .

Measures & Fmts: Community SyGuS competitions: - challenge problems - measure progress - provide infra - support research Rishabh Singh SyGuS Organizer

Measures & Fmts: Community SyGuS competitions: - challenge problems FUN! - measure progress - provide infra - support research Rishabh Singh SyGuS Organizer

al: Standard Numeric Yardsticks Go Goal: Formats core, imperative, precisions, std error defs Tools reference / baseline eval, infrastructure Benchmark Suites diverse domains, objectives, challenge catalog . . .

al: Standard Numeric Yardsticks Go Goal: Formats core, imperative, precisions, std error defs Vision: - reproducible, fair comparisons Tools reference / baseline eval, infrastructure - lower barrier to entry for new research Benchmark Suites - compose existing tools for new problems diverse domains, objectives, challenge catalog - build community (regular competitions?) . . .

FPB FPBenc ench Formats: FPCore, FPImp Tools: reference evaluators, infrastructure Eval: growing suite, anecdotes, adoption Future: more types, more benches, RFCs http://fpbench.org

FPBench Formats: FPCore (FPCore (u v T) :name "doppler1" s-exprs :cite (darulova-kuncak-2014) :fpbench-domain science :type binary64 meta :pre (and (<= -100 u 100) (<= 20 v 20000) spec (<= -30 T 50)) :rosa-ensuring 1e-12 pure (let ([t1 (+ 331.4 (* 0.6 T))]) (/ (* (- t1) v) (* (+ t1 u) expr (+ t1 u)))))

FPBench Formats: FPCore (FPCore (t0 w0 N) :name "Pendulum" :fpbench-domain science :pre (and (< -2 t0 2) (< -5 w0 5)) :example ([N 1000]) (let ([h 0.01] [L 2.0] [m 1.5] [g 9.80665]) common C/Fortran ops (while (< n N) ([t t0 (let ([k1w (* (/ (- g) L) (sin t))]) (let ([k2t (+ w (* (/ h 2) k1w))]) loops (+ t (* h k2t))))] [w w0 (let ([k2w (* (/ (- g) L) (sin (+ t (* (/ h 2) w))))]) (+ w (* h k2w)))] [n 0 (+ n 1)]) t)))

FPBench Formats: FPCore (FPCore (x (sr* sl*) 0.0 :name (let ((cosi "Odometry" (let ((arg :description (let ((delta_theta "Compute the position of a robot from the speed of the wheels.\nInputs: (let ((inv_l 0.1) Speed `sl`, `sr` of the left and right wheel, in rad/s." (delta_dl (let ((c 12.34)) (* c sl))) :cite (delta_dr (let ((c 12.34)) (* c sr)))) (damouche-martel-chapoutot-fmics15) (* (- delta_dr delta_dl) inv_l)))) :fpbench-domain (+ theta (* delta_theta 0.5))))) controls (cos arg))) :type (delta_d binary32 (let ((delta_dr (let ((c 12.34)) (* c sr))) :pre (delta_dl (let ((c 12.34)) (* c sl)))) (and (< 0.05 sl (* 2 PI)) (< 0.05 sr (* 2 PI))) (* (+ delta_dl delta_dr) 0.5)))) :example (+ x (* delta_d cosi)))) ((sr* 0.0785398163397) (sl* 0.0525398163397)) (sini (while 0.0 (< t 1000) (let ((arg ((delta_dl 0.0 (let ((c 12.34)) (* c sl))) (let ((delta_theta (delta_dr 0.0 (let ((c 12.34)) (* c sr))) (let ((inv_l 0.1) (delta_d (delta_dl (let ((c 12.34)) (* c sl))) 0.0 (delta_dr (let ((c 12.34)) (* c sr)))) (let ((delta_dr (let ((c 12.34)) (* c sr))) (* (- delta_dr delta_dl) inv_l)))) (delta_dl (let ((c 12.34)) (* c sl)))) (+ theta (* delta_theta 0.5))))) (* (+ delta_dl delta_dr) 0.5))) (sin arg))) (delta_theta (y 0.0 0.0 (let ((inv_l 0.1) (let ((sini (delta_dl (let ((c 12.34)) (* c sl))) (let ((arg (delta_dr (let ((c 12.34)) (* c sr)))) (let ((delta_theta (* (- delta_dr delta_dl) inv_l))) (let ((inv_l 0.1) (arg (delta_dl (let ((c 12.34)) (* c sl))) 0.0 (delta_dr (let ((c 12.34)) (* c sr)))) (let ((delta_theta (* (- delta_dr delta_dl) inv_l)))) (let ((inv_l 0.1) (+ theta (* delta_theta 0.5))))) (delta_dl (let ((c 12.34)) (* c sl))) (sin arg))) (delta_dr (let ((c 12.34)) (* c sr)))) (delta_d (* (- delta_dr delta_dl) inv_l)))) (let ((delta_dr (let ((c 12.34)) (* c sr))) (+ theta (* delta_theta 0.5)))) (delta_dl (let ((c 12.34)) (* c sl)))) (cosi (* (+ delta_dl delta_dr) 0.5)))) 0.0 (+ y (* delta_d sini)))) (let ((arg (theta (let ((delta_theta -0.985 (let ((inv_l 0.1) (let ((delta_theta (delta_dl (let ((c 12.34)) (* c sl))) (let ((inv_l 0.1) (delta_dr (let ((c 12.34)) (* c sr)))) (delta_dl (let ((c 12.34)) (* c sl))) (* (- delta_dr delta_dl) inv_l)))) (delta_dr (let ((c 12.34)) (* c sr)))) (+ theta (* delta_theta 0.5))))) (* (- delta_dr delta_dl) inv_l)))) (cos arg))) (+ theta delta_theta))) (t 0 (+ t 1)) (j 0 (if (== j 50) 0 (+ j 1))) (tmp 0.0 (if (== j 50) sl tmp)) (sl sl* (if (== j 50) sr sl)) (sr sr* (if (== j 50) (let ((tmp sl)) tmp) sr)))

FPBench Formats: FPCore (FPCore (x (sr* sl*) 0.0 :name (let ((cosi "Odometry" (let ((arg :description (let ((delta_theta "Compute the position of a robot from the speed of the wheels.\nInputs: (let ((inv_l 0.1) Speed `sl`, `sr` of the left and right wheel, in rad/s." (delta_dl (let ((c 12.34)) (* c sl))) :cite (delta_dr (let ((c 12.34)) (* c sr)))) (damouche-martel-chapoutot-fmics15) (* (- delta_dr delta_dl) inv_l)))) :fpbench-domain (+ theta (* delta_theta 0.5))))) controls (cos arg))) :type (delta_d binary32 (let ((delta_dr (let ((c 12.34)) (* c sr))) :pre (delta_dl (let ((c 12.34)) (* c sl)))) (and (< 0.05 sl (* 2 PI)) (< 0.05 sr (* 2 PI))) (* (+ delta_dl delta_dr) 0.5)))) :example (+ x (* delta_d cosi)))) ((sr* 0.0785398163397) (sl* 0.0525398163397)) (sini (while 0.0 (< t 1000) (let ((arg ((delta_dl 0.0 (let ((c 12.34)) (* c sl))) (let ((delta_theta (delta_dr 0.0 (let ((c 12.34)) (* c sr))) (let ((inv_l 0.1) (delta_d (delta_dl (let ((c 12.34)) (* c sl))) 0.0 (delta_dr (let ((c 12.34)) (* c sr)))) (let ((delta_dr (let ((c 12.34)) (* c sr))) (* (- delta_dr delta_dl) inv_l)))) Things can get a little… verbose. (delta_dl (let ((c 12.34)) (* c sl)))) (+ theta (* delta_theta 0.5))))) (* (+ delta_dl delta_dr) 0.5))) (sin arg))) (delta_theta (y 0.0 0.0 (let ((inv_l 0.1) (let ((sini (delta_dl (let ((c 12.34)) (* c sl))) (let ((arg (delta_dr (let ((c 12.34)) (* c sr)))) (let ((delta_theta (* (- delta_dr delta_dl) inv_l))) (let ((inv_l 0.1) (arg (delta_dl (let ((c 12.34)) (* c sl))) 0.0 (delta_dr (let ((c 12.34)) (* c sr)))) (let ((delta_theta (* (- delta_dr delta_dl) inv_l)))) (let ((inv_l 0.1) (+ theta (* delta_theta 0.5))))) (delta_dl (let ((c 12.34)) (* c sl))) (sin arg))) (delta_dr (let ((c 12.34)) (* c sr)))) (delta_d (* (- delta_dr delta_dl) inv_l)))) (let ((delta_dr (let ((c 12.34)) (* c sr))) (+ theta (* delta_theta 0.5)))) (delta_dl (let ((c 12.34)) (* c sl)))) (cosi (* (+ delta_dl delta_dr) 0.5)))) 0.0 (+ y (* delta_d sini)))) (let ((arg (theta (let ((delta_theta -0.985 (let ((inv_l 0.1) (let ((delta_theta (delta_dl (let ((c 12.34)) (* c sl))) (let ((inv_l 0.1) (delta_dr (let ((c 12.34)) (* c sr)))) (delta_dl (let ((c 12.34)) (* c sl))) (* (- delta_dr delta_dl) inv_l)))) (delta_dr (let ((c 12.34)) (* c sr)))) (+ theta (* delta_theta 0.5))))) (* (- delta_dr delta_dl) inv_l)))) (cos arg))) (+ theta delta_theta))) (t 0 (+ t 1)) (j 0 (if (== j 50) 0 (+ j 1))) (tmp 0.0 (if (== j 50) sl tmp)) (sl sl* (if (== j 50) sr sl)) (sr sr* (if (== j 50) (let ((tmp sl)) tmp) sr)))

Toward Standard Formats and Benchmark Suites for Floating Point - PowerPoint PPT Presentation

Toward Standard Formats and Benchmark Suites for Floating Point Tools Zach Tatlock Collaborators Nasrine Damouche Pavel Panchekha Matthieu Martel Alex Sanchez-Stern Eva Darulova Chen Qiu Heiko Becker Sorin Lerner Debasmita Lohar Bill

Toward a Standard Benchmark Format and Suite for Floating-Point Analysis Nasrine Damouche,

Benchmark suites to measure Motivation computer performance Benchmarking overview

Floating-point numbers Fractional binary numbers IEEE floating-point standard Floating-point

Acceleration Targets: A Study of Popular Benchmark Suites Lisa Wu and Martha A. Kim, Department

PARSEC vs. SPLASH-2: A Quantitative Comparison of Two Multithreaded Benchmark Suites Christian

Floating point Today ! IEEE Floating Point Standard ! Rounding ! Floating Point Operations !

WELCOME TO SOUTHERN AFRICA LEAdERSHIp SUMMIT THEME: bENCHMARk HOSTED BY: MEN OF ISSACHAR VISION

Establishing Realistic Investment Earnings Benchmarks What is a Benchmark? A benchmark is a

Formal verification of floating-point algorithms John Harrison Intel Corporation Floating

TOTAL NUMBER OF SUITES 26 7 JUNIOR SUITES 2 JUNIOR SUITES FOR PEOPLE WHO NEED SPECIAL CARE 4

TOTAL NUMBER OF SUITES 26 7 JUNIOR SUITES 2 JUNIOR SUITES FOR PEOPLE WHO NEED SPECIAL CARE 4

15-213 The course that gives CMU its Zip! Floating Point Sept 6, 2006 Topics Topics

Sequence File Formats Sequence File Formats Different formats for different uses

2/10/2020 Today: Floating Point Background: Fractional binary numbers IEEE floating point

9/20/2018 Today: Floating Point Background: Fractional binary numbers IEEE floating point

SUITES P A R K H Y A T T C H I C A G O the SUITES P A R K S U I T E E X E C U T I V E S U I

Lecture: Metrics to Evaluate Performance Topics: Benchmark suites, Performance equation,

Formal Verification Methods 5: Floating-point verification John Harrison Intel Corporation

Real Number Representation 1 Topics Terminology IEEE standard for floating-point

A Machine-Checked Theory of Floating Point Arithmetic John Harrison Intel Corporation, EY2-03

Deluxe Studio Suites Anemi DELUXE Studios Our Deluxe Studio suites have been carefully designed

Floating Point CSE 238/2038/2138: Systems Programming Instructor: Fatma CORUT ERGN Slides

Chapter 1 Linear Programming Paragraph 7 Standard Formats MPS, LP, and the CPLEX callable

Lecture 3 Floating Point Representations 1 Floating-point arithmetic We often incur