Pavel Alex James Zach Panchekha Sanchez-Stern Wilcox Tatlock - PowerPoint PPT Presentation

Automatically Improving Accuracy for Floating Point Expressions error ⇒ Pavel Alex James Zach Panchekha Sanchez-Stern Wilcox Tatlock

Floating Point’s Wild Success

Floating Point’s Wild Success F ≈ R Often floating point is close to real arithmetic But not always !

Floating Point’s Wild Success But not always ! Numerous articles retracted [Altman ’99, ’03] Financial regulations [Euro ’98] Market distortions [McCullough ’99, Quinn ’83]

Rounding Error in Sculpture Blake Courter @bcourter

Rounding Error in Sculpture Blake Courter @bcourter Rounding error ⇒

Rounding Error in Sculpture Blake Courter @bcourter Rounding error ⇒

Existing options Big Float - Unreliable + More Reliable + Reliable - Slow Code + Fast Code + Fast Code - Expert Task

e’ e Heuristic search to find expert transformations

e’ e Heuristic search to find expert transformations Worked Example How Herbie Works Evaluation

e’ e e R e F Heuristic search to find expert transformations Worked Example How Herbie Works Evaluation

Rounding Error in Quadratic √ b 2 − 4 ac − b + 2 a 64 error

Rounding Error in Quadratic √ b 2 − 4 ac − b + 2 a What is rounding error? exact computed J e K R J e K F 64                error 7 ULPs

Rounding Error in Quadratic √ b 2 − 4 ac − b + 2 a What is rounding error? exact computed J e K R J e K F 64 log(ULPs)                7 ULPs log(ULPs) estimates # of incorrect bits

Rounding Error in Quadratic √ b 2 − 4 ac − b + 2 a 64 log(ULPs) b

Rounding Error in Quadratic  c b − b if b ∈ A a     √ b 2 − 4 ac  − b + √ if b ∈ B  b 2 − 4 ac − b +  2 a  ⇒ 2 c 2 a if b ∈ C √  b 2 − 4 ac − b −     − c if b ∈ D   b  A B C D 64 log(ULPs) b

Rounding Error in Quadratic  c b − b A if b ∈ A a     √ b 2 − 4 ac  − b + √ if b ∈ B  b 2 − 4 ac − b +  2 a  ⇒ 2 c 2 a if b ∈ C √  b 2 − 4 ac − b −     − c if b ∈ D   b  A B C D 64 log(ULPs) Overflow J b 2 K F b If is large, overflows and the the whole expression returns . ∞ b

Rounding Error in Quadratic  c b − b A if b ∈ A a     √ b 2 − 4 ac  − b + B √ if b ∈ B  b 2 − 4 ac − b +  2 a  ⇒ 2 c 2 a if b ∈ C √  b 2 − 4 ac − b −     − c if b ∈ D   b  Pretty Accurate A B C D 64 log(ULPs) b

Rounding Error in Quadratic  b − b c A if b ∈ A a     √ b 2 − 4 ac  − b + B √ if b ∈ B  b 2 − 4 ac − b +  2 a  ⇒ 2 c 2 a C if b ∈ C √  b 2 − 4 ac − b −     − c if b ∈ D   b  Catastrophic Cancellation A B C D 64 log(ULPs) If is large, but and are small, b a c p and the difference b 2 − 4 ac b ≈ is rounded off. b

Rounding Error in Quadratic  b − b c A if b ∈ A a     √ b 2 − 4 ac  − b + B √ if b ∈ B  b 2 − 4 ac − b +  2 a  ⇒ 2 c 2 a C if b ∈ C √  b 2 − 4 ac − b −     − c if b ∈ D D   b  Overflow again A B C D 64 log(ULPs) b

Rounding Error in Quadratic  b − b c A if b ∈ A a     √ b 2 − 4 ac  − b + B √ if b ∈ B  b 2 − 4 ac − b +  2 a  ⇒ 2 c 2 a C if b ∈ C √  b 2 − 4 ac − b −     − c if b ∈ D D   b  A B C D 64 64 log(ULPs) ⇒ b b

e’ e e R e F Heuristic search to find expert transformations Worked Example How Herbie Works Evaluation

e’ e e R e F Heuristic search to find expert transformations Worked Example How Herbie Works Evaluation

Herbie Architecture regimes sample cands e’ e e R e F focus generate more candidates

Herbie Architecture Ground truth regimes sample cands e’ e e R e F focus generate more candidates

Herbie Architecture regimes sample cands e’ e e R e F focus Localize error generate more candidates

Herbie Architecture regimes sample cands e’ e e R e F focus generate more Heuristic search candidates

Herbie Architecture regimes sample cands e’ e e R e F focus generate Keep all good more candidates candidates

Herbie Architecture Combine candidates sample regimes cands e’ e e R e F focus generate more candidates

Herbie Architecture Ground truth regimes sample cands e’ e e R e F focus generate more candidates

Determine ground truth 64 random bits X = sample ( domain ( e )) e.g. X = { 1 . 2 · 10 − 17 , − 3 . 8 · 10 204 , 173 . 5 , . . . } Round ( J e K R ( X )) J e K F ( X ) Compute in F Get 64-bit prefix with MPFR. error Subtle ! See paper. e.g. { 13 . 2 b , 51 . 7 b , 1 b , . . . }

Herbie Architecture Ground truth regimes sample cands e’ e e R e F focus generate more candidates

Herbie Architecture regimes sample cands e’ e e R e F focus Localize error generate more candidates

Focus: Estimate Error Source Overflow Cancellation 1. For each op in f e √ 2. Evaluate args in R b 2 − 4 ac − b + 3. Apply to them f R 2 a 4. Apply to them f F 5. Compare + F + R Round ( x + R y ) ( x + F y ) + K R K R J J p − b y = b 2 − 4 ac x =

Herbie Architecture regimes sample cands e’ e e R e F focus Localize error generate more candidates

Herbie Architecture regimes sample cands e’ e e R e F focus generate more Heuristic search candidates

Herbie Architecture regimes sample cands e’ e e R e F focus rewrite Create candidates series simplify

Apply rewrites to √ b 2 − 4 ac Recursive rewrites: − b + - Database of rules 2 a - Flexible - Chains of rewrites No cancellation − x 0 − x in denominator x + y x 2 − y 2 √ ! ( − b ) 2 − ( b 2 − 4 ac ) 2 / 2 a √ − b − b 2 − 4 ac x − y ( x − y ) + z √ √ � � (0 − b ) + b 2 − 4 ac b 2 − 4 ac b − 0 − x − ( y − z ) 2 a 2 a … 120 more … Rule DB

Herbie Architecture regimes sample cands e’ e e R e F focus rewrite Create candidates series simplify

Herbie Architecture regimes sample cands e’ e e R e F focus rewrite Approximate expr series simplify

Series Expansions Idea: near -identities √ b 2 − 4 ac − b + √ 2 a 1 − x ≈ 1 − x/ 2 (for ) x ≈ 0 − b + b (1 − 4 ac/ 2 b 2 ) 2 a Bounded Laurent series: - Transcendental functions - Singularities - Number of terms to take

Herbie Architecture regimes sample cands e’ e e R e F focus rewrite Approximate expr series simplify

Herbie Architecture regimes sample cands e’ e e R e F focus rewrite series Cancel & clean up simplify

Simplify Expressions √ ! ( − b ) 2 − ( b 2 − 4 ac ) 2 Difficult! [Caviness ’70] / 2 a √ − b − b 2 − 4 ac - many possible rewrites - huge search space √ ! b 2 − ( b 2 − 4 ac ) 2 = / 2 a - avoid undoing progress! √ − b − b 2 − 4 ac ✓ b 2 − ( b 2 − 4 ac ) ◆ = / 2 a √ − b − b 2 − 4 ac E-graphs [Nelson ’79] - Terminate early ✓ ◆ 4 ac / 2 a = - Prune useless nodes √ − b − b 2 − 4 ac - Restrict rewrites 2 c = √ b 2 − 4 ac − b −

Herbie Architecture regimes sample cands e’ e e R e F focus rewrite series simplify Cancel & clean up

Herbie Architecture regimes sample cands e’ e e R e F focus rewrite Keep all good series candidates simplify

Herbie Architecture Combine candidates sample regimes cands e’ e e R e F focus rewrite series simplify

Regime Inference √ c b − b − c 2 c b 2 − 4 ac − b + √ a b b 2 − 4 ac 2 a − b −

Regime Inference  c b − b if b ∈ ( ∞ , − 1 . 15 E 122] Dynamic programming:  a    √ - Bounds quickly b 2 − 4 ac  − b + if b ∈ ( − 1 . 125 E 122 , 1 . 06 E − 304]   2 a  - Tune: binary search 2 c if b ∈ (1 . 06 E − 304 , 4 . 62 E 63] √  b 2 − 4 ac − b − - Pick best variable     − c if b ∈ (4 . 62 E 63 , ∞ )   b  A B C D

Herbie Architecture Combine candidates sample regimes cands e’ e e R e F focus rewrite series simplify

e’ e e R e F Heuristic search to find expert transformations Worked Example How Herbie Works Evaluation

e’ e e R e F Heuristic search to find expert transformations Worked Example How Herbie Works Evaluation

Evaluating Herbie A. Does accuracy improve? B. Does it reproduce expert transformations? C. Is the output code fast? D. Does it work in the real world?