Probabilistic Rounding Error Analysis for Sums Eric Hallman North - PowerPoint PPT Presentation

Probabilistic Rounding Error Analysis for Sums Eric Hallman North Carolina State University Spring 2020 Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 1 / 32

Introduction The problem: for x 1 , . . . , x n ∈ R , compute n � s n := x i i =1 Compute the sum using recursive summation: s 1 := x 1 , ˆ s i := fl(ˆ ˆ s i − 1 + x i ) , i = 2 : n . The goal: bound the error | ˆ s n − s n | . Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 2 / 32

Introduction Classical model for floating-point arithmetic: Model (Classical) For any floating point numbers a and b, ⊙ ∈ { + , − , × , /, √} , fl( a ⊙ b ) = ( a ⊙ b )(1 + δ ) , | δ | ≤ u , where u is the unit roundoff. Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 3 / 32

Outline Background 1 Deterministic Error Bounds Probabilistic Error Analysis Intermediate Sums 2 New Bounds 3 Experiments/Conclusions 4 Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 4 / 32

Deterministic Bound Computing the partial sums: s 1 = x 1 , ˆ s 2 = (ˆ ˆ s 1 + x 2 )(1 + δ 2 ) = x 1 (1 + δ 2 ) + x 2 (1 + δ 2 ) , s 3 = (ˆ ˆ s 2 + x 3 )(1 + δ 3 ) = x 1 (1 + δ 2 )(1 + δ 3 ) + x 2 (1 + δ 2 )(1 + δ 3 ) + x 3 (1 + δ 3 ) , . . .   n n � � s n = ˆ x i (1 + δ j )   i =1 j =max { 2 , i } Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 5 / 32

Deterministic Bound Lemma (Deterministic Error Bound) If | δ i | ≤ u for i = 1 : n, and nu < 1 , then n � (1 + δ i ) = 1 + θ n , i =1 where nu | θ n | ≤ 1 − nu =: γ n . Consequently, n � | ˆ s n − s n | ≤ γ n | x i | . i =1 γ n ≈ nu as long as nu ≪ 1. Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 6 / 32

Deterministic Bound Higham (2002): Whenever we write γ n there is an implicit assumption that nu < 1 , which is true in virtually any circumstance that might arise with IEEE single or double precision arithmetic. Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 7 / 32

Low-Precision Arithmetic Normalized non-zero floating point numbers: x = ( − 1) s (1 . f ) 2 · 2 e Precision Sign Exp Float u 1 . 11 · 10 − 16 Double (f64) 1 11 52 5 . 96 · 10 − 8 Single (f32) 1 8 23 4 . 88 · 10 − 4 Half (f16) 1 5 10 3 . 13 · 10 − 2 Quarter (??) 1 3 4 Problem sizes getting larger Half precision increasingly common When nu > 1, bounds using γ n become useless Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 8 / 32

Outline Background 1 Deterministic Error Bounds Probabilistic Error Analysis Intermediate Sums 2 New Bounds 3 Experiments/Conclusions 4 Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 9 / 32

Probabilistic Model Solution: abandon worst-case bounds, try probabilistic analysis instead Model (Probabilistic) For any floating point numbers a and b, ⊙ ∈ { + , − , × , /, √} , fl( a ⊙ b ) = ( a ⊙ b )(1 + δ ) , | δ | ≤ u , where u is the unit roundoff. The quantities δ for each computation are independent random variables with mean zero. This model is wrong. But is it useful ? Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 10 / 32

Probabilistic Model Central Limit Theorem: if e = � n i =1 δ i t i and | δ i | ≤ u , then � n � 1 / 2 � t 2 | e | ≤ λ u i i =1 with high probability for large n . λ modest in size, controls probability i =1 | t i | can be factor of √ n larger Deterministic bound u � n Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 11 / 32

Probabilistic Model Wilkinson (1961): In general, the statistical distribution of the rounding errors will reduce considerably the function of n occurring in the relative errors. We might expect in each case that this function should be replaced by something which is no bigger than its square root and is usually appreciably smaller. Ultimate goal: replace nu in error bounds with something that grows like √ nu Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 12 / 32

Probabilistic Bound Theorem (Hoeffding’s Inequality) Let X 1 , . . . , X n be independent random variables satisfying | X i | ≤ c i , i = 1 : n . The sum S = � n i =1 X i satisfies ξ 2 � � Pr ( | S − E [ S ] | ≥ ξ ) ≤ 2 exp − . 2 � n i =1 c 2 i Assumptions: random variables are independent and bounded . Holds for all n , not just as n → ∞ Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 13 / 32

Probabilistic Bound Lemma (Higham/Mary 2018) Assume the probabilistic model for roundoff errors. If | δ i | ≤ u for i = 1 : n, and nu < 1 , then n � (1 + δ i ) = 1 + ˜ θ n , i =1 where λ √ nu + nu 2 − 1 ≈ λ √ nu � � | ˜ θ n | ≤ ˜ γ n ( λ ) := exp 1 − u with failure probability at most − λ 2 (1 − u ) 2 � � Q ( λ ) = 2 exp . 2 Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 14 / 32

Probabilistic Bound Deterministic bound: n n � � | ˆ s n − s n | ≤ γ n | x i | ≈ nu | x i | . i =1 i =1 Probabilistic bound: WFP at most Q ( λ ), n n | x i | ≈ λ √ nu � � | ˆ s n − s n | ≤ ˜ γ n ( λ ) | x i | . i =1 i =1 Probabilistic approximation holds while λ √ nu ≪ 1 Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 15 / 32

Probabilistic Bound How does the Higham/Mary bound perform in practice? Much stronger than deterministic bound Slightly pessimistic on random [0 , 1] data Way off the mark on random [ − 1 , 1] data Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 16 / 32

Running Error Bound Idea (Higham 2002): rewrite the computed sums as s i = ˆ s i − 1 + x i ˆ , | δ i | ≤ u . 1 + δ i Result: the deterministic running error bound � � n n � � � � | ˆ s n − s n | = δ i ˆ s i � ≤ u | ˆ s i | � � � � � i =1 i =1 This can be much smaller than the a priori bounds! Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 17 / 32

Running Error Bound Higham 2002: In designing or choosing a summation method to achieve high accuracy, the aim should be to minimize the absolute values of the intermediate sums [ ˆ s i ]. Our goal: find the probabilistic version of the running error bound Problem: Quantities ˆ s i depend on δ terms Difficult to apply concentration bounds directly to � n i =1 δ i ˆ s i Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 18 / 32

Intermediate Sums Our strategy: rewrite error as   n n � �  . s n − s n = ˆ  s i δ i (1 + δ j ) i =2 j = i +1 Deterministic bound: n � | ˆ s n − s n | ≤ u (1 + γ n ) | s i | . i =2 Two small wrinkles in getting the probabilistic version: � n δ i independent � δ i j = i +1 (1 + δ j ) independent The term (1 + γ n ) is a problem when nu > 1 Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 19 / 32

Martingales Use a martingale! Definition (Martingale) A squence of random variables X 1 , X 2 , . . . is a martingale with respect to δ 1 , δ 2 , . . . if for i ≥ 1 X i is a function of δ 1 , . . . , δ i − 1 , 1 E [ | X i | ] < ∞ 2 E [ X i +1 | δ 1 , . . . , δ i − 1 ] = X i . 3 Examples: unbiased random walk, gambler playing a fair game The increments ( X i +1 − X i ) do not need to be independent! Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 20 / 32

Martingales We can relax the requirements of Hoeffding’s Inequality. Theorem (Azuma’s Inequality) Suppose a martingale { X 1 , . . . , X n } satisfies | X i − X i − 1 | ≤ c i , i = 2 : n . Then ξ 2 � � Pr ( | X n − X 1 | ≥ ξ ) ≤ 2 exp − . 2 � n i =2 c 2 i Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 21 / 32

Martingales To construct the martingale, work backwards: X 1 = s n , X 2 = X 1 + s n δ n , X 3 = X 2 + s n − 1 δ n − 1 (1 + δ n ) , . . . n � X n = X n − 1 + s 2 δ 2 (1 + δ j ) j =3 Apply Azuma’s inequality with c i = | s n − i +2 | u (1 + γ n ) , � n � 1 / 2 � s 2 ξ = λ u (1 + γ n ) i i =2 Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 22 / 32

Martingales Deterministic bound: n � | ˆ s n − s n | ≤ u (1 + γ n ) | s i | i =2 Probabilistic bound: � n � 1 / 2 � s 2 | ˆ s n − s n | ≤ λ u (1 + γ n ) i i =2 � � − λ 2 with failure probability at most 2 exp 2 Not good enough: (1 + γ n ) blows up when nu > 1! Eric Hallman (North Carolina State University) Probabilistic Rounding Error Analysis Spring 2020 23 / 32