Proving Equivalence Between Imperative and MapReduce Implementations - PowerPoint PPT Presentation

Proving Equivalence Between Imperative and MapReduce Implementations Using Program Transformations VPT 2018 Thessaloniki 20 April 2018 Bernhard Beckert, Timo Bingman, Moritz Kiefer, Peter Sanders, Mattias Ulbrich , Alexander Weigl www.kit.edu KIT – Die Forschungsuniversit¨ at in der Helmholtz-Gemeinschaft

Motivation Simple algorithm implementations are understandable , . . . Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 2/27

Motivation Simple algorithm implementations are understandable , . . . . . . but not efficient. Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 2/27

Motivation Simple algorithm implementations are understandable , . . . . . . but not efficient. Efficient implementations are often complex , error-prone Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 2/27

Motivation Simple algorithm implementations are understandable , . . . . . . but not efficient. Efficient implementations are often complex , error-prone = ⇒ Prove equivalence between reference and efficient implementation Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 2/27

Motivation Simple algorithm implementations are understandable , . . . . . . but not efficient. Efficient implementations are often complex , error-prone = ⇒ Prove equivalence between reference and efficient implementation Verify MapReduce against imperative reference implementation. Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 2/27

Motivation Simple algorithm implementations are understandable , . . . . . . but not efficient. Efficient implementations are often complex , error-prone = ⇒ Prove equivalence between reference and efficient implementation Verify MapReduce against imperative reference implementation. Challenge for relational reasoning Programs not (necessarily) structurally close Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 2/27

Idea Combine Rewriting and Relational Reasoning ... and be open to automation Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 3/27

MapReduce distributed programming framework / paradigm first used large scale by google using concepts from functional programming to allow implicit parallisation. algorithms are quite different to their IMP counterparts Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 4/27

Recap: MapReduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 5/27

Recap: MapReduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . MAP 3 1 4 1 5 9 2 6 5 3 5 9 0 Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 5/27

Recap: MapReduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . MAP 3 1 4 1 5 9 2 6 5 3 5 9 0 3 9 1 4 6 5 1 3 5 2 0 9 5 Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 5/27

Recap: MapReduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . MAP 3 1 4 1 5 9 2 6 5 3 5 9 0 3 9 1 4 6 5 1 3 5 REDUCE 2 0 9 5 Σ Σ Σ 15 18 20 Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 5/27

Recap: MapReduce Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 6/27

Recap: MapReduce MAP Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 6/27

Recap: MapReduce MAP SHUFFLE REDUCE Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 6/27

Assumption The results produced by used reducers do not depend on the order in the array. Then we can consider the deterministic non-distributed setting. [Commutativity of Reducers, Chen et al. 2016] Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 7/27

Our Approach Imperative MapReduce algorithm algorithm Goal: Imperative ∼ = MapReduce Algorithm Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 8/27

Our Approach . . . Imperative intermed. intermed. intermed. MapReduce B B algorithm 1 2 algorithm n Goal: Imperative ∼ = MapReduce Algorithm User-provided intermediate steps that guide the proof. Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 8/27

Our Approach . . . Imperative intermed. intermed. intermed. MapReduce B B algorithm 1 2 algorithm n translates to λ 1 λ 4 λ 2 λ 3 λ 5 Goal: Imperative ∼ = MapReduce Algorithm User-provided intermediate steps that guide the proof. Translate into equivalent functional expressions Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 8/27

Our Approach . . . Imperative intermed. intermed. intermed. MapReduce B B algorithm 1 2 algorithm n translates to ∼ ∼ ∼ ∼ λ 1 λ 4 = λ 2 = λ 3 = = λ 5 Goal: Imperative ∼ = MapReduce Algorithm User-provided intermediate steps that guide the proof. Translate into equivalent functional expressions Prove equivalences there Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 8/27

Our Approach . . . Imperative intermed. intermed. intermed. MapReduce ∼ ∼ ∼ ∼ = = B = B = algorithm 1 2 algorithm n translates implies to ∼ ∼ ∼ ∼ λ 1 λ 4 = λ 2 = λ 3 = = λ 5 Goal: Imperative ∼ = MapReduce Algorithm User-provided intermediate steps that guide the proof. Translate into equivalent functional expressions Prove equivalences there Hence, equivalence on the original programs Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 8/27

Our Approach Imperative Interaction Language . . . Imperative intermed. intermed. intermed. MapReduce ∼ ∼ ∼ ∼ = = B = B = algorithm 1 2 algorithm n translates implies to ∼ ∼ ∼ ∼ λ 1 λ 4 = λ 2 = λ 3 = = λ 5 Formalized Functional Language Goal: Imperative ∼ = MapReduce Algorithm User-provided intermediate steps that guide the proof. Translate into equivalent functional expressions Prove equivalences there Hence, equivalence on the original programs Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 8/27

Languages Interaction (IL) Formalised Functional (FFL) Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 9/27

Languages Interaction (IL) Formalised Functional (FFL) typed λ calculus (+ µ ) while Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 9/27

Languages Interaction (IL) Formalised Functional (FFL) typed λ calculus (+ µ ) while integers, rationals, sequences (arrays) Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 9/27

Languages Interaction (IL) Formalised Functional (FFL) typed λ calculus (+ µ ) while integers, rationals, sequences (arrays) P(xs) { sum := 0 λ xs . fold( λ sum .λ i . sum + xs [ i ]) for i := 0..n 0 range(0 , n ) sum := sum + xs[i] return sum; } Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 9/27

Languages Interaction (IL) Formalised Functional (FFL) typed λ calculus (+ µ ) while integers, rationals, sequences (arrays) P(xs) { sum := 0 λ xs . fold( λ sum .λ i . sum + xs [ i ]) for i := 0..n 0 range(0 , n ) sum := sum + xs[i] return sum; } semantics by translation to FFL Big steps semantics ⇒ bs Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 9/27

Languages Interaction (IL) Formalised Functional (FFL) typed λ calculus (+ µ ) while integers, rationals, sequences (arrays) P(xs) { sum := 0 λ xs . fold( λ sum .λ i . sum + xs [ i ]) for i := 0..n 0 range(0 , n ) sum := sum + xs[i] return sum; } semantics by translation to FFL Big steps semantics ⇒ bs Implemented in Coq Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 9/27

Languages Interaction (IL) Formalised Functional (FFL) typed λ calculus (+ µ ) while integers, rationals, sequences (arrays) P(xs) { sum := 0 λ xs . fold( λ sum .λ i . sum + xs [ i ]) for i := 0..n 0 range(0 , n ) sum := sum + xs[i] return sum; } semantics by translation to FFL Big steps semantics ⇒ bs Implemented in Coq ∀ xs v . ( P xs ) ⇒ bs v ↔ ( P ′ xs ) ⇒ bs v ∀ xs . P ( xs ) == P’ ( xs ) Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 9/27

Two Types of Rules Context-Independent Rules local and uniform rewriting rules on subexpressions for paradigm shifting : (e.g., from loop to map) Context-Dependent Rules (more) global and flexible relational reasoning using coupling predicates maintaining control structure , adapt data Mattias Ulbrich – Equivalence Between Imperative and MapReduce Implementations 10/27