Optimal Broadcasting Strategies for Conjunctive Queries over - PowerPoint PPT Presentation

Optimal Broadcasting Strategies for Conjunctive Queries over Distributed Data Bas Ketsman, Frank Neven Hasselt University & transnational University of Limburg

Outline 1. Setting and Context 2. Oblivious Broadcasting Functions 3. Correctness & Optimality 4. Broadcast Dependency Sets 5. Conclusion & Future Work 2

True [Ameloot, Neven, Van den Bussche, 2011] Generalization [Ameloot, Neven, K., Zinn, 2014] Context CALM conjecture: “Monotonic = No-coordination” [Hellerstein, 2010] 3

Generalization [Ameloot, Neven, K., Zinn, 2014] Context CALM conjecture: “Monotonic = No-coordination” [Hellerstein, 2010] ▶ True [Ameloot, Neven, Van den Bussche, 2011] 3

Context CALM conjecture: “Monotonic = No-coordination” [Hellerstein, 2010] ▶ True [Ameloot, Neven, Van den Bussche, 2011] ▶ Generalization [Ameloot, Neven, K., Zinn, 2014] 3

Computing nodes Setting A network N is a set of computing nodes. 4

Setting A distribution is a mapping from nodes onto instances. ComplainsAbout ( d , e ) WorksFor ( d , e ) WorksFor ( a , b ) LivesIn ( b , c ) ComplainsAbout ( g , h ) WorksFor ( a , a ) ComplainsAbout ( a , b ) LivesIn ( e , f ) 5

Setting Communication: Asynchronous 6

Let every node broadcast all of its data; Periodically run locally on every node Monotonic ⊆ No-coordination Running Example: Q ( x , y , z ) ← ComplainsAbout ( x , y ) , WorksFor ( x , y ) , LivesIn ( y , z ) ComplainsAbout ( d , e ) WorksFor ( d , e ) WorksFor ( a , b ) LivesIn ( b , c ) ComplainsAbout ( g , h ) WorksFor ( a , a ) ComplainsAbout ( a , b ) LivesIn ( e , f ) 7

Periodically run locally on every node Monotonic ⊆ No-coordination Running Example: Q ( x , y , z ) ← ComplainsAbout ( x , y ) , WorksFor ( x , y ) , LivesIn ( y , z ) ComplainsAbout ( d , e ) WorksFor ( d , e ) WorksFor ( a , b ) LivesIn ( b , c ) ComplainsAbout ( g , h ) WorksFor ( a , a ) ComplainsAbout ( a , b ) LivesIn ( e , f ) ▶ Let every node broadcast all of its data; 7

Monotonic ⊆ No-coordination Running Example: Q ( x , y , z ) ← ComplainsAbout ( x , y ) , WorksFor ( x , y ) , LivesIn ( y , z ) ComplainsAbout ( d , e ) WorksFor ( d , e ) WorksFor ( a , b ) LivesIn ( b , c ) ComplainsAbout ( g , h ) WorksFor ( a , a ) ComplainsAbout ( a , b ) LivesIn ( e , f ) ▶ Let every node broadcast all of its data; ▶ Periodically run Q locally on every node 7

[…] […] Monotonic ⊆ No-coordination Running Example: Q ( x , y , z ) ← ComplainsAbout ( x , y ) , WorksFor ( x , y ) , LivesIn ( y , z ) ComplainsAbout ( d , e ) WorksFor ( d , e ) ComplainsAbout ( a , b ) WorksFor ( a , b ) WorksFor ( a , b ) LivesIn ( b , c ) ComplainsAbout ( g , h ) WorksFor ( a , a ) ComplainsAbout ( d , e ) WorksFor ( d , e ) ComplainsAbout ( a , b ) LivesIn ( e , f ) LivesIn ( e , f ) ▶ Let every node broadcast all of its data; ▶ Periodically run Q locally on every node 7

[…] […] Monotonic ⊆ No-coordination Running Example: Q ( x , y , z ) ← ComplainsAbout ( x , y ) , WorksFor ( x , y ) , LivesIn ( y , z ) ( a , b , c ) ( a , b , c ) ComplainsAbout ( d , e ) WorksFor ( d , e ) ComplainsAbout ( a , b ) WorksFor ( a , b ) WorksFor ( a , b ) LivesIn ( b , c ) ( d , e , f ) ComplainsAbout ( g , h ) WorksFor ( a , a ) ComplainsAbout ( d , e ) WorksFor ( d , e ) ComplainsAbout ( a , b ) LivesIn ( e , f ) LivesIn ( e , f ) ( d , e , f ) ( a , b , c ) ( d , e , f ) ▶ Let every node broadcast all of its data; ▶ Periodically run Q locally on every node 7

[…] […] Monotonic ⊆ No-coordination Running Example: Q ( x , y , z ) ← ComplainsAbout ( x , y ) , WorksFor ( x , y ) , LivesIn ( y , z ) ComplainsAbout ( d , e ) WorksFor ( d , e ) ComplainsAbout ( a , b ) g WorksFor ( a , b ) n WorksFor ( a , b ) i t s a LivesIn ( b , c ) c d a ComplainsAbout ( g , h ) o r B WorksFor ( a , a ) e v ComplainsAbout ( d , e ) i a N WorksFor ( d , e ) ComplainsAbout ( a , b ) LivesIn ( e , f ) LivesIn ( e , f ) ▶ Let every node broadcast all of its data; ▶ Periodically run Q locally on every node 7

Full CQs without self-joins 1. 2. 3. x y R y x x y z R x y S y z R x y x y R x y S y z Current Work No-coordination + Broadcast all No-coordination + Selective broadcasting 8

Current Work No-coordination + Broadcast all No-coordination + Selective broadcasting Full CQs without self-joins 1. Q ( x , y , z ) ← R ( x , y ) , S ( y , z ) 2. Q ( x , y ) ← R ( x , y ) , S ( y , z ) 3. Q ( x , y ) ← R ( x , y ) , R ( y , x ) 8

Outline 1. Setting and Context 2. Oblivious Broadcasting Functions 3. Correctness & Optimality 4. Broadcast Dependency Sets 5. Conclusion & Future Work 9

f f f Oblivious Broadcasting Functions Definition Let f be a total function from instances to instances. We call f an oblivious broadcasting function (OBF) if f is generic, and f ( I ) ⊆ I for every instance I . Q Q Q 10

Running Example: Naive Broadcasting Q ( x , y , z ) ← ComplainsAbout ( x , y ) , WorksFor ( x , y ) , LivesIn ( y , z ) OBF: broadcast everything ComplainsAbout ( d , e ) WorksFor ( d , e ) WorksFor ( a , b ) LivesIn ( b , c ) ComplainsAbout ( g , h ) WorksFor ( a , a ) ComplainsAbout ( a , b ) LivesIn ( e , f ) 11

Running Example: Relation-Based Q ( x , y , z ) ← ComplainsAbout ( x , y ) , WorksFor ( x , y ) , LivesIn ( y , z ) OBF: don’t broadcast ComplainsAbout ( x , y ) ComplainsAbout ( d , e ) WorksFor ( d , e ) WorksFor ( a , b ) LivesIn ( b , c ) ComplainsAbout ( g , h ) WorksFor ( a , a ) ComplainsAbout ( a , b ) LivesIn ( e , f ) 12

Running Example: Relation-Based Q ( x , y , z ) ← ComplainsAbout ( x , y ) , WorksFor ( x , y ) , LivesIn ( y , z ) OBF: don’t broadcast ComplainsAbout ( x , y ) ComplainsAbout ( d , e ) WorksFor ( d , e ) LivesIn ( e , f ) WorksFor ( a , b ) WorksFor ( a , b ) LivesIn ( b , c ) LivesIn ( b , c ) WorksFor ( a , a ) ComplainsAbout ( g , h ) WorksFor ( a , a ) WorksFor ( d , e ) ComplainsAbout ( a , b ) WorksFor ( a , b ) LivesIn ( e , f ) LivesIn ( b , c ) LivesIn ( e , f ) WorksFor ( a , a ) WorksFor ( d , e ) 12

Running Example: Relation-Based Q ( x , y , z ) ← ComplainsAbout ( x , y ) , WorksFor ( x , y ) , LivesIn ( y , z ) OBF: don’t broadcast ComplainsAbout ( x , y ) ( d , e , f ) ComplainsAbout ( d , e ) WorksFor ( d , e ) LivesIn ( e , f ) WorksFor ( a , b ) WorksFor ( a , b ) LivesIn ( b , c ) LivesIn ( b , c ) WorksFor ( a , a ) ComplainsAbout ( g , h ) WorksFor ( a , a ) WorksFor ( d , e ) ComplainsAbout ( a , b ) LivesIn ( e , f ) WorksFor ( a , b ) LivesIn ( b , c ) LivesIn ( e , f ) ( a , b , c ) WorksFor ( a , a ) WorksFor ( d , e ) 12

Distributed Output def f H B f H H c c eval Oblivious Broadcasting Functions Let N be a network, I an instance, H a distribution of I over N . Definition Let f be a total function from instances to instances. We call f an oblivious broadcasting function (OBF) if f is generic, and f ( I ) ⊆ I for every instance I . Broadcast Facts def ∪ B ( f , H ) = f ( H ( c )) . c ∈N 13

Oblivious Broadcasting Functions Let N be a network, I an instance, H a distribution of I over N . Definition Let f be a total function from instances to instances. We call f an oblivious broadcasting function (OBF) if f is generic, and f ( I ) ⊆ I for every instance I . Broadcast Facts def ∪ B ( f , H ) = f ( H ( c )) . c ∈N Distributed Output def ∪ eval ( Q , f , H ) = Q ( H ( c ) ∪ B ( f , H )) c ∈N 13

Outline 1. Setting and Context 2. Oblivious Broadcasting Functions 3. Correctness & Optimality 4. Broadcast Dependency Sets 5. Conclusion & Future Work 14

Broadcast enough f f f Correctness Definition An OBF is correct for CQ Q if Q ( I ) = eval ( Q , f , H ) for every instance I and distribution H for I . 15

f f f Correctness Definition An OBF is correct for CQ Q if Q ( I ) = eval ( Q , f , H ) for every instance I and distribution H for I . Q Broadcast enough Q Q 15

Correctness Definition Let Q be a CQ, f and g be two distinct facts. We say that f and g are compatible, written f ∼ Q g , if there is a valuation V for Q that requires them both. 16

Correctness Definition Let Q be a CQ, f and g be two distinct facts. We say that f and g are compatible, written f ∼ Q g , if there is a valuation V for Q that requires them both. Example Q ( x , y , z ) ← ComplainsAbout ( x , y ) , WorksFor ( x , y ) , LivesIn ( y , z ) ▶ ComplainsAbout ( a , b ) ∼ Q LivesIn ( b , c ) ▶ ComplainsAbout ( a , b ) ̸∼ Q LivesIn ( a , c ) 16

Correctness Definition Let Q be a CQ, f and g be two distinct facts. We say that f and g are compatible, written f ∼ Q g , if there is a valuation V for Q that requires them both. Lemma Let Q be a CQ and f be an OBF. Then, the following are equivalent: 1. f is correct for Q ; and 2. there are no instances I , J , and facts f , g , with f ∼ Q g , g ̸∈ I , f ̸∈ J such that f ̸∈ f ( I ∪ { f } ) and g ̸∈ f ( J ∪ { g } ) . 16

def Definition An OBF f for a CQ is optimal if for every other OBF g for and for every instance I and distribution H . No such OBF exists B f H c f H c B f H B g H Optimality Ideally: “One OBF that is always at least as good as all others” 17