On Sketching Quadratic Forms Bo Qin The Hong Kong University of - PowerPoint PPT Presentation

On Sketching Quadratic Forms Bo Qin The Hong Kong University of Science and Technology January 16, 2016 Joint with: Alexandr Andoni, Jiecao Chen, Robert Krauthgamer, David Woodruff and Qin Zhang Bo Qin On Sketching Quadratic Forms

Outline 1 Sketching Quadratic Forms in Two Models 2 Sketches for General and PSD Matrices 3 Sketches for Laplacian Matrices 4 Cut and Spectral sketches for Laplacians Bo Qin On Sketching Quadratic Forms

Sketching Quadratic Forms Given a matrix A ∈ R n × n , compute a sketch sk ( A ) , which suffices to estimate the quadratic form x T Ax for every query vector x ∈ R n . (1 + ǫ ) -approximation: Output ∈ (1 ± ǫ ) x T Ax Goal: Sketch sk ( A ) of small size Bo Qin On Sketching Quadratic Forms

Two Models “For all” model: sk ( A ) succeeds on all queries x simul- taneously “For each” model: for every fixed query x , the sketch succeeds with constant (or high) probability Bo Qin On Sketching Quadratic Forms

Outline 1 Sketching Quadratic Forms in Two Models 2 Sketches for General and PSD Matrices 3 Sketches for Laplacian Matrices 4 Cut and Spectral sketches for Laplacians Bo Qin On Sketching Quadratic Forms

Lower Bounds for General and PSD Matrices General and PSD Matrices: Any sketch sk ( A ) of A ∈ R n × n satisfying the “for all” guaran- teemust use Ω( n 2 ) bits of space • For General Matrices , Ω( n 2 ) bits is required even in the “for each” model Bo Qin On Sketching Quadratic Forms

PSD Matrices: “For Each” Model • For a positive semidefinite (PSD) matrix A , ∀ x ∈ R n , x T Ax = || A 1 / 2 x || 2 (If A is a PSD matrix, A has the unique square root) Bo Qin On Sketching Quadratic Forms

PSD Matrices: “For Each” Model • For a positive semidefinite (PSD) matrix A , ∀ x ∈ R n , x T Ax = || A 1 / 2 x || 2 (If A is a PSD matrix, A has the unique square root) • Johnson-Lindenstrauss lemma: There exists a random ε − 2 × n matrix T of i.i.d. entries from {± ε } such that, for every fixed x ∈ R n , (1 − ε ) || A 1 / 2 x || 2 ≤ || TA 1 / 2 x || 2 ≤ (1 + ε ) || A 1 / 2 x || 2 , with probability at least 2 / 3 . Bo Qin On Sketching Quadratic Forms

PSD Matrices: “For Each” Model • For a positive semidefinite (PSD) matrix A , ∀ x ∈ R n , x T Ax = || A 1 / 2 x || 2 (If A is a PSD matrix, A has the unique square root) • Johnson-Lindenstrauss lemma: There exists a random ε − 2 × n matrix T of i.i.d. entries from {± ε } such that, for every fixed x ∈ R n , (1 − ε ) || A 1 / 2 x || 2 ≤ || TA 1 / 2 x || 2 ≤ (1 + ε ) || A 1 / 2 x || 2 , with probability at least 2 / 3 . • O ( n/ε 2 ) -size Sketch: sk ( A ) = TA 1 / 2 (It is optimal !) Bo Qin On Sketching Quadratic Forms

Main Results “for all” model “for each” model Matrix family upper bound lower bound upper bound lower bound ˜ ˜ O ( n 2 ) Ω( n 2 ) O ( n 2 ) Ω ( n 2 ) General ˜ ˜ O ( n 2 ) Ω ( n 2 ) O ( n ǫ − 2 ) Ω ( n ε − 2 ) PSD Table: Here, ˜ O ( f ) denotes f · (log f ) O (1) . Our results are in bold. Bo Qin On Sketching Quadratic Forms

Outline 1 Sketching Quadratic Forms in Two Models 2 Sketches for General and PSD Matrices 3 Sketches for Laplacian Matrices 4 Cut and Spectral sketches for Laplacians Bo Qin On Sketching Quadratic Forms

Sketching Quadratic Forms of Laplacians Laplacian: An important subclass of PSD matrices The Laplacian L G of a graph G = ( V, E ) is defined as L G = D − A, where D is the diagonal weighted degree matrix of G , and A is the weighted adjacency matrix of G . Bo Qin On Sketching Quadratic Forms

Sketching Quadratic Forms of Laplacians Laplacian: An important subclass of PSD matrices The Laplacian L G of a graph G = ( V, E ) is defined as L G = D − A, where D is the diagonal weighted degree matrix of G , and A is the weighted adjacency matrix of G . • Spectral query: x ∈ R n • Cut query: x ∈ { 0 , 1 } n , x T L G x = w ( S, V \ S ) where S ⊂ V satisfying each vertex u ∈ S iff x u = 1 Bo Qin On Sketching Quadratic Forms

Laplacians: “For All” Model Can one achieve smaller sketches for Laplacians? • Graph Sparsificaiton: BK96, ST04, ST11, SS11, FHHP11, KP12, BSS14, etc. Spectral Sparsifiers—Sketches for Laplacians in “For All” Model Select a reweighted subgraph H of O ( n/ε 2 ) edges ∀ x ∈ R n , x T L H x ∈ (1 ± ε ) x T L G x [BSS14]: O ( n/ε 2 ) edges is optimal! Bo Qin On Sketching Quadratic Forms

Laplacians: “For All” Model Can one achieve smaller sketches for Laplacians? • Graph Sparsificaiton: BK96, ST04, ST11, SS11, FHHP11, KP12, BSS14, etc. • Spectral Sparsifiers—Sketches for Laplacians in “For All” Model • Select a reweighted subgraph H of O ( n/ε 2 ) edges • ∀ x ∈ R n , x T L H x ∈ (1 ± ε ) x T L G x [BSS14]: O ( n/ε 2 ) edges is optimal! Bo Qin On Sketching Quadratic Forms

Laplacians: “For All” Model Can one achieve smaller sketches for Laplacians? • Graph Sparsificaiton: BK96, ST04, ST11, SS11, FHHP11, KP12, BSS14, etc. • Spectral Sparsifiers—Sketches for Laplacians in “For All” Model • Select a reweighted subgraph H of O ( n/ε 2 ) edges • ∀ x ∈ R n , x T L H x ∈ (1 ± ε ) x T L G x • [BSS14]: O ( n/ε 2 ) edges is optimal! Bo Qin On Sketching Quadratic Forms

Smaller Sketch? Sketches for Laplacians: • Arbitrary Data Structure: Beyond subgraphs? • Cut Queries: Can cut-sparsifiers be smaller than spectral- sparsifiers? • “For Each” Model: Smaller sketches than those in “for all” model? Bo Qin On Sketching Quadratic Forms

Laplacians: Lower Bound in “For All” Model Any Data Structure: Theorem (Informal) Any sketch sk ( A ) of A ∈ R n × n satisfying the “for all” guarantee must use Ω( n/ε 2 ) bits of space, even for cut queries. • The lower bound holds for cut queries (for spectral queries as well). Bo Qin On Sketching Quadratic Forms

Laplacians: Lower Bound in “For All” Model Previous bounds in the restricted versions: • [Alon97] Ω( n/ε 2 ) —The sparsifier H has regular degrees and uniform edge weights. • [BSS14] Ω( n/ε 2 ) — H is a spectral sparsifier. Bo Qin On Sketching Quadratic Forms

Laplacians: Lower Bound in “For All” Model Previous bounds in the restricted versions: • [Alon97] Ω( n/ε 2 ) —The sparsifier H has regular degrees and uniform edge weights. • [BSS14] Ω( n/ε 2 ) — H is a spectral sparsifier. � Our lower bound is the first lower bound without assump- tions! Bo Qin On Sketching Quadratic Forms

Main Results “for all” model “for each” model Matrix family upper bound lower bound upper bound lower bound ˜ ˜ O ( n 2 ) Ω( n 2 ) O ( n 2 ) Ω ( n 2 ) General ˜ ˜ O ( n 2 ) Ω ( n 2 ) O ( n ǫ − 2 ) Ω ( n ε − 2 ) PSD ˜ O ( nε − 2 ) [BSS14] Ω( nε − 2 ) [BSS14] Laplacian, SDD ˜ O ( nε − 2 ) [BSS14] Ω ( n ε − 2 ) Laplacian+cut Table: Here, ˜ O ( f ) denotes f · (log f ) O (1) . Our results are in bold. Bo Qin On Sketching Quadratic Forms

Laplacians: “For Each” Model We can do better in the “For Each” Model! Sketches for Laplacians: Cut Queries: Sketches of size ˜ O ( nε − 1 ) bits Spectral Queries: Sketches of size ˜ O ( nε − 1 . 6 ) bits Bo Qin On Sketching Quadratic Forms

Main Results “for all” model “for each” model Matrix family upper bound lower bound upper bound lower bound ˜ ˜ O ( n 2 ) Ω( n 2 ) O ( n 2 ) Ω ( n 2 ) General ˜ ˜ O ( n 2 ) Ω ( n 2 ) O ( n ǫ − 2 ) Ω ( n ε − 2 ) PSD ˜ O ( nε − 2 ) [BSS14] Ω( nε − 2 ) [BSS14] O ( n ε − 1 . 6 ) ˜ Ω ( n ε − 1 ) Laplacian, SDD ˜ ˜ O ( nε − 2 ) [BSS14] Ω ( n ε − 2 ) O ( n ε − 1 ) Ω ( n ε − 1 ) Laplacian+cut Table: Here, ˜ O ( f ) denotes f · (log f ) O (1) . Our results are in bold. Separates the “for each” and “for all” models for Lapla- cians! Bo Qin On Sketching Quadratic Forms

Main Results “for all” model “for each” model Matrix family upper bound lower bound upper bound lower bound ˜ ˜ O ( n 2 ) Ω( n 2 ) O ( n 2 ) Ω ( n 2 ) General ˜ ˜ O ( n 2 ) Ω ( n 2 ) O ( n ǫ − 2 ) Ω ( n ε − 2 ) PSD ˜ O ( nε − 2 ) [BSS14] Ω( nε − 2 ) [BSS14] O ( n ε − 1 . 6 ) ˜ Ω ( n ε − 1 ) Laplacian, SDD ˜ ˜ O ( nε − 2 ) [BSS14] Ω ( n ε − 2 ) O ( n ε − 1 ) Ω ( n ε − 1 ) Laplacian+cut Table: Here, ˜ O ( f ) denotes f · (log f ) O (1) . Our results are in bold. � Separates the “for each” and “for all” models for Lapla- cians! Bo Qin On Sketching Quadratic Forms

Outline 1 Sketching Quadratic Forms in Two Models 2 Sketches for General and PSD Matrices 3 Sketches for Laplacian Matrices 4 Cut and Spectral sketches for Laplacians Bo Qin On Sketching Quadratic Forms

Cut Sketches “For Each”—First Attempt Consider the complete graph • Standard sampling scheme: Sample edges with probability p e = 1 /ε 2 n • Smaller probability fails: Even for “singleton cuts”, w ( { u } , V \ { u } ) ≈ 1 ε 2 ± 1 ε • Singleton cuts are the “most difficult” for concentration Storing all vertex degrees—Only O ( n ) bits of space! Bo Qin On Sketching Quadratic Forms

Constructing Cut Sketches “For Each” Simple Graphs Assume an unweighted graph G = ( V, E ) satisfies min {| S | , | V \ S |} ≥ 1 w ( S, V \ S ) ∀ S ⊂ V, ε Sketch for Answering Cut Queries s.t. w ( S, V \ S ) ≤ 1 ε 2 Bo Qin On Sketching Quadratic Forms