Information Recovery from Pairwise Measurements A Shannon-Theoretic - PowerPoint PPT Presentation

Information Recovery from Pairwise Measurements A Shannon-Theoretic Approach Yuxin Chen † , Changho Suh ∗ , Andrea Goldsmith † Stanford University † KAIST ∗ Page 1

Recovering data from correlation measurements • A large collection of data instances • In many applications, it is ◦ difficult/infeasible to measure each variable directly ◦ feasible to measure pairwise correlation Page 2

Motivating application: multi-image alignment • Structure from motion : estimate 3D structures from 2D image sequences ◦ Key step: joint alignment – input: (noisy) estimates of relative camera poses – goal: jointly recover all camera poses Page 3

Motivating application: graph clustering • Real-world networks exhibit community structures ◦ input: pairwise similarities between members ◦ goal: uncover hidden clusters Page 4

This talk: recovery from pairwise difference measurements • Goal: recover a collection of variables { x i } • Can only measure several pairwise difference x i − x j (broadly defined) Page 5

This talk: recovery from pairwise difference measurements • Goal: recover a collection of variables { x i } • Can only measure several pairwise difference x i − x j (broadly defined) ◦ Examples: — joint alignment – x i : (angle θ i , position z i ) – relative rotation/translation ( θ i − θ j , z i − z j ) Page 5

This talk: recovery from pairwise difference measurements • Goal: recover a collection of variables { x i } • Can only measure several pairwise difference x i − x j (broadly defined) ◦ Examples: — joint alignment – x i : (angle θ i , position z i ) – relative rotation/translation ( θ i − θ j , z i − z j ) — graph partition – x i : membership (which partition it belongs to) � 1 , if i, j ∈ same partition – cluster agreement: x i − x j = 0 , else. Page 5

This talk: recovery from pairwise difference measurements • Goal: recover a collection of variables { x i } • Can only measure several pairwise difference x i − x j (broadly defined) ◦ Examples: — joint alignment – x i : (angle θ i , position z i ) – relative rotation/translation ( θ i − θ j , z i − z j ) — graph partition – x i : membership (which partition it belongs to) � 1 , if i, j ∈ same partition – cluster agreement: x i − x j = 0 , else. — pairwise maps, parity reads, ... Page 5

A fundamental-limit perspective? • A flurry of activity in recovery algorithm design convex program combinatorial spectral method Page 6

A fundamental-limit perspective? • A flurry of activity in recovery algorithm design convex program combinatorial spectral method ◦ What are the fundamental recovery limits? — min. sample complexity? how noisy the measurements can be? Page 6

A fundamental-limit perspective? • A flurry of activity in recovery algorithm design convex program combinatorial spectral method ◦ What are the fundamental recovery limits? — min. sample complexity? how noisy the measurements can be? • So far mostly studied in a model-specific manner ◦ Seek a more unified framework Page 6

Problem setup: a Shannon-theoretic framework x 1 x 3 x 2 x 4 y 26 y x i ∈ { 0 , · · · , M − 1 } x 5 x 6 x 7 • Information network ◦ n vertices ◦ discrete inputs w/ alphabet size: M — could scale with n Page 7

Problem setup: a Shannon-theoretic framework x 1 x 3 y 13 y 12 y 34 x 2 x 4 y 24 y 35 measurements of y 15 y 26 y 27 measurement graph G x 1 − x 2 , x 1 − x 3 , x 1 − x 5 , · · · x 5 x 6 y 67 x 7 • Pairwise difference measurements ◦ truth: x i − x j ◦ measurements: y ij (arbitrary alphabet) ∗ can be corrupted by noise, distortion, ... • Graphical representation ◦ observe y ij ⇐ ⇒ ( i, j ) ∈ G Page 7

Problem setup: a Shannon-theoretic framework p ( y ij | x i -x j ) x 1 - x 2 y 12 channel x 1 x 3 x 1 - x 5 y 15 channel x 2 x 4 x 5 x 6 x 2 - x 7 y 27 channel x 7 x 6 - x 7 y 67 channel • Channel-decoding perspective ◦ each measurement is modeled by an i.i.d. channel ◦ transition prob. P ( y ij | x i − x j ) Page 7

Problem setup: a Shannon-theoretic framework p ( y ij | x i -x j ) x 1 - x 2 y 12 channel x 1 x 3 x 1 - x 5 y 15 channel x 2 x 4 x 5 x 6 x 2 - x 7 y 27 channel x 7 x 6 - x 7 y 67 channel • Goal: recover { x i } exactly (up to global offset) • Unified framework for decoding model ◦ capture similarities among various applications Page 7

What factors dictate hardness of recovery? y 12 ∼ P 1 x 1 − x 2 = 1 channel x 1 x 2 y 12 ∼ P 2 x 1 − x 2 = 2 channel P l := P ( y ij | x i − x j = l ) • Channel distance/resolution ◦ Captured by KL ( P l � P k ) or Hellinger ( P l � P k ) or ... Page 8

What factors dictate hardness of recovery? y 12 ∼ P 1 x 1 − x 2 = 1 channel x 1 x 2 y 12 ∼ P 3 x 1 − x 2 = 3 channel P l := P ( y ij | x i − x j = l ) • Minimum channel distance/resolution min l � = k KL ( P l � P k ) := KL min or min l � = k Hellinger ( P l � P k ) := Hellinger min or ... ◦ Uncoded input Page 8

What factors dictate hardness of recovery? measurement graph G • Graph connectivity o Impossible to recover isolated vertices Page 9

What factors dictate hardness of recovery? measurement graph G • Graph connectivity o Over-sparse connectivity is fragile Page 9

What factors dictate hardness of recovery? measurement graph G • Graph connectivity o Sufficient connectivity removes fragility! Page 9

Agenda Page 10

Main result: Erdos-Renyi random graph Erdos-Renyi graph G ( n, p obs ) . Each edge ( i, j ) is present independently w.p. p obs ( p obs = 1) ( p obs = 0 . 3) Page 11

Main result: Erdos-Renyi random graph Erdos-Renyi graph G ( n, p obs ) . Each edge ( i, j ) is present independently w.p. p obs ( p obs = 1) ( p obs = 0 . 3) • ML decoding works if 2 log n + 4 log M Hellinger min > p obs n Page 11

Main result: Erdos-Renyi random graph Erdos-Renyi graph G ( n, p obs ) . Each edge ( i, j ) is present independently w.p. p obs ( p obs = 1) ( p obs = 0 . 3) • ML decoding works if 2 log n + 4 log M Hellinger min > p obs n • Converse: no method works if log n KL min < p obs n Page 11

Main result: Erdos-Renyi random graph Erdos-Renyi graph G ( n, p obs ) . Each edge ( i, j ) is present independently w.p. p obs ( p obs = 1) ( p obs = 0 . 3) • ML decoding works if 2 log n + 4 log M Hellinger min >  p obs n      non-asymptotic! • Converse: no method works if  log n  KL min  <   p obs n Page 11

Main result: Erdos-Renyi random graph ( p obs = 1) ( p obs = 0 . 3) Page 12

Main result: Erdos-Renyi random graph ( p obs = 1) ( p obs = 0 . 3) • In the hard regime where d P l d P k ≈ 1 : KL min ≈ 2 · Hellinger min Page 12

Main result: Erdos-Renyi random graph ( p obs = 1) ( p obs = 0 . 3) • In the hard regime where d P l d P k ≈ 1 : KL min ≈ 2 · Hellinger min • Recovery conditions Hellinger min > 2 log n + 4 log M ML works if p obs n log n Hellinger min < Impossible if 2 p obs n Page 12

Main result: Erdos-Renyi random graph ( p obs = 1) ( p obs = 0 . 3) • In the hard regime where d P l d P k ≈ 1 : KL min ≈ 2 · Hellinger min • Fundamental recovery condition (assuming M � poly( n ) ) Hellinger min � log n p obs n Page 12

Main result: Erdos-Renyi random graph ( p obs = 1) ( p obs = 0 . 3) • In the hard regime where d P l d P k ≈ 1 : KL min ≈ 2 · Hellinger min • Fundamental recovery condition (assuming M � poly( n ) ) Hellinger min � log n avg-degree × Hellinger min � log n ⇐ ⇒ p obs n Page 12

Intuition Fundamental recovery condition (Erdos-Renyi graphs). avg-degree × Hellinger min � log n 3 4 2 5 [ x i − x j ] 1 ≤ i,j ≤ n 1 6 9 7 8 Page 13

Intuition Fundamental recovery condition (Erdos-Renyi graphs). avg-degree × Hellinger min � log n [ x i − x j ] 1 ≤ i,j ≤ n hypotheses: H 0 : x = [0 , 0 , · · · , 0] H 1 : x = [1 , 0 , · · · , 0] • H 0 and H 1 differ only at the highlighted region ( ≈ avg-degree pieces of info) Page 13

Intuition Fundamental recovery condition (Erdos-Renyi graphs). avg-degree × Hellinger min � log n [ x i − x j ] 1 ≤ i,j ≤ n hypotheses: H 0 : x = [0 , 0 , · · · , 0] H 2 : x = [0 , 1 , · · · , 0] • H 0 and H 2 differ only at the highlighted region ( ≈ avg-degree pieces of info) Page 13

Intuition Fundamental recovery condition (Erdos-Renyi graphs). avg-degree × Hellinger min � log n (1) [ x i − x j ] 1 ≤ i,j ≤ n hypotheses: H 0 : x = [0 , 0 , · · · , 0] H n : x = [0 , 0 , · · · , 1] • n minimally-separated hypotheses ⇒ needs at least log n bits ◦ the consequence of uncoded inputs Page 13

Minimal sample complexity Fundamental recovery condition (Erdos-Renyi graphs). avg-degree × Hellinger min � log n • Sample complexity: n · avg-degree Page 14

Minimal sample complexity Fundamental recovery condition (Erdos-Renyi graphs). avg-degree × Hellinger min � log n • Sample complexity: n · avg-degree n log n Min sample complexity ≍ Hellinger min Page 14

How general this limit is? Fundamental recovery condition (Erdos-Renyi graphs). avg-degree × Hellinger min � log n • Can we go beyond Erdos-Renyi graphs? Page 15

Main results: homogeneous graphs random geometric graph (generalized) ring Page 16