Proximity in the Age of Distraction: Robust Approximate Nearest - - PowerPoint PPT Presentation

Proximity in the Age of Distraction: Robust Approximate Nearest Neighbor Search

Sariel Har-Peled UIUC Sepideh Mahabadi MIT

Nearest Neighbor Problem

Nearest Neighbor

Dataset of 𝑜 points 𝑄 in a metric space (𝑌, 𝑒𝑌), e.g. ℝ𝑒

Nearest Neighbor

Dataset of 𝑜 points 𝑄 in a metric space (𝑌, 𝑒𝑌), e.g. ℝ𝑒 A query point 𝑟 comes online

𝑟

Nearest Neighbor

Dataset of 𝑜 points 𝑄 in a metric space (𝑌, 𝑒𝑌), e.g. ℝ𝑒 A query point 𝑟 comes online Goal:

• Find the nearest data point 𝑞∗

𝑟 𝑞∗

Nearest Neighbor

Dataset of 𝑜 points 𝑄 in a metric space (𝑌, 𝑒𝑌), e.g. ℝ𝑒 A query point 𝑟 comes online Goal:

• Find the nearest data point 𝑞∗
• Do it in sub-linear time and small space

𝑟 𝑞∗

Approximate Nearest Neighbor

Dataset of 𝑜 points 𝑄 in a metric space (𝑌, 𝑒𝑌), e.g. ℝ𝑒 A query point 𝑟 comes online Goal:

• Find the nearest data point 𝑞∗
• Do it in sub-linear time and small space
• Approximate Nearest Neighbor

─ If optimal distance is 𝑠, report a point in distance c𝑠 for c = 1 + 𝜗

𝑟 𝑞∗ 𝑞

Approximate Nearest Neighbor

Dataset of 𝑜 points 𝑄 in a metric space (𝑌, 𝑒𝑌), e.g. ℝ𝑒 A query point 𝑟 comes online Goal:

• Find the nearest data point 𝑞∗
• Do it in sub-linear time and small space
• Approximate Nearest Neighbor

─ If optimal distance is 𝑠, report a point in distance c𝑠 for c = 1 + 𝜗 ─ For Hamming (and 𝑀1) query time is 𝑜1/𝑃(𝑑) [IM98] ─ and for Euclidean (𝑀2) it is 𝑜

1 𝑃(𝑑2) [AI08]

𝑟 𝑞∗ 𝑞

Applications of NN

Searching for the closest object

Robust NN Problem

Robustness

The data points are:

Robustness

The data points are:

• corrupted, noisy
• Image denoising

Robustness

The data points are:

• corrupted, noisy
• Image denoising
• Incomplete
• Recommendation: Sparse

matrix

1 - 0 - -

• 0 1 - 0 -
• -
• 1 1 -

Movies Users

Robustness

The data points are:

• corrupted, noisy
• Image denoising
• Incomplete
• Recommendation: Sparse

matrix

• Irrelevant
• Occluded image

1 - 0 - -

• 0 1 - 0 -
• -
• 1 1 -

Movies Users

The Robust NN problem

• Dataset of 𝑜 points 𝑄 in ℝ𝑒

n=3 𝑞1 = (3,4,0,5) 𝑞2 = (3,2,1,2) 𝑞3 = (2,3,3,1)

The Robust NN problem

• Dataset of 𝑜 points 𝑄 in ℝ𝑒
• A parameter 𝒍

n=3,k=2 𝑞1 = (3,4,0,5) 𝑞2 = (3,2,1,2) 𝑞3 = (2,3,3,1)

The Robust NN problem

• Dataset of 𝑜 points 𝑄 in ℝ𝑒
• A parameter 𝒍
• A query point 𝑟 comes online
• Find the closest point after

removing 𝒍 coordinates 𝑟 = (1,2, 1,5) n=3,k=2 𝑞1 = (3,4,0,5) 𝑞2 = (3,2,1,2) 𝑞3 = (2,3,3,1)

The Robust NN problem

• Dataset of 𝑜 points 𝑄 in ℝ𝑒
• A parameter 𝒍
• A query point 𝑟 comes online
• Find the closest point after

removing 𝒍 coordinates 𝑟 = (1,2, 1,5) n=3,k=2 𝑞1 = (3,4,0,5) dist=1 𝑞2 = (3,2,1,2) dist=0 𝑞3 = (2,3,3,1) dist=2

The Robust NN problem

• Dataset of 𝑜 points 𝑄 in ℝ𝑒
• A parameter 𝒍
• A query point 𝑟 comes online
• Find the closest point after

removing 𝒍 coordinates 𝑟 = (1,2, 1,5) n=3,k=2 𝑞1 = (3,4,0,5) dist=1 𝑞2 = (3,2,1,2) dist=0 𝑞3 = (2,3,3,1) dist=2

The Robust NN problem

• Dataset of 𝑜 points 𝑄 in ℝ𝑒
• A parameter 𝒍
• A query point 𝑟 comes online
• Find the closest point after

removing 𝒍 coordinates 𝑟 = (1,2, 1,5) n=3,k=2 𝑞1 = (3,4,0,5) dist=1 𝑞2 = (3,2,1,2) dist=0 𝑞3 = (2,3,3,1) dist=2

• Different set of coordinates for different points
• Applying this naively would require 𝑒

𝑙 ≈ 𝑒𝑙

Budgeted Version

• Dataset of 𝑜 points 𝑄 in ℝ𝑒
• 𝑒 weights

𝑥 = (𝑥1, 𝑥2, … , 𝑥𝑒) ∈ 0,1 𝑒 𝑥 = 0.5, 0.5, 0.8, 0.3 n=3 𝑞1 = (1,4,0,3) 𝑞2 = (3,2,4,2) 𝑞3 = (4,6,3,4)

Budgeted Version

• Dataset of 𝑜 points 𝑄 in ℝ𝑒
• 𝑒 weights

𝑥 = (𝑥1, 𝑥2, … , 𝑥𝑒) ∈ 0,1 𝑒

• A query point 𝑟 comes online
• Find the closest point after

removing a set of coordinates 𝐶

• f weight at most 𝟐.

𝑥 = 0.5, 0.5, 0.8, 0.3 𝑟 = (1,2, 5,5) n=3 𝑞1 = (1,4,0,3) 𝑞2 = (3,2,4,2) 𝑞3 = (4,6,3,4)

Budgeted Version

• Dataset of 𝑜 points 𝑄 in ℝ𝑒
• 𝑒 weights

𝑥 = (𝑥1, 𝑥2, … , 𝑥𝑒) ∈ 0,1 𝑒

• A query point 𝑟 comes online
• Find the closest point after

removing a set of coordinates 𝐶

• f weight at most 𝟐.

𝑥 = 0.5, 0.5, 0.8, 0.3 𝑟 = (1,2, 5,5) n=3 𝑞1 = (1,4,0,3) 𝑞2 = (3,2,4,2) 𝑞3 = (4,6,3,4)

Budgeted Version

• Dataset of 𝑜 points 𝑄 in ℝ𝑒
• 𝑒 weights

𝑥 = (𝑥1, 𝑥2, … , 𝑥𝑒) ∈ 0,1 𝑒

• A query point 𝑟 comes online
• Find the closest point after

removing a set of coordinates 𝐶

• f weight at most 𝟐.

𝑥 = 0.5, 0.5, 0.8, 0.3 𝑟 = (1,2, 5,5) n=3 𝑞1 = (1,4,0,3) dist=4 𝑞2 = (3,2,4,2) dist=1 𝑞3 = (4,6,3,4) dist=3

Budgeted Version

• Dataset of 𝑜 points 𝑄 in ℝ𝑒
• 𝑒 weights

𝑥 = (𝑥1, 𝑥2, … , 𝑥𝑒) ∈ 0,1 𝑒

• A query point 𝑟 comes online
• Find the closest point after

removing a set of coordinates 𝐶

• f weight at most 𝟐.

𝑥 = 0.5, 0.5, 0.8, 0.3 𝑟 = (1,2, 5,5) n=3 𝑞1 = (1,4,0,3) dist=4 𝑞2 = (3,2,4,2) dist=1 𝑞3 = (4,6,3,4) dist=3

Results

Bicriterion Approximation, for 𝑀1 norm

• Suppose that for 𝑞∗ ⊂ 𝑄 we have 𝑒𝑗𝑡𝑢 𝑟, 𝑞∗ = 𝑠 after

ignoring 𝑙 coordinates

Results

Bicriterion Approximation, for 𝑀1 norm

• Suppose that for 𝑞∗ ⊂ 𝑄 we have 𝑒𝑗𝑡𝑢 𝑟, 𝑞∗ = 𝑠 after

ignoring 𝑙 coordinates

• For 𝜀 ∈ (0,1)
• Report a point 𝑞 s.t. 𝑒𝑗𝑡𝑢 𝑟, 𝑞 = 𝑃(𝑠/𝜀) after

ignoring 𝑃(𝑙/𝜀) coordinates.

• Query time equals to 𝑜𝜀 queries in 2-ANN data-

structure

Results

Bicriterion Approximation, for 𝑀1 norm

• Suppose that for 𝑞∗ ⊂ 𝑄 we have 𝑒𝑗𝑡𝑢 𝑟, 𝑞∗ = 𝑠 after

ignoring 𝑙 coordinates

• For 𝜀 ∈ (0,1)
• Report a point 𝑞 s.t. 𝑒𝑗𝑡𝑢 𝑟, 𝑞 = 𝑃(𝑠/𝜀) after

ignoring 𝑃(𝑙/𝜀) coordinates.

• Query time equals to 𝑜𝜀 queries in 2-ANN data-

structure Why not single criterion?

• Equivalent to exact near neighbor in Hamming: there is a

point within distance 𝑠 of the query iff there is a point within distance 0 after ignoring 𝑙 = 𝑠 coordinates

Results

distance #ignored coordinates Query Time #Queries Query type Opt 𝑠 𝑙

Results

distance #ignored coordinates Query Time #Queries Query type Opt 𝑠 𝑙 𝑀1 𝑃(𝑠 𝜀) 𝑃(𝑙 𝜀) 𝑜𝜀 2-ANN

Results

distance #ignored coordinates Query Time #Queries Query type Opt 𝑠 𝑙 𝑀1 𝑃(𝑠 𝜀) 𝑃(𝑙 𝜀) 𝑜𝜀 2-ANN 𝑀𝐪

𝑃(𝑠 𝑑 + 1 𝜀

1/p

)

𝑃(𝑙 𝑑 + 1

𝜀 )

𝑜𝜀 𝑑1/p-ANN

Results

distance #ignored coordinates Query Time #Queries Query type Opt 𝑠 𝑙 𝑀1 𝑃(𝑠 𝜀) 𝑃(𝑙 𝜀) 𝑜𝜀 2-ANN 𝑀𝐪

𝑃(𝑠 𝑑 + 1 𝜀

1/p

)

𝑃(𝑙 𝑑 + 1

𝜀 )

𝑜𝜀 𝑑1/p-ANN (1 + 𝜗)- approximation 𝑠(1 + 𝜗) 𝑃( 𝑙

𝜗𝜀)

O(𝑜𝜀

𝜗 )

1 + 𝜗 −ANN

Results

distance #ignored coordinates Query Time #Queries Query type Opt 𝑠 𝑙 𝑀1 𝑃(𝑠 𝜀) 𝑃(𝑙 𝜀) 𝑜𝜀 2-ANN 𝑀𝐪

𝑃(𝑠 𝑑 + 1 𝜀

1/p

)

𝑃(𝑙 𝑑 + 1

𝜀 )

𝑜𝜀 𝑑1/p-ANN (1 + 𝜗)- approximation 𝑠(1 + 𝜗) 𝑃( 𝑙

𝜗𝜀)

O(𝑜𝜀

𝜗 )

1 + 𝜗 −ANN Budgeted Version 𝑃(𝑠) Weight of 𝑃(1) 𝑜𝜀 2-ANN +𝑃(𝑜𝜀𝑒4)

Algorithm

High Level Algorithm

• Theorem. If for a point 𝑞∗ ⊂ 𝑄 , the 𝑀1 distance of 𝑟 and 𝑞∗ is

at most 𝑠 after removing 𝑙 coordinates, there exists an algorithm which reports a point 𝑞 whose distance to 𝑟 is 𝑃(𝑠/𝜀) after removing 𝑃(𝑙/𝜀) coordinates.

High Level Algorithm

• Cannot apply randomized dimensionality reduction e.g.

Johnson-Lindenstrauss

• Theorem. If for a point 𝑞∗ ⊂ 𝑄 , the 𝑀1 distance of 𝑟 and 𝑞∗ is

at most 𝑠 after removing 𝑙 coordinates, there exists an algorithm which reports a point 𝑞 whose distance to 𝑟 is 𝑃(𝑠/𝜀) after removing 𝑃(𝑙/𝜀) coordinates.

High Level Algorithm

• Cannot apply randomized dimensionality reduction e.g.

Johnson-Lindenstrauss

• A set of randomized maps 𝒈𝟐, 𝒈𝟑, … 𝒈𝒏: ℝ𝒆 → ℝ𝒆′
• All of them map far points from query to far points
• At least one of them maps a close point to a close point
• Theorem. If for a point 𝑞∗ ⊂ 𝑄 , the 𝑀1 distance of 𝑟 and 𝑞∗ is

at most 𝑠 after removing 𝑙 coordinates, there exists an algorithm which reports a point 𝑞 whose distance to 𝑟 is 𝑃(𝑠/𝜀) after removing 𝑃(𝑙/𝜀) coordinates.

High Level Algorithm

• Cannot apply randomized dimensionality reduction e.g.

Johnson-Lindenstrauss

• A set of randomized maps 𝒈𝟐, 𝒈𝟑, … 𝒈𝒏: ℝ𝒆 → ℝ𝒆′
• All of them map far points from query to far points
• At least one of them maps a close point to a close point
• W.l.o.g. assume that the query is the origin
• Find the data point with minimum norm.
• Theorem. If for a point 𝑞∗ ⊂ 𝑄 , the 𝑀1 distance of 𝑟 and 𝑞∗ is

at most 𝑠 after removing 𝑙 coordinates, there exists an algorithm which reports a point 𝑞 whose distance to 𝑟 is 𝑃(𝑠/𝜀) after removing 𝑃(𝑙/𝜀) coordinates.

A Randomized Map

• Embed all the points using a random mapping 𝒈: ℝ𝒆 → ℝ𝒆′:

A Randomized Map

• Embed all the points using a random mapping 𝒈: ℝ𝒆 → ℝ𝒆′:
• Repeat 𝑢 = 𝑃(ln 𝑜) times
• Sample each coordinate in [𝑒] with probability 𝜀/𝑙

A Randomized Map

• Embed all the points using a random mapping 𝒈: ℝ𝒆 → ℝ𝒆′:
• Repeat 𝑢 = 𝑃(ln 𝑜) times
• Sample each coordinate in [𝑒] with probability 𝜀/𝑙
• E.g. 𝑒 = 5
• round 1: coordinates (1,3,4) sampled
• round 2: coordinate (4) sampled
• 𝑤 = 3,6,1,2,4 maps to 𝑔 𝑤 = (3,1,2,2)

A Randomized Map

• Embed all the points using a random mapping 𝒈: ℝ𝒆 → ℝ𝒆′:
• Repeat 𝑢 = 𝑃(ln 𝑜) times
• Sample each coordinate in [𝑒] with probability 𝜀/𝑙
• E.g. 𝑒 = 5
• round 1: coordinates (1,3,4) sampled
• round 2: coordinate (4) sampled
• 𝑤 = 3,6,1,2,4 maps to 𝑔 𝑤 = (3,1,2,2)

𝔽 𝒆′ = 𝑷(𝒆 𝐦𝐨 𝒐 ⋅ 𝜺 𝒍)

A Randomized Map

• Embed all the points using a random mapping 𝒈: ℝ𝒆 → ℝ𝒆′:
• Repeat 𝑢 = 𝑃(ln 𝑜) times
• Sample each coordinate in [𝑒] with probability 𝜀/𝑙
• E.g. 𝑒 = 5
• round 1: coordinates (1,3,4) sampled
• round 2: coordinate (4) sampled
• 𝑤 = 3,6,1,2,4 maps to 𝑔 𝑤 = (3,1,2,2)
• Simple setup: Consider a vector 𝑤 where each coordinate is either 0 or ∞

𝔽 𝒆′ = 𝑷(𝒆 𝐦𝐨 𝒐 ⋅ 𝜺 𝒍)

A Randomized Map

• Embed all the points using a random mapping 𝒈: ℝ𝒆 → ℝ𝒆′:
• Repeat 𝑢 = 𝑃(ln 𝑜) times
• Sample each coordinate in [𝑒] with probability 𝜀/𝑙
• E.g. 𝑒 = 5
• round 1: coordinates (1,3,4) sampled
• round 2: coordinate (4) sampled
• 𝑤 = 3,6,1,2,4 maps to 𝑔 𝑤 = (3,1,2,2)
• Simple setup: Consider a vector 𝑤 where each coordinate is either 0 or ∞
• Close point:
• 𝑤 has at most 𝑙 large coordinates
• Probability of avoiding large coordinates is at least 1 − 𝜀

𝑙 𝑙⋅ln 𝑜

≈ 𝑜−𝜀 𝔽 𝒆′ = 𝑷(𝒆 𝐦𝐨 𝒐 ⋅ 𝜺 𝒍)

A Randomized Map

• Embed all the points using a random mapping 𝒈: ℝ𝒆 → ℝ𝒆′:
• Repeat 𝑢 = 𝑃(ln 𝑜) times
• Sample each coordinate in [𝑒] with probability 𝜀/𝑙
• E.g. 𝑒 = 5
• round 1: coordinates (1,3,4) sampled
• round 2: coordinate (4) sampled
• 𝑤 = 3,6,1,2,4 maps to 𝑔 𝑤 = (3,1,2,2)
• Simple setup: Consider a vector 𝑤 where each coordinate is either 0 or ∞
• Close point:
• 𝑤 has at most 𝑙 large coordinates
• Probability of avoiding large coordinates is at least 1 − 𝜀

𝑙 𝑙⋅ln 𝑜

≈ 𝑜−𝜀

• Far point
• 𝑤 has at least 𝑙/𝜀 large coordinates
• Probability of missing large coordinates is at most 1 − 𝜀

𝑙 (𝑙/𝜀)⋅ln 𝑜

≈ 1/𝑜 𝔽 𝒆′ = 𝑷(𝒆 𝐦𝐨 𝒐 ⋅ 𝜺 𝒍)

Outline

• Embed all the points using a random mapping 𝑔: ℝ𝑒 → ℝ𝑒′
• With probability 𝑜−𝜀
• all far points will be mapped to far points under 𝑀1 distance
• a close by point will be mapped to a close by point under 𝑀1 distance.

Outline

• Embed all the points using a random mapping 𝑔: ℝ𝑒 → ℝ𝑒′
• With probability 𝑜−𝜀
• all far points will be mapped to far points under 𝑀1 distance
• a close by point will be mapped to a close by point under 𝑀1 distance.
• We can use ANN as a black-box to find it

Outline

• Embed all the points using a random mapping 𝑔: ℝ𝑒 → ℝ𝑒′
• With probability 𝑜−𝜀
• all far points will be mapped to far points under 𝑀1 distance
• a close by point will be mapped to a close by point under 𝑀1 distance.
• We can use ANN as a black-box to find it
• Repeat this embedding O(𝑜𝜀 log 𝑜) times and report the

best.

Algorithm

𝑆𝑒

𝑟

Ignore k-coords

Algorithm

t times

f

Sample every coordinate w.p 𝜺/𝒍

𝑆𝑒

𝑟

𝑆𝑒′

𝑟

Ignore k-coords 𝑀1

Algorithm

t times

f

Sample every coordinate w.p 𝜺/𝒍

𝑆𝑒

𝑟

𝑆𝑒′

𝑟

𝐵𝑂𝑂 Ignore k-coords 𝑀1

Algorithm

t times

f

Sample every coordinate w.p 𝜺/𝒍

. . . 𝑜𝜀 times 𝑆𝑒

𝑟

𝑆𝑒′

𝑟

𝐵𝑂𝑂

t times

f

Sample every coordinate w.p 𝜺/𝒍

𝑆𝑒

𝑟

𝑆𝑒′

𝑟

𝐵𝑂𝑂 Ignore k-coords 𝑀1

Algorithm

Check the distance of all 𝑜𝜀 candidates and report the closest one after ignoring 𝑙 coordinates

t times

f

Sample every coordinate w.p 𝜺/𝒍

. . . 𝑜𝜀 times 𝑆𝑒

𝑟

𝑆𝑒′

𝑟

𝐵𝑂𝑂

t times

f

Sample every coordinate w.p 𝜺/𝒍

𝑆𝑒

𝑟

𝑆𝑒′

𝑟

𝐵𝑂𝑂 Ignore k-coords 𝑀1

Analysis

Truncation

Coordinates are not necessarily 𝟏 and ∞

Truncation

Coordinates are not necessarily 𝟏 and ∞

Let 𝒈′ be obtained by sampling every coordinate with probability 𝝊 = 𝜺/𝒍

Truncation

Coordinates are not necessarily 𝟏 and ∞

Let 𝒈′ be obtained by sampling every coordinate with probability 𝝊 = 𝜺/𝒍

• 𝔽 𝑔′ 𝑤

1 = 𝜐 𝑤 1

Truncation

Coordinates are not necessarily 𝟏 and ∞

Let 𝒈′ be obtained by sampling every coordinate with probability 𝝊 = 𝜺/𝒍

• 𝔽 𝑔′ 𝑤

1 = 𝜐 𝑤 1

• 𝐖𝐛𝐬 𝑔′ 𝑤

1 = 𝜐 1 − 𝜐 𝑤 2 2

Truncation

Coordinates are not necessarily 𝟏 and ∞

Let 𝒈′ be obtained by sampling every coordinate with probability 𝝊 = 𝜺/𝒍

• 𝔽 𝑔′ 𝑤

1 = 𝜐 𝑤 1

• 𝐖𝐛𝐬 𝑔′ 𝑤

1 = 𝜐 1 − 𝜐 𝑤 2 2

Need to bound the influence of every coordinate.

Truncation

Coordinates are not necessarily 𝟏 and ∞

Let 𝒈′ be obtained by sampling every coordinate with probability 𝝊 = 𝜺/𝒍

• 𝔽 𝑔′ 𝑤

1 = 𝜐 𝑤 1

• 𝐖𝐛𝐬 𝑔′ 𝑤

1 = 𝜐 1 − 𝜐 𝑤 2 2

Need to bound the influence of every coordinate.

• Truncate every coordinate at 𝒔/𝒍 ,i.e., 𝒘𝒋 = 𝐧𝐣𝐨{𝒘𝒋, 𝒔/𝒍}

Truncation

Coordinates are not necessarily 𝟏 and ∞

Let 𝒈′ be obtained by sampling every coordinate with probability 𝝊 = 𝜺/𝒍

• 𝔽 𝑔′ 𝑤

1 = 𝜐 𝑤 1

• 𝐖𝐛𝐬 𝑔′ 𝑤

1 = 𝜐 1 − 𝜐 𝑤 2 2

Need to bound the influence of every coordinate.

• Truncate every coordinate at 𝒔/𝒍 ,i.e., 𝒘𝒋 = 𝐧𝐣𝐨{𝒘𝒋, 𝒔/𝒍}
• 𝒔 −Light point: a point with norm ≤ 𝑠 after truncation
• R− Heavy point: a point with norm ≥ 𝑆 after truncation

Truncation

Coordinates are not necessarily 𝟏 and ∞

Let 𝒈′ be obtained by sampling every coordinate with probability 𝝊 = 𝜺/𝒍

• 𝔽 𝑔′ 𝑤

1 = 𝜐 𝑤 1

• 𝐖𝐛𝐬 𝑔′ 𝑤

1 = 𝜐 1 − 𝜐 𝑤 2 2

Need to bound the influence of every coordinate.

• Truncate every coordinate at 𝒔/𝒍 ,i.e., 𝒘𝒋 = 𝐧𝐣𝐨{𝒘𝒋, 𝒔/𝒍}
• 𝒔 −Light point: a point with norm ≤ 𝑠 after truncation
• R− Heavy point: a point with norm ≥ 𝑆 after truncation
• Close point: a point with norm ≤ 𝑠 after ignoring 𝑙 coordinates
• Far point: a point with norm ≥ 𝑠/𝜀 after ignoring 𝑙/𝜀 coordinates

Truncation

Coordinates are not necessarily 𝟏 and ∞

Let 𝒈′ be obtained by sampling every coordinate with probability 𝝊 = 𝜺/𝒍

• 𝔽 𝑔′ 𝑤

1 = 𝜐 𝑤 1

• 𝐖𝐛𝐬 𝑔′ 𝑤

1 = 𝜐 1 − 𝜐 𝑤 2 2

Need to bound the influence of every coordinate.

• Truncate every coordinate at 𝒔/𝒍 ,i.e., 𝒘𝒋 = 𝐧𝐣𝐨{𝒘𝒋, 𝒔/𝒍}
• 𝒔 −Light point: a point with norm ≤ 𝑠 after truncation
• R− Heavy point: a point with norm ≥ 𝑆 after truncation
• Close point: a point with norm ≤ 𝑠 after ignoring 𝑙 coordinates
• Far point: a point with norm ≥ 𝑠/𝜀 after ignoring 𝑙/𝜀 coordinates

Claim:

• A close point is 2𝑠 −light.

Truncation

Coordinates are not necessarily 𝟏 and ∞

Let 𝒈′ be obtained by sampling every coordinate with probability 𝝊 = 𝜺/𝒍

• 𝔽 𝑔′ 𝑤

1 = 𝜐 𝑤 1

• 𝐖𝐛𝐬 𝑔′ 𝑤

1 = 𝜐 1 − 𝜐 𝑤 2 2

Need to bound the influence of every coordinate.

• Truncate every coordinate at 𝒔/𝒍 ,i.e., 𝒘𝒋 = 𝐧𝐣𝐨{𝒘𝒋, 𝒔/𝒍}
• 𝒔 −Light point: a point with norm ≤ 𝑠 after truncation
• R− Heavy point: a point with norm ≥ 𝑆 after truncation
• Close point: a point with norm ≤ 𝑠 after ignoring 𝑙 coordinates
• Far point: a point with norm ≥ 𝑠/𝜀 after ignoring 𝑙/𝜀 coordinates

Claim:

• A close point is 2𝑠 −light.
• A far point is 𝑠

𝜀 −heavy.

Truncation

Coordinates are not necessarily 𝟏 and ∞

Let 𝒈′ be obtained by sampling every coordinate with probability 𝝊 = 𝜺/𝒍

• 𝔽 𝑔′ 𝑤

1 = 𝜐 𝑤 1

• 𝐖𝐛𝐬 𝑔′ 𝑤

1 = 𝜐 1 − 𝜐 𝑤 2 2

Need to bound the influence of every coordinate.

• Truncate every coordinate at 𝒔/𝒍 ,i.e., 𝒘𝒋 = 𝐧𝐣𝐨{𝒘𝒋, 𝒔/𝒍}
• 𝒔 −Light point: a point with norm ≤ 𝑠 after truncation
• R− Heavy point: a point with norm ≥ 𝑆 after truncation
• Close point: a point with norm ≤ 𝑠 after ignoring 𝑙 coordinates
• Far point: a point with norm ≥ 𝑠/𝜀 after ignoring 𝑙/𝜀 coordinates

Claim:

• A close point is 2𝑠 −light.
• A far point is 𝑠

𝜀 −heavy.

• Analyze the behavior of the maps over the truncated points instead.

𝑀1 Norm

Using truncation

• Bound the variance and prove concentration for 𝒈′ by Chebyshev

𝑀1 Norm

Using truncation

• Bound the variance and prove concentration for 𝒈′ by Chebyshev

𝒈 is a concatenation of 𝒖 = 𝑷(𝒎𝒐 𝒐) such 𝒈′

𝑀1 Norm

Using truncation

• Bound the variance and prove concentration for 𝒈′ by Chebyshev

𝒈 is a concatenation of 𝒖 = 𝑷(𝒎𝒐 𝒐) such 𝒈′

• 𝔽 𝑔 𝑤

1 = 𝑢𝜐 𝑤 1

• Prove concentration for 𝒈 using Chernoff

𝑀1 Norm

Using truncation

• Bound the variance and prove concentration for 𝒈′ by Chebyshev

𝒈 is a concatenation of 𝒖 = 𝑷(𝒎𝒐 𝒐) such 𝒈′

• 𝔽 𝑔 𝑤

1 = 𝑢𝜐 𝑤 1

• Prove concentration for 𝒈 using Chernoff

distance #ignored coordinates Query Time #Queries Query type Opt 𝑠 𝑙 𝑀1 𝑃(𝑠 𝜀) 𝑃(𝑙 𝜀) 𝑜𝜀 2-ANN

Generalizations

 𝑴𝒒 norm

Generalizations

 𝑴𝒒 norm

• Minimize the 𝒘 𝒒

𝒒 norm, i.e., 𝒋 𝒘𝒋 𝒒 similar to the 𝑀1 norm

Generalizations

 𝑴𝒒 norm

• Minimize the 𝒘 𝒒

𝒒 norm, i.e., 𝒋 𝒘𝒋 𝒒 similar to the 𝑀1 norm

𝑀𝒒

𝑃(𝑠 𝑑 + 1 𝜀

1/𝒒

)

𝑃(𝑙 𝑑 + 1

𝜀 )

𝑜𝜀 𝑑1/𝐪-ANN

Generalizations

 𝑴𝒒 norm

• Minimize the 𝒘 𝒒

𝒒 norm, i.e., 𝒋 𝒘𝒋 𝒒 similar to the 𝑀1 norm

 Budgeted

𝑀𝒒

𝑃(𝑠 𝑑 + 1 𝜀

1/𝒒

)

𝑃(𝑙 𝑑 + 1

𝜀 )

𝑜𝜀 𝑑1/𝐪-ANN

Generalizations

 𝑴𝒒 norm

• Minimize the 𝒘 𝒒

𝒒 norm, i.e., 𝒋 𝒘𝒋 𝒒 similar to the 𝑀1 norm

 Budgeted

• Map:
• sample coordinate 𝒋 with probability proportional to 𝟐/𝒙𝒋

𝑀𝒒

𝑃(𝑠 𝑑 + 1 𝜀

1/𝒒

)

𝑃(𝑙 𝑑 + 1

𝜀 )

𝑜𝜀 𝑑1/𝐪-ANN

Generalizations

 𝑴𝒒 norm

• Minimize the 𝒘 𝒒

𝒒 norm, i.e., 𝒋 𝒘𝒋 𝒒 similar to the 𝑀1 norm

 Budgeted

• Map:
• sample coordinate 𝒋 with probability proportional to 𝟐/𝒙𝒋
• To maintain the expectation multiply sampled coordinates by 𝒙𝒋

𝑀𝒒

𝑃(𝑠 𝑑 + 1 𝜀

1/𝒒

)

𝑃(𝑙 𝑑 + 1

𝜀 )

𝑜𝜀 𝑑1/𝐪-ANN

Generalizations

 𝑴𝒒 norm

• Minimize the 𝒘 𝒒

𝒒 norm, i.e., 𝒋 𝒘𝒋 𝒒 similar to the 𝑀1 norm

 Budgeted

• Map:
• sample coordinate 𝒋 with probability proportional to 𝟐/𝒙𝒋
• To maintain the expectation multiply sampled coordinates by 𝒙𝒋
• Truncation:
• Truncate coordinate 𝒋 with by value

𝒔 𝒅/𝒙𝒋−𝟐

𝑀𝒒

𝑃(𝑠 𝑑 + 1 𝜀

1/𝒒

)

𝑃(𝑙 𝑑 + 1

𝜀 )

𝑜𝜀 𝑑1/𝐪-ANN

Generalizations

 𝑴𝒒 norm

• Minimize the 𝒘 𝒒

𝒒 norm, i.e., 𝒋 𝒘𝒋 𝒒 similar to the 𝑀1 norm

 Budgeted

• Map:
• sample coordinate 𝒋 with probability proportional to 𝟐/𝒙𝒋
• To maintain the expectation multiply sampled coordinates by 𝒙𝒋
• Truncation:
• Truncate coordinate 𝒋 with by value

𝒔 𝒅/𝒙𝒋−𝟐

• E.g. a coordinate of cost approaching 0 will be truncated to 0

𝑀𝒒

𝑃(𝑠 𝑑 + 1 𝜀

1/𝒒

)

𝑃(𝑙 𝑑 + 1

𝜀 )

𝑜𝜀 𝑑1/𝐪-ANN

Generalizations

 𝑴𝒒 norm

• Minimize the 𝒘 𝒒

𝒒 norm, i.e., 𝒋 𝒘𝒋 𝒒 similar to the 𝑀1 norm

 Budgeted

• Map:
• sample coordinate 𝒋 with probability proportional to 𝟐/𝒙𝒋
• To maintain the expectation multiply sampled coordinates by 𝒙𝒋
• Truncation:
• Truncate coordinate 𝒋 with by value

𝒔 𝒅/𝒙𝒋−𝟐

• E.g. a coordinate of cost approaching 0 will be truncated to 0

𝑀𝒒

𝑃(𝑠 𝑑 + 1 𝜀

1/𝒒

)

𝑃(𝑙 𝑑 + 1

𝜀 )

𝑜𝜀 𝑑1/𝐪-ANN Budgeted Version 𝑃(𝑠) Weight of 𝑃(1) 𝑜𝜀 2-ANN +𝑃(𝑜𝜀𝑒4)

Conclusion

distance #ignored coordinates Query Time #Queries Query type Opt 𝑠 𝑙 𝑀1 𝑃(𝑠 𝜀) 𝑃(𝑙 𝜀) 𝑜𝜀 2-ANN 𝑀𝒒

𝑃(𝑠 𝑑 + 1 𝜀

1/𝒒

)

𝑃(𝑙 𝑑 + 1

𝜀 )

𝑜𝜀 𝑑1/𝐪-ANN (1 + 𝜗)- approximation 𝑠(1 + 𝜗) 𝑃( 𝑙

𝜗𝜀)

O(𝑜𝜀

𝜗 )

1 + 𝜗 −ANN Budgeted Version 𝑃(𝑠) Weight of 𝑃(1) 𝑜𝜀 2-ANN +𝑃(𝑜𝜀𝑒4)

Conclusion

Open Problems

• Improve the dependence on 𝜀
• Prove lower bounds

distance #ignored coordinates Query Time #Queries Query type Opt 𝑠 𝑙 𝑀1 𝑃(𝑠 𝜀) 𝑃(𝑙 𝜀) 𝑜𝜀 2-ANN 𝑀𝒒

𝑃(𝑠 𝑑 + 1 𝜀

1/𝒒

)

𝑃(𝑙 𝑑 + 1

𝜀 )

𝑜𝜀 𝑑1/𝐪-ANN (1 + 𝜗)- approximation 𝑠(1 + 𝜗) 𝑃( 𝑙

𝜗𝜀)

O(𝑜𝜀

𝜗 )

1 + 𝜗 −ANN Budgeted Version 𝑃(𝑠) Weight of 𝑃(1) 𝑜𝜀 2-ANN +𝑃(𝑜𝜀𝑒4)

Conclusion

Open Problems

• Improve the dependence on 𝜀
• Prove lower bounds

distance #ignored coordinates Query Time #Queries Query type Opt 𝑠 𝑙 𝑀1 𝑃(𝑠 𝜀) 𝑃(𝑙 𝜀) 𝑜𝜀 2-ANN 𝑀𝒒

𝑃(𝑠 𝑑 + 1 𝜀

1/𝒒

)

𝑃(𝑙 𝑑 + 1

𝜀 )

𝑜𝜀 𝑑1/𝐪-ANN (1 + 𝜗)- approximation 𝑠(1 + 𝜗) 𝑃( 𝑙

𝜗𝜀)

O(𝑜𝜀

𝜗 )

1 + 𝜗 −ANN Budgeted Version 𝑃(𝑠) Weight of 𝑃(1) 𝑜𝜀 2-ANN +𝑃(𝑜𝜀𝑒4)

Thank You!