neighbor graph Artur Czumaj DIMAP (Centre for Discrete Mathematics - - PowerPoint PPT Presentation

Artur Czumaj

DIMAP (Centre for Discrete Mathematics and it Applications) Department of Computer Science

University of Warwick

Joint work with Christian Sohler

Sublinear-time approximation of the cost of a metric k-nearest neighbor graph

k-nearest neighbor graph

The 𝑙-nearest neighbor (𝑙-NN) graph is a basic data structure with applications in

– spectral clustering (and unsupervised learning) – non-linear dimensionality reduction – density estimation – and many more…

Computing a 𝑙-NN graph takes Ω(𝑜2) time in the worst case

– many data structures for approximate nearest neighbor queries for specific distances measures known – heuristics

k-NN graph

A k-NN graph for a set of objects with a distance measure is a directed graph, such that

• every vertex corresponds to exactly one object
• every vertex has exactly 𝑙 outgoing edges
• the outgoing edges point to its 𝑙 closest neighbors

Every point 𝑣 connects to 𝑙 nearest neighbors

Access to the input: Distance oracle model

Distance Oracle Model for Metric Spaces:

• Input: 𝑜-point metric space (𝑌, 𝑒)

– we will often view the metric space as a complete edge- weighted graph 𝐼 = 𝑊, 𝐹 with 𝑊 = {1, … , 𝑜}, and weights satisfying triangle inequality

Query: return distance between any 𝑣 and 𝑤 time ~ number of queries

𝑙-NN graph cost approximation

Problem: Given oracle access to an 𝑜-point metric space, compute a (1 + 𝜁)-approximation to the cost (denoted by cost(𝑌))

• f the 𝑙-NN graph

cost(𝑌) denotes the sum of edge weights

(Simple) lower bounds

Observation Computing a 𝑙-NN graph requires Ω(𝑜2) queries Proof: Consider a (1,2)-metric with all distances 2 except for a single random pair with distance 1

• we must find this edge to compute 𝑙-NN graph
• finding a random edge requires Ω(𝑜2) time

1

(Simple) lower bounds

Observation Finding a (2 − 𝜁)-approx. 𝑙-NN requires Ω 𝑜2/log 𝑜 queries Proof:

• Take a random (1,2)-metric where each distance is

chosen independently at random to be 1 with probability Θ(

𝑙 log 𝑜 𝑜

)

• Whp. every vertex has ≥ 𝑙 neighbors with distance 1
• (2 − 𝜁)-approximate 𝑙-NN contains Ω(𝑜𝑙) of the 1s

(Simple) lower bounds

Observation Approximating the cost of 1-NN requires Ω(𝑜2) queries Proof:

• Via lower bound for perfect matching [Badoiu, C, Indyk, Sohler, ICALP’05]
• Pick a random perfect matching and set all matching

distances to 0 and all other distances to 1

• Distinguish this from an instance where one matching

edge has value 1

• Finding this edge requires Ω(𝑜2) queries

(Simple) lower bounds

• Pick a random perfect matching and set all matching

distances to 0 and all other distances to 1

• Distinguish this from an instance where one matching

edge has value 1

• Finding this edge requires Ω(𝑜2) queries

All matching edges except possibly a single one have weights 0 One random edge may have weight 1 All non-matching distances are 1

distance 0 or 1

1-NN cost is either 0 or 1 depending on a single edge

(Simple) lower bounds

Observation Approximating the cost of 1-NN requires Ω(𝑜2) queries Proof:

• Via lower bound for perfect matching [Badoiu, C, Indyk, Sohler, ICALP’05]
• Pick a random perfect matching and set all matching

distances to 0 and all other distances to 1

• Distinguish this from an instance where one matching

edge has value 1

• Finding this edge requires Ω(𝑜2) queries

Approximating cost(X)

The “simple lower bounds” show that

• finding a low-cost 𝑙-NN graph is hopeless
• estimating cost(X) is hopeless

– at least for small 𝑙

Can we do anything?

Approximating cost(X)

The “simple lower bounds” show that

• finding a low-cost 𝑙-NN graph is hopeless
• estimating cost(X) is hopeless

– at least for small 𝑙

A similar situation has been known for some other problems: MST, degree estimation, etc Chazelle et al: MST cost in graphs can be (1 + 𝜁)-approx. in poly 𝑋𝑒/𝜁 time, d-max-deg, W-max-weight

• C. Sohler: MST cost in metric case – approx. in ෨

𝑃(𝑜) time

Approximating cost(X): New results

Theorem 1 (1 + 𝜁)-approximation of cost(X) with ෨ 𝑃(𝑜2/𝑙𝜁2) queries Theorem 2 Ω(𝑜2/𝑙) queries are necessary to approximate cost(X) within any constant factor

Approximating cost(X): New results

Theorem 1 (1 + 𝜁)-approximation of cost(X) with ෨ 𝑃(𝑜2/𝑙𝜁2) queries Theorem 2 Ω(𝑜2/𝑙) queries are necessary to approximate cost(X) within any constant factor This is very bad for small 𝑙; and very good for large 𝑙. What can we do for small 𝑙?

Approximating cost(X): New results

The lower bound for small 𝑙 holds when the instances are very “spread-out” and are “disjoint” Can we get a faster algorithm when we allow the approximation guarantee to depend on the MST cost?

Approximating cost(X): New results

Theorem 3 With ෨ 𝑃𝜁(𝑜𝑙3/2) queries one can approximate cost(X) with error 𝜁 ⋅ (cost(X) + MST(X)) Corollary 4 With ෨ 𝑃𝜁(min{𝑜𝑙3/2, 𝑜2/𝑙}) queries one can approximate cost(X) with error 𝜁 ⋅ (cost(X) + MST(X)) Theorem 5 Any algorithm that approximates cost(X) with error 𝜁 ⋅ cost(X) + MST(X) requires Ω(min{𝑜𝑙3/2/𝜁, 𝑜2/𝑙}) queries.

Approximating cost(X): New results

We have tight bounds for estimation of cost(X) When we want a (1 + 𝜁)–approximation:

• ෩

Θ𝜁(n2/k) queries are sufficient and necessary When happy with 𝜁(cost(X) + MST(X)) additive error:

• ෩

Θ𝜁(min{𝑜𝑙3/2, 𝑜2/𝑙}) queries are sufficient/necessary

• it’s sublinear for every 𝑙, always at most ෨

𝑃𝜁(𝑜8/5)

Approximating cost(X): New results

We have tight bounds for estimation of cost(X) When we want a (1 + 𝜁)–approximation:

• ෩

Θ𝜁(n2/k) queries are sufficient and necessary When happy with 𝜁(cost(X) + MST(X)) additive error:

• ෩

Θ𝜁(min{𝑜𝑙3/2, 𝑜2/𝑙}) queries are sufficient/necessary

Techniques:

• Upper bounds: clever random sampling
• Lower bounds: analysis of some clustering inputs (more complex part)

Approximating cost(X): New results

We have tight bounds for estimation of cost(X) When we want a (1 + 𝜁)–approximation:

• ෩

Θ𝜁(n2/k) queries are sufficient and necessary Efficient for large 𝑙 Relies on a random sampling: of close neighbors and far neighbors in 𝑙-NN graph

Upper bound for (𝟐 + 𝜻)-approximation

Two “hard” instances

• A cluster of 𝑜 − 1 points and a single point far away
• A cluster of 𝑜 − (𝑙 + 1) points and 𝑙 + 1 points far

away and close to each other Our algorithm must be able to distinguish between these two instances

Upper bound for (𝟐 + 𝜻)-approximation

Each point 𝑣 approximates length to its 𝑙/2th–nearest neighbor 𝑛𝑣

• ෨

𝑃(𝑜/𝑙) queries per point; ෨ 𝑃(𝑜2/𝑙) queries in total Short edges in 𝑙-NN: (𝑣, 𝑤) s.t. 𝑒 𝑣, 𝑤 ≤ 10𝑛𝑣 Long edges in 𝑙-NN: all other edges We separately estimate total length of short edges and total length of long edges

Upper bound for (𝟐 + 𝜻)-approximation

Short edges in 𝑙-NN: (𝑣, 𝑤) s.t. 𝑒 𝑣, 𝑤 ≤ 10𝑛𝑣 Long edges in 𝑙-NN: all other edges

• We separately estimate total length of short edges and

total length of long edges by some random sampling methods Summing estimations of short and long edges gives a (1 + 𝜁)–approximation of cost(X)

Lower bound for (𝟐 + 𝜻)-approximation

Consider two problem instances:

– inner-cluster distance ~ 0; outer-cluster distance ~1

• 𝑜

𝑙+1 clusters of size 𝑙 + 1 each

– cost(X)~0

• 𝑜

𝑙+1 − 1 clusters of size 𝑙 + 1 each; one cluster of size

𝑙, one cluster of size 1

– cost(Y) ≫ 0

We prove that one requires Ω(𝑜2/𝑙) queries to distinguish between these two problem instances

Lower bound for (𝟐 + 𝜻)-approximation

clusters of size 𝑙 + 1 each cost(X) ~ 0

Lower bound for (𝟐 + 𝜻)-approximation

clusters of size 𝑙 + 1 each cost(X) ~ 0 In other instance: remove a random point from its cluster cost(Y) ≫ 0

Lower bound for (𝟐 + 𝜻)-approximation

clusters of size 𝑙 + 1 each cost(X) ~ 0 In other instance: remove a random point from its cluster cost(Y) ≫ 0 To find that single point one needs Ω(𝑜2/𝑙) queries

• 𝑃(𝑜) samples to hit it first
• 𝑃(𝑜/𝑙) further samples to detect no neighbors

Lower bound for (𝟐 + 𝜻)-approximation

Consider two problem instances:

– inner-cluster distance ~ 0; outer-cluster distance ~1

• 𝑜

𝑙+1 clusters of size 𝑙 + 1 each

– cost(X)~0

• 𝑜

𝑙+1 − 1 clusters of size 𝑙 + 1 each; one cluster of size

𝑙, one cluster of size 1

– cost(Y) ≫ 0

We prove that one requires Ω(𝑜2/𝑙) queries to distinguish between these two problem instances

Approximating with error 𝜻(cost(X) + MST(X))

Simplifying assumptions:

• All distances are of the form 1 + 𝜁 𝑗
• All distances are between 1 and poly(𝑜)

The cost of a 𝑙-NN graph & threshold graphs

Threshold graphs [Chazelle, Rubinfeld, Trevisan, SICOMP 2005;

C, Sohler, SICOMP 2009]

Let 𝐻 be the 𝑙-NN graph of an 𝑜-point metric space

• 𝐻(𝑗) = (𝑊, 𝐹(𝑗)) contain edges of 𝐻 of weight ≤ 1 + 𝜁 𝑗
• deg(𝑗) 𝑣 = outdegree of 𝑣 in 𝐻(𝑗)

Simple formula for 𝑙-NN cost: cost 𝑌 = 𝑜𝑙 + 𝜁 ෍

𝑗

1 + 𝜁 𝑗 ⋅ ෍

𝑣

(𝑙 − deg 𝑗 (𝑣))

The cost of a 𝑙-NN graph

cost 𝑌 = 𝑜𝑙 + 𝜁 ෍

𝑗

1 + 𝜁 𝑗 ⋅ ෍

𝑣

(𝑙 − deg 𝑗 (𝑣)) Implies that it suffices to estimate σ𝑣(𝑙 − deg 𝑗 (𝑣))

The cost of a 𝑙-NN graph

cost 𝑌 = 𝑜𝑙 + 𝜁 ෍

𝑗

1 + 𝜁 𝑗 ⋅ ෍

𝑣

(𝑙 − deg 𝑗 (𝑣)) Implies that it suffices to estimate σ𝑣(𝑙 − deg 𝑗 (𝑣))

• deg 𝑗 (𝑣) is given implicitly only; we need to sample

distances to all neighbors to compute deg 𝑗 (𝑣) exactly

The cost of a 𝑙-NN graph

cost 𝑌 = 𝑜𝑙 + 𝜁 ෍

𝑗

1 + 𝜁 𝑗 ⋅ ෍

𝑣

(𝑙 − deg 𝑗 (𝑣)) Implies that it suffices to estimate σ𝑣(𝑙 − deg 𝑗 (𝑣)) What about simple random sampling?

sample 𝑡 points and for each sample 𝑠 of their neighbors to estimate 𝑙 − deg 𝑗 (𝑣)

Won’t work well: – Cluster with 𝑜 − 1 points and a single outlier – We need to find the single outlier – Sample size must be 𝛁(𝑜)

The cost of a 𝑙-NN graph

cost 𝑌 = 𝑜𝑙 + 𝜁 ෍

𝑗

1 + 𝜁 𝑗 ⋅ ෍

𝑣

(𝑙 − deg 𝑗 (𝑣)) Implies that it suffices to estimate σ𝑣(𝑙 − deg 𝑗 (𝑣)) Idea (inspired by MST approximation due to C&Sohler, 2009):

• If 𝑣 has many (≫ 𝑙) neighbors of distance ≤ 1 + 𝜁 𝑗

 𝑣 doesn’t contribute to our cost function

– random sampling can detect it

• Otherwise, we can afford slower random sampling

Approximating number of vertices of given degree

For a given point 𝑣, sample vertices uniformly at random

• If the number of sampled vertices with distance ≤

1 + 𝜁 𝑗 to 𝑣 is sufficiently large then 𝑣 is not-useful

• Otherwise, double the sample size and repeat until it

becomes 𝑡 = 𝑃(𝑜 log 𝑜 /𝑙)

– return 𝑣 as useful

• If 𝑣 has 𝑒 points at distance ≤ 1 + 𝜁 𝑗, then w.h.p., in

expected time 𝑃(𝑜 log 𝑜/ (𝑙 + 𝑒)):

– if 𝑒 ≥ 4𝑙 then we mark 𝑣 as non-useful – If 𝑒 ≤ 𝑙 then we mark 𝑣 as useful

Approximating number of vertices of given degree

Using this approach, we can take a sample of size 𝑃(𝑜 1 + 𝜁 𝑗/𝜁𝑃(1)MST(X)) to get a desired error Expected running time evaluation on a randomly chosen vertex is 𝑃(𝑙 log 𝑜 𝑁𝑇𝑈(𝑌)/ 1 + 𝜁 𝑗) Theorem We can approximate the cost of a 𝑙-NN graph in time 𝑃(𝑜𝑙2/𝜁𝑃(1)) with an error of 𝜁 cost(X) + 𝑁𝑇𝑈(𝑌)

• 𝑙2 can be improved to 𝑙3/2

Lower bound

clusters of size 𝑙 + 1 each cost(X) ~ 0, MST(X) ~ 𝑜/𝑙

Lower bound

clusters of size 𝑙 + 1 each cost(X) ~ 0, MST(X) ~ 𝑜/𝑙 In other instance: in some clusters move 𝑙 points to some other cluster

Lower bound

clusters of size 𝑙 + 1 each cost(X) ~ 0, MST(X) ~ 𝑜/𝑙 cost(Y) ≫ 0, MST(Y) ~ 𝑜/𝑙 In other instance: in some clusters move 𝑙 points to some other cluster

Lower bound

clusters of size 𝑙 + 1 each cost(X) ~ 0, MST(X) ~ 𝑜/𝑙 cost(Y) ≫ 0, MST(Y) ~ 𝑜/𝑙 To distinguish between the instances one needs Ω𝜁(𝑜𝑙3/2) queries (for small 𝑙); for large 𝑙 it’s still Ω(𝑜2/𝑙) queries In other instance: in some clusters move 𝑙 points to some other cluster

Approximating cost(X): Summary of results

We have tight bounds for estimation of cost(X) When we want a (1 + 𝜁)–approximation:

• ෩

Θ𝜁(n2/k) queries are sufficient and necessary When happy with 𝜁(cost(X) + MST(X)) additive error:

• ෩

Θ𝜁(min{𝑜𝑙3/2, 𝑜2/𝑙}) queries are sufficient/necessary

Open problems

Sublinear algorithms in metric spaces:

• more problems approximated in sublinear-time
• if we add some other parameters to the error bound

(like MST) – can we approximate more problems in sublinear time

THANK YOU!