Faster Cover Trees Mike Izbicki and Christian R. Shelton UC - PowerPoint PPT Presentation

Faster Cover Trees Mike Izbicki and Christian R. Shelton UC Riverside Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 1 / 21

Outline Why care about faster cover trees? Making cover trees faster. Experimental setup Simpler definition reduces the number of nodes The nearest ancestor invariant Better cache performance Constructing and querying the tree in parallel Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 2 / 21

Methods for fast nearest neighbor queries: provable arbitrary high speedup metric dimensions quadtree yes no no k d-tree yes no somewhat hashing yes no yes ball tree no yes somewhat yes yes yes cover tree (Beygelzimer, Kakade, and Langford, 2006) Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 3 / 21

Other uses of cover trees Any learning algorithm that cares about distance can be made faster using cover trees. Examples: k -nearest neighbor Support vector machines (Segata and Blanzieri, 2010) Dimensionality reduction (Lisitsyn et. al. , 2010) Reinforcement learning (Tziortziotis et. al. , 2014) Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 4 / 21

Outline Why care about faster cover trees? Making cover trees faster. ◮ Experimental setup ◮ Simpler definition reduces the number of nodes ◮ The nearest ancestor invariant ◮ Better cache performance ◮ Constructing and querying the tree in parallel Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 5 / 21

Experimental setup Three data sources: MLPack benchmarks with Euclidean distance Protein dataset with the random walk graph distance Yahoo! 1.5 million creative common images with the earth movers distance Benchmarking procedure: Construct a cover tree on the dataset For each data point in the dataset, find the nearest neighbor Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 6 / 21

The simplified cover tree level 3 10 level 2 8 12 level 1 7 9 The covering invariant. For every node p , define the function covdist ( p ) = 2 level ( p ) . For each child q of p d ( p , q ) ≤ covdist ( p ) The separating invariant. For every node p , define the function sepdist ( p ) = 2 level ( p ) − 1 . For all distinct children q 1 and q 2 of p d ( q 1 , q 2 ) ≥ sepdist ( p ) Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 7 / 21

The simplified cover tree level 3 10 level 2 8 12 level 1 7 9 Advantages of the simplified cover tree: Maintains all runtime guarantees of the original cover tree. Significantly easier to understand and implement. The original cover tree was described in terms of an infinitely large tree, only a subset of which actually gets implemented. Requires exactly n nodes instead of O ( n ) nodes. Fewer nodes means a faster constant factor for all algorithms. Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 7 / 21

The simplified cover tree fraction of nodes in the original cover tree required for the simplified cover tree 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 yearpredict twitter tinyImages mnist corel covtype artificial40 faces Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 8 / 21

The nearest ancestor cover tree level 3 10 10 level 2 8 12 8 12 level 1 7 11 9 13 7 9 11 13 A nearest ancestor cover tree is a simplified cover tree where every point p satisfies the additional invariant that if q 1 is an ancestor of p and q 2 is a sibling of q 1 , then d ( p , q 1 ) ≤ d ( p , q 2 ) Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 9 / 21

The nearest ancestor cover tree level 3 10 10 level 2 8 12 8 12 level 1 7 11 9 13 7 9 11 13 Insertions require rebalancing. No runtime guarantees on the rebalance step. In practice, queries are much faster and construction is only slightly slower. Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 9 / 21

Comparing cover trees on construction time Izbicki and Shelton (UC Riverside) number of distance comparisons in tree construction only (normalized by the original cover tree) 0 1 2 3 4 5 yearpredict twitter 19.1 tinyImages mnist Faster Cover Trees corel covtype artificial40 faces Original cover tree Simplified cover tree July 7, 2015 Nearest ancestor cover tree 10 / 21

Comparing cover trees on construction and query time Izbicki and Shelton (UC Riverside) number of distance comparisons in tree construction and query (normalized by n 2 ) 0.2 0.4 0.6 0.8 1.2 0 1 yearpredict twitter tinyImages mnist Faster Cover Trees corel covtype artificial40 faces Original cover tree Simplified cover tree July 7, 2015 Nearest ancestor cover tree 11 / 21

All of the cover trees scale similarly This experiment uses the protein data and the random walk graph kernel. total distance comparisons (millions) 1600 Nearest ancestor cover tree on construction and query 1400 1200 Simplified cover tree Original cover tree 1000 800 600 400 200 0 0 50 100 150 200 250 number of data points (thousands) Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 12 / 21

Cache oblivious cover tree Need to consider cache accesses for fast, modern data structures image from: http://1024cores.net Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 13 / 21

Cache oblivious cover tree Arrange nodes in memory according to a preorder traversal of the tree (van Emde Boas et al. , 1966; Demaine, 2002) image from: Wikipedia Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 14 / 21

The cache efficiency of three cover tree implementations 1 (cache misses / cache accesses) Without van embde boas 0.8 With van embde boas cache miss rate 0.6 0.4 0.2 0 yearpredict twitter tinyImages mnist corel covtype artificial40 faces Measured using Linux’s perf stat utility on an Amazon AWS instance Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 15 / 21

Merging cover trees Merging cover trees gives us a parallel tree construction algorithm Sometimes, merging cover trees is easy : level 3 10 level 2 8 12 level 1 7 9 11 13 No runtime bound on the merge operation, but it is fast in practice Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 16 / 21

Merging cover trees Merging cover trees gives us a parallel tree construction algorithm Sometimes, merging cover trees is hard : level 3 10 level 2 8 11.5 level 1 7 9 11 13 No runtime bound on the merge operation, but it is fast in practice Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 17 / 21

The effect of parallel tree construction on small datasets normalized tree construction time number of processors 2 +1 2 +0 1 2 2 − 1 Our cover tree 4 2 − 2 8 16 2 − 3 2 − 4 yearpredict twitter tinyImages mnist (77sec) (107sec) (65sec) (12sec) Experiments run on an Amazon AWS instance with 16 true cores Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 18 / 21

Parallel tree construction really matters on larger data sets On large datasets with an expensive metric, parallelism is more useful Yahoo! Flickr dataset with 1.5 million images and earth mover distance num cores simplified tree nearest ancestor tree time speedup time speedup 1 70.7 min 1.0 210.9 min 1.0 2 36.6 min 1.9 94.2 min 2.2 4 18.5 min 3.8 48.5 min 4.3 8 10.2 min 6.9 25.3 min 8.3 16 6.7 min 10.5 12.0 min 17.6 Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 19 / 21

The effect of parallel tree construction and query (both construction and query ) 2 +1 normalized total runtime 1 1 2 +0 1 MLPack’s cover tree Reference cover tree 2 2 − 1 Our cover tree 2 − 2 4 2 − 3 8 16 2 − 4 yearpredict twitter tinyImages mnist (277min) (51min) (34min) (30min) Experiments run on an Amazon AWS instance with 16 true cores Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 20 / 21

Summary You should use cover trees. We made them easier to implement and faster. All the code is licensed under the BSD3 and available at: http://github.com/mikeizbicki/hlearn Izbicki and Shelton (UC Riverside) Faster Cover Trees July 7, 2015 21 / 21