Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix - PowerPoint PPT Presentation

Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, Sanjiv Kumar

Overview

Application: recommender systems

Application: recommender systems

Application: recommender systems

Application: recommender systems

Application: recommender systems

MIPS: paruitioning and quantization Paruitioning: Split database into disjoint subsets ● Search only the most promising subsets ●

MIPS: paruitioning and quantization Paruitioning: Split database into disjoint subsets ● Search only the most promising subsets ● Quantization: Reduce the number of bits used to describe data ● points. Leads to smaller index size and faster inner ● product calculations.

MIPS: paruitioning and quantization Paruitioning: Split database into disjoint subsets ● Search only the most promising subsets ● Quantization: Reduce the number of bits used to describe data ● points. Leads to smaller index size and faster inner ● product calculations.

Quantization overview: codebooks Given a set of vectors x 1 , x 2 , …, x n , we want to create a quantized dataset x̃ 1 , x̃ 2 , …, x̃ n . Quantize to an element of the codebook, C θ

Example codebook: vector quantization Parameters are a set of centers c 1 , c 2 , …, c k . Codebook C θ is the set of all centers: {c 1 , c 2 , …, c k }.

Example codebook: vector quantization Parameters are a set of centers c 1 , c 2 , …, c k . Codebook C θ is the set of all centers: {c 1 , c 2 , …, c k }. Product quantization: splits the space into multiple subspaces ● uses a vector quantization codebook for each subspace. ●

Quantization basics: assignment To assign a datapoint to a codeword, we select the codeword that minimizes a loss function.

Traditional loss function choice Classic approach: reconstruction error.

Traditional loss function choice Classic approach: reconstruction error. By Cauchy-Schwaruz:

Some inner product errors are worse than others Consider a query q and database points x 1 , … , x n Rank points by inner product

Some inner product errors are worse than others Consider a query q and database points x 1 , … , x n Rank points by inner product

Some inner product errors are worse than others Consider a query q and database points x 1 , … , x n Rank points by inner product a n- a n- a 1 a 2 a 3 a 4 a 5 … a n-1 a n 3 2 Low inner High inner product product

Some inner product errors are worse than others Consider a query q and database points x 1 , … , x n Rank points by inner product MIPS Results a n- a n- a 1 a 2 a 3 a 4 a 5 … a n-1 a n 3 2 Low inner High inner product product

Some inner product errors are worse than others Consider a query q and database points x 1 , … , x n Rank points by inner product MIPS Results a n- a n- a 1 a 2 a 3 a 4 a 5 … a n-1 a n 3 2 Low inner High inner product product Peruurbations of low inner products are unlikely to result in changes to top-k

Some inner product errors are worse than others Consider a query q and database points x 1 , … , x n Rank points by inner product MIPS Results a n- a n- a 1 a 2 a 3 a 4 a 5 … a n-1 a n 3 2 Low inner High inner product product Peruurbations of low Peruurbations of high inner inner products are products change top-k and unlikely to result in lead to recall loss changes to top-k

Some inner product errors are worse than others Consider a query q and database points x 1 , … , x n Rank points by inner product MIPS Results a n- a n- a 1 a 2 a 3 a 4 a 5 … a n-1 a n 3 2 Low inner High inner product product Takeaway: to maximize Peruurbations of low Peruurbations of high inner recall, emphasize inner products are products change top-k and reducing quantization unlikely to result in lead to recall loss error for high inner changes to top-k products

Visualization of query distribution x

Visualization of query distribution x Quantization error: litule impact on MIPS recall

Visualization of query distribution x Quantization error: some impact on MIPS recall

Visualization of query distribution x Quantization error: signifjcant impact on MIPS recall

Visualization of query distribution x x Reconstruction loss Quantization error: signifjcant impact on MIPS recall

Score-aware quantization loss Traditional quantization loss: Score-aware loss: N By earlier intuition, w should put more weight on higher . Example weight function: w(t) = 1 (t ≥T).

Evaluating and minimizing score-aware loss Expand expectation:

Evaluating and minimizing score-aware loss r ⟂ r || error x x̃

Evaluating and minimizing score-aware loss Integral evaluates to a weighted sum of r || and r ⟂ : For w that weight higher inner products more,

Visualization of result c 1 gives lower inner product error than c 2 even though ||x - c 1 || > ||x - c 2 || Reason: x - c 1 is oruhogonal, not parallel, to x

Applications to quantization Given a family of codewords C, we now want to solve the following optimization problem. We work out an approach for effjcient approximate optimization in the large-scale setuing for: 1. Vector Quantization 2. Product Quantization

Constant-bitrate comparison GloVe: 100 dimensions, 1183514 points Cosine distance dataset; normalize dataset to unit-norm during training time 25 codebooks, 16 centers each 50 codebooks, 16 centers each

Glove: QPS-recall experiment setup Pruning via K-means tree Quantized Scoring Exact re-scoring 2000 centers; all but the Compute approximate Top b inner products closest a centers to the inner products via with from AH are query are pruned quantized database re-computed exactly; top (product quantization 10 are returned as MIPS with anisotropic loss) results Higher a , b result in higher recall, lower QPS

Glove: QPS-recall pareto frontier

Glove: QPS-recall pareto frontier Source code: https://github.com/google-research/google-research/tree/master/scann