On the Downstream Performance of Compressed Word Embeddings Avner - PowerPoint PPT Presentation

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. On the Downstream Performance of Compressed Word Embeddings Avner May, Jian Zhang, Tri Dao, Chris Ré Stanford University

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Word Embeddings 2

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Word Embeddings Important for strong NLP performance 2

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Word Embeddings Important for strong NLP performance Take a lot of memory 2

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Word Embedding Compression 3

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. What determines whether a compressed embedding matrix will perform well on downstream tasks? 4

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. What determines whether a compressed embedding matrix will perform well on downstream tasks? Train model 4

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. What determines whether a compressed embedding matrix will perform well on downstream tasks? Train model 4

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. What determines whether a compressed embedding matrix will perform well on downstream tasks? Train model ?? Train model 4

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Motivating Observation Existing ways of measuring compression quality often fail to explain relative downstream performance. 5

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Motivating Observation Existing ways of measuring compression quality often fail to explain relative downstream performance. Better compression quality measure 5

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Motivating Observation Existing ways of measuring compression quality often fail to explain relative downstream performance. Better compression Worse downstream quality measure performance 5

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Our Contributions: Outline 6

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Our Contributions: Outline 1 Define a new measure of compression quality. 6

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Our Contributions: Outline 1 Define a new measure of compression quality. Prove generalization bounds using this measure. 2 6

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Our Contributions: Outline 1 Define a new measure of compression quality. Prove generalization bounds using this measure. 2 Show strong empirical correlation w. downstream performance. 3 6

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Our Contributions: Outline 1 Define a new measure of compression quality. Prove generalization bounds using this measure. 2 Show strong empirical correlation w. downstream performance. 3 Use measure to select compressed embeddings. 4 6

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Our Contributions: Outline 1 Define a new measure of compression quality. Prove generalization bounds using this measure. 2 Show strong empirical correlation w. downstream performance. 3 Use measure to select compressed embeddings. 4 Up to 2x lower selection error rates than the next best measure. 6

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Defining the Measure : Intuition from Linear Regression Observation: Predictions are determined by data matrix’s left singular vectors. 7

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Defining the Measure : Intuition from Linear Regression Observation: Predictions are determined by data matrix’s left singular vectors. = Embed. Singular Value matrix Decomposition 7

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Defining the Measure : Intuition from Linear Regression Observation: Predictions are determined by data matrix’s left singular vectors. = Embed. Singular Value matrix Decomposition 7

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Defining the Measure : Intuition from Linear Regression Observation: Predictions are determined by data matrix’s left singular vectors. = Embed. Singular Value Regression label y matrix Decomposition 7

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Defining the Measure : Intuition from Linear Regression Observation: Predictions are determined by data matrix’s left singular vectors. Linear regressor predictions = Embed. Singular Value Regression label Project y onto span of y matrix Decomposition left singular vectors 7

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Defining the Measure : Eigenspace Overlap Score (EOS) Intuition: Measures similarity between the span of left singular vectors. 8

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Defining the Measure : Eigenspace Overlap Score (EOS) Intuition: Measures similarity between the span of left singular vectors. EOS Eigenspace Compressed Uncompressed overlap score embed. SVD embed. SVD 8

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Theoretical Results : Linear Regression Theorem (informal) : Expected difference in test mean-squared error attained by compressed vs. uncompressed embeddings is determined by EOS . 9

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Theoretical Results : Linear Regression Theorem (informal) : Expected difference in test mean-squared error attained by compressed vs. uncompressed embeddings is determined by EOS . Higher EOS 9

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Theoretical Results : Linear Regression Theorem (informal) : Expected difference in test mean-squared error attained by compressed vs. uncompressed embeddings is determined by EOS . Better downstream Higher EOS performance 9

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Empirical Correlation: Beyond Linear Regression EOS attains strong correlation with downstream model accuracy . 10

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Empirical Correlation: Beyond Linear Regression EOS attains strong correlation with downstream model accuracy . Higher accuracy EOS Higher quality 10

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Empirical Correlation: Beyond Linear Regression EOS attains strong correlation with downstream model accuracy . Higher accuracy EOS Higher quality 10

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Empirical Correlation: Beyond Linear Regression EOS attains strong correlation with downstream model accuracy . Higher accuracy Neg. PIP Loss [1] EOS Higher quality Higher quality [1] Yin and Shen, On the Dimensionality of Word Embeddings . NeurIPS 2018. 10

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. EOS as a Selection Criterion EOS attains up to 2x lower selection error rates than 2 nd best. 11

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. EOS as a Selection Criterion EOS attains up to 2x lower selection error rates than 2 nd best. Selection Error Rate (%) NLP Tasks [1] Avron et al., ICML 2017. [2] Yin and Shen. NeurIPS 2018. [3] Zhang et al., AISTATS 2019. 11

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. EOS as a Selection Criterion EOS attains up to 2x lower selection error rates than 2 nd best. Selection Error Rate (%) NLP Tasks [1] Avron et al., ICML 2017. [2] Yin and Shen. NeurIPS 2018. [3] Zhang et al., AISTATS 2019. 11

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Our Contributions: Summary 12

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Our Contributions: Summary 1 Defined a new measure of compression quality. 12

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Our Contributions: Summary 1 Defined a new measure of compression quality. Proved generalization bounds using this measure. 2 12

On the Downstream Performance of Compressed Word Embeddings. NeurIPS Spotlight 12/12/19. Our Contributions: Summary 1 Defined a new measure of compression quality. Proved generalization bounds using this measure. 2 Showed strong empirical correlation w. downstream perf. 3 12