Locally private learning without interaction requires separation - PowerPoint PPT Presentation

Locally private learning without interaction requires separation Vitaly Feldman Research with Amit Daniely Hebrew University

Local Differential Privacy (LDP) 𝑨 1 [KLNRS ‘08] 𝜗 -LDP if for every user 𝑗, message 𝑘 is sent using a local 𝜗 𝑗,𝑘 -DP randomizer 𝑨 2 𝐵 𝑗,𝑘 and ≤ 𝜗 𝜗 𝑗,𝑘 𝑨 3 𝑘 Server 𝑨 𝑜

Non-interactive LDP 𝑨 1 𝑨 2 𝑨 3 Server 𝑨 𝑜

PAC learning PAC model [Valiant ‘84] : Let 𝐷 be a set of binary classifiers over 𝑌 𝐵 is a PAC learning algorithm for 𝐷 if ∀𝑔 ∈ 𝐷 and distribution 𝐸 over 𝑌 , given i.i.d. examples 𝑦 𝑗 , 𝑔 𝑦 𝑗 for 𝑦 𝑗 ∼ 𝐸 , 𝐵 outputs ℎ such that w.h.p. 𝑦∼𝐸 ℎ 𝑦 ≠ 𝑔(𝑦) ≤ 𝛽 𝐐𝐬 Distribution-specific learning: 𝐸 is fixed and known to 𝐵 4

Statistical query model [Kearns ‘93] 𝜚 1 𝑤 1 𝑄 distribution over 𝑎 𝜚 2 𝑎 = 𝑌 × {±1} 𝑤 2 𝑄 is the distribution of 𝑄 (𝑦, 𝑔 𝑦 ) for 𝑦 ∼ 𝐸 𝜚 𝑟 SQ algorithm 𝑤 𝑟 SQ oracle 𝑤 1 − 𝐅 𝑨∼𝑄 𝜚 1 𝑨 ≤ 𝜐 𝜚 1 : 𝑎 → 0,1 𝜐 is tolerance of the query; 𝜐 = 1/ 𝑜 [KLNRS ‘08] Simulation with success prob. 1 − 𝛾 (𝜗 ≤ 1) 𝛾 𝜗 -LDP with 𝑛 messages ⇒ 𝑃 𝑛 queries with 𝜐 = Ω 𝑛 • 𝑟 log 𝑟/𝛾 𝑟 queries with tolerance 𝜐 ⇒ 𝜗 -LDP with 𝑜 = 𝑃 samples/messages • 𝜐𝜗 2 Non-interactive if and only if queries are non-adaptive

Known results 𝐷 is SQ-learnable efficiently (non-adaptively) if and only if learnable efficiently with 𝜗 -LDP (non-interactively) Examples: Yes: halfspaces/linear classifiers [Dunagan,Vempala ‘04] • No: parity functions [Kearns ‘93] • Yes, non-adaptively: Boolean conjunctions • [KLNRS 08] There exists 𝐷 that is 1. SQ/LDP-learnable efficiently over the uniform distribution on 0,1 𝑒 but 2. requires exponential num. of samples to learn non-interactively by an LDP algorithm [KLNRS 08] : Does separation hold for distribution-independent learning? Masked parity 6

Margin Complexity - - - - - - Margin complexity of 𝐷 over 𝑌 - 𝐍𝐃(𝐷) : - - smallest 𝑁 such that exists an embedding Ψ: 𝑌 → 𝐂 𝑒 (1) under - - - 1 - which every 𝑔 ∈ 𝐷 is linearly separable with margin 𝛿 ≥ 𝑁 + + + Positive examples Ψ 𝑦 𝑔 𝑦 = +1} + + + + + Negative examples Ψ 𝑦 𝑔 𝑦 = −1} + + + + 7

Lower bound Thm: Let 𝐷 be a negation-closed set of classifiers. Any non-interactive 1 -LPD algorithm that learns 𝐷 with error 𝛽 < 1/2 and success probability Ω 1 needs 𝑜 = Ω 𝐍𝐃 𝐷 2/3 Corollaries: Decision lists over 0,1 𝑒 : 𝑜 = 2 Ω 𝑒 1/3 • [Buhrman,Vereshchagin,de Wolf ‘07] 𝑒 (Interactively) learnable with 𝑜 = poly 𝛽𝜗 [Kearns ’93] Linear classifiers over 0,1 𝑒 : 𝑜 = 2 Ω 𝑒 • [Goldmann,Hastad,Razborov ‘92; Sherstov ‘07] 𝑒 (Interactively) learnable with 𝑜 = poly 𝛽𝜗 [Dunagan,Vempala ’04] 8

Upper bound Thm: For any 𝐷 and distribution 𝐸 there exists a non-adaptive 𝜗 -LPD algorithm that learns 𝐷 over 𝐸 with error 𝛽 and success probability 1 − 𝛾 using 𝑜 = poly 𝐍𝐃 𝐷 ⋅ log 1/𝛾 𝛽𝜗 Instead of fixed 𝐸 access to public unlabeled samples from 𝐸 • (interactive) LDP access to unlabeled samples from 𝐸 • Lower bound holds against the hybrid model 9

Lower bound technique Thm: Let 𝐷 be a negation-closed set of classifiers. If exists a non-adaptive SQ algorithm that uses 𝑟 queries of tolerance 1/𝑟 to learn 𝐷 with error 𝛽 < 1/2 and success probability Ω 1 then 𝐍𝐃 𝐷 = 𝑃 𝑟 3/2 Correlation dimension of 𝐷 - CSQdim(𝐷) [F. ’08] : smallest 𝑢 for which exist 𝑢 functions ℎ 1 , … , ℎ 𝑢 : 𝑌 → [−1,1] such that for every 𝑔 ∈ 𝐷 and distribution 𝐸 exists 𝑗 such that ≥ 1 𝑦∼𝐸 𝑔 𝑦 ℎ 𝑗 𝑦 𝐅 𝑢 Thm: [F. ’08; Kallweit,Simon ‘11] : 𝐍𝐃 𝐷 ≤ CSQdim 𝐷 3/2 10

Proof If exists a non-adaptive SQ algorithm 𝐵 that uses 𝑟 queries of tolerance 1/𝑟 to learn 𝐷 with error 𝛽 < 1/2 then CSQdim 𝐷 ≤ 𝑟 Let 𝜚 1 , … , 𝜚 𝑟 : 𝑌 × ±1 → 0,1 be the (non-adaptive) queries of 𝐵 Decompose 𝜚 𝑦, 𝑧 = 𝜚 𝑦, 1 + 𝜚 𝑦, −1 + 𝜚 𝑦, 1 − 𝜚 𝑦, −1 ⋅ 𝑧 2 2 𝑕 ℎ 𝑦∼𝐸 𝑔 𝑦 ℎ 𝑗 𝑦 𝑦∼𝐸 𝜚 𝑗 (𝑦, 𝑔 𝑦 ) = 𝐅 𝐅 𝑦∼𝐸 𝑕 𝑗 𝑦 + 𝐅 ≤ 1 If 𝐅 𝑟 then 𝐅 𝑦∼𝐸 𝜚 𝑗 (𝑦, −𝑔 𝑦 ) 𝑦∼𝐸 𝑔 𝑦 ℎ 𝑗 𝑦 𝑦∼𝐸 𝜚 𝑗 (𝑦, 𝑔 𝑦 ) ≈ 𝐅 If this holds for all 𝑗 ∈ [𝑟] , then the algorithm cannot distinguish between 𝑔 and −𝑔 Cannot achieve error < 1/2 11

Upper bound Thm: For any 𝐷 and distribution 𝐸 there exists a non-adaptive 𝜗 -LPD algorithm that learns 𝐷 over 𝐸 with error 𝛽 < 1/2 and success probability 1 − 𝛾 using 𝑜 = poly 𝐍𝐃 𝐷 ⋅ log 1/𝛾 𝛽𝜗 Margin complexity of 𝐷 over 𝑌 - 𝐍𝐃(𝐷) : smallest 𝑁 such that exists an embedding Ψ: 𝑌 → 𝐂 𝑒 (1) under which every 𝑔 ∈ 𝐷 is 1 linearly separable with margin 𝛿 ≥ 𝑁 Thm [Arriaga,Vempala ’99; Ben -David,Eiron,Simon ‘02] : For every every 𝑔 ∈ 𝐷, random projection into 𝐂 𝑒 (1) for 𝑒 = 𝑃(𝐍𝐃 𝐷 2 log(1/𝛾)) 1 𝟑 𝐍𝐃 𝐷 ensures that with prob. 1 − 𝛾 , 1 − 𝛾 fraction of points are linearly separable with margin 𝛿 ≥ 12

Algorithm Perceptron: if sign( 𝑥 𝑢 , 𝑦 ) ≠ 𝑧 then update 𝑥 𝑢+1 ← 𝑥 𝑢 + 𝑧𝑦 Expected update: (𝑦,𝑧)∼𝑄 𝑧𝑦 | sign( 𝑥 𝑢 , 𝑦 ) ≠ 𝑧 𝐅 || scalar ≥ 𝛽 (𝑦,𝑧)∼𝑄 𝑧𝑦 ⋅ 𝟚(sign( 𝑥 𝑢 , 𝑦 ) ≠ 𝑧) / 𝐅 (𝑦,𝑧)∼𝑄 sign( 𝑥 𝑢 , 𝑦 ) ≠ 𝑧 𝐐𝐬 || 𝑦,𝑧 ∼𝑄 𝑦𝑧 + 𝐅 𝑦,𝑧 ∼𝑄 𝑦 sign 𝑥 𝑢 , 𝑦 𝐅 (𝑦,𝑧)∼𝑄 𝑦 ⋅ 𝑧 − sign 𝑥 𝑢 , 𝑦 = 𝐅 2 2 independent of the label non-adaptive Estimate the mean vector with ℓ 2 error • LDP [Duchi,Jordan,Wainright ‘ 13] • SQs [F.,Guzman,Vempala ‘15] 13

Conclusions • New approach to lower bounds for non-interactive LDP o Reduction to margin-complexity lower bounds • Lower bounds for classical learning problems • Same results for communication constrained protocols o Also equivalent to SQ • Interaction is necessary for learning • Open: o Distribution-independent learning in poly 𝐍𝐃 𝐷 o Lower bounds against 2 + round protocols o Stochastic convex optimization https://arxiv.org/abs/1809.09165 14