Pr Private Stochastic Convex Optimization wi with Optimal Ra Rate - PowerPoint PPT Presentation

Pr Private Stochastic Convex Optimization wi with Optimal Ra Rate Raef Bassily Vitaly Feldman Kunal Talwar Abhradeep Guha Thakaurta UC Santa Cruz Ohio State University Google Brain Google Brain Google Brain

This work Differentially private (DP) algorithms for stochastic convex optimization with optimal excess population risk

Stochastic Convex Optimization (SCO) Unknown distribution (population) 𝒠 over data universe 𝒶 Convex parameter space 𝒟 ⊂ ℝ 2 𝑀 4 /𝑀 4 setting: 𝒟 and 𝜖ℓ are bounded in 𝑀 4 -norm Convex loss function ℓ: 𝒟 × 𝒶 → ℝ Dataset 𝑇 = 𝑨 & , … , 𝑨 ) ∼ 𝒠 )

Stochastic Convex Optimization (SCO) Unknown distribution (population) 𝒠 over data universe 𝒶 Convex parameter space 𝒟 ⊂ ℝ 2 𝑀 4 /𝑀 4 setting: 𝒟 and 𝜖ℓ are bounded in 𝑀 4 -norm Convex loss function ℓ: 𝒟 × 𝒶 → ℝ Dataset 𝑇 = 𝑨 & , … , 𝑨 ) ∼ 𝒠 ) A SCO algorithm, given 𝑇 , outputs 7 𝜄 ∈ 𝒟 s.t. Excess Pop. Risk ≜ 𝔽 <∼𝒠 ℓ 7 𝜄, 𝑨 − min A∈𝒟 𝔽 <∼𝒠 ℓ 𝜄, 𝑨 is as small as possible & Well-studied problem: optimal rate ≈ )

Private Stochastic Convex Optimization (PSCO) Unknown distribution (population) 𝒠 over data universe 𝒶 Convex parameter space 𝒟 ⊂ ℝ 2 𝑀 4 /𝑀 4 setting: 𝒟 and 𝜖ℓ are bounded in 𝑀 4 -norm Convex loss function ℓ: 𝒟 × 𝒶 → ℝ Dataset 𝑇 = 𝑨 & , … , 𝑨 ) ∼ 𝒠 ) Goal: 𝜗, 𝜀 -DP algorithm 𝒝 FGHI that, given 𝑇 , outputs 7 𝜄 ∈ 𝒟 s.t. Excess Pop. Risk ≜ 𝔽 <∼𝒠 ℓ 7 𝜄, 𝑨 − min A∈𝒟 𝔽 <∼𝒠 ℓ 𝜄, 𝑨 is as small as possible

Main Result & 2 Optimal excess population risk for PSCO is ≈ max ) , L ) Optimal non-private Optimal private population risk empirical risk [ B S T 14]

Main Result & 2 Optimal excess population risk for PSCO is ≈ max ) , L ) When 𝑒 = Θ 𝑜 (common in modern ML) & Opt. risk for PSCO ≈ ) = opt. risk for SCO asymptotically no cost of privacy

Algorithms Two algorithms under mild smoothness assumption on ℓ : Ø A variant of mini-batch noisy SGD: Ø Objective Perturbation (entails rank assumption on ∇ 4 ℓ )

Algorithms Two algorithms under mild smoothness assumption on ℓ : Ø A variant of mini-batch noisy SGD: Ø Objective Perturbation (entails rank assumption on ∇ 4 ℓ ) • The objective function in both algorithms is the empirical risk .

Algorithms Two algorithms under mild smoothness assumption on ℓ : Ø A variant of mini-batch noisy SGD: Ø Objective Perturbation (entails rank assumption on ∇ 4 ℓ ) • The objective function in both algorithms is the empirical risk . • Generalization error is bounded via uniform stability : o For the first algorithm: uniform stability of SGD [HRS15, F V19]. o For the second algorithm: uniform stability due to regularization.

Algorithms • General non-smooth loss: Ø A new, efficient, noisy stochastic proximal gradient algorithm: o Based on Moreau-Yosida smoothing o A gradient step w.r.t. the smoothed loss is equivalent to a proximal step w.r.t. the original loss .

Results vs. Prior Work on DP-ERM This work & 2 Optimal excess population risk for PSCO is ≈ max ) , • L ) Previous work • Focused on the empirical version (DP-ERM): [CMS11, KS T 12, B S T 14, TT Z15, …] • Optimal empirical risk is previously known [ B S T 14], but not optimal population risk . 2 Q/R 2 • Best known population risk using DP-ERM algorithms ≈ max ) , L ) [ B S T 14].

Poster #163 Full version: arXiv:1908.09970