Poisson Subsampled Rnyi Differential Privacy Yuqing Zhu joint work - PowerPoint PPT Presentation

Poisson Subsampled Rényi Differential Privacy Yuqing Zhu joint work with Yu-Xiang Wang 1

Privacy Amplification by Sampling 𝑃 𝛿𝜗, 𝛿𝜀 -DP 𝜗, 𝜀 -DP Sampling probability 𝛅 [KLNRS’08], [Li et al., 2011] 𝜗, 𝛽 -Rényi DP What’s the optimal bound ? Strong composition tool 2

Example: The Noisy SGD Algorithm Song et al. 2013; Bassily et al. 2014 ! 1 X r f i ( ✓ t ) + Z t ✓ t +1 ✓ t � ⌘ t |I| i ∈ I 1.Randomly chosen minibatch (Poisson subsampling) 2.Then add Gaussian noise (Gaussian mechanism) RDP analysis for subsampled Gaussian mechanism (Abadi et al., 2016) Really what makes Deep Learning with Differential Privacy practical 3

Exact RDP of Subsampled Mechanism Let M be any randomized algorithm that obeys ( α , ϵ ( α )) − RDP γ be the subsampling probability and for integer α≥ 2 Asymptotic rate 𝜗 (∘𝒕𝒃𝒏𝒒𝒎𝒇 𝛽 ≤ Ο(𝛽𝛿 4 𝜗 2 ) This asymptotic rate holds for any mechanism M ! 4

Exact RDP of Subsampled Mechanism Let M be any randomized algorithm that obeys ( α , ϵ ( α )) − RDP γ be the subsampling probability and for integer α≥ 2 𝝑 𝐍∘𝑻𝒃𝒏𝒒𝒎𝒇 𝜷 ≤ 𝟐 𝜷 𝐦𝐩𝐡{ 𝟐 − 𝜹 𝜷B𝟐 𝜷𝜹 − 𝜹 + 𝟐 + 𝜷 𝟑 𝜹 𝟑 𝟐 − 𝜹 𝜷B𝟑 𝒇 𝝑 𝟑 𝜷 +𝟒 F 𝜷 𝟐 − 𝜹 𝜷Bℓ 𝜹 ℓ 𝒇 ℓB𝟐 𝝑(ℓ) } ℓ ℓI𝟒 This bound is optimal, up to a factor of 3 on a low order term 5

Exact Amplification Bound for RDP Let M be any randomized algorithm that obeys ( α , ϵ ( α )) − RDP γ be the subsampling probability and for integer α≥ 2 𝝑 𝐍∘𝑻𝒃𝒏𝒒𝒎𝒇 𝜷 ≤ 𝟐 𝜷 𝐦𝐩𝐡{ 𝟐 − 𝜹 𝜷B𝟐 𝜷𝜹 − 𝜹 + 𝟐 + 𝜷 𝟑 𝜹 𝟑 𝟐 − 𝜹 𝜷B𝟑 𝒇 𝝑 𝟑 𝜷 +𝟒 F 𝜷 𝟐 − 𝜹 𝜷Bℓ 𝜹 ℓ 𝒇 ℓB𝟐 𝝑(ℓ) } ℓ ℓI𝟒 Get rid of it Matches the lower bound when M is Gaussian or Laplace mechanism 6

� Overall ( 𝜗 , 𝜀 )-DP over composition 𝑷(𝑳) 𝑷( 𝑳 ) Subsampled Gaussian Mechanism 𝜏 = 5, 𝛿 = 1𝑓 − 3 7

Low Privacy Regime Subsampled Gaussian Mechanism 𝜏 = 1, 𝛿 = 1𝑓 − 3 8

Thank you! Poster Number Pacific Ballroom #178 Code available: https://github.com/yuxiangw/autodp Or just use: pip install autodp Get Paper 9