Lecture 12 - Filtering Welcome! , = (, ) , - PowerPoint PPT Presentation

INFOMAGR – Advanced Graphics Jacco Bikker - November 2019 - February 2020 Lecture 12 - “Filtering” Welcome! 𝑱 𝒚, 𝒚 ′ = 𝒉(𝒚, 𝒚 ′ ) 𝝑 𝒚, 𝒚 ′ + න 𝝇 𝒚, 𝒚 ′ , 𝒚 ′′ 𝑱 𝒚 ′ , 𝒚 ′′ 𝒆𝒚′′ 𝑻

Today’s Agenda: ▪ Noise ▪ Ingredients ▪ State of the Art

Today’s Agenda: ▪ Noise ▪ Ingredients ▪ State of the Art

Advanced Graphics – Filtering 9 Noise Path Tracing The rendering equation allows us to accurately simulate light transport: ▪ Given a sensible scene description ▪ Given a sensible camera model ▪ Ignoring some complex effects. We evaluate it using Monte Carlo integration: en.wikipedia.org/wiki/Path_tracing ➔ The result is always an estimate, with variance.

Advanced Graphics – Filtering 10 Noise Path Tracing The rendering equation allows us to accurately simulate light transport: ▪ Given a sensible scene description ▪ Given a sensible camera model ▪ Ignoring some complex effects. We evaluate it using Monte Carlo integration: www.mitsuba-renderer.org ➔ The result is always an estimate, with variance.

Advanced Graphics – Filtering 11 Noise Noise in the Real World Noise in a digital photo can be the result of short exposure. www.photoreview.com.au/tips/shooting/how-to-control-image-noise

Advanced Graphics – Filtering 12 Noise Noise in the Real World Noise in a digital photo can be the result of short exposure. We solve this by lowering our ISO settings, so we gather more photons per pixel. (that obviously has consequences) www.photoreview.com.au/tips/shooting/how-to-control-image-noise

Advanced Graphics – Filtering 13 Noise Reducing Noise in Path Tracing We can increase ‘exposure’ in a pathtracer by taking more samples. aras-p.info/blog/2018/03/28/Daily-Pathtracer-Part-1-Initial-C-/

Advanced Graphics – Filtering 14 Noise Reducing Noise in Path Tracing We can increase ‘exposure’ in a pathtracer by taking more samples. Or we can use Importance Sampling.

Advanced Graphics – Filtering 15 Noise Reducing Noise in Path Tracing We can increase ‘exposure’ in a pathtracer by taking more samples. Or we can use Importance Sampling. Or Next Event Estimation. However, at interactive rates it will be quite hard to reach a converged image.

Advanced Graphics – Filtering 16 Noise well-lit, diffuse: low noise Indirect light: depth of field: moderate noise moderate noise ? ?

Advanced Graphics – Filtering 17 Noise Not All Noise is Equal In general, if we can importance sample it, noise will be low: ▪ sample BRDF proportional to cos 𝜄 ; ▪ sample a light proportional to potential contribution. The ‘fireflies’ are the result of a random bounce, followed by a specular bounce, which then hit the light: a direction is sampled with a low probability, but yields a high value.

Advanced Graphics – Filtering 18 Noise 𝑂 ≈ 1 න 𝑔 𝑦 𝑒𝑦 = 𝐹 𝑔 𝑦 𝑂 ෍ 𝑔 𝑌 𝑗 𝑗=1 𝑂 𝑔 𝑌 𝑗 ≈ 1 𝐹 𝑔 𝑦 𝑂 ෍ 𝑞 𝑌 𝑗 𝑗=1 𝑚𝑏𝑠𝑕𝑓 𝑤𝑏𝑚𝑣𝑓 𝑚𝑏𝑠𝑕𝑓 𝑤𝑏𝑚𝑣𝑓 𝑡𝑛𝑏𝑚𝑚 𝑤𝑏𝑚𝑣𝑓 = 𝑤𝑓𝑠𝑧 𝑚𝑏𝑠𝑕𝑓 𝑚𝑏𝑠𝑕𝑓 𝑤𝑏𝑚𝑣𝑓 ≈ 1

Advanced Graphics – Filtering 19 Noise Not All Noise is Equal A good RNG is important when producing samples.

Advanced Graphics – Filtering 20 Noise Not All Noise is Equal A good RNG is important when producing samples. We can do better than a high quality uniform RNG however. Pseudo-random numbers can greatly improve the uniformity, while keeping the relevant properties of random numbers*. Relevant property: avoid correlation . *: C. Waechter. Quasi-Monte Carlo Light Transport Simulation by Efficient Ray Tracing. PhD thesis, Ulm, 2009.

Advanced Graphics – Filtering 21 Noise Not All Noise is Equal Our eyes expect noise. ‘Good noise’ is uniform, high- frequent, and… blue*. *: www.arnoldrenderer.com/arnold/research

Advanced Graphics – Filtering 22 Noise

Advanced Graphics – Filtering 23 Noise

Advanced Graphics – Filtering 24 Noise

Advanced Graphics – Filtering 25 Noise

Advanced Graphics – Filtering 26 Noise Suppressing Fireflies “A firefly is easily recognized in the final image: it is a pixel with a value that differs significantly from its neighbors.” ▪ Is this always true? ▪ How to fix it? ▪ Is that still correct? Firefly suppression introduces bias in our estimator. ▪ Spread out the removed energy over the image / neighborhood ▪ Just wait it out (additional samples will improve the average) ▪ Do some adaptive sampling (detect high variance) ▪ Just Just acc accept it it.

Today’s Agenda: ▪ Noise ▪ Ingredients ▪ State of the Art

Advanced Graphics – Filtering 28 Ingredients Reducing the Problem - Filtering Core idea: Exploit the fact that illumination is typically low-frequent: Nearby pixels tend to converge to similar values, so we should be able to use information gathered for one pixel to improve the estimate of the next. Note: Unless neighboring pixels actually converge to the same value, filtering will introduce bias.

Ƹ Advanced Graphics – Filtering 29 Ingredients Filter kernels For the actual filtering, we apply a kernel. σ 𝑘∈𝒪 𝑗 𝑑 𝑘 𝑥(𝑗, 𝑘) Pixel FilteredValue( 𝑗 𝑦 , 𝑗 𝑧 , ℎ𝑏𝑚𝑔𝑋𝑗𝑒𝑢ℎ ) 𝑑 𝑗 = 𝑡𝑣𝑛 = 0 σ 𝑘∈𝒪 𝑗 𝑥(𝑗, 𝑘) 𝑡𝑣𝑛𝑛𝑓𝑒𝑋𝑓𝑗𝑕ℎ𝑢 = 0 for 𝑘 𝑦 = 𝑗 𝑦 - ℎ𝑏𝑚𝑔𝑋𝑗𝑒𝑢ℎ to 𝑗 𝑦 + ℎ𝑏𝑚𝑔𝑋𝑗𝑒𝑢ℎ for 𝑘 𝑧 = 𝑗 𝑧 - ℎ𝑏𝑚𝑔𝑋𝑗𝑒𝑢ℎ to 𝑗 𝑧 + ℎ𝑏𝑚𝑔𝑋𝑗𝑒𝑢ℎ 𝑡𝑣𝑛 += ReadPixel( 𝑘 𝑦 , 𝑘 𝑧 ) * weight( 𝑘 𝑦 , 𝑘 𝑧 ) 𝑡𝑣𝑛𝑛𝑓𝑒𝑋𝑓𝑗𝑕ℎ𝑢 += weight( 𝑘 𝑦 , 𝑘 𝑧 ) return 𝑡𝑣𝑛 / 𝑡𝑣𝑛𝑛𝑓𝑒𝑋𝑓𝑗𝑕ℎ𝑢

Advanced Graphics – Filtering 30 Ingredients Filter kernels For the actual filtering, we apply a kernel. Pixel FilteredValue( 𝑗 𝑦 , 𝑗 𝑧 , ℎ𝑏𝑚𝑔𝑋𝑗𝑒𝑢ℎ ) 𝑡𝑣𝑛 = 0 𝑡𝑣𝑛𝑛𝑓𝑒𝑋𝑓𝑗𝑕ℎ𝑢 = 0 for 𝑘 𝑦 = 𝑗 𝑦 - ℎ𝑏𝑚𝑔𝑋𝑗𝑒𝑢ℎ to 𝑗 𝑦 + ℎ𝑏𝑚𝑔𝑋𝑗𝑒𝑢ℎ for 𝑘 𝑧 = 𝑗 𝑧 - ℎ𝑏𝑚𝑔𝑋𝑗𝑒𝑢ℎ to 𝑗 𝑧 + ℎ𝑏𝑚𝑔𝑋𝑗𝑒𝑢ℎ 𝑡𝑣𝑛 += ReadPixel( 𝑘 𝑦 , 𝑘 𝑧 ) * weight( 𝑘 𝑦 , 𝑘 𝑧 ) 𝑡𝑣𝑛𝑛𝑓𝑒𝑋𝑓𝑗𝑕ℎ𝑢 += weight( 𝑘 𝑦 , 𝑘 𝑧 ) return 𝑡𝑣𝑛 / 𝑡𝑣𝑛𝑛𝑓𝑒𝑋𝑓𝑗𝑕ℎ𝑢 Here, weight or 𝑥 is the weight function. We could simply use the Gaussian kernel: 2 − 𝑞 𝑗 −𝑞 𝑘 𝑥 𝑗, 𝑘 = exp , where 𝑞 𝑗 and 𝑞 𝑘 are screen space positions and 𝜏 𝑒 is 2 2𝜏 𝑒 the spatial standard deviation of the Gaussian kernel.

Advanced Graphics – Filtering 31 Ingredients Filter kernels A Gaussian filter (as well as other low-pass filters) blurs out high frequency details. We can improve on this using a non-linear bilateral filter*. 2 2 − 𝑞 𝑗 − 𝑞 𝑘 × exp − 𝑑 𝑗 − 𝑑 𝑘 𝑥 𝑗, 𝑘 = exp 2 2 2𝜏 𝑒 2𝜏 𝑠 *: Tomasi & Manduchi , Bilateral filtering for gray and color images. ICCV ’98.

Advanced Graphics – Filtering 32 Ingredients Filter kernels The bilateral filter takes the color of nearby pixels into account. We can take this further, by taking an arbitrary set of features into account. The cross bilateral filter*: 2 𝐿 2 − 𝑞 𝑗 − 𝑞 𝑘 − 𝑔 𝑙,𝑗 − 𝑔 𝑙,𝑘 𝑥 𝑗, 𝑘 = exp × ෑ exp 2 2 2𝜏 𝑒 2𝜏 𝑙 𝑙=1 Here, 𝑔 𝑙,𝑗 is the 𝑙’ th feature vector at pixel 𝑗 and 𝜏 𝑙 is the bandwidth parameter for feature 𝑙 . Note that we can use noise-free features to smooth noisy features. Example of a low-noise feature: normals at the primary intersection point. Example of a noisy feature: indirect illumination at the primary intersection point. *: Eisemann & Durand. Flash photography enhancement via intrinsic relighting. ACM Trans. Graph. 23, 3 (Aug. 2004).

Advanced Graphics – Filtering 33 Ingredients Indirect illumination as a feature: A path tracer allows us to conveniently split direct from indirect, and bounce 1 from bounce 2. Separating illumination into layers allows us to filter each layer separately. This prevents bleeding, and allows for layer-specific kernel sizes. We can also separate albedo from illumination.

Advanced Graphics – Filtering 34 Ingredients X = Adding this separation to an existing renderer: ▪ store albedo at the primary intersection (simple material property); ▪ at the end of the pipeline: illumination = sample / max( epsilon, albedo ).

Advanced Graphics – Filtering 35 Ingredients Example of denoising using a cross-bilateral filter in Blender. sinmantyx.wordpress.com/2015/12/02/denoising-cycles-renders-in-blender-theory