Adaptive Incident Radiance Field Sampling and Reconstruction Using - PowerPoint PPT Presentation

Adaptive Incident Radiance Field Sampling and Reconstruction Using Deep Reinforcement Learning Yuchi Huo, Rui Wnag, Ruzhang Zheng, Hualin Xu, Hujun Bao, Sung-eui Yoon KAIST and CAD&CG

Global Illumination Path Guiding Filtering • Animation, preview … • Design, Physical Simulation, Data Generation … 2

Global Illumination Path Guiding Filtering • Animation, preview … • Design, Physical Simulation, Data Generation … 3

Filtering V.S. Path Guiding 1 30 2 60 Reference 8 120 min KPCN Ours KPCN Ours min 4

Path Guiding - A Sampling Problem 5

Path Guiding - A Sampling Problem Evaluate Sample QUALITY NETWORK Reward high reward if Reconstruction à GT RECONSTRUCTION NETWORK 6

Training R- and Q-networks Dataset Dataset Ground Truth Feature … Input Input Target Q-network R-network Output Target Reconstructio n Reward Output Reward Generation 7

Radiance Field Reconstruction using Deep Leaning θ Reconstruction: ü To guide path tracing φ ü To generate preview ü Others v Extremely sparse samples v Memory/computation overhead v Hard to find ground-truth 8

Radiance Field Reconstruction using Deep Reinforcement Leaning θ R-network : Q-network: φ Adaptive samplin Reconstruct 4D r adiance field in b g and refining the radiance field, trai oth image and di ned by DRL rection spaces 9

Adaptive Sampling and Refining Q-network: R-network 10

Adaptive Sampling and Refining (a) Sampling in Direction Space θ =0 cos %& 0.5 0.5π φ=0 2π R-network 11

Adaptive Sampling and Refining (b) Quality values of adap (a) Sampling in Direction Space tive actions θ =0 Refine Q-network cos %& 0.5 or Prediction Resample 0.5π φ=0 2π R-network 12

Adaptive Sampling and Refining (c) Adaptive sampling and (b) Quality values of adap (a) Sampling in Direction refining in Direction Space Space tive actions θ =0 Refine Q-network cos %& 0.5 or Prediction Resample 0.5π φ=0 2π R-network 13

R-network • Explore 4D radiance field in: • Image-direction • Direction-image • Direction • Image 14

R-network • Image-direction network: Direction Space Feature Maps Image Space Features: & &- & &- 𝑺 + 𝑺 + 𝑯 + 𝑮𝒆 + 𝑮𝒆 + 6 (𝒚 2 ) 3 6 Normal, Radiance … 𝑮𝒆 0 𝑀 25 w φ CONVs (3 layers) θ … Rearrange … … CONVs … (6 layers) & h 𝑮𝒆 0 𝒚 2 &- ∀𝑗 ∈ Γ 𝑮𝒆 0 & 𝑺 + 𝑯 + Image part Direction part 15

Q-network φ φ • Actions: θ • Refinement θ Refinement • Resampling • Q-value(reward): • Decline of the φ φ Difference between GT and R-network output θ Resampling θ 16

Q-network Direction Space = 𝑮𝒆 + Feature Maps w = 𝑮𝒆 2 Q-value of CONVs Refining FCs Flatten (3 layers) h (4 layers) Q-value of Resampling ∀𝑗 ∈ Γ = = @(=) 𝑯 + 𝑺 + 𝑰 + 𝑺 + ∀𝑘 ∈ Ψ Image Space Features: Normal, Radiance, Hierarchies … 17

Results (path guiding) 18

Results (path guiding) 19

Results (path guiding) 20

Results (direct filtering) 21

Results (Filtering v.s. Path Guiding) 1 30 2 60 Reference 8 120 min KPCN Ours KPCN Ours min 22

THANK YOU