Smart camera design for realtime High Dynamic Range High Dynamic - PowerPoint PPT Presentation

Smart camera design for realtime Smart camera design for realtime High Dynamic Range High Dynamic Range imaging imaging P.J. Lapray, B. Heyrman, M. Rossé & D. Ginhac P.J. Lapray, B. Heyrman, M. Rossé & D. Ginhac Introduction What is HDR imaging ? LE2I UMR 5158, Univ Burgundy, Dijon, France What’s our goals ? Email: Pierre-Jean.Lapray@u-bourgogne.fr Our hardware platform Real-time Thursday, April 5th 2012 HDR Solution HDR capture WASC’12 Memory Management HDR Blending Tone mapping Demo Future

Smart camera design for Introduction 1 realtime High Dynamic What is HDR imaging ? Range imaging What’s our goals ? P.J. Lapray, Our hardware platform B. Heyrman, M. Rossé & D. Ginhac Real-time HDR Solution 2 Introduction HDR capture What is HDR imaging ? Memory Management What’s our goals ? Our hardware HDR Blending platform Real-time Tone mapping HDR Solution HDR capture Memory Demo 3 Management HDR Blending Tone mapping Future 4 Demo Future

Smart camera design for Summary realtime High Dynamic Range imaging Introduction 1 P.J. Lapray, What is HDR imaging ? B. Heyrman, M. Rossé & What’s our goals ? D. Ginhac Our hardware platform Introduction What is HDR Real-time HDR Solution imaging ? 2 What’s our goals ? HDR capture Our hardware platform Memory Management Real-time HDR Solution HDR Blending HDR capture Tone mapping Memory Management HDR Blending Demo Tone mapping 3 Demo Future 4 Future 3

What is HDR imaging ? Smart camera High Dynamic Range design for realtime Dynamic Range is measured in High Dynamic Range Exposure Value (EV) differences or imaging stops between the brightest and the P.J. Lapray, B. Heyrman, darkest parts of the image. An increase M. Rossé & D. Ginhac of one stop is doubling the amount of Introduction light of the image What is HDR imaging ? Capture limitation What’s our goals ? Our hardware platform A standard camera is able to capture only a fraction of the Real-time visual information. HDR Solution HDR capture Memory Management HDR Blending Tone mapping Demo Future 4

What is HDR imaging ? Smart camera design for realtime High Dynamic Range For a digital camera, number of stops = bit precision of imaging the ADC (ex : 10 stops for a 10-bit camera) camera P.J. Lapray, B. Heyrman, Real scenes includes sunlit and shaded areas. When M. Rossé & D. Ginhac capturing such a scene, we can perceive pixels that are Introduction saturated in the image. What is HDR imaging ? What’s our goals ? Our hardware platform Real-time VS HDR Solution HDR capture Memory Management HDR Blending Tone mapping Demo Future 5

What is HDR imaging ? Smart camera Limitation on display design for realtime Human eyes perceives a greater Dynamic Range than a High Dynamic Range digital camera (12 orders of magnitude) imaging P.J. Lapray, The standard screens can not transmit to the human eye B. Heyrman, this dynamic range. M. Rossé & D. Ginhac Introduction What is HDR imaging ? What’s our goals ? Our hardware platform Real-time HDR Solution HDR capture Memory Management HDR Blending Tone mapping Demo Future 6

What is HDR imaging ? Smart camera design for realtime At left, an HDR image consisting High Dynamic Range of details in dark and illuminated imaging areas P.J. Lapray, B. Heyrman, Below, the acquisitions made by M. Rossé & D. Ginhac a camera. Introduction What is HDR imaging ? What’s our goals ? Our hardware platform Real-time HDR Solution HDR capture Memory Management HDR Blending Tone mapping Demo Future 7

What is HDR imaging ? Smart camera design for realtime High Dynamic Another example of B&W HDR Range imaging image P.J. Lapray, (Images acquired by Thales B. Heyrman, M. Rossé & Angenieux) D. Ginhac Introduction What is HDR imaging ? What’s our goals ? Our hardware platform Real-time HDR Solution HDR capture Memory Management HDR Blending Tone mapping Demo Future 8

What is HDR imaging ? Smart camera Final goal design for realtime High Dynamic By limiting the exposure time, the resulting image contains the Range imaging details in high illumination areas. By increasing the exposure P.J. Lapray, time, the resulting image contains the details in the dark areas. B. Heyrman, M. Rossé & D. Ginhac Introduction What is HDR imaging ? What’s our goals ? Our hardware platform Real-time HDR Solution HDR capture Memory Management HDR Blending Tone mapping Demo Future 9

What’s our goal ? Smart camera design for realtime High Dynamic Range imaging P.J. Lapray, B. Heyrman, Build a dedicated hardware camera on FPGA M. Rossé & D. Ginhac Perform multiple captures, HDR blending, tone mapping Introduction and displaying HDR contents What is HDR imaging ? 60 images/s image processing in real-time What’s our goals ? Our hardware 1.3 Megapixels platform Real-time HDR Solution HDR capture Memory Management HDR Blending Tone mapping Demo Future 10

Our hardware platform Smart camera design for realtime High Dynamic A Virtex 5 FPGA development Range imaging board P.J. Lapray, B. Heyrman, e2V sensor : 1.3 Megapixel, 60 M. Rossé & D. Ginhac images/s, high sensitivity, low power, global shutter mode Introduction What is HDR Several communication interfaces : imaging ? What’s our goals ? Ethernet, SDRAM (256MB), serial Our hardware platform interface, DVI... Real-time HDR Solution HDR capture Memory Management HDR Blending Tone mapping Demo Future 11

Smart camera design for Summary realtime High Dynamic Range imaging Introduction 1 P.J. Lapray, What is HDR imaging ? B. Heyrman, M. Rossé & What’s our goals ? D. Ginhac Our hardware platform Introduction What is HDR Real-time HDR Solution imaging ? 2 What’s our goals ? HDR capture Our hardware platform Memory Management Real-time HDR Solution HDR Blending HDR capture Tone mapping Memory Management HDR Blending Demo Tone mapping 3 Demo Future 4 Future 12

HDR capture Smart camera The sensor is able to send successively 2 images with 2 design for realtime different integration times at 60 frames/s High Dynamic Range The integration time varies rapidly during the capture imaging P.J. Lapray, B. Heyrman, M. Rossé & D. Ginhac Introduction What is HDR imaging ? What’s our goals ? Our hardware platform Real-time HDR Solution HDR capture Memory Management HDR Blending Tone mapping Demo Future 13

Memory Management Smart camera Frame buffering for HDR creating design for realtime While we receive one frame from the sensor, we read the High Dynamic Range last frame from the SDRAM memory and we write the imaging current frame into memory P.J. Lapray, B. Heyrman, M. Rossé & Finally, we have a 2 streams of Low Dynamic Range D. Ginhac images in parallel Introduction What is HDR imaging ? What’s our goals ? Our hardware platform Real-time HDR Solution HDR capture Memory Management HDR Blending Tone mapping Demo Future 14

HDR Blending Smart camera Steps of Debevec et al. algorithm design for realtime Having two images : one underexposed and one High Dynamic Range overexposed imaging P.J. Lapray, Knowing the two exposure times B. Heyrman, M. Rossé & Knowing the response curve of the sensor D. Ginhac Applyiing Debevec algorithm for each pixel Introduction What is HDR We obtain an HDR image encoded with IEEE754 floating imaging ? What’s our goals ? point standard Our hardware platform Real-time HDR Solution HDR capture Memory Management HDR Blending Tone mapping Demo Future 15

HDR Blending Smart camera design for realtime High Dynamic Range imaging Debevec et al. algorithm P.J. Lapray, � P B. Heyrman, j = 1 ω ( Z ij )( g ( Z ij ) − ln ∆ t ij ) M. Rossé & ln E i = (1) D. Ginhac � P j = 1 ω ( Z ij ) Introduction Where ω ( z ) is the weighting function. It is a simple hat What is HDR imaging ? equation. E i is the irradiance, Z ij is the pixel value of pixel What’s our goals ? Our hardware platform location number i in image j and ∆ t ij is the exposure duration. Real-time The response curve g is determined by resolving a complex HDR Solution HDR capture quadratic function in C++. Memory Management HDR Blending Tone mapping Demo Future 16

HDR Blending Smart camera design for realtime High Dynamic Range imaging P.J. Lapray, B. Heyrman, M. Rossé & D. Ginhac Introduction What is HDR imaging ? What’s our goals ? Our hardware platform Real-time HDR Solution HDR capture Memory Management HDR Blending Tone mapping Demo Future 17

HDR Blending Smart camera design for realtime Debevec et al. algorithm High Dynamic Range imaging Device : xc5vfx70t-1ff1136 P.J. Lapray, B. Heyrman, Number of Slice LUTs : 5647/44800 (12%) M. Rossé & D. Ginhac Number of Slice Registers : 5975/44800 (13%) Introduction What is HDR Number of Block RAM/FIFO : 6/148 (4%) imaging ? What’s our goals ? Our hardware Number of DSP48Es : 4/128 (3%) platform Real-time HDR Solution Maximum frequency : 184.536 MHz HDR capture Memory Management Table : Summary of hardware synthesis report HDR Blending Tone mapping Demo Future 18