Photometric stereo for the measurement of surface texture and - PowerPoint PPT Presentation

Photometric stereo for the measurement of surface texture and shape YAN YAN yany14@mails.tsinghua.edu.cn

Main point • Introduction of Background • Building the sensor • optical system • Photometric stereo algorithm • Results • Video of MIT Gelsight

Introduction of Background • Tactile information : 1. one of the most important medium to access outside information 2. premise for robot arm to perform accurate operation • Measurement of surface texture and shape: 1. build 3-D figure from a few pictures acquired by sensor 2. get depth information 3. robot arm decide the force to apply according to depth information

Building the sensor • A sensor that converts information about surface shape and pressure into image • It consists: a slab of clear elastomer, the reflective skin(metal skin , which is sputtered by Semiconductor technology) • When pressed ,the skin distorts to take on the shape of the objects’ surface, when viewed from behind , the skin appears as a replica of the surface

Optical system • Lights: three LEDs evenly spaced on the plate which is 25cm from the center of the sensor at an elevation angle of α degrees • Camera: micro lens camera which is 40 cm from the center of the sensor

Photometric stereo algorithm • Surface height function: 𝑎 = 𝑔(𝑦,𝑧) then the gradient (𝑞, 𝑟) at position 𝑦,𝑧 is ,- ,- 𝑞 = 𝑟 = ,. ,/ • We assume that the intensity on the surface depends only on its surface normal , then 𝑆 is reflectance function 𝐽 𝑦,𝑧 = 𝑆(𝑞,𝑟) • To reduce ambiguities ,we use three images from different illumination conditions 𝐽 𝑦,𝑧 = 𝑆(𝑞 𝑦,𝑧 ,𝑟(𝑦, 𝑧)) Where 𝐽 𝑦,𝑧 = (𝐽 2 𝑦,𝑧 ,𝐽 3 𝑦,𝑧 , 𝐽 4 𝑦,𝑧 ) 𝑆 𝑞, 𝑟 = (𝑆 2 𝑞,𝑟 ,𝑆 3 𝑞,𝑟 ,𝑆 4 𝑞,𝑟 ) • So we can build a lookup table ,when given the intensity 𝐽 , we get the gradient according to the lookup table. The lookup table is populated using a calibration target with known geometry , I use a sphere. Make sure the light environment is the same .

Photometric stereo algorithm • When given an intensity , find the reference 𝐽 5 that is closest to the target and also get the course gradient 𝑞 5 , 𝑟 5 • A nearest-neighbor search problem . First ,it is accelerated with data structure k-d tree, then KNN algorithm is applied to find the closest . • To fill in the missing data , we approximate the reflectance function with a low- order spherical harmonic model , where k=2 . :>@ ;>: R N = ∑ ∑ 𝑏 :,; 𝑍 :,; (𝑂) :>5 ;>?: • Estimating the nine coefficients 𝑏 :,; with least square method . • As the gradient 𝑞 5 ,𝑟 5 from lookup table returns is course , to refine the estimate, we approximate the reflectance function in the neighborhood of 𝑞 5 , 𝑟 5 using a first-order Taylor series expansion : ,G2 ,G2 ,H ,I 𝑞 − 𝑞 5 ,G3 ,G3 ,I R p,q = R 𝑞 5 ,𝑟 5 + J 𝑞 5 ,𝑟 5 𝑟 − 𝑟 5 𝐾 = ( ) (1) ,H ,G4 ,G4 ,H ,I

Photometric stereo algorithm • R p, q i s the target intensity , R 𝑞 5 ,𝑟 5 is the reference intensity from the lookup table , 𝑞 5 ,𝑟 5 is the course gradient , p,q is the refined gradient 𝑞 + 𝑞 5 = 𝐾 K R p,q − R 𝑞 5 ,𝑟 5 𝐾 K = (𝐾 L 𝐾 + 𝛿𝐽) ?2 𝐾 L (2) 𝑟 𝑟 5 • In the regions of large curvature , the first-order Taylor series approximation may not be accurate , so we use the 𝑞 5 ,𝑟 5 instead. So the (2) can be improved as 𝑞 + 𝑞 5 = 𝑥𝐾 K R p,q − R 𝑞 5 ,𝑟 5 𝑟 𝑟 5 • After we get the accurate gradient , then we can compute the depth . First we define an error function 𝐹 𝑎, 𝑞, 𝑟 = (𝑎 . − 𝑞) 3 +(𝑎 / − 𝑟) 3 the error function E is also a map defined on every point (x , y), then the problem of finding optimal 𝑎 can be formulated as a minimization of cost function 𝑑𝑝𝑡𝑢 𝑎 = ∬𝐹 𝑎,𝑞,𝑟 𝑒𝑦𝑒𝑧 • Use Frankot Chellappa Algorithm ,we finally get the depth information.

Results • Results of the algorithm • Show of the robot hand

Thank you !