Depth and Surface Normal Estimation from a Single Image Mian Wei - PowerPoint PPT Presentation

1 Depth and Surface Normal Estimation from a Single Image Mian Wei University of Toronto

2 Indirect-Invariant What is the problem?

3 Indirect-Invariant Given one image

4 N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from RGBD images,” in Proc. Eur. Conf. Comput. Vision , 2012, pp. 746–760.

5 Indirect-Invariant Estimate the following:

6 Eigen, D. and Fergus, R. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. ICCV 2015

7 Eigen, D. and Fergus, R. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. ICCV 2015

8 Indirect-Invariant Why is this hard?

9 Indirect-Invariant Multiple ambiguities

10 Indirect-Invariant Scale ambiguity

11

12

13 Indirect-Invariant Bas-relief ambiguity P. Belhumeur, D. Kriegman, and A. Yuille, “The Bas-Relief Ambiguity,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1040-1046, 1997.

14 Indirect-Invariant Let’s play a game

15 Indirect-Invariant Spot the Difference

16 P. Belhumeur, D. Kriegman, and A. Yuille, “The Bas-Relief Ambiguity,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1040-1046, 1997.

17 P. Belhumeur, D. Kriegman, and A. Yuille, “The Bas-Relief Ambiguity,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1040-1046, 1997.

18 Indirect-Invariant All the same

19 P. Belhumeur, D. Kriegman, and A. Yuille, “The Bas-Relief Ambiguity,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1040-1046, 1997.

20 P. Belhumeur, D. Kriegman, and A. Yuille, “The Bas-Relief Ambiguity,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1040-1046, 1997.

21 Indirect-Invariant Family of transformation

22 Indirect-Invariant Generalized Bas-Relief

23 Indirect-Invariant Change shape and illumination

24 Indirect-Invariant Yield same image

25 Indirect-Invariant Existing works

26 Indirect-Invariant Multi-view Stereo Hartley,R. and Zisserman, A. 2000. Multiple view geometry in computer vision , Cambridge University Press: Cambridge, UK.

27

28

29 Indirect-Invariant Photometric Stereo Woodham, R.J. (1980), Photometric method for determining surface orientation from multiple images, Optical Engineering 19 (1) 139-144.

30 Indirect-Invariant Collimated Light Sources

31 Indirect-Invariant Light rays parallel

32

33

34

35 Indirect-Invariant Shape from Focus S. Nayar and N. Yasuo, “Shape From Focus,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 16, no. 8, pp. 824-831, 1994.

36

37

38

39

40

41

42

43

44

45

46

47

48 Indirect-Invariant Light Fall-off Stereo M. Liao, L. Wang, R. Yang, and M. Gong. Light fall-off stereo. In Proceedings of CVPR, pages 1–8, 2007.

49

50

51 Indirect-Invariant Specialized Hardware

52 Indirect-Invariant Laser Scanner

53 Indirect-Invariant Active Illumination

54 Indirect-Invariant Time of Flight

55 Indirect-Invariant Estimating Depth D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from a single image using a multi-scale deep network. NIPS 2014

56

57 Indirect-Invariant Train 2 networks

58 Indirect-Invariant Global coarse-scale network

59 Indirect-Invariant Local fine-scale network

60 Indirect-Invariant Global coarse-scale network

61 Indirect-Invariant Learns a coarse depth map

62

63

64

65 Indirect-Invariant Used as input to local network

66 Indirect-Invariant Intuition:

67 Indirect-Invariant Coarse info learnt already

68 Indirect-Invariant Focus on learning finer info

69

70

71 Indirect-Invariant Scale ambiguity

72 Indirect-Invariant Scale invariant error function

73 n D ( y , y * ) = 1 ∑ (log y i − log y * + α ( y i , y * i )) 2 i 2 n i = 1 n i ) = 1 α ( y i , y * ∑ (log y * i − log y i ) n i = 1

74 n D ( ay , ay * ) = 1 ∑ (log ay i − log ay * + α ( ay i , ay * i )) 2 i 2 n i = 1 n D ( ay , ay * ) = 1 ∑ (log a − log a + log y i − log y * + α ( ay i , ay * i )) 2 i 2 n i = 1 n D ( ay , ay * ) = 1 ∑ (log y i − log y * + log a − log a + α ( y i , y * i )) 2 i 2 n i = 1 D ( ay , ay * ) = D ( y , y * )

75 Indirect-Invariant Loss Function

76 Indirect-Invariant Scale invariant

77 n n L ( y , y * ) = 1 − λ ∑ d 2 ∑ ) 2 n 2 ( d i i n i = 1 i = 1 d i = log y i − log y * i

78 Indirect-Invariant 2 Datasets

79 Indirect-Invariant NYUDepthV2 N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from RGBD images,” in Proc. Eur. Conf. Comput. Vision , 2012, pp. 746–760.

80 Indirect-Invariant Indoor Rooms

81

82 Indirect-Invariant KITTI A. Greiger, P. Lenz, C. Stiller, and R. Urtasun. Vision meets robotics: The kitti dataset. International Journal of Robotics Research (IJRR). 2013.

83 Indirect-Invariant Outdoor images taken on a car

84

85 Indirect-Invariant How do you get ground truth?

86 Indirect-Invariant NYUDepthV2

87 Indirect-Invariant Kinect

88

89 Indirect-Invariant KITTI

90

91 Indirect-Invariant Time of Flight

92 Indirect-Invariant Times how long light travels

93 Indirect-Invariant From light source to camera

94 Indirect-Invariant Results

95

96

97

98 Indirect-Invariant Estimating Surface Normals X. Wang, D. F. Fouhey, and A. Gupta. Designing deep networks for surface normal estimation. CVPR 2015

99 Indirect-Invariant Similar to Eigen

100