stereo vision approaches for human to robot handover
play

Stereo Vision Approaches for Human to Robot Handover Aleksej - PowerPoint PPT Presentation

Apr 01, 2023 •350 likes •1.48k views

MIN Faculty Department of Informatics Stereo Vision Approaches for Human to Robot Handover Aleksej Logacjov University of Hamburg Faculty of Mathematics, Informatics and Natural Sciences Department of Informatics Technical Aspects of

  1. Basics: Image Rectification Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Olofsson, A. 2010] [Olofsson, A. 2010] ◮ Transform 2D search in 1D ◮ Linear time complexity [Kuhl, A., 2005] ◮ Has to be done for each image pair A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 11 / 44

  2. Basics: Image Rectification Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Olofsson, A. 2010] [Olofsson, A. 2010] ◮ Transform 2D search in 1D ◮ Linear time complexity [Kuhl, A., 2005] ◮ Has to be done for each image pair A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 11 / 44

  3. Basics: Image Rectification Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Olofsson, A. 2010] [Olofsson, A. 2010] ◮ Transform 2D search in 1D ◮ Linear time complexity [Kuhl, A., 2005] ◮ Has to be done for each image pair A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 11 / 44

  4. Basics: Image Rectification Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 12 / 44

  5. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Camera Cal- ibration ◮ Find matching pixels in both images Image Rec- ◮ Calc. disparity d = X L − X R f.e. tification pixel ◮ Problems: Occlusion, sensor Stereo Cor- noise ... [Olofsson, A. 2010] respondence ◮ Still open research [Luo, W. et al., 2016] Depth Calculation A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 13 / 44

  6. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Camera Cal- ibration ◮ Find matching pixels in both images Image Rec- ◮ Calc. disparity d = X L − X R f.e. tification pixel ◮ Problems: Occlusion, sensor Stereo Cor- noise ... [Olofsson, A. 2010] respondence ◮ Still open research [Luo, W. et al., 2016] Depth Calculation A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 13 / 44

  7. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Camera Cal- ibration ◮ Find matching pixels in both images Image Rec- ◮ Calc. disparity d = X L − X R f.e. tification pixel ◮ Problems: Occlusion, sensor Stereo Cor- noise ... [Olofsson, A. 2010] respondence ◮ Still open research [Luo, W. et al., 2016] Depth Calculation A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 13 / 44

  8. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Camera Cal- ibration ◮ Find matching pixels in both images Image Rec- ◮ Calc. disparity d = X L − X R f.e. tification pixel ◮ Problems: Occlusion, sensor Stereo Cor- noise ... [Olofsson, A. 2010] respondence ◮ Still open research [Luo, W. et al., 2016] Depth Calculation A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 13 / 44

  9. Basics: Depth Calculation Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Camera Cal- ibration Image Rec- ◮ F.e. pixel in disparity map: calc. tification distance z = T · f d ◮ Straight forward approach with Stereo Cor- linear complexity respondence Depth Calculation A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 14 / 44

  10. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Middleburry Dataset] ◮ Find corresponding pixels A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 15 / 44

  11. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Middleburry Dataset] ◮ Find corresponding pixels A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 15 / 44

  12. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Middleburry Dataset] ◮ Find corresponding pixels A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 15 / 44

  13. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 2 basic approaches: ◮ local ◮ global A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 16 / 44

  14. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 2 basic approaches: ◮ local ◮ global A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 16 / 44

  15. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 2 basic approaches: ◮ local ◮ Window based ◮ Fast, simple, sensitive to noise and occlusion [Olofsson, A. 2010] ◮ global A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 16 / 44

  16. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 2 basic approaches: ◮ local ◮ Window based ◮ Fast, simple, sensitive to noise and occlusion [Olofsson, A. 2010] ◮ global ◮ For whole image at once ◮ Better in noise/occlusion handling [Olofsson, A. 2010] A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 16 / 44

  17. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 2 basic approaches: ◮ local ◮ Window based ◮ Fast, simple, sensitive to noise and occlusion [Olofsson, A. 2010] ◮ global ◮ For whole image at once ◮ Better in noise/occlusion handling [Olofsson, A. 2010] Focus on local A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 16 / 44

  18. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Cost is needed ◮ F.e. pixel, compute cost accord. to each possible disparity C ( x 1 , y 1 , 0 ) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 17 / 44

  19. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Cost is needed ◮ F.e. pixel, compute cost accord. to each possible disparity C ( x 1 , y 1 , 0 ) , C ( x 1 , y 1 , 1 ) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 17 / 44

  20. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Cost is needed ◮ F.e. pixel, compute cost accord. to each possible disparity C ( x 1 , y 1 , 0 ) , C ( x 1 , y 1 , 1 ) , C ( x 1 , y 1 , 2 ) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 17 / 44

  21. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Cost is needed ◮ F.e. pixel, compute cost accord. to each possible disparity C ( x 1 , y 1 , 0 ) , C ( x 1 , y 1 , 1 ) , C ( x 1 , y 1 , 2 ) , C ( x 1 , y 1 , 3 ) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 17 / 44

  22. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Cost is needed ◮ F.e. pixel, compute cost accord. to each possible disparity C ( x 1 , y 1 , 0 ) , C ( x 1 , y 1 , 1 ) , C ( x 1 , y 1 , 2 ) , C ( x 1 , y 1 , 3 ) , C ( x 1 , y 1 , 4 ) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 17 / 44

  23. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Cost is needed ◮ F.e. pixel, compute cost accord. to each possible disparity C ( x 1 , y 1 , 0 ) , C ( x 1 , y 1 , 1 ) , C ( x 1 , y 1 , 2 ) , C ( x 1 , y 1 , 3 ) , C ( x 1 , y 1 , 4 ) C ( x 1 , y 2 , − 1 ) , C ( x 1 , y 2 , 0 ) , C ( x 1 , y 2 , 1 ) , C ( x 1 , y 2 , 2 ) , C ( x 1 , y 2 , 3 ) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 17 / 44

  24. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Cost is needed ◮ F.e. pixel, compute cost accord. to each possible disparity C ( x 1 , y 1 , 0 ) , C ( x 1 , y 1 , 1 ) , C ( x 1 , y 1 , 2 ) , C ( x 1 , y 1 , 3 ) , C ( x 1 , y 1 , 4 ) C ( x 1 , y 2 , − 1 ) , C ( x 1 , y 2 , 0 ) , C ( x 1 , y 2 , 1 ) , C ( x 1 , y 2 , 2 ) , C ( x 1 , y 2 , 3 ) ◮ Cost Computation A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 17 / 44

  25. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Window centered at pixel ◮ Taking neighbors into account A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 18 / 44

  26. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work d ◮ Disparity Space Image (DSI) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 19 / 44

  27. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work C d d Min ◮ Disparity Space ◮ For each pixel Image (DSI) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 19 / 44

  28. Basics: Stereo Correspondence Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Example window-based cost computation [Olofsson, A. 2010] A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 20 / 44

  29. Stereo Correspondence Algorithms Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 3 different approaches to compute the matching cost : ◮ (Sum of) Absolute Intensity Difference C SAD ( x , y , d ) ◮ Deep Learning Approach [Luo, W. et al., 2016] ◮ Efficient Large-Scale-Stereo-Matching (ELAS) [Geiger, A. 2012] A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 21 / 44

  30. (Sum of) Absolute Intensity Difference Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Absolute Intensity Difference (AD): C AD ( x , y , d ) = | I L ( x , y ) − I R ( x − d , y ) | ◮ (Sum of) = Window-based ◮ Sum of AD: C SAD ( x , y , d ) = Σ ( u , v ) ∈ N ( x , y ) | I L ( u , v ) − I R ( u − d , v ) | ◮ With Neighborhood N(x,y) of (x,y) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 22 / 44

  31. (Sum of) Absolute Intensity Difference Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Absolute Intensity Difference (AD): C AD ( x , y , d ) = | I L ( x , y ) − I R ( x − d , y ) | ◮ (Sum of) = Window-based ◮ Sum of AD: C SAD ( x , y , d ) = Σ ( u , v ) ∈ N ( x , y ) | I L ( u , v ) − I R ( u − d , v ) | ◮ With Neighborhood N(x,y) of (x,y) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 22 / 44

  32. (Sum of) Absolute Intensity Difference Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Absolute Intensity Difference (AD): C AD ( x , y , d ) = | I L ( x , y ) − I R ( x − d , y ) | ◮ (Sum of) = Window-based ◮ Sum of AD: C SAD ( x , y , d ) = Σ ( u , v ) ∈ N ( x , y ) | I L ( u , v ) − I R ( u − d , v ) | ◮ With Neighborhood N(x,y) of (x,y) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 22 / 44

  33. (Sum of) Absolute Intensity Difference Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Pros: Fast, simple. According to [Scharstein, D. et al., 2002] one of the fastest classical approach. ◮ Cons: Bad accuracy (place 8 of 20 according to [Scharstein, D. et al., 2002]) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 23 / 44

  34. (Sum of) Absolute Intensity Difference Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Pros: Fast, simple. According to [Scharstein, D. et al., 2002] one of the fastest classical approach. ◮ Cons: Bad accuracy (place 8 of 20 according to [Scharstein, D. et al., 2002]) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 23 / 44

  35. Deep Learning Approach [Luo, W. et al., 2016] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Luo, W. et al., 2016] A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 24 / 44

  36. Deep Learning Approach [Luo, W. et al., 2016] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Luo, W. et al., 2016] ◮ Put one image patch, centered at pixel (x,y) as input (9x9) ◮ Put an image patch, of size (max_disparity,9) as sec. input ◮ Network computes in one iteration , the cost for all given disparities A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 25 / 44

  37. Deep Learning Approach [Luo, W. et al., 2016] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work [Luo, W. et al., 2016] ◮ Put one image patch, centered at pixel (x,y) as input (9x9) ◮ Put an image patch, of size (max_disparity,9) as sec. input ◮ Network computes in one iteration , the cost for all given disparities A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 25 / 44

  38. Deep Learning Approach [Luo, W. et al., 2016] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Training: ◮ Random image patches from the Kitti dataset ◮ Cross entropy loss for multi class classification (disparities) ◮ 6.5 hours training ◮ Testing/Benchmarking: ◮ On Kitti and Middleburry dataset A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 26 / 44

  39. Deep Learning Approach [Luo, W. et al., 2016] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Training: ◮ Random image patches from the Kitti dataset ◮ Cross entropy loss for multi class classification (disparities) ◮ 6.5 hours training ◮ Testing/Benchmarking: ◮ On Kitti and Middleburry dataset A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 26 / 44

  40. Deep Learning Approach [Luo, W. et al., 2016] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Pros: ◮ Very fast, compared to other learning approaches (1sec on NVIDIA Titan-X) ◮ As accurate as other learning approaches ◮ Cons: ◮ No comparison to non learning state-of-the art approaches ◮ After calculation, cost aggregation, smoothing done (time consuming) ◮ No CPU runtime A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 27 / 44

  41. Deep Learning Approach [Luo, W. et al., 2016] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Pros: ◮ Very fast, compared to other learning approaches (1sec on NVIDIA Titan-X) ◮ As accurate as other learning approaches ◮ Cons: ◮ No comparison to non learning state-of-the art approaches ◮ After calculation, cost aggregation, smoothing done (time consuming) ◮ No CPU runtime A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 27 / 44

  42. Efficient Large-Scale-Stereo-Matching (ELAS) [Geiger, A. 2012] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Not trying to find matching f.e. pixel in first run ◮ They can be ambiguous ◮ Find robust matchings with matching support points algorithm ◮ These pixels: support points ◮ Find support points by calc. L1-distance between feature vectors ◮ Using this support points to calc. the remaining matchings A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 28 / 44

  43. Efficient Large-Scale-Stereo-Matching (ELAS) [Geiger, A. 2012] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Not trying to find matching f.e. pixel in first run ◮ They can be ambiguous ◮ Find robust matchings with matching support points algorithm ◮ These pixels: support points ◮ Find support points by calc. L1-distance between feature vectors ◮ Using this support points to calc. the remaining matchings A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 28 / 44

  44. Efficient Large-Scale-Stereo-Matching (ELAS) [Geiger, A. 2012] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Not trying to find matching f.e. pixel in first run ◮ They can be ambiguous ◮ Find robust matchings with matching support points algorithm ◮ These pixels: support points ◮ Find support points by calc. L1-distance between feature vectors ◮ Using this support points to calc. the remaining matchings A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 28 / 44

  45. Efficient Large-Scale-Stereo-Matching (ELAS) [Geiger, A. 2012] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Pros: ◮ Very fast, (0.7 sec on i7 CPU with 2.66 GHz ) ◮ Performs well on higher resolution images (900x750) ◮ Better accuracy than other state-of-the-art approaches ◮ Cons: ◮ Non trivial algorithm ◮ 0.7 sec maybe to slow for human-robot-handover A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 29 / 44

  46. Efficient Large-Scale-Stereo-Matching (ELAS) [Geiger, A. 2012] Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work iCub using ELAS [Nguyen, P. D., et al. 2018] A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 30 / 44

  47. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ All 3 approaches can be used in real-time applications ◮ With powerful hardware (except SAD) ◮ Maybe this is not given for some humanoid robots ◮ Focusing on creating a good disparity map for each pixel ◮ They are also focusing on handling noise (except SAD) ◮ But we don’t need both (at least not so much) ◮ We only need the pixels corresponding to object ◮ We don’t need to consider a lot of different disparities ◮ Only objects which are nearer than approx. 30cm A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 31 / 44

  48. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ All 3 approaches can be used in real-time applications ◮ With powerful hardware (except SAD) ◮ Maybe this is not given for some humanoid robots ◮ Focusing on creating a good disparity map for each pixel ◮ They are also focusing on handling noise (except SAD) ◮ But we don’t need both (at least not so much) ◮ We only need the pixels corresponding to object ◮ We don’t need to consider a lot of different disparities ◮ Only objects which are nearer than approx. 30cm A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 31 / 44

  49. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ All 3 approaches can be used in real-time applications ◮ With powerful hardware (except SAD) ◮ Maybe this is not given for some humanoid robots ◮ Focusing on creating a good disparity map for each pixel ◮ They are also focusing on handling noise (except SAD) ◮ But we don’t need both (at least not so much) ◮ We only need the pixels corresponding to object ◮ We don’t need to consider a lot of different disparities ◮ Only objects which are nearer than approx. 30cm A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 31 / 44

  50. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work We need 3 things: 1. Object detection and tracking (create bounding box ) 2. Cutting out the bounding box 3. Use z = T · f to calc. disparity boundaries d A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 32 / 44

  51. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work We need 3 things: 1. Object detection and tracking (create bounding box ) 2. Cutting out the bounding box 3. Use z = T · f to calc. disparity boundaries d A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 32 / 44

  52. Object Detection and Object Tracking Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Object Detection (YOLOv3)[Redmon, J., 2018] ◮ Has to be done once at the beginning ◮ After that, tracking ◮ Object Tracking (CSRT)[OpenCV] ◮ Fast and accurate tracking NICO example A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 33 / 44

  53. Object Detection and Object Tracking Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Object Detection (YOLOv3)[Redmon, J., 2018] ◮ Has to be done once at the beginning ◮ After that, tracking ◮ Object Tracking (CSRT)[OpenCV] ◮ Fast and accurate tracking NICO example A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 33 / 44

  54. Object Detection and Object Tracking Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Object Detection (YOLOv3)[Redmon, J., 2018] ◮ Has to be done once at the beginning ◮ After that, tracking ◮ Object Tracking (CSRT)[OpenCV] ◮ Fast and accurate tracking NICO example A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 33 / 44

  55. Object Detection and Object Tracking Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ Object Detection (YOLOv3)[Redmon, J., 2018] ◮ Has to be done once at the beginning ◮ After that, tracking ◮ Object Tracking (CSRT)[OpenCV] ◮ Fast and accurate tracking NICO example A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 33 / 44

  56. Cutting out the bounding box Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Full view A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 34 / 44

  57. Cutting out the bounding box Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Cut out view Full view A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 34 / 44

  58. Cutting out the bounding box Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Cut out view Full view ◮ Detection runtime: approx 2 sec ◮ x coordinates important for disparity A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 34 / 44

  59. Cutting out the BB (Disparity Comparison) Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work Original image left [Middlebury Dataset] Original image right [Middlebury Dataset] ◮ size: 640x438 A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 35 / 44

  60. Cutting out the BB (Disparity Comparison) Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ELAS [Geiger, A. 2012] SAD A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 36 / 44

  61. Cutting out the BB (Disparity Comparison) Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ELAS [Geiger, A. 2012] SAD ◮ size: 224x376 (approx 3 times smaller) A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 37 / 44

  62. Cutting out the BB (Disparity Comparison) Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ SAD: ◮ Big: 0.014 sec ◮ Small: 0.006 sec (2 times faster) ◮ ELAS: ◮ Big: 0.24 sec ◮ Small: 0.06 sec (4 times faster) ◮ Time to cut out: 0.0005 sec A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 38 / 44

  63. Calc. the disparity boundaries Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ z = T · f d ◮ d = T · f z ◮ Knowing T , f and max. reachable dist. z max : ◮ Calc. smallest disparity d min = T · f z max ◮ Similar to smallest distance z min ◮ Lead to a smaller DSI d d A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 39 / 44

  64. Calc. the disparity boundaries Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ z = T · f d ◮ d = T · f z ◮ Knowing T , f and max. reachable dist. z max : ◮ Calc. smallest disparity d min = T · f z max ◮ Similar to smallest distance z min ◮ Lead to a smaller DSI d d A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 39 / 44

  65. Calc. the disparity boundaries Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ z = T · f d ◮ d = T · f z ◮ Knowing T , f and max. reachable dist. z max : ◮ Calc. smallest disparity d min = T · f z max ◮ Similar to smallest distance z min ◮ Lead to a smaller DSI d d A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 39 / 44

  66. Calc. the disparity boundaries Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work ◮ z = T · f d ◮ d = T · f z ◮ Knowing T , f and max. reachable dist. z max : ◮ Calc. smallest disparity d min = T · f z max ◮ Similar to smallest distance z min ◮ Lead to a smaller DSI d d A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 39 / 44

  67. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 1. (*)Calibrate the cameras 2. Get both video streams from the cams 3. Rectify the frames 4. (*)At some frame, detect the object in both frames (YOLOv3 approx. 2 sec) 5. Track it (CSRT approx. 0.05 sec) 6. Cut out the BB (approx. 0.0005 sec) 7. Stereo Correspondence Algorithm (SAD: between 0.006 and 0.014 sec) 8. Calculate the depth of the nearest pixels (*) = Only 1x A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 40 / 44

  68. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 1. (*)Calibrate the cameras 2. Get both video streams from the cams 3. Rectify the frames 4. (*)At some frame, detect the object in both frames (YOLOv3 approx. 2 sec) 5. Track it (CSRT approx. 0.05 sec) 6. Cut out the BB (approx. 0.0005 sec) 7. Stereo Correspondence Algorithm (SAD: between 0.006 and 0.014 sec) 8. Calculate the depth of the nearest pixels (*) = Only 1x A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 40 / 44

  69. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 1. (*)Calibrate the cameras 2. Get both video streams from the cams 3. Rectify the frames 4. (*)At some frame, detect the object in both frames (YOLOv3 approx. 2 sec) 5. Track it (CSRT approx. 0.05 sec) 6. Cut out the BB (approx. 0.0005 sec) 7. Stereo Correspondence Algorithm (SAD: between 0.006 and 0.014 sec) 8. Calculate the depth of the nearest pixels (*) = Only 1x A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 40 / 44

  70. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 1. (*)Calibrate the cameras 2. Get both video streams from the cams 3. Rectify the frames 4. (*)At some frame, detect the object in both frames (YOLOv3 approx. 2 sec) 5. Track it (CSRT approx. 0.05 sec) 6. Cut out the BB (approx. 0.0005 sec) 7. Stereo Correspondence Algorithm (SAD: between 0.006 and 0.014 sec) 8. Calculate the depth of the nearest pixels (*) = Only 1x A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 40 / 44

  71. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 1. (*)Calibrate the cameras 2. Get both video streams from the cams 3. Rectify the frames 4. (*)At some frame, detect the object in both frames (YOLOv3 approx. 2 sec) 5. Track it (CSRT approx. 0.05 sec) 6. Cut out the BB (approx. 0.0005 sec) 7. Stereo Correspondence Algorithm (SAD: between 0.006 and 0.014 sec) 8. Calculate the depth of the nearest pixels (*) = Only 1x A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 40 / 44

  72. Improvements for Human-to-Robot Handover Motivation Basics Stereo Correspondence Algorithms Improvements for Human-to-Robot Handover Future Work 1. (*)Calibrate the cameras 2. Get both video streams from the cams 3. Rectify the frames 4. (*)At some frame, detect the object in both frames (YOLOv3 approx. 2 sec) 5. Track it (CSRT approx. 0.05 sec) 6. Cut out the BB (approx. 0.0005 sec) 7. Stereo Correspondence Algorithm (SAD: between 0.006 and 0.014 sec) 8. Calculate the depth of the nearest pixels (*) = Only 1x A. Logacjov – Stereo Vision Approaches for Human to Robot Handover 40 / 44

Recommend

3D Vision: Stereo Marc Pollefeys, Torsten Sattler Spring 2016

3D Vision: Stereo Marc Pollefeys, Torsten Sattler Spring 2016

3D Vision: Stereo Marc Pollefeys, Torsten Sattler Spring 2016 http://cvg.ethz.ch/teaching/3dvision/ Dense Correspondence / Stereo Tsukuba dataset http://cat.middlebury.edu/stereo/ Stereo Standard stereo geometry Stereo matching

478 views • 35 slides
1 Basic Stereo Derivations Correspondence It is fundamentally ambiguous, even with stereo

1 Basic Stereo Derivations Correspondence It is fundamentally ambiguous, even with stereo

Stereo Vision Reading: Chapter 11 Stereo matching computes depth from two or more images Subproblems: Calibrating camera positions. Finding all corresponding points (hardest part) Computing depth or surfaces. Slide credits

107 views • 7 slides
Stereo Matching 16-385 Computer Vision (Kris Kitani) Carnegie Mellon University What is stereo

Stereo Matching 16-385 Computer Vision (Kris Kitani) Carnegie Mellon University What is stereo

Stereo Matching 16-385 Computer Vision (Kris Kitani) Carnegie Mellon University What is stereo rectification? Reproject image planes onto a common plane parallel to the line between camera centers Recall: Stereo Rectification What can we

597 views • 26 slides
Stereo Vision Reading: Chapter 11 Stereo matching computes depth from two or more images

Stereo Vision Reading: Chapter 11 Stereo matching computes depth from two or more images

Stereo Vision Reading: Chapter 11 Stereo matching computes depth from two or more images Subproblems: Calibrating camera positions. Finding all corresponding points (hardest part) Computing depth or surfaces. Slide credits

468 views • 42 slides
Video-Rate Stereo Vision on a Reconfigurable Hardware Ahmad Darabiha Department of Electrical

Video-Rate Stereo Vision on a Reconfigurable Hardware Ahmad Darabiha Department of Electrical

Video-Rate Stereo Vision on a Reconfigurable Hardware Ahmad Darabiha Department of Electrical and Computer Engineering University of Toronto Introduction What is Stereo Vision? The ability of finding the depth information encoded

384 views • 26 slides
Stereo Tues Mar 27 Kristen Grauman UT Austin Outline Human stereopsis Epipolar

Stereo Tues Mar 27 Kristen Grauman UT Austin Outline Human stereopsis Epipolar

CS 376: Computer Vision - lecture 17 3/26/2018 Stereo Tues Mar 27 Kristen Grauman UT Austin Outline Human stereopsis Epipolar geometry and the epipolar constraint Case example with parallel optical axes General case with

300 views • 18 slides
Human-Robot Interaction CMSC 691 Spring 2016 2 u What is an interaction with a robot? u What is

Human-Robot Interaction CMSC 691 Spring 2016 2 u What is an interaction with a robot? u What is

2/9/16 Human-Robot Interaction CMSC 691 Spring 2016 2 u What is an interaction with a robot? u What is a robot? u What counts as interaction? u Robot(ic)s, for our purposes, is where computation meets the physical world Principles of u

364 views • 4 slides
Human-robot interaction Human-robot interaction in autism in autism S. Casalini, G. Dalle Mura,

Human-robot interaction Human-robot interaction in autism in autism S. Casalini, G. Dalle Mura,

ICRA07 2007 IEEE International Conference on Robotics and Automation Full Day Workshop on Roboethics Rome, April 14th, 2007 Human-robot interaction Human-robot interaction in autism in autism S. Casalini, G. Dalle Mura, M. L. Sica, A.

729 views • 34 slides
Stereo Vision, Multi-View Object and Scene Reconstruction Veronica SCURTU ARTEMIS Department 1

Stereo Vision, Multi-View Object and Scene Reconstruction Veronica SCURTU ARTEMIS Department 1

29/05/2018 Stereo Vision, Multi-View Object and Scene Reconstruction Veronica SCURTU ARTEMIS Department 1 30/05/2018 Institut Mines-Tlcom IMA 4509 - Stereo vision 3D Visual content Our goal: Estimate 3D world properties from

1.27k views • 122 slides
SPORT Handover Conference 08/03/18 A NEW VISION FOR SPORT We are committed to providing

SPORT Handover Conference 08/03/18 A NEW VISION FOR SPORT We are committed to providing

SPORT Handover Conference 08/03/18 A NEW VISION FOR SPORT We are committed to providing University of Portsmouth students with the best possible sports package. We are exploring a new partnership between Sport & Recreation and the

89 views • 8 slides
Geometry and Structure from Motion Computer Vision Fall 2018 Columbia University Stereo

Geometry and Structure from Motion Computer Vision Fall 2018 Columbia University Stereo

Geometry and Structure from Motion Computer Vision Fall 2018 Columbia University Stereo epipolar lines (x 2 , y 1 ) (x 1 , y 1 ) Two images captured by a purely horizontal translating camera ( rectified stereo pair) x 2 -x 1 = the disparity

650 views • 64 slides
Stereo Vision I Introduction to Computer Vision CSE 152 Lecture 13 CSE152, Spr 07 Intro

Stereo Vision I Introduction to Computer Vision CSE 152 Lecture 13 CSE152, Spr 07 Intro

Stereo Vision I Introduction to Computer Vision CSE 152 Lecture 13 CSE152, Spr 07 Intro Computer Vision Announcements Midterm Percentage Score Out of Mean 67 52. 78 pts. Median 62 48 High 96 75 Low 24 18 CSE152,

315 views • 17 slides
Continually Improving Grounded Natural Language Understanding through Human-Robot Dialog Jesse

Continually Improving Grounded Natural Language Understanding through Human-Robot Dialog Jesse

Continually Improving Grounded Natural Language Understanding through Human-Robot Dialog Jesse Thomason University of Texas at Austin Ph.D. Defense Human-Robot Dialog 2 Human-Robot Dialog alert me if her heart rate decreases bring

1.73k views • 142 slides
Evaluation of Different Methods for Using Colour Information in Global Stereo Matching Approaches

Evaluation of Different Methods for Using Colour Information in Global Stereo Matching Approaches

Evaluation of Different Methods for Using Colour Information in Global Stereo Matching Approaches Michael Bleyer 1 , Sylvie Chambon 2 , Uta Poppe 1 and Margrit Gelautz 1 1 Vienna University of Technology, Austria 2 Laboratoire Central des Ponts et

880 views • 58 slides
Human-Robot Interaction Elective in Artificial Intelligence Lecture 9 Motion control for HRI

Human-Robot Interaction Elective in Artificial Intelligence Lecture 9 Motion control for HRI

Human-Robot Interaction Elective in Artificial Intelligence Lecture 9 Motion control for HRI Luca Iocchi DIAG, Sapienza University of Rome, Italy Outline Robot motion: main feature characterizing HRI from HCI Human-robot distance

437 views • 21 slides
COMPUTER VISION FOR ROBOT NAVIGATION Sanketh Shetty Computer Vision and Robotics Laboratory

COMPUTER VISION FOR ROBOT NAVIGATION Sanketh Shetty Computer Vision and Robotics Laboratory

COMPUTER VISION FOR ROBOT NAVIGATION Sanketh Shetty Computer Vision and Robotics Laboratory University of Illinois Urbana-Champaign MEET THE ROBOTS WHAT DOES A ROBOT CARE ABOUT WHEN NAVIGATING? Current Location and Destination (possibly

658 views • 30 slides
Human-Robot Interaction Programming HRI is hard! Represent interaction in time &

Human-Robot Interaction Programming HRI is hard! Represent interaction in time &

B ayesian I nteraction P rimitives: A SLAM Approach to Human-Robot Interaction Joseph Campbell, Heni Ben Amor School of Computing, Informatics and Decision Systems Engineering Arizona State University 1st Conference on Robot Learning (CoRL

1.33k views • 12 slides
Rights & Obligations Katsushi Ikeuchi Microsoft Who the hell is this guy? Physics-based

Rights & Obligations Katsushi Ikeuchi Microsoft Who the hell is this guy? Physics-based

Rights & Obligations Katsushi Ikeuchi Microsoft Who the hell is this guy? Physics-based Vision Robot Vision MS 2015 Shape-from-shading Bin-picking UTokyo Single view Photometric stereo Photometric 1996 Vision Algorithm

643 views • 39 slides
Stereo Vision 16-385 Computer Vision (Kris Kitani) Carnegie Mellon University Whats different

Stereo Vision 16-385 Computer Vision (Kris Kitani) Carnegie Mellon University Whats different

Stereo Vision 16-385 Computer Vision (Kris Kitani) Carnegie Mellon University Whats different between these two images? Objects that are close move more or less? The amount of horizontal movement is inversely proportional to The amount

738 views • 50 slides
experiment PsyIntEC Feasability demonstration project targeting Human-Robot Interfacing and

experiment PsyIntEC Feasability demonstration project targeting Human-Robot Interfacing and

Karlskrona, Sweden experiment PsyIntEC Feasability demonstration project targeting Human-Robot Interfacing and Safety Dr. Johan Hagelbck Project idea Joint human-robot work cell. A human co-worker collaboratively solves a reference

638 views • 40 slides
Rights of Robots Luke Branson Robot Rights = A.I. Rights Robot- a machine resembling a human

Rights of Robots Luke Branson Robot Rights = A.I. Rights Robot- a machine resembling a human

Rights of Robots Luke Branson Robot Rights = A.I. Rights Robot- a machine resembling a human being and A.I. - an area of computer science that emphasizes able to replicate certain human movements and the creation of intelligent machines that

415 views • 17 slides
C18 Computer Vision Lecture 8 Recovering 3D from two images II: stereo correspondences,

C18 Computer Vision Lecture 8 Recovering 3D from two images II: stereo correspondences,

C18 Computer Vision Lecture 8 Recovering 3D from two images II: stereo correspondences, triangulation. Victor Adrian Prisacariu http://www.robots.ox.ac.uk/~victor Computer Vision: This time 5. Imaging geometry, camera calibration. 6.

637 views • 41 slides
Today Recap: epipolar constraint Stereo image rectification Stereo: Stereo

Today Recap: epipolar constraint Stereo image rectification Stereo: Stereo

3/29/2011 Today Recap: epipolar constraint Stereo image rectification Stereo: Stereo solutions Correspondence and Calibration Computing correspondences Non-geometric stereo constraints Mon, March 28 Calibration

342 views • 13 slides
Real-Time Monocular SLAM Andrew Davison Robot Vision Group Department of Computing Imperial

Real-Time Monocular SLAM Andrew Davison Robot Vision Group Department of Computing Imperial

Real-Time Monocular SLAM Andrew Davison Robot Vision Group Department of Computing Imperial College London March 30, 2011 Robot Vision in Real-Time Performance in robot vision is advancing fast . What are the reasons? Continued exponential

525 views • 34 slides

More recommend