Haishan Wu School of Computer Science, Fudan University Shanghai, - PowerPoint PPT Presentation

Haishan Wu School of Computer Science, Fudan University Shanghai, China 1 �

When simulation meets empirical data � Different sensors Computer vision techniques: pattern recognition, feature extraction, image processing, stereo reconstruction, machine learning Measurement data 2 �

Related works on motion measurement methods � ž On small groups — Human, fish, ants, bees, cells, bacteria ž On large groups — Bacterial swarms, Drosophila groups, bird flock, human crowd, bat flock There is no robust implementation publicly available for tracking large groups of targets yet. 3 �

General Procedure Tracking each individual in Detecting each individual 2D or 3D throughout the whole data sequences Maintaining the identity individually is very difficult. (false positive Target Detection is a task-specific detections, occlusions etc.). and challenging problem. Computation speed should also be considered. Computer vision techniques that Computer vision techniques that may be used: multi-view geometry, may be used: image segmentation, multi-target tracking, combinatorial object detection, image denoising optimization, feature extraction and feature extraction 4 � matching

Very useful references ž Four excellent books 1. Multiple view geometry in computer vision 2. Computer Vision: Algorithms and Applications （ the latest draft can be downloaded in http://szeliski.org/Book/) 3. Pattern Recognition and Machine Learning 4. Multiple-target tracking with radar applications 5 �

Multi-view geometry and stereo reconstruction Epipolar geometry Image rectification 6 �

Framework for 3D tracking :1 Image sequences of multi-camera setup, often acquired by two or more CCD-cameras in a convergent arrangement Tracking of particles in 2D image space with different approaches (filter + data association) Stereoscopic correspondences problem, matching of the 2D tracks to generate particle trajectories in object space, requires system calibration and orientation 7 �

Framework for 3D tracking: 2 Image sequences of multi-camera (more than two) setup Establishment of particle correspondences with consequent use of epipolar constraints and determination of 3D particle positions, treating each single time step separately, requires system calibration. Tracking is performed in 3D object space, one 3D point cloud as input for each time step 8 �

Framework for 3D tracking :3 Establishment of particle correspondences with epipolar constraints and determination of 3D particle positions, in addition storage of all relevant data from particle detection process 9 �

My work ž Drosophila group tracking in 3D ž Fish school tracking in 2D (I bring my poster here ) +,-./012345,60$(% %"" &"" '"" !"" ("" #"" )"" $"" *"" 10 � !"" #"" $"" %""" %&"" %!"" %#""

Tracking fruit fly group in 3D � Why it is a challenging problem: Tracking or matching Frequent failures will lead occlusions make to the many tracking broken or methods failed. � incomplete trajectories in 3D space � Appearance feature will be also ineffectual for tracking or matching across various views � 11 �

Tracking fruit fly group in 3D (cont.) ž By using two high-speed cameras, we solved the abovementioned problems by using one compact framework: linear assignment problem (LAP) 12

Tracking fruit fly group in 3D (cont.) The first LAP � ž 2 steps: single particle state updating and data association ž State updating: alpha-beta or Kalman filter � 13 �

Project 1: Tracking fruit fly group in 3D (cont.) The first LAP � To handle these difficulties: � ž The first LAP is: � 14 �

Tracking fruit fly group in 3D (cont.) The second LAP � ž Now our problem is how to assign (match) a 2D trajectory in one view to that of another view ž The appearance feature of targets, however, is useless in our experiments, so how to define the matching cost? ž Observations: l Two true matching trajectories will submit to epipolar constrains l The longer the length of matched trajectories, the larger the possibility of correct matching is. 15 �

Project 1: Tracking fruit fly group in 3D (cont.) The second LAP � ž Our matching cost: MECL(maximum epipolar co-motion length) � 16 �

Tracking fruit fly group in 3D (cont.) The third LAP � ž 3D trajectories can be recovered by using multi- view geometry technique. ž However, tracking errors make these trajectories broken into segments ž We solve this problem by formulating it as pairwise tracklet matching problem � 17 �

Tracking fruit fly group in 3D (cont.) The third LAP � ž We incorporate temporal and kinematic information into linking cost. l temporal information l kinematic information � 18 �

Tracking fruit fly group in 3D (cont.) 19

Large scale fish school tracking (I bring the poster here ) 20 �

Acknowledgements � ž Thank Prof. Tamas Vicsek for offering me the opportunity to give a talk here. Many thanks also go to Zsuzsa Ákos. ž Thank Qiyuan Tian, Qi Xu, Yihao Zhou, Miaohui Wang, Ye Liu, Jinlong Shi, Iain Couzin for their valuable comments in Drosophila group tracking. ž Thank Linguo Li and Wei Li at School of Life Sciences of Fudan University for providing fruit flies for the experiments. ž Thank Simon Leblanc in Princeton University for providing the fish school videos 21 �