skystitch
play

SKYSTITCH A Cooperative Multi-UAV-based Real-time Video - PowerPoint PPT Presentation

Jul 11, 2023 •289 likes •1.11k views

SKYSTITCH A Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching Xiangyun Meng, Wei Wang and Ben Leong National University of Singapore Motivation Aerial video surveillance has become ubiquitous Search &

  1. SKYSTITCH A Cooperative Multi-UAV-based Real-time Video Surveillance System with Stitching Xiangyun Meng, Wei Wang and Ben Leong National University of Singapore

  2. Motivation Aerial video surveillance has become ubiquitous • Search & rescue • TV live broadcast • Border monitoring

  3. Motivation We always prefer… Higher resolution Larger field of view (More details) (Better awareness)

  4. The problem Single aircraft Limited resolution Limited field of view

  5. Increasing #aircraft?

  6. Increasing #aircraft? Hard to correlate multiple videos!

  7. Stitch them together!

  8. Stitch them together! Video Stitching Video Streaming

  9. Challenges Image stitching is Artefacts affect computationally perceptual expensive experience

  10. Our contributions 4X improvement on Sensor Distributed stitching speed GPU Architecture Hints H Ensuring good quality Failure Sanity Fusion under dynamic conditions Recovery Check

  11. Image stitching in a nutshell Image alignment 1. Feature extraction 2. Feature matching 3. Estimation of H 4. Compositing H

  12. Image stitching in a nutshell Image alignment 1. Feature extraction 2. Feature matching 3. Estimation of H 4. Compositing H

  13. Image stitching in a nutshell Image alignment 1. Feature extraction 2. Feature matching 3. Estimation of H 4. Compositing H

  14. Image stitching in a nutshell Image alignment 1. Feature extraction 2. Feature matching 3. Estimation of H 4. Compositing H

  15. Conventional architecture Video frames Feature Feature RANSAC Compositing Extraction Matching Ground Station Stitching Pipeline

  16. Conventional architecture Video frames + Features Feature Feature Feature Feature Feature RANSAC Compositing Extraction Extraction Extraction Extraction Matching 6ms / image Ground Station Stitching Pipeline

  17. Offloading feature extraction 300 CPU Benchmark 250 6X faster than CPU benchmark GPU Benchmark 2X faster than GPU benchmark 200 SkyStitch Time (ms) Scalability: constant 150 100 50 0 2 3 4 5 6 7 8 9 10 11 12 # Video sources

  18. Speed optimization Video streams + Features Feature RANSAC Compositing Matching Ground Station Stitching Pipeline

  19. Speeding up feature matching ¨ Bruteforce feature matching: inefficient and error- prone ¨ 1K features ➔ 1 million comparisons ➔ 16 ms + + + + + + + + + + + + + + + + + + + + + + + + + +

  20. Exploiting flight status information Camera Attitude Accelerometer Gyroscope Compass GPS Camera Heading Camera Location

  21. Exploiting flight status information ¨ Idea: estimate the matched feature’s location + + + + + + + + + + + + + + + + + + + + + + + + + +

  22. Exploiting flight status information ¨ Idea: search for the matched feature around the estimated location

  23. Exploiting flight status information ¨ Idea: search for the matched feature around the estimated location r r < 30 pixels for a 1280x1024 image

  24. Speeding up feature matching 20 18 GPU Benchmark 70X faster than CPU benchmark 16 SkyStitch 4X faster than GPU benchmark 14 Time (ms) 12 Note: SkyStitch’s feature 10 matching runs on CPU! 8 Potentially much faster if 6 implemented on GPU. 4 2 0 2 3 4 5 6 7 8 9 10 11 12 # Video sources

  25. Speed optimization Video streams + Features Feature GPU GPU RANSAC Matching++ Compositing 20X faster 30 ms for compositing 12 HD images Ground Station

  26. Putting things together… 40 CPU Benchmark 35 GPU Benchmark Stitching rate (fps) 30 SkyStitch 25 SkyStitch: 22 fps 20 15 10 5 GPU Benchmark: 4.2 fps CPU Benchmark: 1.4 fps 0 2 3 4 5 6 7 8 9 10 11 12 # Video sources

  27. Speed is not everything Perspective Frame Perspective distortion drops jerkiness

  28. Frame drops ¨ When we get a bad homography, we have to drop the frame Frame n L H n Frame n × Stitched frame n+1

  29. Failure recovery ¨ Instead of dropping the frame, we predict a good one Computing an optical flow F homography F on each UAV UAV 1 Frame n Frame n+1 1, n + 1 L H n Predict H n+1 : − 1 H n M 1, n F − 1 M 1, n + 1 − 1 H n + 1 = M 2, n + 1 F 2, n + 1 M 2, n 1, n + 1 F Frame n Frame n+1 2, n + 1 UAV 2 ( M is the orthorectification matrix) × Stitched frame n+1

  30. Failure recovery ¨ Instead of dropping the frame, we predict a good one Computing an optical flow F homography F on each UAV UAV 1 Frame n Frame n+1 1, n + 1 J H n H n + 1 Predict H n+1 : − 1 H n M 1, n F − 1 M 1, n + 1 − 1 H n + 1 = M 2, n + 1 F 2, n + 1 M 2, n 1, n + 1 F Frame n Frame n+1 2, n + 1 UAV 2 ( M is the orthorectification matrix) ✔ Stitched frame n+1

  31. Perspective jerkiness

  32. Quantifying jerkiness Frame 1 R + tn T Roll, pitch, yaw H 1 Frame 1

  33. Stitching each pair of frame Frame 1 Frame 2 Frame 3 Frame 4 H 1 H 2 H 3 H 4 Frame 1 Frame 2 Frame 3 Frame 4

  34. Stitching is noisy Pitch angles in estimated homographies 3 Only Stitching 2.5 2 1.5 Angle (deg) 1 0.5 0 -0.5 -1 -1.5 0 50 100 150 200 250 Frames

  35. Observation We have two homography F 1, n + 1 solutions for a pair of frames Frame n Frame n+1 H n H n + 1 One is from stitching F 2, n + 1 Frame n Frame n+1 The other is from prediction Stitched Stitched frame n frame n+1

  36. Keep doing prediction Frame 1 Frame 2 Frame 3 Frame 4 H 1 H 2 H 3 H 4 Frame 1 Frame 2 Frame 3 Frame 4 Stitching Prediction Prediction Prediction

  37. Optical flow is drifty Pitch angles in estimated homographies 3 Only Optical Flow 2.5 2 1.5 Angle (deg) 1 0.5 0 -0.5 -1 -1.5 0 50 100 150 200 250 Frames

  38. Stitching vs. Prediction Short Term Long Term Stitching Noisy Stable Prediction Smooth Drifty

  39. Idea: fuse them n 1 Matched features H stitching t 1 R 1 Multiplicative R’ Solve Translation Extended R 2 Kalman Filter t 2 H prediction H’ t’ n 2

  40. Fusion Pitch angles in estimated homographies 3 Only Stitching 2.5 Only Optical Flow 2 Fused 1.5 Angle (deg) 1 0.5 0 -0.5 -1 -1.5 0 50 100 150 200 250 Frames

  42. Implementation UAVs: Two DIY Quadcopters Ground station: A single Linux desktop 16k lines of C/C++ DIY Quadcopter ($1200 USD each)

  43. Demo 1 Demo 1 Synchronized shutter Very strong wind Camera views UAV 1 UAV 2

  44. Demo 2 Demo 2 Simulation of stitching 12 video streams

  45. Conclusion & Future work Present Future Test in more complex Real-time scenarios performance (20 fps @ 12 videos) Maximizing field of view High quality (High success rate, Network optimization low jerkiness)

  46. Thank you!

  47. Unused slides

  49. Performance of existing solutions Image alignment 1. Feature extraction 2. Feature matching 3. Estimation of H 4. Compositing H 6 ms per image 3 ms per image pair 11 ms per image pair (GPU) (GPU) (CPU) Test setup: Intel Core i7 2600K; GeForce GTX 670; OpenCV 2.4.8 with CUDA; 1000 features per image

  50. Performance of existing solutions ¨ Each stage could be a computational bottleneck ¨ Optimize each stage one by one

  51. Offloading feature extraction Video streams + Features Feature RANSAC Compositing Matching Ground Station Stitching Pipeline

  52. Exploiting flight status information Cameras can be tilted due to wind turbulence Quadcopter attitude Flight controller Orthorectification R Camera M = KBR − 1 B − 1 K − 1 Orthorectified Video video frames frames Warp each image as if camera is always pointed vertically downwards

  53. Speed optimization Video streams + Features Feature RANSAC Compositing Matching++ Ground Station

  54. Speeding up RANSAC ¨ Existing RANSAC homography estimator ¤ Each iteration: Solve a 4-point homography n Do SVD on a 9x9 matrix to find the eigenvector corresponding to eigenvalue zero. ¤ Takes 11 ms for 512-iteration RANSAC on a 3.4GHz Core i7 ¤ GPU SVD? n SVD is not well suited for GPU architecture (shown later)

  55. Speeding up RANSAC ¨ Idea: no need to do SVD at all! ¤ Just find the null vector for the 9x9 matrix ¤ Gauss-Jordan elimination is sufficient n Well suited to GPU architecture n Much simpler code n No branching n Takes 0.6 ms for 512-iteration RANSAC

  56. Speeding up RANSAC ¨ Idea: maximize parallelism and minimize I/O ¤ Compute ALL pairwise homographies in one pass 4 − point Candidate correspondences homographies All matches Inlier mask Scores p q 1 2 3 4 H p q Pair 1 2 3 4 H p q ... ... Pair i 1 , j 1 i 1 , j 1 p q 1 2 3 4 H p q Reprojection 1 2 3 4 H ... ... ... Error Gauss Jordan p q ... ... ... Pair i k , j k Pair p q Elimination ... i k , j k p q H 0 H n − 1 1 2 3 4 H p q H 0 H n − 1 Uploaded Uploaded Downloaded Downloaded from CPU from CPU from GPU from GPU

  57. Speeding up RANSAC 180 benchmark, CPU 160 benchmark, GPU RANSAC time (ms) 140 SkyStitch 24X faster than CPU benchmark 120 18X faster than GPU benchmark 100 based on Jacobian SVD 80 60 40 20 0 2 4 6 8 10 12 Number of video sources (512 RANSAC iterations)

  58. Multiple video sources H 1 Camera 1 Camera 2 H 4 H 2 H 3 Camera 4 Camera 3

Recommend

More recommend