Hand Gesture Recognition By Jonathan Pritchard Outline Motivation - PowerPoint PPT Presentation

Hand Gesture Recognition By Jonathan Pritchard

Outline • Motivation • Methods o Kinematic Models o Feature Extraction • Implemented Algorithm • Results

Motivation • Virtual Reality – Manipulation of virtual objects with one’s hands. • Robotics/Telepresence – Precise control of machinery from remote locations. • Sign Language – Help the disabled interact with computers. ASL can be used as test bed for different algorithms. Murthy, G. R. S., & Jadon, R. S. (2009). A review of vision based hand gestures recognition. International Journal of Information Technology and Knowledge Management , 2(2), 405-410.

Kinematic Models • Simplifying assumptions about hand motion used to limit the degrees of freedom in the model • Many model based approaches use a form of causal tracking to ease computation. o Filtering used to estimate state (pose, gesture covariance matrix) based on previous state(s) • Wire Frame and Silhouette models J. M. Rehg and T. Kanade . “Visual tracking of high DOF articulated structures: an application to human hand tracking”. In J. -O. Eklundh, editor, Proc. 3rd European Conf. on Computer Vision , volume II of Lecture Notes in Computer Science 801 , pages 35 – 46. Springer-Verlag, May 1994.

Kinematic Models: Wire Frame Stereo Vision - Hand 3D Model Features Identified Filtering Pose Estimation J. M. Rehg and T. Kanade. “Visual tracking of high DOF articulated structures: an application to human hand tracking”. In J.-O. Eklundh, editor, Proc. 3rd European Conf. on Computer Vision , volume II of Lecture Notes in Computer Science 801 , pages 35 – 46. Springer-Verlag, May 1994.

Kinematic Models: Silhouette Silhouette matched to gesture 3D Model outline with error minimizing (Truncated Quadratics) Kalman Filtering Silhouettes From 3D Model Stenger, B., Mendonca, P. & Cipolla , R. “Model - Based 3D Tracking of an Articulated Hand”. In IEEE Conference on Computer Vision and Pattern Recognition, (2001) 310 – 315.

Feature Extraction • “Getting your man without finding his body parts” • Low level image features used to extract information without estimating pose • Not nearly as robust as model based approaches, but far simpler and faster to compute. R. Polana and R. Nelson, “Low level recognition of human motion”, in Proc. of IEEE Workshop on Motion of Non-Rigid and Articulated Objects, Austin, 1994 , pp. 77 – 82.

Feature Extraction: Number of Fingers Image separated into Threshold applied to saturation HSL color spaces space, only largest connected contour kept Circle centered at centroid used Wrist removed, centroid calculated to calculate number of fingers New, J. R., Hasanbelliu , E. and Aguilar, M. “Facilitating User Interaction with Complex Systems via Hand Gesture Recognition.” In Proc. of Southeastern ACM Conf., Savannah, (2003).

Feature Extraction: Fingertips Orientation found by HSV color space used comparing oriented Fingertips detected through to obtain binary image histograms algorithm looking at top edge of hand, and it’s derivative J. Raheja, K. Das & A. Chaudhary “ Fingertip Detection: A Fast Method with Natural Hand“ . International Journal of Embedded Systems and Computer Engineering , Vol. 3, No. 2, July-December 2011, pp 85-88

Implemented Algorithm • Fingertip Detection using MATLAB image processing toolbox. • Combination of previous feature extraction algorithms o HSV color space used to threshold binary image o Detect orientation of hand, find outline of top o Use outline values, derivative filter, and knowledge of hand orientation to locate fingertips.

Preliminary Results Binary Top Image Outline Result Fingertips

Preliminary Results Binary Top Image Outline Result Fingertips

Preliminary Results Binary Top Image Outline Result Fingertips

Preliminary Results Binary Top Image Outline Result Fingertips

Continued Work • Orientation invariance through wrist detection • Calculate centroid of binary image o Reject detected fingertips that are too close to centroid • Subtract wrist for more accurate centroid calculation.

Questions? M. Randall “Questions”, XKCD, no. 1256 Available: http://imgs.xkcd.com/comics/questions.png