Kinematic Model of the Hand using Computer Vision Edgar Sim o - PowerPoint PPT Presentation

Kinematic Model of the Hand using Computer Vision Edgar Sim´ o Serra Barcelona, Spain Barcelona - April, 2011

Summary Obtain hand movements through computer vision Reconstruct hand kinematics from data

Index 1 Overview 2 Kinematic Hand Model 3 Hand Detection 4 Kinematic Synthesis 5 Non-linear solver 6 Results

Overview

Motivation EMG Signals ⇐ ⇒ Hand Movement Mapping of Electromyographic (EMG) signals Understanding hand movements Prosthesis design Exoskeleton design

Objectives 1 Design of a theoretical kinematic model of the hand 2 Detection of real hand characteristics with computer vision 3 Solver to adjust the theoretical model to the experimental data

Kinematic Hand Model

State of the Art Robotic Grippers Prosthetic Hands Match function not form Simplified kinematic models

State of the Art Robotic Grippers Prosthetic Hands Match function not form Simplified kinematic models

State of the Art Robotic Grippers Prosthetic Hands Match function not form Simplified kinematic models

Hand Anatomy 5 Fingers 14 Joints 27 Bones Complex articulated system

Hand Topology 3 common joints (wrist) Two fingers with 4 joints (index, middle) Three fingers with 5 joints (third, fourth, thumb) Thumb has different joint structure

Hand Topology (a) Full hand. (b) Index finger. (c) Middle finger. (d) Third finger. (e) Fourth finger. (f) Thumb.

Hand Model 26 degrees of freedom All revolute joints Strong similarity with bone structure

Detection of Hand Characteristics

State of the Art Contact approach Low portability High cost Need for calibration Low resolution

State of the Art Contact approach Low portability High cost Need for calibration Low resolution Non-contact Approach Focus on realtime Low precision Applications in augmented reality

Approach Tracking of nails Non-deformable Fixed to hand structure Easily identifiable Not easy to identify rotation Markers for robust design Allows obtaining of pose (position and rotation)

Markers Simple for robustness Convey pose information

Markers Simple for robustness Convey pose information

Detecting Markers 1 Detect candidate points with Corner and edge detector (Harris 1998) 2 Match candidate points with heuristics 3 Use DAISY descriptors to do tracking (Tola 2010)

Detecting Markers 1 Detect candidate points with Corner and edge detector (Harris 1998) 2 Match candidate points with heuristics 3 Use DAISY descriptors to do tracking (Tola 2010)

P n P: Pose Estimation Problem u = [ A ][ R | t ] p Very important problem in computer vision Identify camera position given n 2D-3D correspondences u ↔ p

Solving the Pose Estimation Specific case 4 coplanar points: P4P Single unique solution (Yuan 1989) Usage of combination of algorithms Orthogonal Iteration (Hager et al. 2000) Robust Pose Estimation (Schweighofer et al. 2006)

Error Analysis Pose (0 degrees) Pose (10 degrees) Pose (20 degrees) 100 100 100 50 50 50 0 0 0 0 50 100 150 200 0 50 100 150 0 50 100 150 Pose (30 degrees) Pose (40 degrees) Pose (50 degrees) 100 100 100 50 50 50 0 0 0 0 50 100 150 −50 0 50 100 150 −50 0 50 100 150 Pose (60 degrees) Pose (70 degrees) Pose (80 degrees) 100 100 100 50 50 50 0 0 0 −50 0 50 100 150 −50 0 50 100 150 −50 0 50 100 150

Error Analysis T. Error (0 degrees) T. Error (10 degrees) T. Error (20 degrees) Interval (99%) 100 100 100 Error (percent) Error (percent) Error (percent) Mean 50 50 50 0 0 0 0 5 10 15 0 5 10 15 0 5 10 15 Noise Standard Deviation (px) Noise Standard Deviation (px) Noise Standard Deviation (px) T. Error (30 degrees) T. Error (40 degrees) T. Error (50 degrees) 100 100 100 Error (percent) Error (percent) Error (percent) 50 50 50 0 0 0 0 5 10 15 0 5 10 15 0 5 10 15 Noise Standard Deviation (px) Noise Standard Deviation (px) Noise Standard Deviation (px) T. Error (60 degrees) T. Error (70 degrees) T. Error (80 degrees) 100 100 100 Error (percent) Error (percent) Error (percent) 50 50 50 0 0 0 0 5 10 15 0 5 10 15 0 5 10 15 Noise Standard Deviation (px) Noise Standard Deviation (px) Noise Standard Deviation (px)

Error Analysis R. Error (0 degrees) R. Error (10 degrees) R. Error (20 degrees) Rotation Error (percent) Rotation Error (percent) Rotation Error (percent) Interval (99%) 60 60 60 Mean 40 40 40 20 20 20 0 0 0 0 5 10 15 0 5 10 15 0 5 10 15 Noise Standard Deviation (px) Noise Standard Deviation (px) Noise Standard Deviation (px) R. Error (30 degrees) R. Error (40 degrees) R. Error (50 degrees) Rotation Error (percent) Rotation Error (percent) Rotation Error (percent) 60 60 60 40 40 40 20 20 20 0 0 0 0 5 10 15 0 5 10 15 0 5 10 15 Noise Standard Deviation (px) Noise Standard Deviation (px) Noise Standard Deviation (px) R. Error (60 degrees) R. Error (70 degrees) R. Error (80 degrees) Rotation Error (percent) Rotation Error (percent) Rotation Error (percent) 60 60 60 40 40 40 20 20 20 0 0 0 0 5 10 15 0 5 10 15 0 5 10 15 Noise Standard Deviation (px) Noise Standard Deviation (px) Noise Standard Deviation (px)

Detection of Hand Reliable and robust Low error in pose estimation with little noise Experimental dataset for adjusting theoretical hand model

Kinematic Synthesis

Kinematic Synthesis Generalized inverse kinematics problem Changing structural parameters to solve problem Motion-to-form problem Very complex and generally unsolved problem

State of the Art Simplifying the problem Gro¨ ebner bases Elimination theory Linear product decomposition

State of the Art Simplifying the problem Gro¨ ebner bases Elimination theory Linear product decomposition Revolute joint structures solved 2R: 2 solutions (A. Perez and J.M. McCarthy 2003)

State of the Art Simplifying the problem Gro¨ ebner bases Elimination theory Linear product decomposition Revolute joint structures solved 2R: 2 solutions (A. Perez and J.M. McCarthy 2003) 3R: 13 solutions in an interval (Lee et al. 2004)

State of the Art Simplifying the problem Gro¨ ebner bases Elimination theory Linear product decomposition Revolute joint structures solved 2R: 2 solutions (A. Perez and J.M. McCarthy 2003) 3R: 13 solutions in an interval (Lee et al. 2004) 4R: Unsolved

State of the Art Simplifying the problem Gro¨ ebner bases Elimination theory Linear product decomposition Revolute joint structures solved 2R: 2 solutions (A. Perez and J.M. McCarthy 2003) 3R: 13 solutions in an interval (Lee et al. 2004) 4R: Unsolved ... 26R: Unsolved (our problem)

Kinematic Chain S 1 M S 3 S 2 a 12 [G] α 12 [H] F Θ 12 d 12 a 23 α 23 Articulated system formed by joints Different topologies Serial Parallel Loop Tree (hand model) S 1 S 3 S 4 2 S 2 1 [G] S 4 Branch 2 F Common Joints Branch 1

Pl¨ ucker Coordinates s 0 Line representation: S = s + ǫ ���� ���� moment orientation Only 2 representations per line ( s and − s )

Dual Quaternions → Dual Quaternion ˆ Homogeneous Matrix [ H ] �− Q � �� � � �� � 8 components 16 components Element of Clifford algebra Unit dual quaternions Represents 3D displacements, points and lines q ∗ = 1 and ˆ q 0 = 0 Two constraints: ˆ q ˆ q · ˆ

Exponential Map Based on Lie Algebra S Allows joints to be written as 1-parameter screws Joint parameters: ˆ θ = θ + ǫ d d Prismatic: θ = 0 Θ Revolute: d = 0 ˆ ˆ θ θ 2 S = cos ˆ S (ˆ ˆ θ θ ) = e 2 + sin 2S

Forward Kinematics Determines end effector position based on joints Direct calculation for serial kinematic topologies T ( θ ) = ˆ ˆ S 1 (ˆ θ 1 )ˆ S 2 (ˆ θ 2 ) · · · ˆ S n (ˆ θ n )ˆ G ˆ ˆ 2 S 1 e 2 S 2 · · · e ˆ 2 S n ˆ θ 1 θ 2 θ n = e G ˆ ˆ ˆ ˆ θ 1 θ 1 θ n θ n 2 S n )ˆ = ( cos 2 + sin 2 S 1 ) · · · ( cos 2 + sin G

Forward Kinematics ∆ˆ θ = θ − θ ref + ǫ ( d − d ref ) ˆ θ ) = ˆ S 1 (∆ˆ θ 1 )ˆ S 2 (∆ˆ θ 2 ) · · · ˆ S n (∆ˆ Q ( ∆ˆ θ n ) ∆ˆ ∆ˆ 2 S 1 e θ 1 2 S 2 · · · e θ 2 ∆ˆ 2 S n θ n = e = ( cos ∆ˆ + sin ∆ˆ θ 1 θ 1 2 S 1 ) 2 · · · ( cos ∆ˆ + sin ∆ˆ θ n θ n 2 S n ) 2

Design Equations Input: task defined by m − 1 relative transformations ˆ P 1 j = ˆ T j ˆ T − 1 1 , j = 2 , . . . , m Variables: joint axes and joint angles Equations: S 1 j (∆ˆ ˆ θ 1 j )ˆ S 2 j (∆ˆ θ 2 j )ˆ S 3 j (∆ˆ θ 3 j ) ˆ 4 j (∆ˆ 4 j ) · · · ˆ 7 j (∆ˆ 7 j ) − ˆ S k θ k S k θ k P k 1 j = 0 , k ∈ 1 , 2 , S 1 j (∆ˆ ˆ θ 1 j )ˆ S 2 j (∆ˆ θ 2 j )ˆ S 3 j (∆ˆ θ 3 j ) ˆ 4 j (∆ˆ 4 j ) · · · ˆ 8 j (∆ˆ 8 j ) − ˆ S k θ k S k θ k P k 1 j = 0 , k ∈ 3 , 4 , 5 , j = 2 , ..., m

Non-linear Solver

System Dimension Relations between variables and equations decrease dimension Equations use 3 algebraic sets: Set Comp Ind Notes 1 1 Periodic with period 2 π θ s · s 0 = 0 , � s � = 1 S = s + ǫ s 0 6 4 q 0 = 0 , ˆ q ∗ = 1 ˆ q 0 Q = ˆ q + ǫ ˆ 8 6 ˆ q · ˆ q ˆ