Skeletal Posture Estimation Shane Transue, Phuc Nguyen, Tam Vu, and - PowerPoint PPT Presentation

Thermal-Depth Fusion for Occluded Body Skeletal Posture Estimation Shane Transue, Phuc Nguyen, Tam Vu, and Min-Hyung Choi University of Colorado IEEE Conference on Connected Health: Applications, Systems and Engineering Technologies

N ON -C ONTACT R ESPIRATORY M ONITORING Radar-based Tidal-volume Estimation Sleep-based Supervised Respiration  Rate, tidal volume, apnea, COPD  Automated Radar Solutions  Orthogonal radar monitoring  Region-based chest movements [P. Nguyen et al, IEEE INFOCOM’16]  Automated Camera Solutions  Monitors chest surface deformations Camera-based Tidal-volume Estimation  Computes changes in volume/behavior Automated Solutions Require Posture  Patient chest orientation  Occlusion detection [S. Transue et al, IEEE/ACM CHASE’16] 2/20

D EPTH AND T HERMAL P OSTURE E STIMATION Occluded Skeletal Tracking  Problem: Identifying occluded joints  Blankets, clothing, hide joint positions  Depth-image is ambiguous  No ground-truth training/labeling/scoring Blanket induced occluded body (example)  Prior: Depth-based Skeletal Tracking  Prior: Thermal Posture Imaging (low) [F. Achilles et al., MICCAI’16] 3/20

THERMAL DEPTH FUSION IMAGING Core Idea: Fuse Depth + Thermal  Depth for 3D surface reconstruction  Thermal for patient heat tracking Monitoring Devices:  Microsoft Kinect2 (512x424@30[fps])  FLIR C2 (80x60@15[fps])  Alignment Bracket Prototype Device and Experimental Design 4/20

M ODELING T HERMAL V OLUME P OSTURE Occlusion Implications Model Assumptions  Ambiguous Depth + Thermal data  Predefined Skeletal Structure (b)  Partial skeletal data (c)  Enclosed volume (patient on surface)  Disconnected skeletal components (c)  Thermal volume reconstruction (a) 5/20 IEEE/ACM Chase 2017

T HERMAL P OSTURE G ROUND -T RUTH Occluded Skeletal Tracking  Visual markers are occluded  Skeletal structure may be incomplete  Require a method for capturing thermal markers Solution: Thermal Motion Tracking  Borrows from traditional motion-capture  Markers are defined by thermal spheres  Interchangeable Thermal Suit  Heated markers Thermal Posture Tracking  Markers are detachable (training only)  Flexible Design  Fixed joint count 6/20

D EPTH + T HERMAL P OSTURE M ODELING (1) Proposed thermal posture estimation: Thermal + Depth to CNN Classification Depth + Thermal Fusion 7/20

D EPTH + T HERMAL P OSTURE M ODELING (2) Proposed thermal posture estimation: Thermal + Depth to CNN Classification TEGI Heat Depth + Thermal Fusion Propagation 8/20

D EPTH + T HERMAL P OSTURE M ODELING (3) Proposed thermal posture estimation: Thermal + Depth to CNN Classification TEGI Heat Thermal Volume Depth + Thermal Fusion Propagation Reconstruction 9/20

D EPTH + T HERMAL P OSTURE M ODELING (4) Proposed thermal posture estimation: Thermal + Depth to CNN Classification TEGI Heat Thermal Volume Depth + Thermal Occluded Estimate Fusion Propagation Reconstruction Posture 10/20

T HERMAL V OLUME R ECONSTRUCTION Posture Volume Assumptions  Posture is enclosed  Human Body is a Connected-component  Propagate from known location (head)  Generate internal structure (enclosed volume) Infrared Image of Posture Solution: Thermal Sphere Hierarchy (enclosed volume under surface)  Sphere-packing  Boundary Conditions (1) Surface Boundary (2) Thermal threshold 2D Thermal Sphere Packing 11/20

2D-3D I NVERSE T HERMAL P ROPAGATION How can we map 2D surface thermal information to 3D voxels? Solution: Thermal Extended Gaussian Images  Maps 2D thermal data to 3D volumes  Parametrized by distance, heat, etc.  Computed by spherical projection TEGI Point-to-volume Mapping 12/20

3D THERMAL VOLUME RESULT Volume Reconstruction Pipeline (1) Surface thermal-cloud (2) Volume enclosure (3) Sphere-packing and heat propagation (4) Voxel-grid representation Depth + Thermal Enclosed Volume Thermal Volume Heat Propagation 13/20

3D T HERMAL M ONITORING ( VIDEO ) 14/20

POSTURE MONITORING APPLICATION 15/20

LABELING, TRAINING, AND CLASSIFICATION Training: Correlate skeletal posture to volumetric thermal data  Volumetric data provided as 3D image to CNN  Classification based on 3D distribution  Coarse-grain posture from classifications Training Components (training-data + labeling) Runtime data 16/20

P OSTURE C LASSIFICATION R ESULTS Note: Classification labels correspond to confusion matrix results 17/20

CLASSIFICATION RESULTS Standard Posture Confusion Matrices Patient-specific training/classification Cross-patient training/classification 18/20

CONCLUSION AND FUTURE WORK Conclusion  Fuse Depth + Fusion for occluded patient tracking  3D Patient heat distribution  Occluded skeletal estimation  Sleep-study patient tracking/posture  Automated respiratory monitoring  Long-term studies Future Work  Feature-based Training (RDF)  Improve image resolution  Address challenges/ambiguity  Other Depth + Thermal applications 19/20

T HANK Y OU