Neurodynamics of expression coding in the core face network - PowerPoint PPT Presentation

Neurodynamics of expression coding in the core face network Yuanning Li, Michael J. Ward, Witold J. Lipski, R. Mark Richardson, and Avniel Singh Ghuman Carnegie Mellon University University of PiDsburgh

Does neural activity in fusiform code for facial expression information? • Contradictory evidence and theories can be found in the literature about the coding in fusiform face area (FFA). Classical model Recent proposed model (Haxby et al., 2000) (Duchaine & Yovel, 2015) general structural and shape FFA coding invariant aspect of faces information of faces FFA contributes No Yes to expression recognition Time of activity not specified ~170 ms after stim onset 2

Does neural activity in fusiform code for facial expression information? • Contradictory evidence and theories can be found in the literature about the coding in fusiform face area (FFA). Classical model Recent proposed model (Haxby et al., 2000) (Duchaine & Yovel, 2015) general structural and shape FFA coding invariant aspect of faces information of faces FFA contributed No Yes to expression recognition Time of activity not specified ~170 ms after stim onset 2

Does neural activity in fusiform code for facial expression information? • meta-analysis: 53 studies found on Neurosynth.org with full brain functional mapping and comparison between emotions. 3

Does neural activity in fusiform code for facial expression information? • meta-analysis: 53 studies found on Neurosynth.org with full brain functional mapping and comparison between emotions. • 14/53 report significant contrast in fusiform. 3

Does neural activity in fusiform code for facial expression information? • meta-analysis: 53 studies found on Neurosynth.org with full brain functional mapping and comparison between emotions. • 14/53 report significant contrast in fusiform. 3

Research questions • Can facial expression information be decoded from fusiform? • What are the spatiotemporal dynamics of such encoding in fusiform? 4

Research questions • Can facial expression information be decoded from fusiform? • What are the spatiotemporal dynamics of such encoding in fusiform? intracranial EEG: • 19 subjects, 29 electrodes directly recording from the human fusiform • Sensitive multivariate classification approach 4

Methods: intracranial EEG • 19 human epileptic patients • 29 fusiform electrodes selected • anatomical: electrode located in fusiform area • functional: face sensitivity over other categories in event- related potential (ERP) and broadband activity (BB) 5

Methods: intracranial EEG • 19 human epileptic patients • 29 fusiform electrodes selected • anatomical: electrode located in fusiform area • functional: face sensitivity over other categories in event- related potential (ERP) and broadband activity (BB) ERP ~200 ms after stim onset BB R L 5

Methods: • Cognitive task: gender discriminant • 40 individuals (20 male), 5 expressions (neutral, angry, happy, fear, sad) 6

Methods: • Cognitive task: gender discriminant • 40 individuals (20 male), 5 expressions (neutral, angry, happy, fear, sad) • Data analysis • sliding time window • multivariate pattern classification • consider both ERP and BB 6

Results: expression decoding • mean binary expression classification across all fusiform electrodes • peak accuracy 52.34% at 190 ms after stim onset (p < 0.05, Bonferroni corrected) 53 52 accuracy % 51 50 49 -100 0 100 200 300 400 500 600 R L time (ms) 7

Results: spatiotemporal dynamics • pick the electrodes with significant facial expression decoding (permutation test) 17/29 electrodes have significant expression decoding 56 54 accuracy % 52 50 48 -100 0 100 200 300 400 500 600 R L time (ms) 8

Research questions • Can facial expression information be decoded from fusiform? Yes, fusiform activity encodes facial expressions • What are the spatiotemporal dynamics of such encoding in fusiform? 9

Research questions • Can facial expression information be decoded from fusiform? Yes, fusiform activity encodes facial expressions • What are the spatiotemporal dynamics of such encoding in fusiform? 9

Results: spatiotemporal dynamics • No significant difference between the time courses of left fusiform and right fusiform left vs. right 56 left right 54 accuracy % 52 50 48 -100 0 100 200 300 400 500 600 time (ms) R L 10

Results: spatiotemporal dynamics • Significant difference between the timecourses of posterior fusiform and anterior fusiform posterior vs. anterior *** 56 posterior anterior 54 accuracy % 52 50 48 -100 0 100 200 300 400 500 600 time (ms) R L 11

Results: spatiotemporal dynamics • Fusiform electrodes cluster into posterior and anterior clusters posterior vs. anterior 600 late anterior 500 peak time (ms) 400 early 300 posterior 200 100 -65 -60 -55 -50 -45 -40 -35 y axis (mm) R L 12

Research questions • Can facial expression information be decoded from fusiform? Yes, bilateral fusiform activity encodes facial expressions • What are the spatiotemporal dynamics of such encoding in fusiform? Posterior fusiform encodes expressions at the early stage Anterior fusiform encodes expressions at the late stage 13

Discussion • Timing is an important factor in analyzing facial expression processing. • Early (100-200 ms): core processing, intrinsic coding for structural and general shape info • Late (300-500 ms): reciprocal, more deliberative processing (Freiwald & Tsao 2010) Note: (Ghuman et al., 2014) FFA encodes face category in the early stage and individual faces in the late stage. 14

Discussion • Spatial heterogeneity may explain the discrepancy of expression encoding in fusiform from the literature, esp. in fMRI studies. posterior fusiform anterior fusiform 54 accuracy % 52 50 48 -100 0 100 200 300 400 500 600 time (ms) R L 15

Acknowledgments Institutions: Coauthors: • Dr. Avniel Singh Ghuman (UPMC, CNBC) • Dr. R. Mark Richardson (UPMC, CNBC) • Dr. Witold Lipski (UPMC) • Michael Ward (UPMC) iEEG data collection and preprocessing: Funding support: • EMU staff (UPMC Presbyterian) • Matthew Boring (CNUP, CNBC) • Ari Kappel (UPMC) 16

Thank you!

Future directions • Identity X Expression • FFA encodes face individuation (late stage, 200-500 ms after stim. onset) 
 (Ghuman et al., 2014) 58 identity expression 54 accuracy % 50 46 -100 0 100 200 300 400 500 600 time (ms)

Future directions • What facial features underlie such spatiotemporal processing?

Methods: intracranial EEG • 19 human epileptic patients • 29 fusiform electrodes selected • anatomical: electrode located in fusiform area • functional: face sensitivity over other categories in event-related potential (ERP) and broadband activity (BB) 1.5 1 d' 0.5 0 -100 0 100 200 300 400 500 time (ms) R L

Results: face sensitivity • mean face sensitivity across all fusiform electrodes (face vs. non-face) posterior vs. anterior posterior anterior 1.5 1 d' 0.5 0 -100 0 100 200 300 400 500 time (ms) R L

Results: face sensitivity • mean face sensitivity across all fusiform electrodes (face vs. non-face) left vs. right left right 1.5 1 d' 0.5 0 -100 0 100 200 300 400 500 R L time (ms)

Results: representational dissimilarity matrix (RDM) early late (50-250 ms) (250-450 ms) AF AF AN AN p bilateral HA HA NE NE SA SA AF AN HA NE SA AF AN HA NE SA AF AF AN AN left HA HA NE NE SA SA AF AN HA NE SA AF AN HA NE SA AF AF AN AN right HA HA NE NE R L SA SA AF AN HA NE SA AF AN HA NE SA

Results: representational dissimilarity matrix (RDM) early late (50-250 ms) (250-450 ms) AF AF AN AN p bilateral HA HA NE NE SA SA AF AN HA NE SA AF AN HA NE SA AF AF AN AN anterior HA HA NE NE SA SA AF AN HA NE SA AF AN HA NE SA AF AF AN AN posterior HA HA NE NE SA SA R L AF AN HA NE SA AF AN HA NE SA