Introduction to Brain Imaging: fMRI and MEG/EEG The Algonauts - PowerPoint PPT Presentation

Introduction to Brain Imaging: fMRI and MEG/EEG The Algonauts Workshop Yalda Mohsenzadeh Computer Science and Artificial Intelligence Lab. MIT, Cambridge, USA July 2019

Convnets: Brain Inspired Architectures V1 V2 V4 IT Zebra 2 (Khaligh-Razavi et al. 2014, Yamins et al. 2014, Guclu et al. 2015, Cichy et al. 2016 )

What is fMRI? 3

Magnetic Resonance Imaging (MRI) 4

Functional Magnetic Resonance Imaging (fMRI) BOLD % 4 Time (sec) 3 BOLD % 2 1 0 0 5 10 15 20 25 Time (sec) Stimulus on Neurons fire Bold response 5

Data Structure Experiment S1 S2 Data Time … 6

General Linear Model: Constructing BOLD signals HRF Baseline signal ß 1 Response to S1 ß 2 Response to S2 ß 3 error(t) = signal (t) – prediction (t) 7

General Linear Model: Constructing BOLD signals ß 1 x + ß 2 x + ß 3 x + err = + err = 8

General Linear Model: Constructing BOLD signals Bold signal Y = X × B + e Task related Noise variations ß 1 variations Time ß 2 x + = ß 3 � Find B such that Min � �𝟐 𝑼 𝑼 Design Matrix 9

Visual Recognition in the Brain • What brain regions are engaged in visual processing? • What kind of representations are held in these regions? • What algorithms are being carried out by these regions? 10

Visual Recognition in the Brain EVC IT 11

Stimulus set 12

N=15 2.5 – 6 seconds + 1/2 sec + 2.5 – 6 seconds + 1/2 sec + 2.5 – 6 seconds + PRESS BUTTON + 2.5 – 6 seconds + + 1/2 sec 13

Representational Similarity Analysis Stimulus Brain representation Representational pattern (e.g. fMRI pattern dissimilarities) (e.g. voxels, neurons, model units) (e.g. images, sounds, other experimental conditions) Compute dissimilarity (e.g. 1 - correlation) Representational dissimilarity matrix (RDM) 14 (Kriegeskorte & Kievit 2013, Diedrichsen et al. 2011, Laakso & Cottrell 2000, Kriegeskorte et al. 2008)

fMRI Track RDMs EVC IT face Bodies object scene Dissimilarity Dissimilarity 15

What is MEG? 16

Magnetoencephalography (MEG) / Electroencephalography (EEG) 306 Channel SQUID sensor array EEG 17

N=15 700 to 1000 m sec + 500 m sec + 700 to 1000 m sec + 500 m sec + 700 to 1000 m sec + PRESS BUTTON + 700 to 1000 m sec + + 500 m sec 18

Stimulus on 19

MEG Neural Data Decoding Representational MEG pattern vector at time t Pairwise classification dissimilarity matrix at time t Train set Test set vs. (i,j) SVM Classifier -300 ms 900 ms t (Carlson et al. 2013; Cichy et al. 2014; Isik et al. 2014; Clarke et al. 2014; Kaneshiro et al. 2015) 20

Time-Resolved MEG RDMs 21

Temporal Generalization N th raw pattern MEG pattern vector Train a SVM Time-Time at time t vector at time t y classifier using N-1 decoding matrix raw vectors at t x 1 2 N N-1 (t x, t y ) SVM Classifier 22 (Carlson et al. 2013; Cichy et al. 2014; Isik et al. 2014; Clarke et al. 2014; Kaneshiro et al. 2015; King et al. 2016)

Possible Neural Architectures (King et al., 2016) 23

A Neural Architecture with Recurrent Interactions Architecture Feedforward Recurrent Un-occluded Occluded 60% (Rajaei, Mohsenzadeh, Ebrahimpour, Khaligh-Razavi, 2019) 24

Distance Measures Decoding Distance Pearson Distance Time (ms) Time (ms) Euclidean Distance 25 Time (ms)

Reliability of Distance Measures Decoding Euclidean Pearson Reliability Time (ms) 26

MEG/ ROI fMRI fusion Early Visual Cortex (EVC) Inferior Temporal (IT) MEG-fMRI representational similarity in EVC MEG-fMRI representational similarity in IT MEG fMRI MEG fMRI -300 ms 900 ms t -300 ms 900 ms t Calculate Spearman’s rho Calculate Spearman’s rho Cichy et al. 2014, 2016; Mohsenzadeh et al., (2018, 2019) 27

ROI-based fMRI-MEG fusion Early Late EVC IT Early Visual Cortex (EVC) Spearman’s rho Inferior Temporal (IT) Time (ms) 28

MEG Track RDMs Early Late face Dissimilarity Dissimilarity Bodies object scene 29

Summary 30

Brain Imaging Methods MRI 31

Brain Imaging Methods fMRI MRI 32

Brain Imaging Methods fMRI EEG MRI MEG 4 3 BOLD % 2 1 0 0 5 10 15 20 25 33 Time (sec)

Representational Similarity Analysis Brain representation Behavior (e.g. fMRI pattern dissimilarities) (e.g. dissimilarity judgments) Computational model representation Stimulus description (e.g. face-detector model) (e.g. pixel-based dissimilarity) 34 (Kriegeskorte & Kievit 2013, Diedrichsen et al. 2011, Laakso & Cottrell 2000, Kriegeskorte et al. 2008)

Acknowledgments 35

36