SEGMENTING HEAD & NECK MR IMAGES Antal Nagy Department of - - PowerPoint PPT Presentation

SEGMENTING HEAD & NECK MR IMAGES

Antal Nagy

Department of Image Processing and Computer Graphics University of Szeged

27th Summer School on Image Processing

Timişsoara, România, 2019

• Medical Image
• Tomographic Image Acquisition Systems
• MR protocols
• Segmentation
• Delineating certain areas
• Organs at Risk (OAR) for radiotherapy
• Healthy organs must be protected from radiation
• Tumors
• Abnormal mass of tissue

Segmenting Head & Neck MR images

2

Introduction

• Transmission Tomography
• Imaging procedure
• Cross-sections of the 3D object

are determined from its projection images

• Projection images created by rays
• Transmitted through and absorbed by the object
• Emission Tomography
• Whole space is filled with some

homogeneous absorbing material

• The function to be reconstructed

represents an object emitting radioactive rays into the surrounding space

• Observe metabolic processes

Segmenting Head & Neck MR images

3

Tomographic Acquisition Systems

Courtesy of http://www.whatisnuclearmedicine.com Courtesy of https://medical-dictionary.thefreedictionary.com

• Created by using
• Powerful magnet
• Magnetic protons which align with the field
• Radio waves
• Knocks down the atoms
• Disrupts their polarity
• Sensors
• Detect the time to return to the original alignment
• MRI shows the fluid characteristic
• f different tissues (soft tissues)
• Air and bone are black

Segmenting Head & Neck MR images

4

Magnetic Resonance Imaging

• Tissue Characteristics
• T1 weighted
• Longitudinal movement of protons
• Protons in fat realign quickly with high energy
• Normal anatomical details
• T2 weighted
• Transverse movement of proton
• Protons in water dephase slowly
• Pathological details

Segmenting Head & Neck MR images

5

Magnetic Resonance Imaging

Historical Overview

Segmenting Head & Neck MR images

6

Intensity of gamma radiation Liver scan after the „vector-gradient” process Kidney study 3D Phase and Amplitude images

SEGAMS nuclear medicine system

Syllabus

• MR Head and Neck

Studies

• Study parameters
• Segments
• Preprocessing Steps
• Noise reduction
• Inhomogeneity

correction

• Histogram matching
• Generating Atlas

Information

• Segmentation Methods
• Different methods
• Different features

Segmenting Head & Neck MR images

7

• Head and neck region is suggested by physicians
• There are no tools/results
• „Tubular” organs
• Starting at a certain slice number
• There is a vertical coherence between the slices
• Organs
• Carotid arteries (left, right)
• Jugular veins (left, right)
• Trachea
• Spinal cord
• Parotid glands (left, right)
• Sternocleidomastoid

muscle (SCM)

• Varying size, shape and

deformation

Segmenting Head & Neck MR images

8

Motivation

INPUTS

„If you know the enemy and know yourself, you need not fear the result of a hundred battles” – Sun Tzu

Segmenting Head & Neck MR images

9

• MR studies used in daily routine
• Body Planes
• Axial slices
• 0.5 mm x 0.5 mm x 7 mm
• MR Protocols
• T1 FATSAT
• T1 FSE
• T1 FSPOSTGAD
• T2 FRFSE

Segmenting Head & Neck MR images

10

MR Head & Neck Input Images

• Body Planes
• Sagital
• T1 FSE POSTGAD
• T2 FRFSE
• Coronal
• T1 FSE POSTGAD
• T2 FRFSE
• Physicians use all
• Positioning
• Sagital and coronal
• Reporting
• Axial

Segmenting Head & Neck MR images

11

MR Head & Neck Input Images

coronal sagital axial

• Axial, coronal, and sagital studies are in different spaces
• They are created in different time even
• Can not be used in same way as the physicians do
• Solution

Make your own input

• Jan Sijbers (Univ. of Antwerp):

Superresolution reconstruction of magnetic resonance images

Segmenting Head & Neck MR images

12

MR Head & Neck Input Images

• Contouring the target organs
• Carried out
• By 2 well trained physicians
• Separately
• On all studies
• On each slices
• Using different modalities
• T2 FRFSE and T1 FSPostGad
• Application
• Validating the segmentation

algorithms’ results

• Creating organ atlas

Segmenting Head & Neck MR images

13

Organ Delineation

• Problems
• Organs delignated in different ways
• On axial slices
• Organ boundary
• Different interpretation of

the head and neck region

• Starting and ending slices
• Overlapping
• Neighboring organs
• Pairs of organs
• E.g. removed during operation
• Tumor deformation

Segmenting Head & Neck MR images

14

Organ Delineation

PREPROCESSING STEPS

Battle with Artifacts

Segmenting Head & Neck MR images

15

• Multiplicative noise with low frequency
• Due to the un-calibrated coils
• Solutions
• N3
• Nonparametric non-uniform normalization
• Retrospective bias correction algorithm
• N4
• De-convolving the intensity histogram

by a Gaussian

• Remapping the intensities
• Spatially smoothing this result

by a B-spline modeling of the bias field itself

• http://stnava.github.io/ANTs/

Segmenting Head & Neck MR images

16

Intensity Inhomogeneity within Studies

• The absolute intensity values

do not have a fixed meaning

• n MRI images
• Solution
• Intensity transformation
• Histogram matching
• Between two MRI studies
• Nyul, L. G., Udupa, J. K., & Zhang, X. (2000).

New variants of a method of MRI scale standardization. IEEE Transactions on Medical Imaging, 19(2), 143-150. DOI: 10.1109/42.836373

Segmenting Head & Neck MR images

17

Intensity Inhomogeneity between Studies

• Anisotropic diffusion
• Edge preserving filtering
• Be careful with the parameters
• Important details

can be washout

Segmenting Head & Neck MR images

18

Noise Reduction

CREATING ATLAS

„Navigare necesse est …”

Segmenting Head & Neck MR images

19

• Problems with manual contouring
• Time consuming
• Organs hard to delineate on certain modalities
• Possibilities of the application
• f the image registration
• Searching for geometry correspondence

between images

• Image resampling on other image grid
• Superimposition
• Joint images combined displaying
• Image fusion

Segmenting Head & Neck MR images

20

Task

Courtesy of Attila Tanács

• Different modalities of the same patients
• Segmented organs of a modality

can be transformed to other mobilities

• Voxel intensities in the same

3D position help the decision

• Manual segmentation
• Automatic clustering

Segmenting Head & Neck MR images

21

Image Registration Tasks

Courtesy of Attila Tanács

• Multiple patients’ data on the same modality
• MRI Images
• Segmented organs
• Transforming them

into a common reference space

• Statistical atlas
• Atlas can be

transformed into a new study space

Segmenting Head & Neck MR images

22

Image Registration Tasks

Courtesy of Attila Tanács

• Same patient, different modalities
• No movement during the acquisition
• Rigid body transformation
• Deformation can be seen
• Non-linear transformation (B-Spline)
• Different patients, same modalities
• Non-linear transformation is necessary with linear fitting preliminary
• Differences between organ shape/size
• Different positions of the head and neck in the studies
• Varying position of the organs

Segmenting Head & Neck MR images

23

Image Registration Algorithms

• Statistic on organ position
• Where is it? Most common
• Steps
• Selecting reference study
• Target area should be covered
• No anomalies
• Registering other studies to the reference
• Scaled rigid body + B-Spline non-linear refinement
• Finding the transformation
• Mutual information
• No anomalies
• Transforming the physicians'’ segmentations

into reference space as well with the resulting transformation

Segmenting Head & Neck MR images

24

Statistical Atlas

• Transforming the statistical atlas into study to be

segmented

• Finding the transformation

between the new study and the reference study

• Using the inverse of

transformations on the

• rgan atlas information

Segmenting Head & Neck MR images

25

Application of the Atlas

Courtesy of Attila Tanács

Early result

• Fitting the jugular vein atlas
• Physician segmentation (white)
• Yellow shows the carotid artery
• Statistical atlas (purple spot)
• Bright area means
• Larger probability
• Not easy to register

Segmenting Head & Neck MR images

26

Application of the Atlas

Courtesy of Attila Tanács

Early result

Application of the Atlas

Segmenting Head & Neck MR images

27

Spinal cord (green), trachea (yellow), Carotid (red), Jugular (blue), Parotid (brown), SCM (cyan)

Elastix ITK composite

Courtesy of Attila Tanács

Application of the Atlas

Segmenting Head & Neck MR images

28

Hard to judge visually → Numerical evaluation is needed

Elastix ITK composite

Courtesy of Attila Tanács

Atlas coverage in the organ regions

Segmenting Head & Neck MR images

29

The atlases generated by the elastix registration provide clearly better coverages

Courtesy of Attila Tanács

elastix Composite Organ Max Mean (SD) Max Mean (SD) Spinal cord 100.0% 61.5% (25.1%) 96.7% 36.6% (24.7%) Trachea 87.7% 48.9% (22.9%) 84.8% 42.8% (21.4%) Carotid (left) 83.4% 31.1% (18.2%) 54.1% 19.9% (11.3%) Carotid (right) 69.3% 26.6% (16.3%) 44.7% 18.7% (10.7%) Jugular (left) 76.9% 37.4% (18.8%) 70.7% 23.1% (16.1%) Jugular (right) 82.1% 35.5% (18.4%) 60.8% 23.6% (13.5%) Parotid (left) 98.8% 54.2% (27.2%) 89.4% 37.1% (23.9%) Parotid (right) 99.9% 56.2% (27.7%) 98.3% 39.1% (26.8%) SCM (left) 100.0% 58.1% (29.1%) 92.4% 41.5% (25.6%) SCM (right) 99.7% 51.5% (28.3%) 96.0% 38.1% (26.3%) Carotid as jugular 52.8% 9.4% (11.2%) 28.6% 4.9% ( 5.7%) Jugular as carotid 49.5% 4.7% (7.6%) 39.9% 4.9% (7.2%)

Carotid and jugular separation

Segmenting Head & Neck MR images

30

Problem Tubular structures close to each other Similar intensity values Experience Overlap occurs mainly in the outer, less likely regions of the atlases Left two images: High overlapping (mean 14%) Right two images: low overlapping (1,26%) Average overlapping: 4,7 % (SD 7,6%)

Courtesy of Attila Tanács

Errors induced by inverse registration

Segmenting Head & Neck MR images

31 Original organs’ voxel positions and distance between the forward and backward transformation The difference can be more than one voxel due to the registration approach

Courtesy of Attila Tanács

elastix Composite Distance (SD) X Y Z Distance (SD) Spinal cord 4.77 (2.00) 0.72 1.78 4.03 8.43 (5.09) Trachea 6.86 (3.03) 0.97 2.17 6.14 7.60 (5.11) Carotid (left) 4.79 (2.34) 1.21 1.18 4.12 7.10 (5.41) Carotid (right) 4.57 (2.20) 1.12 1.39 3.86 6.53 (4.36) Jugular (left) 4.38 (2.64) 1.43 1.25 3.55 6.60 (5.19) Jugular (right) 4.41 (1.89) 1.35 1.51 3.43 7.78 (4.86) Parotid (left) 4.02 (3.06) 1.25 1.67 3.00 9.28 (9.11) Parotid (right) 4.83 (2.57) 1.47 2.25 3.38 8.74 (5.40) SCM (left) 4.08 (1.79) 1.53 1.57 2.96 6.24 (5.03) SCM (right) 4.26 (1.75) 1.44 1.70 3.09 7.50 (3.84)

• Attila Tanács: Generation and Evaluation of an MRI

Statistical Organ Atlas in the Head-Neck Region, ISPA 2017

Segmenting Head & Neck MR images

32

Publication

SEGMENTATION ALGORITHMS

Segmenting Head & Neck MR images

33

• Preprocessing
• BIAS correction
• Histogram matching
• Extracting DSIFT
• Machine learning
• Model fitting
• Evaluation of the test set
• Post processing
• Correction in two phase
• On 2D slices
• 3D coherence

Segmenting Head & Neck MR images

34

Spinal cord

Upper region Lower region

• Taking the gradient magnitude
• n the given scale in all points
• Calculating histogram in 8 directions

for each 4x4 block in the 16x16 neighborhood

• f the point
• Gaussian weighting on the histogram values
• Result
• 4x4x8 = 128 dimensional descriptor
• DSIFT
• Getting this feature in each point

Segmenting Head & Neck MR images

35

Scale Invariant Feature Transformation (SIFT)

• Extracting DSIFT features
• Training set
• Positive examples
• Spinal cord area
• Negative examples
• Spinal cord atlas
• Around the spinal cord area using dilation
• More strict
• Test set
• Spinal cord atlas
• Not using the segmentation information
• Support Vector Machine (SVM)
• Efficient in case of
• high dimensional data
• If the test set size is smaller than the dimension
• Using kernel filters

Segmenting Head & Neck MR images

36

Machine learning

Courtesy of Dominik Hirling

• The result of the SVM is the starting input
• False positive voxels
• False negative voxels
• First phase
• Eliminating the true positive voxels
• Selecting the wrong slices
• Second phase
• 3D interpolating the wrong slices

using the neighbor correct slices

Segmenting Head & Neck MR images

37

Postprocessing

Courtesy of Dominik Hirling

• First phase
• Input segmentations from trained
• Atlas area
• Around the spinal cord area
• Assignment of the objects with the maximal size
• Closing, hole filling
• Comparing the two results
• If there is any overlapping
• Choose the results coming from more strict model
• If there is no overlapping
• Take all voxels which coming form other model

and can be found in the result of the strict model before the hole filing and the maximal object selection

• If there is no such a voxel take the no strict area
• Give results in most cases

Segmenting Head & Neck MR images

38

Postprocessing

Courtesy of Dominik Hirling

• First phase

Segmenting Head & Neck MR images

39

Postprocessing

Courtesy of Dominik Hirling

• Second phase
• Mark slice of the firs phase results’ if
• The size of the area is very-very small
• The distance between the center of

the atlas area and object far away from each other

• Interpolating slice between the last

two not marked slices

Segmenting Head & Neck MR images

40

Postprocessing

Courtesy of Dominik Hirling

Results

Segmenting Head & Neck MR images

41

RME a r SPC p F1 Jaccard var N38 10.61% 97.54% 91.11% 99.42% 98.20% 94.42% 90.99% 10.09% N42 10.46% 97.53% 92.59% 98.91% 96.96% 94.60% 89.72% 11.06% N31 21.89% 96.62% 80.75% 99.38% 97.08% 86.45% 85.67% 20.40% N41 23.76% 96.13% 84.61% 98.72% 92.25% 87.43% 80.14% 24.56% N39 17.26% 96.82% 86.21% 99.05% 95.87% 90.22% 85.79% 18.14% N51 15.62% 97.65% 87.93% 98.88% 96.49% 91.47% 86.97% 15.62% Avg. 17.43% 96.78% 86.33% 99.12% 95.77% 90.80% 89.02% 13.32%

Results

Segmenting Head & Neck MR images

42

N38 N42 RME: RME: 11.45% 15.27% 14.83% 8.73% 17.17% 16.31%

Courtesy of Dominik Hirling

Results

Segmenting Head & Neck MR images

43

N31 N41 RME: RME: 24.82% 21.65% 26.52% 27.23% 22.29% 49.55%

Courtesy of Dominik Hirling

Physicians’ Agreement

Segmenting Head & Neck MR images

44

J(A,B) 67.35% 44.87% Var(A,B) 48.48% 122.86% J(A,B) 63.20% 51.57% Var(A,B) 58.24% 93.91% Jaccard-index: 𝐾 𝐵, 𝐶 = 𝐵 ∩ 𝐶 𝐵 ∪ 𝐶 (∗ 100) Variability: 𝑤𝑏𝑠 𝐵, 𝐶 = 𝐵 ∪ 𝐶 − 𝐵 ∩ 𝐶 𝐵 ∩ 𝐶 (∗ 100)

Courtesy of Dominik Hirling

• Dominik Hirling: Segmenting Spinal Cord in the Head and

Neck Area, Local Scientific Student Conference

Segmenting Head & Neck MR images

45

Publication

• Automatic segmentation within the atlas area
• Registered Protocols
• T2-FRFSE, T1-FSE, PostGad T1-FSE
• Image fusion
• Pixel level classification
• Deep neural network
• Training and test data set
• Result is a probability value

Segmenting Head & Neck MR images

46

Background-Foreground Segmentation

Courtesy of László Varga

• For each organ within the atlas region
• 2D slices
• Using 101x101 neighborhood
• Taking the background and foreground voxels with the same probability
• The label is the center voxel
• Segmenting the voxel neighborhood

Segmenting Head & Neck MR images

47

Learning Phase

Parotid gland Non Parotid gland

Courtesy of László Varga

• Organ segmentation according to their complexity

Segmenting Head & Neck MR images

48

Results

DSC= 2 ∗

𝑞∗𝑠 𝑞+𝑠 (∗ 100)

Organ DSC Trachea 0.823 Spinal cord 0.866 Parotid 0.814 SCM 0.814 Carotid 0.624 Jugular 0.659

Courtesy of László Varga

• Trachea, Spinal cord
• Easy to segment
• Lots of results in the literature
• DSC > 0.8
• Comparable with other methods
• SCM, Parotid gland
• Little bit better than the results

in the literature

• DSC  0.8
• Carotid artery, Jugular vein
• Small flexible organ
• Hard to segment
• DSC > 0.6
• No results in the literature

Segmenting Head & Neck MR images

49

Results

Courtesy of László Varga

• László G. Varga, Viktor Szpisják: Automatic background-

foreground segmentation of organs on MRI images of the head-neck area, Neumann Kollokvium 2017

Segmenting Head & Neck MR images

50

Publication

• Aim
• Handling the local changing of the organs
• Input
• Registered T2w, T1w, T1w fat-saturated MRI images
• The result of the global learning
• Random Forest classification
• Features
• Intensity, gradient magnitude, texture features and local region-based features
• S. Urbán, L. Ruskó, and A. Nagy, A Self-learning Tumor Segmentation

Method on DCE-MRI Images. Springer International Publishing, 2016,

• pp. 591–598
• Area of the atlas above 50%
• Detecting Isolation Forest outlier
• F. T. Liu, K. M. Ting, and Z.-H. Zhou, “Isolation forest,” 2008 Eighth IEEE

International Conference on Data Mining, pp. 413–422, 2008.

Segmenting Head & Neck MR images

51

Learning Based Local Segmentation

• 3 different region with morphology
• Foreground
• Initial segmentation
• Don’t care
• Either foreground or background
• Background
• Behind the don’t care but within the atlas
• Iterative Random Forest learning phase
• Applying the model generated in learning phase
• Postprocessing
• Same as in „A Self-learning Tumor Segmentation Method”

Segmenting Head & Neck MR images

52

Learning Based Local Segmentation

Results

Segmenting Head & Neck MR images

53

Courtesy of Urbán Szabolcs

DSC:82%  8% DSC:80%  8% DSC:74%  7%

• Szabolcs Urbán and Attila Tanács: Atlas-based global and

local RF segmentation of head and neck organs on multimodal MRI images, ISPA 2017

Segmenting Head & Neck MR images

54

Publication

• Aim
• Segmenting tubular vessel structures

in the head and neck region

• Input
• MRI T2 FRFSE
• Method
• Preprocessing
• Reduce the artefacts
• Algorithm
• Watershed like region growing
• Postprocessing
• On Binary results

Segmenting Head & Neck MR images

55

Vessel segmentation

• N4 BIAS correction
• Reducing the inhomogeneity
• Histogram matching to the reference image
• Reducing the intensity differences between the images
• Creating joint atlas area
• Carotid artery and Jugular vein
• Dilation of the convex hull of the joint area
• „Noise reduction”
• Clustering the joint atlas area
• Classified voxels
• Preserving small details

Segmenting Head & Neck MR images

56

Preprocessing

• Initialize with the region belongs to the smallest cluster

number

• Taking into account the size of the area and bounding box
• Expand the region with the clusters
• Where the histogram of cluster is decreasing
• Avoiding to melt two regions
• If two region merged
• Edge information with very low gradient
• Disrupt the circle like objects

Segmenting Head & Neck MR images

57

Algorithm

• Assuming smooth borders
• Convex hull
• Do not merge the objects
• Melting
• Edge based splitting with two parameters
• Melting the regions
• Small objects with overlapping
• Removing object where no overlapping

Segmenting Head & Neck MR images

58

Postprocessing

Results

Segmenting Head & Neck MR images

59

Results

Segmenting Head & Neck MR images

60

Results

Segmenting Head & Neck MR images

61

Results

Segmenting Head & Neck MR images

62

Results

Segmenting Head & Neck MR images

63

• Recognize the spots
• Classification with shape features
• HU moments
• Standard shape characteristics
• Preliminary results
• Using 3D connectedness of the spots
• Application of a 3D model
• Surface smoothing
• Evaluation

Segmenting Head & Neck MR images

64

Things to do …

Acknowledgements

Image processing group

• TANÁCS Attila,
• VARGA László,
• URBÁN Szabolcs,
• BALÁZS Péter,
• HIRLING Dominik,

Physicians

• Prof. Dr. PALKÓ András,
• Dr. SZABÓ Endre,
• Dr. CSOMOR Angéla,
• Dr. KEREKES Fanni,
• Dr. DOBOS Judit,
• Dr. URBÁN Olga,
• Dr. MÁTÉKA Ilona,
• Dr. PÁSZTOR Gyula,
• Dr. CSIZMADIA Sándor

Segmenting Head & Neck MR images

65