Learning to map between ferns with differentiable binary embedding - - PowerPoint PPT Presentation

SLIDE 1

DIFFERENTIABLE BINARY EMBEDDING USING FERNS

Learning to map between ferns with differentiable binary embedding networks

Maximilian Blendowski & Mattias P. Heinrich

Institute of Medical Informatics University of Lübeck Short paper @ MIDL 2020

SLIDE 2

DIFFERENTIABLE BINARY EMBEDDING USING FERNS

𝐸𝑗𝑛 1 2 3 4 5 𝑔𝑓𝑏𝑢𝑣𝑠𝑓 𝒈 +4

• 2

+10

• 6

+8 +1 depth 𝑛 = 2 Dimension 𝑒1

𝐺𝑓𝑠𝑜𝑙, … , 𝑒𝑛 𝐺𝑓𝑠𝑜𝑙

Threshold 𝑢1

𝐺𝑓𝑠𝑜𝑙, … , 𝑢𝑛 𝐺𝑓𝑠𝑜𝑙

binary code 𝐺𝑓𝑠𝑜𝑙 𝒈 𝐺𝑓𝑠𝑜1 1,2 −3,1

−2 < −3 ? , +10 < 1?

𝟏𝟏 𝐺𝑓𝑠𝑜2 5,0 2,0

+1 < +2 ? , +4 < 0?

1𝟏 𝐺𝑓𝑠𝑜3 4,3 3, −1

+8 < +3 ? , −6 < −3?

𝟏𝟐

𝑮𝒇𝒔𝒐𝟐 Class A 1 1 1 1 0.5 0.25 0.0 𝑮𝒇𝒔𝒐𝟐 Class B 1 1 1 1 0.5 0.25 0.0 𝑮𝒇𝒔𝒐𝟑 Class A 1 1 1 1 0.5 0.25 0.0 𝑮𝒇𝒔𝒐𝟑 Class B 1 1 1 1 0.5 0.25 0.0 𝑮𝒇𝒔𝒐𝟒 Class A 1 1 1 1 0.5 0.25 0.0 𝑮𝒇𝒔𝒐𝟒 Class B 1 1 1 1 0.5 0.25 0.0

𝑄 𝒈|𝐵 = 0.5 ∗ 0.25 ∗ 0.5 = 0.0625 𝑄 𝒈|𝐶 = 0.25 ∗ 0.4 ∗ 0.125 = 0.0125 𝑄 𝒈|𝐵 > 𝑄 𝒈|𝐶 → 𝑑𝑚𝑏𝑡𝑡𝑗𝑔𝑧 𝒈 𝑏𝑡 𝐵

Random Fern Basics

Ozuysal, Mustafa, et al. "Fast keypoint recognition using random ferns." IEEE transactions on pattern analysis and machine intelligence 32.3 (2009): 448-461.

SLIDE 3

DIFFERENTIABLE BINARY EMBEDDING USING FERNS

3

Standard convolution

𝑏𝑗(: ) 𝑏𝑘(: ) 𝑐𝑗(: ) 𝑐

𝑘(: )

𝑏𝑗 𝑐𝑗 𝑏𝑘 𝑐

𝑘

𝑙 𝑙 ℎ𝑗𝑜 𝑥𝑗𝑜 𝑙2 ∙ 𝑑𝑗𝑜 𝑜𝑣

𝑉𝐺𝑁

ℎ𝑝𝑣𝑢 𝑥𝑝𝑣𝑢

4) fold 𝑔𝑓𝑏𝑢𝑗

𝑝𝑣𝑢

𝑔𝑓𝑏𝑢𝑘

𝑝𝑣𝑢

𝑑𝑝𝑣𝑢 𝑙2 ∙ 𝑑𝑗𝑜

1) unfold

𝑑𝑝𝑣𝑢

𝐷𝑝𝑜𝑤𝑝𝑚𝑣𝑢𝑗𝑝𝑜 𝐺𝑗𝑚𝑢𝑓𝑠 𝑋𝑓𝑗𝑕ℎ𝑢𝑡

Matrix Multiplication

SLIDE 4

DIFFERENTIABLE BINARY EMBEDDING USING FERNS

4

Drop-in replacement

𝑏𝑗(: ) 𝑏𝑘(: ) 𝑐𝑗(: ) 𝑐

𝑘(: )

𝑏𝑗 𝑐𝑗 𝑏𝑘 𝑐

𝑘

𝑙 𝑙 ℎ𝑗𝑜 𝑥𝑗𝑜 𝑙2 ∙ 𝑑𝑗𝑜 𝑜𝑣

1) unfold

𝑉𝐺𝑁

𝑒1

𝑞

𝑒𝑛

𝑞

𝑒1

𝑟

𝑒𝑛

𝑞

2) Generate LUT codes 𝑑𝑗

𝑞 = tanh 𝑉𝐺𝑁 𝑗

𝑒1

𝑞, … , 𝑒𝑛 𝑞

− 𝑢1

𝑞, … , 𝑢𝑛 𝑞

> 0 ? → 01 … 0 + 𝑡𝑞 = 𝑗𝑒𝑦𝑗

𝑞

𝑑𝑗

𝑟 = tanh 𝑉𝐺𝑁 𝑗

𝑒1

𝑟, … , 𝑒𝑛 𝑟

− 𝑢1

𝑟, … , 𝑢𝑛 𝑟

> 0 ? → 00 … 1 + 𝑡𝑟 = 𝑗𝑒𝑦𝑗

𝑟

𝑑

𝑘 𝑞 = tanh 𝑉𝐺𝑁 𝑘

𝑒1

𝑞, … , 𝑒𝑛 𝑞

− 𝑢1

𝑞, … , 𝑢𝑛 𝑞

> 0 ? → 01 … 0 + 𝑡𝑞 = 𝑗𝑒𝑦𝑘

𝑞

𝑑

𝑘 𝑟 = tanh 𝑉𝐺𝑁 𝑘

𝑒1

𝑟, … , 𝑒𝑛 𝑟

− 𝑢1

𝑟, … , 𝑢𝑛 𝑟

> 0 ? → 01 … 1 + 𝑡𝑟 = 𝑗𝑒𝑦𝑘

𝑟

3) weigthed LUT entry combination 𝑗𝑒𝑦𝑗

𝑞

𝑗𝑒𝑦𝑗

𝑟

𝑗𝑒𝑦𝑘

𝑟

𝑗𝑒𝑦𝑘

𝑞

= 𝑀𝑉𝑈 𝑗𝑒𝑦𝑗

𝑞

∙ 𝑥𝑗

𝑞 + ⋯ + 𝑀𝑉𝑈 𝑗𝑒𝑦𝑗 𝑟

∙ 𝑥𝑗

𝑟

= 𝑀𝑉𝑈 𝑗𝑒𝑦𝑘

𝑞

∙ 𝑥

𝑘 𝑞 + ⋯ + 𝑀𝑉𝑈 𝑗𝑒𝑦𝑘 𝑟

∙ 𝑥

𝑘 𝑟

ℎ𝑝𝑣𝑢 𝑥𝑝𝑣𝑢

4) fold

𝑑𝑝𝑣𝑢

𝑀𝑉𝑈

#𝑔𝑓𝑠𝑜𝑡 ∙ 2𝑒𝑓𝑞𝑢ℎ

trainable weights

𝑔𝑓𝑏𝑢𝑗

𝑝𝑣𝑢

𝑔𝑓𝑏𝑢𝑘

𝑝𝑣𝑢

𝑑𝑝𝑣𝑢 Differentiable ferns

Binary codes on 𝑗 Binary codes on 𝑘

𝐺𝑓𝑠𝑜 𝑞 𝐺𝑓𝑠𝑜 𝑟

SLIDE 5

DIFFERENTIABLE BINARY EMBEDDING USING FERNS

5

Evaluation

[1] Hubara, Itay, et al. "Binarized neural networks." Advances in neural information processing systems. 2016. [2] Rastegari, Mohammad, et al. "Xnor-net: Imagenet classification using binary convolutional neural networks." European conference on

computer vision. Springer, Cham, 2016.

[3]Veta, Mitko, et al. "Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challenge." Medical image analysis 54

(2019): 111-121.

[2] [1]

Input Patch

[3]

cin: 3 cout: 64

Kernelsize: 5 Stride: 2 BatchNorm

cin: 64 cout: 64

Kernelsize: 3 Stride: 2 BatchNorm

cin: 64 cout: 64

Kernelsize: 3 Stride: 2 BatchNorm

Global Average Pooling

cin: 64 cout: 2

Kernelsize: 1 Stride: 1

• Tumor:

Yes/no?