sample synthesis method for few-shot object recognition Eli - PowerPoint PPT Presentation

Δ -encoder: An effective sample synthesis method for few-shot object recognition Eli Schwartz*, Leonid Karlinsky*, Joseph Shtok, Sivan Harary, Mattias Marder, Abhishek Kumar, Rogerio Feris, Raja Giryes, Alex M. Bronstein IBM Research AI | Computer Vision and Augmented Reality

Who’s that dog? IBM Research AI | Computer Vision and Augmented Reality

Key idea – training • The model is a variant of Encoder an auto-encoder operating in feature space 𝑎 Decoder IBM Research AI | Computer Vision and Augmented Reality

Key idea – training target reference • The model is a variant of Encoder an auto-encoder operating in feature space delta • The network learns to 𝑎 encode the delta between the reference and the target image Decoder IBM Research AI | Computer Vision and Augmented Reality

Key idea – training target reference • The model is a variant of Encoder an auto-encoder operating in feature space delta • The network learns to 𝑎 encode the delta between the reference and the target image • This delta is used to recover Decoder the target image as a (non-linear) combination of the reference and the delta recovered target IBM Research AI | Computer Vision and Augmented Reality

Key idea – synthesizing Encoder 𝑎 Decoder IBM Research AI | Computer Vision and Augmented Reality

Key idea – synthesizing sampled sampled target reference • At test time we sample Encoder encoded deltas from random sampled delta training image pairs 𝑎 Decoder IBM Research AI | Computer Vision and Augmented Reality

Key idea – synthesizing sampled sampled target reference • At test time we sample Encoder encoded deltas from random sampled delta training image pairs 𝑎 • The sampled deltas are used to create samples for new classes by combining them with the new class Decoder reference examples • These samples are used to train new class a classifier for the new category reference synthesized new class example IBM Research AI | Computer Vision and Augmented Reality

Few-shot classification experiments one-shot classification benchmarks 75 70 65 60 55 50 minImageNet CIFAR-100 Caltech-256 CUB ours previous state-of-the-art mini niIm ImageN geNet et: 58.5 (previous SOA)  59.9 (ours) CIFAR AR-10 100: 63.4 (previous SOA)  66.7 (ours) Caltec ech-256 56: 63.8 (previous SOA)  73.2 (ours) CUB: 69.6 (previous SOA)  69.8 (ours) IBM Research AI | Computer Vision and Augmented Reality

Few-shot classification experiments one-shot classification benchmarks 75 70 65 60 55 50 minImageNet CIFAR-100 Caltech-256 CUB ours previous state-of-the-art mini niIm ImageN geNet et: 58.5 (previous SOA)  59.9 (ours) CIFAR AR-10 100: 63.4 (previous SOA)  66.7 (ours) Caltec ech-256 56: 63.8 (previous SOA)  73.2 (ours) CUB: 69.6 (previous SOA)  69.8 (ours) IBM Research AI | Computer Vision and Augmented Reality

Real vs synthetic examples ablation study IBM Research AI | Computer Vision and Augmented Reality

IBM Research AI | Computer Vision and Augmented Reality