Weakly-Supervised Temporal Localization via Occurrence Count - PowerPoint PPT Presentation

M ODEL Poisson Binomial Counts Y follows a Poisson-binomial distribution Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Recursion Bin k of count distribution at time t Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Recursion Bin k of count distribution at time t Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Recursion 𝒖 = 𝟏 Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Recursion 𝒒 𝒋 = 𝟕% Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Recursion 𝒒 𝒋 = 𝟕% Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Recursion 𝒒 𝒋 = 𝟒𝟏% Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Recursion 𝒒 𝒋 = 𝟒𝟏% Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Recursion 𝒒 𝒋 = 𝟒𝟏% Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Recursion 𝒒 𝒋 = 𝟒𝟏% Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Recursion 𝒒 𝒋 = 𝟒𝟏% Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL No early triggering Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL No early triggering Mass moves to the right Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL No early triggering Mass moves to the right Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL No early triggering Mass moves to the right Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL No early triggering Mass moves to the right Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL No early triggering Mass moves to the right Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL No early triggering Mass moves to the right Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL No early triggering Mass moves to the right Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL No early triggering Mass moves to the right Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL No early triggering Mass moves to the right Consequence: Mass shifts are irreversible Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL No early triggering Mass moves to the right Consequence: Mass shifts are irreversible • prevents the model from triggering early • prevents the model from false alarms Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Mass Convergence Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Mass Convergence Counting Loss Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Mass Convergence Counting Loss Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Mass Convergence Counting Loss Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Mass Convergence Learns to count Counting Loss Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Mass Convergence Learns to count Counting Loss Converge towards 0,1 extremes Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Mass Convergence Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Mass Convergence A detection cannot be split into numerous small 𝐪 𝐣 (∙) contributions Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Mass Convergence As the model learns to count event occurrences: Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Mass Convergence As the model learns to count event occurrences: • p i (∙) converge towards 0,1 extremes Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Mass Convergence As the model learns to count event occurrences: • p i (∙) converge towards 0,1 extremes • A detection cannot be split into numerous small p i (∙) contributions Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Mass Convergence As the model learns to count event occurrences: • p i (∙) converge towards 0,1 extremes • A detection cannot be split into numerous small p i (∙) contributions A single 𝐪 𝐣 ∙ will contain almost all of them mass for an event. Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Properties 1. Almost binary predictions Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Properties 1. Almost binary predictions 2. No early triggering Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Properties 1. Almost binary predictions 2. No early triggering 3. No systematic late bias Not a theoretical property Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Properties 1. Almost binary predictions 2. No early triggering 3. No systematic late bias Not a theoretical property Achieved trough an implementation trick: Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Properties 1. Almost binary predictions 2. No early triggering 3. No systematic late bias Not a theoretical property Achieved trough an implementation trick: Feeding sequences of variable length Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Properties 1. Almost binary predictions 2. No early triggering 3. No systematic late bias Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

M ODEL Properties 1. Almost binary predictions 2. No early triggering 3. No systematic late bias If the model accurately learns to count occurrences and if the events are detectable, then a coherent localization will emerge naturally. Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

Experiments

D RUM D ETECTION Experiment Specifications Detection of three different drum types in drum audio extracts Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

D RUM D ETECTION Experiment Specifications Detection of three different drum types in drum audio extracts • Tight tolerance of 50ms for a prediction to be correct Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

D RUM D ETECTION Experiment Specifications Detection of three different drum types in drum audio extracts • Tight tolerance of 50ms for a prediction to be correct • Comparison with fully-supervised benchmark models Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

D RUM D ETECTION Our approach Signal Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

D RUM D ETECTION Our approach Signal Fourier Mel-spectrogram 1 st order derivative Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

D RUM D ETECTION Our approach Signal Fourier Mel-spectrogram 1 st order derivative CNNs Convolutional Representations Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

D RUM D ETECTION Our approach Signal Fourier Mel-spectrogram 1 st order derivative CNNs Convolutional Representations LSTM Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

D RUM D ETECTION Our approach Signal Fourier Mel-spectrogram 1 st order derivative CNNs Convolutional Representations LSTM FCs Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

D RUM D ETECTION Our approach Signal Fourier Mel-spectrogram 1 st order derivative CNNs Convolutional Representations LSTM FCs Localization Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

D RUM D ETECTION Our approach Signal Fourier Mel-spectrogram 1 st order derivative CNNs Convolutional Representations Trained with our loss LSTM ( using only occurrence counts) FCs Localization Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

D RUM D ETECTION Results Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

D RUM D ETECTION Results State-of-the-art Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall

D RUM D ETECTION Results Great Overall F1-Score State-of-the-art Weakly-Supervised Temporal Localization via Occurrence Count Learning | Julien Schroeter | Kirill Sidorov | David Marshall