. . . Summarizing the performances of a background subtraction - PowerPoint PPT Presentation

. . . Summarizing the performances of a background subtraction algorithm measured on several videos S´ ebastien Pi´ erard and Marc Van Droogenbroeck Department of Electrical Engineering and Computer Science (Montefiore Institute), University of Li` ege, Belgium . . Special Session on“Dynamic Background Reconstruction/Subtraction for Challenging Environments” 1 / 34

Motivation: scoring an algorithm for multiple videos Background subtraction Background subtraction Background subtraction Binay classifier Binay classifier Binay classifier P 1 , R 1 , TPR 1 , ER 1 , F 1 P 2 , R 2 , TPR 2 , ER 2 , F 2 P 3 , R 3 , TPR 3 , ER 3 , F 3 2 / 34

Scoring multiple videos with a unique series of indicators Background subtraction Background subtraction Background subtraction Binay classifier Binay classifier Binay classifier P, R, TPR, ER, F 3 / 34

Outline Performance indicators for one video 1 Summarizing the performance for several videos 2 Summarizing applied on CDNET 2014 3 Conclusion 4 4 / 34

A scenario for the evaluation of background subtraction algorithms Dataset Input Background subtraction Binay classifier Ground truth P, R, TPR, ER, F 5 / 34

Towards performance indicators applicable to a binary classifier Ground truth Prediction y ∈ { c + = foreground , c − = background } y ∈ { c + = foreground , c − = background } ˆ P, R, TPR, ER, F 6 / 34

The confusion matrix Ground truth Output Predicted class ˆ y Positive Negative Positive TP FN Actual class y Negative FP TN 7 / 34

The confusion matrix Ground truth Output Predicted class ˆ y Positive Negative Positive TP FN Actual class y Negative FP TN 8 / 34

The confusion matrix Ground truth Output Predicted class ˆ y Positive Negative Positive TP FN Actual class y Negative FP TN 9 / 34

The confusion matrix Ground truth Output Predicted class ˆ y Positive Negative Positive TP FN Actual class y Negative FP TN 10 / 34

The confusion matrix Ground truth Output Predicted class ˆ y Positive Negative Positive TP FN Actual class y Negative FP TN 11 / 34

The confusion matrix Ground truth Output Predicted class ˆ y Positive Negative Positive TP FN Actual class y Negative FP TN 12 / 34

Experimental performance indicators based on the confusion matrix c Predicted class ˆ y Positive Negative Positive TP FN Actual class y Negative FP TN π + = TP+FN Positive prior TP+FN+FP+TN TP Precision P = TP+FP = PPV Positive Predictive Value TP True Positive Rate TPR = TP+FN = R Recall 2TP F score F = 2TP+FN+FP 13 / 34

ROC vs PR evaluation spaces: there is a bijection! c There are two well-known evaluation spaces: ROC: Receiver Operating Characteristic, defined by (FPR , TPR) PR: Precision/Recall c ROC space PR space 1 1 True Positive Rate (TPR) precision (P) one video, π + = 0 . 3 bijective relationship between the ROC and PR spaces 0 0 False Positive Rate (FPR) recall (R=TPR) 0 1 0 1 14 / 34

Effect of the arithmetic mean ROC space PR space 1 1 second video, π + = 0 . 9 2 True Positive Rate (TPR) 2 arithmetic mean in ROC precision (P) arithmetic mean in PR first video, π + = 0 . 3 1 bijective relationship 1 between the ROC and PR spaces 0 0 False Positive Rate (FPR) recall (R=TPR) 0 1 0 1 There is no bijection between the means anymore! 15 / 34

The“normalized”confusion matrix Predicted class ˆ y Positive Negative Positive pTP pFN Actual class y Negative pFP pTN The proportion of TP, denoted by pTP, is defined as TP TP + FN + FP + TN This has no impact on the calculation of indicators, such as the F score: 2TP 2pTP F = 2TP + FN + FP = 2pTP + pFN + pFP but it leads to a helpful interpretation of experimental indicators in terms of probabilities. 16 / 34

The“normalized”confusion matrix Predicted class ˆ y Positive Negative Positive pTP pFN Actual class y Negative pFP pTN The proportion of TP, denoted by pTP, is defined as TP TP + FN + FP + TN This has no impact on the calculation of indicators, such as the F score: 2TP 2pTP F = 2TP + FN + FP = 2pTP + pFN + pFP but it leads to a helpful interpretation of experimental indicators in terms of probabilities. 17 / 34

Probabilistic meaning of experimental performance indicators Definition (Joint random experiment for one video) Draw one pixel at random (all pixels being equally likely) from the video and jointly observe the ground-truth class Y and the predicted class ˆ Y for this pixel. Prediction ˆ Joint random experiment Y ∆ = ( Y , ˆ Y ) Positive Negative tp = ( c + , c + ) fn = ( c + , c − ) Positive Ground truth Y fp = ( c − , c + ) tn = ( c − , c − ) Negative There are four possible outcomes: { tp , fn , fp , tn } . 18 / 34

Probabilistic indicators Prediction ˆ Joint random experiment Y ∆ = ( Y , ˆ Y ) Positive Negative tp = ( c + , c + ) fn = ( c + , c − ) Positive Ground truth Y fp = ( c − , c + ) tn = ( c − , c − ) Negative The family of probabilistic indicators can be defined based on this random experiment: P (∆ ∈ A| ∆ ∈ B ) with ∅ � A � B ⊆ { tp , fn , fp , tn } (1) It includes ◮ π + = P (∆ ∈ { tp , fn } | ∆ ∈ { tp , fn , fp , tn } ) = P (∆ ∈ { tp , fn } ) ◮ TPR = R = P (∆ = tp | ∆ ∈ { tp , fn } ) ◮ P = PPV = P (∆ = tp | ∆ ∈ { tp , fp } ), ER = P (∆ ∈ { fn , fp } ) ◮ ... but not the F score! 19 / 34

Probabilistic indicators Prediction ˆ Joint random experiment Y ∆ = ( Y , ˆ Y ) Positive Negative tp = ( c + , c + ) fn = ( c + , c − ) Positive Ground truth Y fp = ( c − , c + ) tn = ( c − , c − ) Negative The family of probabilistic indicators can be defined based on this random experiment: P (∆ ∈ A| ∆ ∈ B ) with ∅ � A � B ⊆ { tp , fn , fp , tn } (1) It includes ◮ π + = P (∆ ∈ { tp , fn } | ∆ ∈ { tp , fn , fp , tn } ) = P (∆ ∈ { tp , fn } ) ◮ TPR = R = P (∆ = tp | ∆ ∈ { tp , fn } ) ◮ P = PPV = P (∆ = tp | ∆ ∈ { tp , fp } ), ER = P (∆ ∈ { fn , fp } ) ◮ ... but not the F score! 20 / 34

Outline Performance indicators for one video 1 Summarizing the performance for several videos 2 Summarizing applied on CDNET 2014 3 Conclusion 4 21 / 34

A probabilistic model for summarization Definition (Parametric random experiment for several videos) First, draw one video V at random in the set V , following an arbitrarily chosen distribution P ( V ). Then, draw one pixel at random from V and observe the ground-truth class Y and the predicted class ˆ Y for this pixel. v 1 v 2 v 3 P ( v 1 ) P ( v 2 ) P ( v 3 ) 1. Select one video 2. Select a pixel Compare y and ˆ y Figure: A probabilistic model for summarization: ∆ = ( V , Y , ˆ Y ). 22 / 34

Summarization formulas Notations: ◮ I( v ) = the value of a performance indicator I for a video v ∈ V , ◮ I( V ) = the value of I for a set V of videos. We define a probabilistic indicator I A|B as P (∆ ∈ A| ∆ ∈ B ), and I B as P (∆ ∈ B ). We have I A|B ( V ) = P (∆ ∈ A| ∆ ∈ B ) � = P (∆ ∈ A , V = v | ∆ ∈ B ) v ∈ V � = P ( V = v | ∆ ∈ B ) P (∆ ∈ A| ∆ ∈ B , V = v ) v ∈ V � I A|B ( V ) = P ( V = v | ∆ ∈ B ) I A|B ( v ) (2) v ∈ V For the particular case of an unconditional probabilistic indicator I A = I A|{ tn , fp , fn , tp } , we have � I A ( V ) = P ( V = v ) I A ( v ) (3) v ∈ V 23 / 34

Summarization formulas Notations: ◮ I( v ) = the value of a performance indicator I for a video v ∈ V , ◮ I( V ) = the value of I for a set V of videos. We define a probabilistic indicator I A|B as P (∆ ∈ A| ∆ ∈ B ), and I B as P (∆ ∈ B ). We have I A|B ( V ) = P (∆ ∈ A| ∆ ∈ B ) � = P (∆ ∈ A , V = v | ∆ ∈ B ) v ∈ V � = P ( V = v | ∆ ∈ B ) P (∆ ∈ A| ∆ ∈ B , V = v ) v ∈ V � I A|B ( V ) = P ( V = v | ∆ ∈ B ) I A|B ( v ) (2) v ∈ V For the particular case of an unconditional probabilistic indicator I A = I A|{ tn , fp , fn , tp } , we have � I A ( V ) = P ( V = v ) I A ( v ) (3) v ∈ V 24 / 34

Summarization formulas and properties Formulas: � I A ( V ) = P ( V = v ) I A ( v ) v ∈ V � I A|B ( V ) = P ( V = v | ∆ ∈ B ) I A|B ( v ) v ∈ V 1 P ( V = v ) π + ( v ) TPR( v ) � Example: TPR( V ) = (4) π + ( V ) v ∈ V Properties: 1 Summarization preserves the consistency between indicators, including the bijection between the ROC and PR spaces! 2 As long as an indicator is defined for at least one video, it can be summarized! To prove it, we rewrite I A|B ( V ) as I A∩B ( V ) I A∩B ( V ) I A|B ( V ) = = (5) I B ( V ) v ∈ V P ( V = v ) I B ( v ) � 25 / 34

Summarization formulas and properties Formulas: � I A ( V ) = P ( V = v ) I A ( v ) v ∈ V � I A|B ( V ) = P ( V = v | ∆ ∈ B ) I A|B ( v ) v ∈ V 1 P ( V = v ) π + ( v ) TPR( v ) � Example: TPR( V ) = (4) π + ( V ) v ∈ V Properties: 1 Summarization preserves the consistency between indicators, including the bijection between the ROC and PR spaces! 2 As long as an indicator is defined for at least one video, it can be summarized! To prove it, we rewrite I A|B ( V ) as I A∩B ( V ) I A∩B ( V ) I A|B ( V ) = = (5) I B ( V ) v ∈ V P ( V = v ) I B ( v ) � 26 / 34