Performance Measures and the DukeMTMC Benchmark for Multi-Target - PowerPoint PPT Presentation

Handover Errors Camera 2 Camera 1 A A Truth A A A A A A A A A A A A A A Tracker 1 b b b b b b b b b b b b b c b b time

Handover Errors Camera 2 Camera 1 A A Truth A A A A A A A A A A A A A A Tracker 1 b b b b b b b b b b b b b c b b time

Handover Errors Camera 2 Camera 1 A A Truth A A A A A A A A A A A A A A Tracker 1 b b b b b b b b b b b b b c b b Tracker 2 b b b b b b b b b b b b b b b time

Handover Errors Camera 2 Camera 1 A A Truth A A A A A A A A A A A A A A Tracker 1 b b b b b b b b b b b b b c b b Tracker 2 b b b b b b b b b b b b b b b time

Scoring Functions • Multiple Object Tracking Accuracy (MOTA) • Multiple Camera Tracking Accuracy (MCTA) • Trajectory Scores MT, PT, ML Ignore identity • Handover Errors

Trajectory Scores Truth A A A A A A A A A A time

Trajectory Scores Truth A A A A A A A A A A time Mostly Tracked: GT trajectories which are covered by tracker output for more than 80% in length

Trajectory Scores Truth A A A A A A A A A A Tracker 1 b e b f f e c c g g time Mostly Tracked: GT trajectories which are covered by tracker output for more than 80% in length

Trajectory Scores Truth A A A A A A A A A A Tracker 1 MT = 1 b e b f f e c c g g time Mostly Tracked: GT trajectories which are covered by tracker output for more than 80% in length

Trajectory Scores Truth A A A A A A A A A A Tracker MT = 1 FRG = 4 b g b b b b c c g g time Mostly Tracked: Percentage of GT trajectories which are covered by tracker output for more than 80% in length Fragments: The number of times that a GT trajectory is interrupted in tracking result

Trajectory Scores Truth A A A A A A A A A A Tracker MT = 1 FRG = 4 b e b f f e c c g g time

Trajectory Scores Truth A A A A A A A A A A Tracker MT = 1 FRG = 4 b e b f f e c c g g Tracker++ b g b b b b c b b g time

Trajectory Scores Truth A A A A A A A A A A Tracker MT = 1 FRG = 4 b e b f f e c c g g Tracker++ MT = 1 FRG = 4 b g b b b b c b b g time

Scoring Functions • Multiple Object Tracking Accuracy (MOTA) • Multiple Camera Tracking Accuracy (MCTA) Bizarre combination • Trajectory Scores MT, PT, ML • Handover Errors

MCTA Camera 2 Camera 1 A A Truth A A A A A A A A A A A A A A Tracker 1 b b b b b b b b b b c b b b b b Tracker 2 b b b b b c b b b b b b b b b b time

MCTA Camera 2 Camera 1 A A Truth A A A A A A A A A A A A A A Tracker 1 b b b b b b b b b b c b b b b b Tracker 2 b b b b b c b b b b b b b b b b time Fragmentations Handover | Within camera Tracker 1 Tracker 2

MCTA Camera 2 Camera 1 A A Truth A A A A A A A A A A A A A A Tracker 1 b b b b b b b b b b c b b b b b Tracker 2 b b b b b c b b b b b b b b b b time Fragmentations Handover | Within camera Tracker 1 Tracker 2

Scoring Functions • Multiple Object Tracking Accuracy (MOTA) Mapping not one-to-one • Multiple Camera Tracking Accuracy (MCTA) • Trajectory Scores MT, PT, ML • Handover Errors

MOTA True A A A A A A A A Computed b b b b c c c c time True A A A A A A A A Computed b b b b b b c b time

MOTA True A A A A A A A A Half explanation Computed b b b b c c c c time True A A A A Nearly full A A A A explanation Computed b b b b b b c b time

Outline • Problem Definition • Performance Measures • Existing Measures • Issues • Identity Measures • DukeMTMCT Benchmark • Summary

Identity Measures • Truth-to-Result Matching T 1 T rue C 1 C omputed C 3 C 2 T 2 • Scoring Function FIO PR-MOTA MOTA MCTA FIT IDS MTMC STDA-D Tracker Performance TP MT Handover OP ML ATA (X, FRG) FRG

Identity Measures • Proposed Truth-to-Result Matching Camera 2 Camera 1 A A True A A A A A A A A A A A A A A Computed b b c b b b b c b b b b b b time

Identity Measures • Proposed Truth-to-Result Matching Camera 2 Camera 1 A A True A A A A A A A A A A A A A A Computed b b c b b b b c b b b b b b time A b c

Identity Measures • Proposed Truth-to-Result Matching Camera 2 Camera 1 A A True A A A A A A A A A A A A A A Computed b b c b b b b c b b b b b b time 4 A b 14 16 c Cost on each edge is number of mis-assigned 12 frames between two trajectories 2 0 0

Identity Measures • Proposed Truth-to-Result Matching Camera 2 Camera 1 A A True A A A A A A A A A A A A A A Computed b b b b b b b b b b b b time 4 A b Cost on each edge is number of mis-assigned frames between two trajectories

Identity Measures • Proposed Truth-to-Result Matching Camera 2 Camera 1 A A True A A A A A A A A A A A A A A Computed b b c b b b b c b b b b b b time 4 A b 14 16 c Cost on each edge is number of mis-assigned 12 frames between two trajectories 2 0 0

Identity Measures • Proposed Truth-to-Result Matching Camera 2 Camera 1 A A True A A A A A A A A A A A A A A Computed c c time A 14 c Cost on each edge is number of mis-assigned frames between two trajectories

Identity Measures • Proposed Truth-to-Result Matching Camera 2 Camera 1 A A True A A A A A A A A A A A A A A Computed b b c b b b b c b b b b b b time 4 A b 14 16 c Cost on each edge is number of mis-assigned 12 frames between two trajectories 2 0 0

Identity Measures • Proposed Truth-to-Result Matching Camera 2 Camera 1 A A True A A A A A A A A A A A A A A Computed time A 16 Cost on each edge is number of mis-assigned frames between two trajectories

Identity Measures • Proposed Truth-to-Result Matching Camera 2 Camera 1 A A True A A A A A A A A A A A A A A Computed b b c b b b b c b b b b b b time 4 A b 14 16 c Cost on each edge is number of mis-assigned 12 frames between two trajectories 2 0 0

Identity Measures • Proposed Truth-to-Result Matching Camera 2 Camera 1 True Computed b b b b b b b b b b b b time b Cost on each edge is number of mis-assigned 12 frames between two trajectories

Identity Measures • Proposed Truth-to-Result Matching Camera 2 Camera 1 A A True A A A A A A A A A A A A A A Computed b b c b b b b c b b b b b b time 4 A b 14 16 c Cost on each edge is number of mis-assigned 12 frames between two trajectories 2 0 0

Identity Measures • Truth-to-Result Matching T 1 T rue C 1 C omputed C 3 C 2 T 2 • Scoring Function FIO PR-MOTA MOTA MCTA FIT IDS MTMC STDA-D Tracker Performance TP MT Handover OP ML ATA (X, FRG) FRG

Identity Measures • Truth-to-Result Matching T 1 T rue C 1 C omputed C 3 C 2 T 2 • Scoring Function FIO PR-MOTA MOTA MCTA FIT IDS MTMC STDA-D Tracker Performance TP MT Handover OP ML ATA (X, FRG) FRG

Identity Measures • Truth-to-Result Matching T 1 T rue C 1 C omputed C 3 C 2 T 2 • Scoring Function FIO PR-MOTA MOTA MCTA FIT IDS MTMC STDA-D Tracker Performance TP MT Handover OP ML ATA (X, FRG) FRG

Identity Measures • Proposed Scores • ID Precision • ID Recall • F 1 -score False True False Negatives Positives Positives True Computed Detections Detections

Identity Measures • Proposed Scores • IDP: Fraction of computed detections that are correctly identified. • ID Precision • ID Recall • F 1 -score False True False Negatives Positives Positives True Computed Detections Detections

Identity Measures • Proposed Scores • IDP: Fraction of computed detections that are correctly identified. • ID Precision • ID Recall • IDR: Fraction of ground-truth detections that are correctly identified. • F 1 -score False True False Negatives Positives Positives True Computed Detections Detections

Identity Measures • Proposed Scores • IDP: Fraction of computed detections that are correctly identified. • ID Precision • ID Recall • IDR: Fraction of ground-truth detections that are correctly identified. • IDF1: Ratio of correctly identified detections over the average • F 1 -score number of ground-truth and computed detections False True False Negatives Positives Positives True Computed Detections Detections

Identity Measures • Properties • True and computed identities are mapped 1-1 • Notion of identity switch is not present in ID measures • The truth-to-result matching is the most favorable to the algorithm • Applicable to single- and multi-camera tracking

Identity Measures • Properties • True and computed identities are mapped 1-1 • Notion of identity switch is not present in ID measures • The truth-to-result matching is the most favorable to the algorithm • Applicable to single- and multi-camera tracking A b c

Identity Measures • Properties • True and computed identities are mapped 1-1 • Notion of identity switch is not present in ID measures • The truth-to-result matching is the most favorable to the algorithm • Applicable to single- and multi-camera tracking A b A b c c

Identity Measures • Properties • True and computed identities are mapped 1-1 • Notion of identity switch is not present in ID measures • The truth-to-result matching is the most favorable to the algorithm • Applicable to single- and multi-camera tracking 4 4 A b A b 14 2 16 c c 12 0 2 0 0 Min Cost Matching

Identity Measures • Properties • True and computed identities are mapped 1-1 • Notion of identity switch is not present in ID measures • The truth-to-result matching is the most favorable to the algorithm • Applicable to single- and multi-camera tracking T 1 C 1 C 3 C 2 T 2

Identity Measures • Properties • True and computed identities are mapped 1-1 • Notion of identity switch is not present in ID measures • The truth-to-result matching is the most favorable to the algorithm • Applicable to single- and multi-camera tracking T 1 C 1 C 3 C 2 T 2

Scope and Limitations • ID measures are useful in: • Surveillance • Sport • MTT for autonomous driving • Not necessary when: • Tracking indistinguishable targets (ants, sheep) • Not applicable to: • Tracking targets that merge/split (cells)