Collaborative Image Triage with Humans and Computer Vision and - PowerPoint PPT Presentation

Collaborative Image Triage with Humans Collaborative Image Triage with Humans and Computer Vision and Computer Vision A. Bohannon Introduction Approach Addison Bohannon Image Assignment Joint Classification Applied Math, Statistics, & Scientific Computing System Design Results Set-up Advisors: Analytical Results Simulation 1 Vernon Lawhern Brian Sadler Simulation 2 Army Research Laboratory Army Research Laboratory Conclusions References May 3, 2016

Outline Collaborative Image Triage with Humans Introduction 1 and Computer Vision 2 Approach A. Bohannon Image Assignment Introduction Joint Classification Approach System Design Image Assignment Joint Classification System Design 3 Results Results Set-up Set-up Analytical Results Simulation 1 Analytical Results Simulation 2 Simulation 1 Conclusions References Simulation 2 4 Conclusions

Motivation Collaborative Image Triage with Humans We want to triage a large database of unlabeled images: and Computer Vision � Our purpose is motivated by DOD imagery intelligence A. Bohannon requirements, but other people are interested in this and similar problems: Introduction Approach � Google Images, Facebook, Galaxy Zoo, fold.it Image Assignment � This could be fully automated by computer vision Joint Classification System Design algorithms, but they require: Results � Training data (lots) and time (lots); or Set-up Analytical Results � Knowledge of the generating process of the data Simulation 1 Simulation 2 � This could be done by humans, but... Conclusions � Humans take a lot of time to classify images References � Task may require expertise or security clearance � Humans require salary, benefits, pension, etc.

Related Work How to triage a large image database Collaborative Image Triage � Human augmentation with Humans and Computer � Rapid Serial Visual Presentation (RSVP) for image Vision labeling [Bigdely-Shamlo et al., 2008] A. Bohannon � Human-machine systems Introduction � Serialize RSVP analyst and computer vision (CV) Approach algorithm [Sajda et al., 2010] Image Assignment Joint Classification � Automate image labeling with CV which can query a System Design human analyst for binary decisions [Joshi et al., 2012] Results Set-up � Crowd-sourcing Analytical Results Simulation 1 � Intelligent control of a system which dynamically scales Simulation 2 human participants [Kamar et al., 2012] Conclusions � Homogeneous human agents whose voting reliability is References learned [Karger et al., 2014] � Heterogeneous human agents intelligently assigned heterogeneous tasks [Ho et al., 2013]

Research Objective Collaborative Image Triage with Humans and Computer Vision A. Bohannon Introduction Approach Image Assignment Joint Classification � Goal: To design and implement in software an image System Design triage system which leverages an ensemble of Results Set-up heterogeneous agents to achieve the accuracy of a Analytical Results Simulation 1 naive parallel implementation in significantly less wall Simulation 2 time. Conclusions References � Problem Statement: � How to optimally distribute images among agents? � How to combine responses from multiple agents? � How to design a software system which can support heterogeneous image labeling interfaces in parallel?

Schedule Collaborative � Develop Joint Classification Module (Summer 2015) Image Triage with Humans and Computer � Implement Spectral Meta-Learner algorithm Vision � Develop Assignment Module (15 OCT - 4 DEC) A. Bohannon � Implement branch and bound algorithm (6 NOV) Introduction � Validate branch and bound algorithm (25 NOV) Approach � Mid-year review (14 DEC) Image Assignment Joint Classification � Build Image Labeling System (25 JAN - 26 FEB) System Design � Build base classes Results Set-up � Develop message-passing interface Analytical Results Simulation 1 � Integrate all components into a system (26 FEB) Simulation 2 � Test Image Labeling System (26 FEB - 15 APR) Conclusions References � Testing (1 APR) � Conclusion (15 APR - 13 MAY) � Final presentation (3 MAY) � Final report (13 May)

Generalized Assignment Problem Collaborative On iteration k , we seek the optimal assignment of n images Image Triage among m agents–with a fixed budget, b k j , and reliability, with Humans and Computer r k j –where each assignment has a unique value, v k Vision ji , and A. Bohannon cost, c ji [Kundakcioglu and Alizamir, 2008]: Introduction � � v k Z = max ji x ji s.t. (1) Approach x Image Assignment i ∈ I j ∈ J Joint Classification System Design Results � c ji x ji ≤ b k j , j ∈ J 1 Set-up Analytical Results i ∈ I Simulation 1 � 0-1 integer linear Simulation 2 � x ji = 1 , i ∈ I problem 2 Conclusions j ∈ J References � NP-hard 3 x ji ∈ { 0 , 1 } � Known solution 4 c ji , b k j ∈ Z + techniques 5 v k ji = r k j − s k i + max i ∈ I s k i

Branch and Bound Algorithm Collaborative Algorithm 1: Branch & Bound Image Triage with Humans and Computer Data : Z 0 Vision Result : x A. Bohannon Z = Z 0 , queue = p 0 ; while queue � = ∅ do Introduction Select p i ∈ queue Approach for j ∈ J do Image Assignment Joint Classification Z i j = bound ( p i j ) ; System Design if Z i j > Z then Results if x j is feasible then Set-up Analytical Results Figure: Visualization of branch x = x i j , Z = Z i Simulation 1 j and bound (B&B) algorithm. Simulation 2 else Conclusions add p i Nodes along the m -nary search j to queue References tree represent sub-problems end ( p i j ∼ x ji = 1). end end end

Bounding Function Collaborative We introduce the dual problem [Fisher, 2004], Image Triage with Humans � � � � and Computer d ( λ ) = max v ji x ji − λ i ( 1 − x ji ) , Vision x i ∈ I j ∈ J i ∈ I j ∈ J A. Bohannon to define our bounding function, Introduction Approach min λ d ( λ ) ≥ Z ≥ Z feasible . Image Assignment Joint Classification System Design Results Then, we solve the saddle-point problem directly via Set-up sub-gradient descent [Boyd and Vandenberghe, 2004]: Analytical Results Simulation 1 Simulation 2 x k + 1 = arg max � � � ( v ji − λ k i ) x ji s.t. c ji x ji ≤ b j Conclusions x References i ∈ I j ∈ J i ∈ I   λ k + 1 = λ k � i + α k  1 − x ji i  j ∈ J

Validation Generalized Assignment Problem Solvers Collaborative Feasibility Image Triage with Humans and Computer Solver Probability Vision Sub-gradient 1.0 A. Bohannon Multiplier 1.0 Introduction Greedy 1.0 Approach MATLAB 0.07 Image Assignment Joint Classification System Design Results Set-up Analytical Results Simulation 1 Simulation 2 Conclusions References Time Complexity

Maximum Likelihood Estimation Spectral Meta-Learner Collaborative Consider the set of decisions from m agents for a single Image Triage image i , A i : {− 1 , 1 } m → R . We seek the decision rule with Humans and Computer which maximizes P ( d ( A i ) = y i ) : Vision A. Bohannon � d ( a i ) = arg max j | Y ( a i log P A i j | y i ) , Introduction y i ∈{− 1 , 1 } j ∈ J Approach Image Assignment Joint Classification where Y : {− 1 , 1 } → R is the true label of an image [Dawid System Design and Skene, 1979]. Let π j = 1 2 ( ψ j + η j ) , where Results Set-up ψ j = P ( a j = 1 | y i = 1 ) and η j = P ( a j = − 1 | y i = − 1 ) , then the Analytical Results Simulation 1 decision rule is equivalent to Simulation 2 Conclusions m References d ( a i ) = sign � a i � � log α j + log β j , j j = 1 ( 1 − ψ j )( 1 − η j ) and β j = ψ j ( 1 − ψ j ) ψ j η j where α j = η j ( 1 − η j ) [Parisi et al., 2014].

Joint Classification Collaborative Image Triage with Humans and Computer Vision A. Bohannon This provides three results: Introduction 1 Class label of each image, sign ( d ( a i )) Approach Image Assignment Joint Classification 2 Confidence of the MLE estimate of each image, System Design s i = | d ( a i ) | Results Set-up 3 Reliability of each agent, r j = π j = 1 2 ( ψ j + η j ) Analytical Results Simulation 1 Simulation 2 Conclusions References

Software Map Collaborative Image Triage with Humans and Computer Vision A. Bohannon Introduction Approach Image Assignment Joint Classification System Design Results Set-up Analytical Results Simulation 1 Simulation 2 Conclusions References Figure: Visualization of the software design of the image triage system. Architecture prioritizes software flexibility and independent operation for a network of distributed agents.

Process Flow Collaborative Image Triage with Humans and Computer Vision A. Bohannon Introduction Approach Image Assignment Joint Classification System Design Results Set-up Analytical Results Simulation 1 Simulation 2 Conclusions References Figure: Visualization of process flow on central server. Asynchronous read operations facilitate parallel classification among distributed agents.