Identifying Malicious Players in GWAP-based Disaster Monitoring - PowerPoint PPT Presentation

. Yaxi Chen . . . . . . . . Identifying Malicious Players in GWAP-based Disaster Monitoring Crowdsourcing System Changkun Ou, Yifei Zhan Institute of Computer Science . The Key Laboratory for Computer Systems of University of Munich State Ethnic Afgairs Commission changkun.ou@lmu.de Southwest Minzu University yifei.zhan@campus.lmu.de yaxichen@swun.cn ICAIBD’ 19, Chengdu, China May 26, 2019 Ou et al. 2019 (LMU and SMU) Malicious Detection in GAWP Systems May 26, 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 / 29

. 3 . . . . . . . Outline 1 Background & Motivation 2 Preliminaries Main Results . Player Rating Model (PRM) Disaster Evaluation Model (DEM) Model Initialization 4 Evaluation & Discussion Simulation Limitations 5 Conclusions Ou et al. 2019 (LMU and SMU) Malicious Detection in GAWP Systems May 26, 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 / 29

. 3 . . . . . . . Outline 1 Background & Motivation 2 Preliminaries Main Results . Player Rating Model (PRM) Disaster Evaluation Model (DEM) Model Initialization 4 Evaluation & Discussion Simulation Limitations 5 Conclusions Ou et al. 2019 (LMU and SMU) Malicious Detection in GAWP Systems May 26, 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 / 29

. “Systems that combine humans and computers to solve large-scale . . . . . . . . . Background: Human Computation in 1 Minute What are Human Computation systems? problems that neither can solve alone” (Luis von Ahn, retrived 30 . Apr. 2019) Software systems with humans in the loop, human as explicit (or active) or implicit (or passive) contributors Human Computation systems can be seen as Crowdsourcing markets (Wisdom of crowds). Useful inputs (wisdom) can be gained from a group of persons provided: Diversity of opinion; Idependence; Decentralization; Aggregation. (James Surowiecki, 2005) Game-With-A-Purpose (GWAP) tries to hide actual intent away from players and aggregates human inputs for solving diffjcult problems. Ou et al. 2019 (LMU and SMU) Malicious Detection in GAWP Systems May 26, 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 / 29

. “Systems that combine humans and computers to solve large-scale . . . . . . . . . Background: Human Computation in 1 Minute What are Human Computation systems? problems that neither can solve alone” (Luis von Ahn, retrived 30 . Apr. 2019) Software systems with humans in the loop, human as explicit (or active) or implicit (or passive) contributors Human Computation systems can be seen as Crowdsourcing markets (Wisdom of crowds). Useful inputs (wisdom) can be gained from a group of persons provided: Diversity of opinion; Idependence; Decentralization; Aggregation. (James Surowiecki, 2005) Game-With-A-Purpose (GWAP) tries to hide actual intent away from players and aggregates human inputs for solving diffjcult problems. Ou et al. 2019 (LMU and SMU) Malicious Detection in GAWP Systems May 26, 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 / 29

. “Systems that combine humans and computers to solve large-scale . . . . . . . . . Background: Human Computation in 1 Minute What are Human Computation systems? problems that neither can solve alone” (Luis von Ahn, retrived 30 . Apr. 2019) Software systems with humans in the loop, human as explicit (or active) or implicit (or passive) contributors Human Computation systems can be seen as Crowdsourcing markets (Wisdom of crowds). Useful inputs (wisdom) can be gained from a group of persons provided: Diversity of opinion; Idependence; Decentralization; Aggregation. (James Surowiecki, 2005) Game-With-A-Purpose (GWAP) tries to hide actual intent away from players and aggregates human inputs for solving diffjcult problems. Ou et al. 2019 (LMU and SMU) Malicious Detection in GAWP Systems May 26, 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 / 29

. . . . . . . . . . . . Motivation . Non-profjt organizations (e.g. UNICEF) has lack of resources in monitoring disaster regions, an automated system is essential. Sucessful disaster monitoring requires reliable predictions : system and algorithm design low costs maintains : GWAPs-based crowdsourcing Malicious player detection is critical in disaster monitoring and guarentees the health of a GWAP-based human computation system. Ou et al. 2019 (LMU and SMU) Malicious Detection in GAWP Systems May 26, 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 / 29

. . . . . . . . . . . . Motivation . Non-profjt organizations (e.g. UNICEF) has lack of resources in monitoring disaster regions, an automated system is essential. Sucessful disaster monitoring requires reliable predictions : system and algorithm design low costs maintains : GWAPs-based crowdsourcing Malicious player detection is critical in disaster monitoring and guarentees the health of a GWAP-based human computation system. Ou et al. 2019 (LMU and SMU) Malicious Detection in GAWP Systems May 26, 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 / 29

. 3 . . . . . . . Outline 1 Background & Motivation 2 Preliminaries Main Results . Player Rating Model (PRM) Disaster Evaluation Model (DEM) Model Initialization 4 Evaluation & Discussion Simulation Limitations 5 Conclusions Ou et al. 2019 (LMU and SMU) Malicious Detection in GAWP Systems May 26, 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 / 29

. . . . . . . . . . . . . . System Architecture The system consist of three components: task generating service rating service ranking service . Ou et al. 2019 (LMU and SMU) Malicious Detection in GAWP Systems May 26, 2019 . 7 / 29 . . . . . . . . . . . . . . . . . . . . . . . . Front Web Task Data Clean Rating Training Ranking PlayerDB ResultDB Game Service Generating Service Service Service Service Service Start Crowdsourcing Players Fetch Task Fetch Play Result Report Result Real-time report Generate Disaster Region Task Disaster Level AnonymousID, ROI, Tags Newly tagged satellite Image Assign Store Stakeholders Gaming (NPOs, government, hospitals, etc.) Data Report Disaster Report Reliable Result Store Area Pictures Notify Evaluation Training Result Update Model

. . . . . . . . . . . . . . . System Interface (a) (b) (c) areas; c) Disaster level report in stakeholder view. Ou et al. 2019 (LMU and SMU) Malicious Detection in GAWP Systems May 26, 2019 . . . . . . . . . . . . . . . . . . . . . . . . . 8 / 29 Figure: System interface. a) Player game panel overview; b) Multi-tags selection for selected

. . . . . . . . . . . . . . . Preliminaries Defjnition (Region of Interests, ROI) Figure: Reliable players (red and blue) draw rectangles to indicate area with disaster, however malicious player does not cooperate (black) selects other ROIs. Ou et al. 2019 (LMU and SMU) Malicious Detection in GAWP Systems May 26, 2019 . . . . . . . . . . . . . . 9 / 29 . . . . . . . . . . . An ROI represents a subset of R 2 . The i -th ROI from player p in image k is denoted by ROI p , i , k . Image k 1 Player 1 Player 2 Player 3 ROI 1 , 1 ,k 1 Reliable Malicious Players Player Image k 2 ROI 1 , 2 ,k 1 ROI 2 , 1 ,k 2 Monitored Geographical Region

. . . . . . . . . . . . . . . . Preliminaries (cond.) Defjnition (Tag Vector, TV) number of tags. Ou et al. 2019 (LMU and SMU) Malicious Detection in GAWP Systems May 26, 2019 . . . . . . . . . . . . . . . . . . . . . . . . 10 / 29 Assuming n difgerent tags g 1 , g 2 , ..., g n for a certain image k , the tag vector is defjned by T p , i , k = ( | g 1 | , | g 2 | , ..., | g n | ) ⊤ of ROI p , i , k where g l is the l -th tag where l = 1 , 2 , ..., n , | g l | is the count of g l in a player task object, and n equals to the

. 3 . . . . . . . Outline 1 Background & Motivation 2 Preliminaries Main Results . Player Rating Model (PRM) Disaster Evaluation Model (DEM) Model Initialization 4 Evaluation & Discussion Simulation Limitations 5 Conclusions Ou et al. 2019 (LMU and SMU) Malicious Detection in GAWP Systems May 26, 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 / 29

. . . . . . . . . . . . . . . Player Rating Graph (PRG) Defjnition (System Weight Vector) (1) Lemma (Properties) Ou et al. 2019 (LMU and SMU) Malicious Detection in GAWP Systems May 26, 2019 . . . . . . . . . . . . . . 12 / 29 . . . . . . . . . . . For n difgerent tags g 1 , g 2 , ..., g n . Let | g i | is the count of g i in the system. A system weight vector v = ( p ( g 1 ) , p ( g 2 ) , ..., p ( g n )) ⊤ , where | g i | p ( g i ) = j =1 | g j | , i = 1 , ..., n . ∑ n p ( g i ) holds the properties: 0 ≤ p ( g i ) ≤ 1 ∑ n i =1 p ( g i ) = 1 ∑ s i =1 p ( g r i ) ≤ 1