Organizational Structure, Exploration, and Exploitation on the ELICIT - PowerPoint PPT Presentation

Organizational Structure, Exploration, and Exploitation on the ELICIT Experimental Platform Allan Friedman Ethan Bernstein David Lazer Harvard ELICIT

Outline • Exploration & Exploitation • ELICIT overview • Extensions to ELICIT – Capturing Exploration & Exploitation – Experimental Design Tweaks • Experimental Summary • Preliminary Results Harvard ELICIT

Exploration & Exploitation • Defined by James March (1991) – Exploration: introduce new information – Exploitation: use existing information • Ambidexterity – balancing the two Harvard ELICIT

Ambidexterity & Agility Exploration Exploration Exploitation Exploitation Adaptation Adaptation Robustness Robustness Resilience Resilience Innovation Innovation Agility Agility Harvard ELICIT

Exploration & Exploitation • Defined by James March (1991) – Exploration: introduce new information – Exploitation: use existing information • Ambidexterity – balancing the two • Structure and Exploration vs. Exploitation (Lazer & Friedman, 2007) – Finding: Structural barriers can promote exploration Harvard ELICIT

ELICIT Overview • Lab ‐ based “whodunnit” game • Computer mediated system • Subjects receive “factoids”, have to work with each other to identify a terrorism plot – Who – What – Where – When Harvard ELICIT

New Features to ELICIT Harvard ELICIT

New Features to ELICIT (well, at least new) Harvard ELICIT

Search Functionality • Subjects have the option of seeking out new information for the organization Harvard ELICIT

Myopic & Cooperative Search Harvard ELICIT

Myopic & Cooperative Search • Red Herrings Harvard ELICIT

Myopic & Cooperative Search • Red Herrings • No Silver Bullets Harvard ELICIT

Myopic & Cooperative Search • Red Herrings • No Silver Bullets • Disintegrated Problems • Multiple Factoids Required Harvard ELICIT

Progress Check • We periodically query the user for her best guess of the solution, as well as allowing her to enter it manually. Harvard ELICIT

Experimental Design • Clear incentives for solving the problem • Control information flow through sharing patterns – No websites • Enable annotation to allow theory sharing • Pretest to control for skill • Visible progress check inside the network Harvard ELICIT

Visible Progress Check • Players may see their network neighbors’ most recent progress check updates. – Strong tie / explicit theory sharing Harvard ELICIT

Network Structures • Goal: individual nodes are identical – Macrostructural variations • Question: How do structural impediments to information flow change the exploration/exploitation dynamic? Harvard ELICIT

Caveman vs Rewired Caveman Motivated by Watts’ Small Worlds (1998) Harvard ELICIT

Degree ‐ Preserving Hierarchy Harvard ELICIT

Simple Lattice Harvard ELICIT

Summary of Experiments • 18 Experimental Sessions • 416 subjects • 70 rounds of 25 minutes • 58: Network x Factoid x Round Order • 12 rounds testing Progress Check & AV capacity • 1120 subject ‐ round records  Quantitative Analysis Harvard ELICIT

Challenges with Data • Messy Data • What are the performance metrics? – 1 solution vs. 4 sub ‐ problems – Time – Individual vs. group level • What to control for? • Challenges of understanding network ‐ level data Harvard ELICIT

Preliminary Regression Summary • Rewired Cave performs somewhat better – Other network treatments not significant • Large learning effect • Pretest is a strong predictor of performance • Turning off visible progress check has a large effect Harvard ELICIT

Distribution of Fully Correct Answers Harvard ELICIT

Distribution of Time of 1 st Correct ID Harvard ELICIT

Distribution of Average ID time Harvard ELICIT

Ongoing Data Analysis • Multi ‐ level Models • Knowledge Sharing Patterns • Coming soon: – Controlling for dependencies in WWWW – Understanding the mezzo ‐ level effects of network neighbors – Isomorphic network roles Harvard ELICIT

Conclusions • Integrate Exploration & Exploitation into Agility research • Extend ELICIT to capture search in a complex space • Some evidence that structural barriors to communication can improve performance in some cases • Group experimental data requires careful statistical analysis. Harvard ELICIT

Questions? Thanks to the ELICIT team, especially Mary Ruddy and Szymon Letowski Harvard ELICIT