Robustness and idealizations in agent-based models of scientific - PowerPoint PPT Presentation

Robustness and idealizations in agent-based models of scientific interaction Daniel Frey 1 and Dunja Šešelja 2 July 18-19, RUB, Bochum 1. Faculty of Economics and Social Sciences, Heidelberg University 2. Institute for Philosophy II, Ruhr-University Bochum

Zollman’s 2010 model Changing some assumptions. . . Our results Conclusion 1/34

Zollman’s 2010 model

Zollman’s ABM Modeling science by "bandit problems" A gambler, confronted with slot machines that have different objective probabilities of success. 2/34

Zollman’s ABM Modeling science by "bandit problems" A gambler, confronted with slot machines that have different objective probabilities of success. The research question How do different social structures impact the efficiency of scientific inquiry? 2/34

• pulling a bandit’s arm → testing a hypothesis • the payoff of a slot machine → a successful application of a given hypothesis/method/theory • the objective probability of success (OPS) of a slot machine → OPS of the given hypothesis/method/theory. 3/34

Modeling science by "bandit problems" • scientists are presented with the choice between two hypotheses • they always choose to pursue the hypothesis that seems to be more successful • they update their beliefs via Bayesian reasoning (via beta distributions), based on: 1. their own success 2. the success of some other agents, with whom they are connected in a social network 4/34

Restricting the information flow • unrestricted information flow appears to be harmful • the cycle scores the best, then the wheel, and then the complete graph 5/34

Changing some assumptions. . .

Static vs. dynamic epistemic success 5/34

Static epistemic success Zollman’s parameters • OPS(True theory)=0.5 • OPS(False theory)=0.499 • Success: converging onto the true theory. • Scientists may never get it right. 6/34

Static epistemic success Zollman’s parameters • OPS(True theory)=0.5 • OPS(False theory)=0.499 • Success: converging onto the true theory. • Scientists may never get it right. However. . . • We observe that scientists eventually do get it right! • The question is not if , but rather when . 6/34

Static epistemic success Zollman’s parameters • OPS(True theory)=0.5 • OPS(False theory)=0.499 • Success: converging onto the true theory. • Scientists may never get it right. However. . . • We observe that scientists eventually do get it right! • The question is not if , but rather when . Dynamic notion of success • OPS = probability of gaining corroborating evidence given the current state of inquiry : Current probability of success 6/34

7/34

"Restless bandit" 8/34

"Restless bandit" 9/34

Introducing dynamic epistemic success Current probability of success (CPS) • agent on True theory: CPS ( T ) moves 0.1% towards 1. • agent on False theory: CPS ( T ) moves 0.1% towards 0. More precisely: • APS (True theory)=1 and APS (False theory)=0 • CPS new ( T ) = CPS old ( T ) + f ( d ) • f ( d ) = d / s • d = APS ( T ) − OPS ( T ) 10/34

Basic notions Zollman (2010) Our ABM SPS(T) SPS(T) Updates: beta distribution Updates: beta distribution OPS(T) – static CPS(T) – dynamic APS(T) ∈ { 0 , 1 } – static 11/34

(The lack of) Critical interaction 11/34

Lack of critical interaction Critical aspect of interaction: not represented neither explicitly, nor implicitly Epistemic benefits of criticism criticism exposes errors in research 12/34

Critical interaction 13/34

Introducing critical interaction Assumptions • criticism is truth conducive (Moffett (2007); Betz (2012)) • it occurs between proponents of rivaling theories 14/34

Introducing critical interaction Assumptions • criticism is truth conducive (Moffett (2007); Betz (2012)) • it occurs between proponents of rivaling theories Triggering condition: • every time an agent pursuing T x receives information from an agent pursuing T y , such that T y turns out to be better than she had expected: SPS old ( T y ) < SPS new ( T y ) . Critical interaction • agent on True theory: CPS ( T ) moves 0.1% towards 1. • agent on False theory: CPS ( T ) moves 0.1% towards 0. 14/34

Aggregation problem 14/34

Scientists can distinguish between similarly successful theories Zollman’s parameters • OPS(True theory)=0.5 • OPS(False theory)=0.499 Agents can perfectly determine which theory is better, no matter how similarly successful their applications are. Aggregation problem Often it is impossible for scientists to say which theory is more worthy of pursuit, if they are similarly successful. 15/34

Treating similar theories as equally good Threshold: The rival theory counts as better only if it surpasses one’s own theory by the margin of 0.1. 16/34

(The lack of) Inertia 16/34

Scientists easily abandon their inquiry • agents are easily swayed by new evidence • they abandon their theory if only the evidence suggests the rival is superior Rational inertia in one’s inquiry Taking time to examine whether problems can be resolved. 17/34

Inertia of one’s inquiry Jump threshold: Agents switch theories only after the rival has turned out to be - better for a certain number of rounds (e.g. 10 rounds). 18/34

Our results

Parameters • 10.000 runs for each data point • each simulation: up to 100.000 rounds • A run stops when all agents converge on the better theory for the final time. • We measure how much time steps they need to do so. 19/34

Difficult inquiry (improvements in inquiry happen rarely) 19/34

Capturing Zollman’s (2010) results with dynamic CPS 4000 Average Round Converged 3000 2000 1000 0 2 4 6 8 10 12 scientists cycle complete Figure 1: No critical interaction, no theory threshold, no jump threshold; global improvement in CPS every 100 round. 20/34

Adding critical interaction 4000 Average Round Converged 3000 2000 1000 0 2 4 6 8 10 12 scientists cycle complete Figure 2: Critical interaction , no theory threshold, no jump threshold; global improvement in CPS every 100 round. 21/34

Adding jump threshold 2000 Average Round Converged 1500 1000 500 0 2 4 6 8 10 12 scientists cycle complete Figure 3: No critical interaction, no theory threshold, jump threshold of 10 ; global improvement in CPS every 100 round. 22/34

Difficult inquiry, no theory threshold 2000 Average Round Converged 1500 1000 500 0 2 4 6 8 10 12 scientists cycle complete Figure 4: Critical interaction , no theory threshold, jump threshold of 10 ; global improvement in CPS every 100 round. 23/34

Let’s make inquiry even more difficult. . . (in terms of theory threshold) 23/34

Adding theory threshold (difficult inquiry) 30000 Average Round Converged 20000 10000 0 2 4 6 8 10 12 scientists cycle complete Figure 5: No critical interaction, theory threshold of 0.1 , no jump threshold; global improvement in CPS every 100 round. 24/34

Adding critical interaction (difficult inquiry) 4000 Average Round Converged 3000 2000 1000 0 2 4 6 8 10 12 scientists cycle complete Figure 6: Critical interaction , theory threshold of 0.1 , no jump threshold; global improvement in CPS every 100 round. 25/34

Adding jump threshold (difficult inquiry) 4000 Average Round Converged 3000 2000 1000 0 2 4 6 8 10 12 scientists cycle complete Figure 7: Critical interaction , theory threshold of 0.1 , jump threshold of 10 ; global improvement in CPS every 100 round. 26/34

Summing up Difficult inquiry • Efficiency is obtained by either critical interaction or by being cautious. • The degree of connectedness doesn’t always play a major role. 27/34

Conclusion

• Results of Zollman’s 2010 model are not robust under different assumptions concerning scientific inquiry • Further research: • different decision making procedures • sensitivity analysis • etc. 28/34

When are models informative of real world phenomena? 29/34

Thank you! 30/34

Easy inquiry 60 Average Round Converged 40 20 0 2 4 6 8 10 12 scientists cycle complete Figure 8: No critical interaction, no theory threshold, no jump threshold; global improvement in CPS every 10 rounds 1% .

Bibliography

Bibliography i References Betz, G.: 2012, Debate dynamics: How controversy improves our beliefs , Vol. 357. Springer Science & Business Media. Douglas, H. E.: 2009, Science, Policy, and the Value-Free Ideal . University of Pittsburgh Press. Goldman, A. and T. Blanchard: 2016, ‘Social Epistemology’. In: E. N. Zalta (ed.): The Stanford Encyclopedia of Philosophy . Stanford University, summer 2016 edition.