IP Scoring Rules: Foundations and Applications Jason Konek - - PowerPoint PPT Presentation

▶
ip scoring rules foundations and applications
SMART_READER_LITE
LIVE PREVIEW

IP Scoring Rules: Foundations and Applications Jason Konek - - PowerPoint PPT Presentation

Aug 14, 2023 350 likes •852 views

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References IP Scoring Rules: Foundations and Applications Jason Konek Department of Philosophy University of Bristol 11 th International Symposium on Imprecise

slide-1
SLIDE 1

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

IP Scoring Rules: Foundations and Applications

Jason Konek

Department of Philosophy University of Bristol

11th International Symposium on Imprecise Probabilities 4 July 2019

Jason Konek IP Scoring Rules: Foundations and Applications

slide-2
SLIDE 2

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Aggregating Interval Forecasts

MGIS: Greenland ice sheet will melt by 2200

Jason Konek IP Scoring Rules: Foundations and Applications

slide-3
SLIDE 3

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Aggregating Interval Forecasts

(Kriegler, 2009)

Jason Konek IP Scoring Rules: Foundations and Applications

slide-4
SLIDE 4

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Aggregating Interval Forecasts

(Kriegler, 2009) IP Scoring Rule

Jason Konek IP Scoring Rules: Foundations and Applications

slide-5
SLIDE 5

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Aggregating Interval Forecasts

(Kriegler, 2009) IP Scoring Rule Identify Accuracy- Related Properties

Jason Konek IP Scoring Rules: Foundations and Applications

slide-6
SLIDE 6

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Aggregating Interval Forecasts

(Kriegler, 2009) IP Scoring Rule Identify Accuracy- Related Properties Engineer New Aggregation Procedure

Jason Konek IP Scoring Rules: Foundations and Applications

slide-7
SLIDE 7

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

IP Scoring Rules: generalised type I and type II error

An IP scoring rule is a loss function I which maps an IP distribution C (lower probabilities, lower previsions, sets of probabilities, etc.) and a world w to a non-negative real number, I(C, w).

Jason Konek IP Scoring Rules: Foundations and Applications

slide-8
SLIDE 8

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

IP Scoring Rules: generalised type I and type II error

An IP scoring rule is a loss function I which maps an IP distribution C (lower probabilities, lower previsions, sets of probabilities, etc.) and a world w to a non-negative real number, I(C, w).

I(C, w) measures the extent to which C fails to avoid various types of alethic (accuracy-related) error.

Jason Konek IP Scoring Rules: Foundations and Applications

slide-9
SLIDE 9

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

IP Scoring Rules: generalised type I and type II error

An IP scoring rule is a loss function I which maps an IP distribution C (lower probabilities, lower previsions, sets of probabilities, etc.) and a world w to a non-negative real number, I(C, w).

I(C, w) measures the extent to which C fails to avoid various types of alethic (accuracy-related) error.

Two principal types of alethic error: generalised type I and type II error.

Jason Konek IP Scoring Rules: Foundations and Applications

slide-10
SLIDE 10

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

IP Scoring Rules: generalised type I and type II error

An IP scoring rule is a loss function I which maps an IP distribution C (lower probabilities, lower previsions, sets of probabilities, etc.) and a world w to a non-negative real number, I(C, w).

I(C, w) measures the extent to which C fails to avoid various types of alethic (accuracy-related) error.

Two principal types of alethic error: generalised type I and type II error.

C avoids generalised type I error to the extent that it leaves

  • pen accurate previsions/estimates.

Jason Konek IP Scoring Rules: Foundations and Applications

slide-11
SLIDE 11

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

IP Scoring Rules: generalised type I and type II error

An IP scoring rule is a loss function I which maps an IP distribution C (lower probabilities, lower previsions, sets of probabilities, etc.) and a world w to a non-negative real number, I(C, w).

I(C, w) measures the extent to which C fails to avoid various types of alethic (accuracy-related) error.

Two principal types of alethic error: generalised type I and type II error.

C avoids generalised type I error to the extent that it leaves

  • pen accurate previsions/estimates.

C avoids generalised type II error to the extent that it rules out inaccurate previsions/estimates.

Jason Konek IP Scoring Rules: Foundations and Applications

slide-12
SLIDE 12

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

IP Scoring Rules: generalised type I and type II error

An IP scoring rule is a loss function I which maps an IP distribution C (lower probabilities, lower previsions, sets of probabilities, etc.) and a world w to a non-negative real number, I(C, w).

I(C, w) measures the extent to which C fails to avoid various types of alethic (accuracy-related) error.

Two principal types of alethic error: generalised type I and type II error.

C avoids generalised type I error to the extent that it leaves

  • pen accurate previsions/estimates.

C avoids generalised type II error to the extent that it rules out inaccurate previsions/estimates.

Iα(C, w) = α · min

c∈C I(c, w) + (1 − α) · max c∈C I(c, w)

Jason Konek IP Scoring Rules: Foundations and Applications

slide-13
SLIDE 13

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Impossibility Theorems

I0.7([.3, .7], 0) = I0.7([.3, .7], 1) = 0.21

Jason Konek IP Scoring Rules: Foundations and Applications

slide-14
SLIDE 14

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Impossibility Theorems

I0.7([.3, .7], 0) = I0.7([.3, .7], 1) = 0.21 I0.7([.5, .5], 0) = I0.7([.5, .5], 1) = 0.25

Jason Konek IP Scoring Rules: Foundations and Applications

slide-15
SLIDE 15

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Impossibility Theorems

I0.7([.3, .7], 0) = I0.7([.3, .7], 1) = 0.21 I0.7([.5, .5], 0) = I0.7([.5, .5], 1) = 0.25 Seidenfeld et al. (2012), Mayo-Wilson and Wheeler (2015), and Schoenfield (2015) show that any continuous IP scoring rule I renders some IP distribution dominated.

Jason Konek IP Scoring Rules: Foundations and Applications

slide-16
SLIDE 16

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Impossibility Theorems

I0.7([.3, .7], 0) = I0.7([.3, .7], 1) = 0.21 I0.7([.5, .5], 0) = I0.7([.5, .5], 1) = 0.25 Seidenfeld et al. (2012), Mayo-Wilson and Wheeler (2015), and Schoenfield (2015) show that any continuous IP scoring rule I renders some IP distribution dominated. Scepticism about using IP scoring rules to provide accuracy-centered foundations for IP, elication

Jason Konek IP Scoring Rules: Foundations and Applications

slide-17
SLIDE 17

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Impossibility Theorems

Choose an interval forecast [a, b] of event E.

Jason Konek IP Scoring Rules: Foundations and Applications

slide-18
SLIDE 18

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Impossibility Theorems

Choose an interval forecast [a, b] of event E. Though no IP scoring rule renders all interval forecasts non-dominated, each interval forecast is rendered non-dominated by some IP scoring rule

Jason Konek IP Scoring Rules: Foundations and Applications

slide-19
SLIDE 19

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Impossibility Theorems

Choose an interval forecast [a, b] of event E. Though no IP scoring rule renders all interval forecasts non-dominated, each interval forecast is rendered non-dominated by some IP scoring rule Iα with α = −b + ab + √ ab − a2b − ab2 + a2b2 a − b is the unique IP scoring rule of the form proposed in Konek (2019) that renders [a, b] non-dominated.

Jason Konek IP Scoring Rules: Foundations and Applications

slide-20
SLIDE 20

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Impossibility Theorems

How Iα prioritises avoiding type I over type II error (or vice versa) fixes the stock

  • f available interval forecasts.

Jason Konek IP Scoring Rules: Foundations and Applications

slide-21
SLIDE 21

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Impossibility Theorems

How Iα prioritises avoiding type I over type II error (or vice versa) fixes the stock

  • f available interval forecasts.

Where you report 0.5, I report [0.3, 0.7]

Jason Konek IP Scoring Rules: Foundations and Applications

slide-22
SLIDE 22

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Impossibility Theorems

How Iα prioritises avoiding type I over type II error (or vice versa) fixes the stock

  • f available interval forecasts.

Where you report 0.5, I report [0.3, 0.7] Different interval forecasts, same evidence

Jason Konek IP Scoring Rules: Foundations and Applications

slide-23
SLIDE 23

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Impossibility Theorems

How Iα prioritises avoiding type I over type II error (or vice versa) fixes the stock

  • f available interval forecasts.

Where you report 0.5, I report [0.3, 0.7] Different interval forecasts, same evidence

Reason: different attitudes toward the comparative importance of avoiding type I and type II error

Jason Konek IP Scoring Rules: Foundations and Applications

slide-24
SLIDE 24

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Impossibility Theorems

How Iα prioritises avoiding type I over type II error (or vice versa) fixes the stock

  • f available interval forecasts.

Where you report 0.5, I report [0.3, 0.7] Different interval forecasts, same evidence

Reason: different attitudes toward the comparative importance of avoiding type I and type II error

Certain ways of prioritising type I over type II error-avoidance (captured by α) justify certain IP methods (more imprecise interval forecasts);

Jason Konek IP Scoring Rules: Foundations and Applications

slide-25
SLIDE 25

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Impossibility Theorems

How Iα prioritises avoiding type I over type II error (or vice versa) fixes the stock

  • f available interval forecasts.

Where you report 0.5, I report [0.3, 0.7] Different interval forecasts, same evidence

Reason: different attitudes toward the comparative importance of avoiding type I and type II error

Certain ways of prioritising type I over type II error-avoidance (captured by α) justify certain IP methods (more imprecise interval forecasts); others justify others (more precise forecasts).

Jason Konek IP Scoring Rules: Foundations and Applications

slide-26
SLIDE 26

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Aggregating Interval Forecasts

Jason Konek IP Scoring Rules: Foundations and Applications

slide-27
SLIDE 27

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Current Approaches: Convex IP Pooling

Convex IP Pooling (Stewart and Quintana, 2018): the ag- gregate of [a1, b1] . . . [an, bn] is the convex hull of their union, conv{∪i[ai, bi]}

Jason Konek IP Scoring Rules: Foundations and Applications

slide-28
SLIDE 28

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Current Approaches: Convex IP Pooling

Convex IP Pooling (Stewart and Quintana, 2018): the ag- gregate of [a1, b1] . . . [an, bn] is the convex hull of their union, conv{∪i[ai, bi]} Good:

Jason Konek IP Scoring Rules: Foundations and Applications

slide-29
SLIDE 29

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Current Approaches: Convex IP Pooling

Convex IP Pooling (Stewart and Quintana, 2018): the ag- gregate of [a1, b1] . . . [an, bn] is the convex hull of their union, conv{∪i[ai, bi]} Good: Preserves independence judgments (in a sense)

Jason Konek IP Scoring Rules: Foundations and Applications

slide-30
SLIDE 30

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Current Approaches: Convex IP Pooling

Convex IP Pooling (Stewart and Quintana, 2018): the ag- gregate of [a1, b1] . . . [an, bn] is the convex hull of their union, conv{∪i[ai, bi]} Good: Preserves independence judgments (in a sense) Commutes with Bayesian conditioning (Field conditioning, imaging)

Jason Konek IP Scoring Rules: Foundations and Applications

slide-31
SLIDE 31

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Current Approaches: Convex IP Pooling

Convex IP Pooling (Stewart and Quintana, 2018): the ag- gregate of [a1, b1] . . . [an, bn] is the convex hull of their union, conv{∪i[ai, bi]} Good: Preserves independence judgments (in a sense) Commutes with Bayesian conditioning (Field conditioning, imaging) Bad:

Jason Konek IP Scoring Rules: Foundations and Applications

slide-32
SLIDE 32

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Current Approaches: Convex IP Pooling

Convex IP Pooling (Stewart and Quintana, 2018): the ag- gregate of [a1, b1] . . . [an, bn] is the convex hull of their union, conv{∪i[ai, bi]} Good: Preserves independence judgments (in a sense) Commutes with Bayesian conditioning (Field conditioning, imaging) Bad: Captures consensus, not compromise; washes out info distributed across community

Jason Konek IP Scoring Rules: Foundations and Applications

slide-33
SLIDE 33

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Current Approaches: Convex IP Pooling

Convex IP Pooling (Stewart and Quintana, 2018): the ag- gregate of [a1, b1] . . . [an, bn] is the convex hull of their union, conv{∪i[ai, bi]} Good: Preserves independence judgments (in a sense) Commutes with Bayesian conditioning (Field conditioning, imaging) Bad: Captures consensus, not compromise; washes out info distributed across community Ill-suited to inform future research, serve as an input to decision-theory.

Jason Konek IP Scoring Rules: Foundations and Applications

slide-34
SLIDE 34

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Imprecise Credence

BIGGER PROBLEM: Convex IP pooling delivers dominated aggregates

Jason Konek IP Scoring Rules: Foundations and Applications

slide-35
SLIDE 35

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Imprecise Credence

BIGGER PROBLEM: Convex IP pooling delivers dominated aggregates My interval forecast for MGIS: [0.1, 0.376923]; yours: [0.4, 0.784].

Jason Konek IP Scoring Rules: Foundations and Applications

slide-36
SLIDE 36

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Imprecise Credence

BIGGER PROBLEM: Convex IP pooling delivers dominated aggregates My interval forecast for MGIS: [0.1, 0.376923]; yours: [0.4, 0.784]. I0.7 uniquely renders our respective forecasts non-dominated I0.7([a, b], w) = 0.7 · min

x∈[a,b] I(x, w) + 0.3 · min x∈[a,b] I(x, w)

Jason Konek IP Scoring Rules: Foundations and Applications

slide-37
SLIDE 37

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Imprecise Credence

BIGGER PROBLEM: Convex IP pooling delivers dominated aggregates My interval forecast for MGIS: [0.1, 0.376923]; yours: [0.4, 0.784]. I0.7 uniquely renders our respective forecasts non-dominated I0.7([a, b], w) = 0.7 · min

x∈[a,b] I(x, w) + 0.3 · min x∈[a,b] I(x, w)

But the convex IP pool, [0.1, 0.784], is accuracy dominated by [0.257, 0.652] according to I0.7

Jason Konek IP Scoring Rules: Foundations and Applications

slide-38
SLIDE 38

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Current Approaches

Linear pooling of lower prob- abilities: the aggregate

  • f

[x1, y1] . . . [xn, yn] is [a, b] with a =

i λixi and b = i λiyi,

where

i λi = 1.

Jason Konek IP Scoring Rules: Foundations and Applications

slide-39
SLIDE 39

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Current Approaches

Linear pooling of lower prob- abilities: the aggregate

  • f

[x1, y1] . . . [xn, yn] is [a, b] with a =

i λixi and b = i λiyi,

where

i λi = 1.

Linear pooling of Bayes risk functions (Nau, 2002): the aggregate of [x1, y1] . . . [xn, yn] is an IP distribution C such that betting against C and betting against [x1, y1] . . . [xn, yn] one by one is indistinguishable.

Jason Konek IP Scoring Rules: Foundations and Applications

slide-40
SLIDE 40

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Current Approaches

Linear pooling of lower prob- abilities: the aggregate

  • f

[x1, y1] . . . [xn, yn] is [a, b] with a =

i λixi and b = i λiyi,

where

i λi = 1.

Linear pooling of Bayes risk functions (Nau, 2002): the aggregate of [x1, y1] . . . [xn, yn] is an IP distribution C such that betting against C and betting against [x1, y1] . . . [xn, yn] one by one is indistinguishable. Good: satisfy some prima facie desirable axioms

Jason Konek IP Scoring Rules: Foundations and Applications

slide-41
SLIDE 41

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Current Approaches

Linear pooling of lower prob- abilities: the aggregate

  • f

[x1, y1] . . . [xn, yn] is [a, b] with a =

i λixi and b = i λiyi,

where

i λi = 1.

Linear pooling of Bayes risk functions (Nau, 2002): the aggregate of [x1, y1] . . . [xn, yn] is an IP distribution C such that betting against C and betting against [x1, y1] . . . [xn, yn] one by one is indistinguishable. Good: satisfy some prima facie desirable axioms Bad: deliver dominated aggregates

Jason Konek IP Scoring Rules: Foundations and Applications

slide-42
SLIDE 42

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Epistemic Utility Based Aggregation

EU Aggregation (Unique EU Function): If n individuals have interval forecasts [a1, b1] , . . . , [an, bn] for E, and they all are rendered non-dominated by Iα, then any reasonable aggregate must take the following form:

  • x,

α2x 1 − 2α + α2 − x + 2αx

  • where mini ai ≤ x ≤ maxi ai.

Jason Konek IP Scoring Rules: Foundations and Applications

slide-43
SLIDE 43

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Epistemic Utility Based Aggregation

Permissible aggregates of [0.1, 0.376923], [0.2, 0.576471] and [0.5, 0.844828]

Jason Konek IP Scoring Rules: Foundations and Applications

slide-44
SLIDE 44

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Advantages

Epistemic utility based aggregate using uniform expert weights: [0.0153442, 0.318378]

Jason Konek IP Scoring Rules: Foundations and Applications

slide-45
SLIDE 45

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Advantages

Epistemic utility based aggregate using uniform expert weights: [0.0153442, 0.318378] Non-dominated aggregates;

Jason Konek IP Scoring Rules: Foundations and Applications

slide-46
SLIDE 46

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Advantages

Epistemic utility based aggregate using uniform expert weights: [0.0153442, 0.318378] Non-dominated aggregates; Compromise, not consensus;

Jason Konek IP Scoring Rules: Foundations and Applications

slide-47
SLIDE 47

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Advantages

Epistemic utility based aggregate using uniform expert weights: [0.0153442, 0.318378] Non-dominated aggregates; Compromise, not consensus; Robust against outliers.

Jason Konek IP Scoring Rules: Foundations and Applications

slide-48
SLIDE 48

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Joyce, J. (1998). A nonpragmatic vindication of probabilism. Philosophy of Science 65(4), 575–603. Joyce, J. M. (2009). Accuracy and coherence: Prospects for an alethic epistemology of partial belief. In F. Huber and

  • C. Schmidt-Petri (Eds.), Degrees of Belief, Volume 342.

Dordrecht: Springer. Konek, J. (2019). Epistemic conservativity and imprecise credence. Philosophy and Phenomenological Research. Kriegler, E. (2009). Imprecise probability assessment of tipping points in the climate system. Proceedings of the National Academy of Sciences 106(13), 5041–5046. Mayo-Wilson, C. and G. Wheeler (2015). Accuracy and imprecision: A mildly immodest proposal. Philosophy and Phenomenological Research. Moss, S. (2011). Scoring rules and epistemic compromise. Mind 120(480), 1053–1069.

Jason Konek IP Scoring Rules: Foundations and Applications

slide-49
SLIDE 49

IP Scoring Rules Impossibility Theorems Aggregating Interval Forecasts References

Nau, R. (2002). The aggregation of imprecise probabilities. Journal of Statistical Planning and Inference 105(1), 265–282. Schoenfield, M. (2015). The accuracy and rationality of imprecise

  • credences. Nous.

Seidenfeld, T., M. J. Schervish, and J. B. Kadane (2012). Forecasting with imprecise probabilities. International Journal of Approximate Reasoning 53, 1248–1261. Stewart, R. and I. Quintana (2018). Probabilistic opinion pooling with imprecise probabilities. J Philos Logic 47, 17–45.

Jason Konek IP Scoring Rules: Foundations and Applications