Army Conference on Applied Statistics Santa Fe, NM October 25, 2001 The Role of Expert Knowledge in Uncertainty Quantification (Are We Adding More Uncertainty (Are We Adding More Uncertainty or More Understanding?) or More Understanding?) Jane M. Booker, ESA-WR Mark C. Anderson, DX-5 Mary A. Meyer, D-1 Weapon Response Group
Expert Knowledge Expert Knowledge : what is known by qualified individuals, responding to complex, difficult (technical) questions, obtained through formal expert elicitation. •A snapshot of the expert’s state of knowledge at the time. •Expressed in qualitative and quantitative form. Weapon Response Group
Expert Knowledge = Expertise + Expert Judgment Structure (Expertise ) • Define the problem • Organize and represent the problem solving knowledge, the information flow • Identify the relevant data and information (e.g., models, experimental results, numerical methods. . .) • Identify uncertainties uncertainties and determine how these are to be represented Contents (Judgment) • Provide quantitative and qualitative estimates and uncertainties, and the heuristics, assumptions and information used to arrive at answers to technical questions. Weapon Response Group
Uses of Expertise & Judgment Expertise: • Decision about what variables enter into a statistical analysis • Decision about which data sets to include in an analysis • Assumptions used in selecting a model or method • Decision concerning which forms of uncertainty are appropriate to use (e.g., probability distributions) • Description of experts’ thinking and information sources in arriving at any of the above responses Expert Judgment: • Estimation of an occurrence of an event • Estimation of the uncertainty of parameter • Prediction of the performance of some product or process Weapon Response Group
Uncertainty Quantification Broad Definition — the process of characterizing, estimating, propagating, and analyzing various kinds of uncertainty (including variability) for a complex decision problem. For complex computer and physical models — focuses upon measurement, computational, parameter (including sensitivities of outputs to input values), and modeling uncertainties leading to verification and validation. Weapon Response Group
Two Categories of Uncertainty • Aleatory — Inherent variation, Random, Irreducible (Includes variability) • Error – numerical, • Epistemic — discretization, Lack of knowledge, mistakes Reducible Weapon Response Group
The Modeling Process with Uncertainties Sources of uncertainty Sources of uncertainty – Measurements Observation of Nature • Noise • Resolution Conceptual Modeling • Processing – Mathematical models Mathematical Modeling • Equations • Boundary conditions Numerical Modeling • Initial conditions • Inputs Numerical Implementation – Numerical models • Weak formulations Numerical Evaluation • Discretizations (mesh, time step) • Approximate solution algorithms Surrogate Modeling • Truncation and roundoff – Surrogate models (statistical) Surrogate Implementation • Approximation error • Interpolation error • Extrapolation error Surrogate Evaluation – Model parameters – Scenarios Weapon Response Group
Additional Uncertainty: “Human In The Loop” Sources of uncertainty Sources of uncertainty The expert is making decisions – Measurements about all of these – Mathematical models choices and – Numerical models inducing uncertainties in the – Surrogate models (statistical) process. – Model parameters more s s e l – Scenarios Weapon Response Group
Cognitive and Motivational Biases Contribute Bias: A skewing from a standard or reference point. Can degrade the quality of the information and contribute to uncertainty. Cognitive biases: • Underestimation of uncertainty (false precision) Underestimation of uncertainty (false precision) • • Availability (accounting for rare events) • Anchoring (cannot move from preconceptions) • Inconsistency (forgetting what preceded) more s s e l Motivational biases: • Group think (follow the leader) • Impression Management (politically correct) • Wishful thinking (wanting makes it a reality) • Misrepresentation (bad translation) Weapon Response Group
Role of Expert Knowledge in Uncertainty Quantification — Contributions to Uncertainty Poor Probability Inconsistent Thinking Thinking Underestimation Decision Of Uncertainty Experts Making more s s e l Weapon Response Group
What Tools / Technologies Are Available To Counter These Contributions? I. Formal, structured elicitation of I. Formal, structured elicitation of expertise and expert judgment expertise and expert judgment • Draws from cognitive psychology, decision analysis, statistics, sociology, cultural anthropology, and knowledge acquisition. • Counters common biases arising from human cognition and behavior. • Adds rigor, defensibility, and increased ability to update the judgments. Weapon Response Group
I. Formal, Structured Elicitation of Expertise I. Formal, Structured Elicitation of Expertise and Expert Judgment and Expert Judgment • Minimizes biases Minimizes biases • • Provides documentation Provides documentation • • Utilizes the way people think, work, and Utilizes the way people think, work, and • problem solve problem solve • Provides what is necessary for uncertainty Provides what is necessary for uncertainty • quantification: quantification: -Sources, Sources, - more s -Quantification, Quantification, - s e l -Estimates and Updates, Estimates and Updates, - -Methods of propagation Methods of propagation - Weapon Response Group
II. Mathematics (Theories) Handling Ignorance, II. Mathematics (Theories) Handling Ignorance, Ambiguity, Vagueness and the Way People Think Ambiguity, Vagueness and the Way People Think - Probability Theory (different interpretations within e.g., Frequentist, Subjective/Bayesian) - Possibility Theory (crisp or fuzzy set) - Fuzzy Sets - Dempster-Schafer (Evidence)Theory - Choquet Capacities - Upper and Lower Probabilities - Convex Sets - Interval Analysis Theories - Information Gap Decision Theory (non measure based) Weapon Response Group
Mathematical Theories — — Frameworks for Expert Frameworks for Expert Thinking Thinking Characteristics Characteristics - Set based (crisp or fuzzy) - Axiomatic - Calculus (rules for implementing axioms) - Consistent / coherence - Computationally practical (??) - Measure based (not all!) Goal: Provide Metrics for Uncertainty For combining uncertainties there needs to be a bridge between the various theories. Weapon Response Group
Hierarchy of Theories for Crisp Sets Coherent Upper and Coherent Upper and Convex Sets Convex Sets Lower Previsions Lower Previsions Choquet Capacities Choquet Capacities Specific to General Coherent Upper and Coherent Upper and Lower Probabilities Lower Probabilities Dempster Schafer Theory Dempster Schafer Theory Possibility Theory Possibility Theory Probability Theory Probability Theory Frequentist Frequentist Subjective Subjective Interval Analysis Interval Analysis aleatory epistemic Weapon Response Group
Set Based Theories for Uncertainty Non-Measure Based Fuzzy Sets Fuzzy Sets Information Gap Information Gap Measure Based Crisp Sets Crisp Sets Weapon Response Group
Some Measure Theory Approaches Dempster- -Schafer Theory Schafer Theory Possibility Theory Dempster Possibility Theory Probability Theory Probability Theory Based on two Based on two measure Based on single measure functions — belief and measure functions — function (additivity, possibility & necessity plausibility (monotonic & monotonic) (monotonic & nonaddivity) Pr:2 X → 0,1 [ ] nonaddivity) ( ) = 0 :2 X → 0,1 :2 X → 0,1 [ ] [ ] Pr ∅ Bel Pl Pos:2 X → 0,1 [ ] Nec:2 X → 0,1 [ ] ( ) = 1 ( ) = 0 ( ) = 0 Pr X Bel ∅ Pl ∅ ( ) = 0 ( ) = 0 Pos ∅ Nec ∅ ( ) = 1 ( ) = 1 Bel X Pl X ( ) = 1 ( ) = 1 Pos X Nec X ( ) ∑ ∑ = − � Pr A Pr A Pr A j � A i k i j < k i i ( ) ∑ ∑ ≥ − � Bel A Bel A Bel A j � A ( ) ( ) n + 1 Pr i i k = sup � Pos A PosA i + � + − 1 � j < k A i i i i i i i ( ) n + 1 Bel + � + − 1 � A i i ( ) ( ) = inf ∑ ∑ Nec � A i Nec A = − Pr � A Pr A Pr A j � A i i i i k i j < k i i ( ) ∑ ∑ ≤ − Pl � A Pl A Pl A j � A n + 1 Pr i i k ( ) + � + − 1 � A j < k i i i i n + 1 Pl ( ) + � + − 1 � A i i Weapon Response Group
Recommend
More recommend
