How Expert Knowledge Can Three Case Studies Help Measurements: - PowerPoint PPT Presentation

Using Expert . . . So How Can Expert . . . Why Is This Useful? But How to . . . How Expert Knowledge Can Three Case Studies Help Measurements: First Case Study Second Case Study Three Case Studies Third Case Study Auxiliary Results Vladik Kreinovich Home Page Department of Computer Science Title Page University of Texas at El Paso ◭◭ ◮◮ vladik@utep.edu http://www.cs.utep.edu/vladik ◭ ◮ Page 1 of 63 Go Back Full Screen Close Quit

Using Expert . . . So How Can Expert . . . 1. Using Expert Knowledge Is Important, But How? Why Is This Useful? • A large amount of information comes from measure- But How to . . . ments. Three Case Studies First Case Study • However, in many areas, it is crucial to also use expert Second Case Study knowledge. Third Case Study • With all modern medical tests and measurements, doc- Auxiliary Results tor’s intuition is still crucial. Home Page • In spite of all the successes of self-driving cars, it is still Title Page not possible to fully replace a human driver. ◭◭ ◮◮ • It is therefore important to supplement measurement ◭ ◮ results with expert estimates. Page 2 of 63 • And this is a big problem for metrology: Go Back – in metrology, we can accustomed to work with sta- Full Screen tistically justified estimates, – while expert estimates are not similarly justified. Close Quit

Using Expert . . . So How Can Expert . . . 2. So How Can Expert Knowledge Help Measure- Why Is This Useful? ments? But How to . . . • In measurement practice: Three Case Studies First Case Study – we come up with a parametric model of the corre- Second Case Study sponding class of phenomena, Third Case Study – we test this model – to make sure that it provides Auxiliary Results an adequate description of the phenomena, and Home Page – we use measurements to estimate the parameters Title Page corresponding to a given situation. ◭◭ ◮◮ • How can experts help? ◭ ◮ – experts can provide such a model, and Page 3 of 63 – experts can provide estimates of the corresponding Go Back parameters. Full Screen Close Quit

Using Expert . . . So How Can Expert . . . 3. Why Is This Useful? Why Is This Useful? • In terms of a model: But How to . . . Three Case Studies – the currently used model often comes from a semi- First Case Study empirical study, Second Case Study – such curve-fitting models are not very convincing, Third Case Study this can be over-fitting, Auxiliary Results – experts’ knowledge and intuition can help separate Home Page explainable models from curve-fitting results. Title Page • In terms of expert estimations: ◭◭ ◮◮ – experts may not be accurate as measurements, but ◭ ◮ they are often faster and cheaper to use, Page 4 of 63 – they also supplement measurement results, this mak- Go Back ing the resulting estimates more accurate. Full Screen Close Quit

Using Expert . . . So How Can Expert . . . 4. But How to Incorporate Expert Knowledge into Why Is This Useful? a Metrological Framework But How to . . . • From the common sense viewpoint, expert knowledge Three Case Studies is useful. First Case Study Second Case Study • But how can include their estimates into a metrological Third Case Study framework, with its precise justifications? Auxiliary Results • A natural idea is to treat an expert as a measuring Home Page instrument: to calibrate the expert. Title Page • Thus, we can get a statistically justified estimate for ◭◭ ◮◮ the accuracy of expert-generated numbers. ◭ ◮ • Moreover, we can use this calibration to improve the Page 5 of 63 expert’s estimates. Go Back • This is similar to how, once know the instrument’s bias, we can subtract it and get more accurate results. Full Screen Close Quit

Using Expert . . . So How Can Expert . . . 5. Three Case Studies Why Is This Useful? • To illustrate the above general ideas, we provide three But How to . . . case studies. Three Case Studies First Case Study • In the first case study, we show that application of Second Case Study usual linear calibration to experts can be helpful. Third Case Study • In the second case study, we provide an example of Auxiliary Results useful non-linear calibration. Home Page • The third case study explains how expert knowledge Title Page can make semi-empirical models more convincing. ◭◭ ◮◮ ◭ ◮ Page 6 of 63 Go Back Full Screen Close Quit

Using Expert . . . So How Can Expert . . . Part I Why Is This Useful? First Case Study: But How to . . . Measurement-Type “Calibration” Three Case Studies of Expert Estimates Improves First Case Study Second Case Study Their Accuracy and Their Third Case Study Usability – Pavement Engineering Auxiliary Results Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 7 of 63 Go Back Full Screen Close Quit

Using Expert . . . So How Can Expert . . . 6. Experts Are Often Used for Estimation Why Is This Useful? • Sometimes, experts are used because no measuring in- But How to . . . struments can replace these experts. Three Case Studies First Case Study • For example, in dermatology, estimates of a skilled ex- Second Case Study pert are more accurate results than of any algorithm. Third Case Study • This is one of the main reasons why, Auxiliary Results – in spite of numerous expert systems, Home Page – human doctors are still needed and still valued. Title Page • In other cases, in principle, we can use automatic sys- ◭◭ ◮◮ tems, but experts are still much cheaper to use. ◭ ◮ • An example of such situation is pavement engineering. Page 8 of 63 • In principle, we can use an expensive automatic vision- Go Back based system to gauge the condition of the pavement. Full Screen • However, it is much cheaper – and faster – to use hu- man raters. Close Quit

Using Expert . . . So How Can Expert . . . 7. Expert Estimates Are Often Very Imprecise Why Is This Useful? • Humans rarely have a skill of accurately evaluating the But How to . . . values of different quantities. Three Case Studies First Case Study • For example, it is well known that humans drastically Second Case Study overestimate small probabilities. Third Case Study • Correspondingly, underestimate the probabilities which Auxiliary Results are close to 1. Home Page • Since most people’s estimates are very inaccurate, it is Title Page difficult to find good expert estimators. ◭◭ ◮◮ • It is well known that there is a high competition to get ◭ ◮ into medical schools. Page 9 of 63 • Even in pavement engineering, finding a good rater is Go Back difficult. Full Screen Close Quit

Using Expert . . . So How Can Expert . . . 8. It Is Difficult to Find Good Experts: Example Why Is This Useful? from Pavement Engineering But How to . . . • According to a current standard, the condition of a Three Case Studies pavement is evaluated by using a special index. First Case Study Second Case Study • This Pavement Condition Index (PCI) combines differ- Third Case Study ent possible pavement faults. Auxiliary Results • To gauge the accuracy of a rater candidate, Home Page – many locations across the US Title Page – use criteria developed by the Metropolitan Trans- ◭◭ ◮◮ portation Commission (MTC) of California. ◭ ◮ • A crucial part of the rater certification is a field survey Page 10 of 63 exam. Go Back • In this exam, a rater evaluates 24 test sites that have Full Screen been previously evaluated by expert raters. Close Quit

Using Expert . . . So How Can Expert . . . 9. Pavement Engineering (cont-d) Why Is This Useful? • Candidate’s PCI values are then compared with the But How to . . . PCI values of the expert rater. Three Case Studies First Case Study • The expert’s values are taken as the ground truth (GT). Second Case Study • To certify, the rater must satisfy the following two cri- Third Case Study teria: Auxiliary Results Home Page – at least for 50% of the evaluated sites, the difference should not exceed 8 points, and Title Page – at least for 88% of the evaluated sites, the difference ◭◭ ◮◮ should not exceed 18 points. ◭ ◮ • MTC provided a sample of 18 typical candidates. Page 11 of 63 • Out of these candidates, only 5 (28%) satisfy both cri- Go Back teria and thus, pass the exam and can be used as raters. Full Screen Close Quit

Using Expert . . . So How Can Expert . . . 10. Problems Why Is This Useful? • What can we do to increase the number of available But How to . . . experts? Three Case Studies First Case Study • And for those who have been selected as experts, can Second Case Study we improve the accuracy of their estimates? Third Case Study Auxiliary Results Home Page Title Page ◭◭ ◮◮ ◭ ◮ Page 12 of 63 Go Back Full Screen Close Quit