A possible Validation Metrics for the ACTD C. W. Gear July 19, 2004 (a modification of a presentation of June 16 by CWG and Skip Knowles) 1
Background • Typically, calculations are visually compared with an “acceptable” experimental data set or an analytic solution – Quality of the calculation is determined by “expert” judgment – Different experts may reach different conclusions – No audit trail defining how the conclusions were reached • Which aspects of the calculations were deemed to be important? • What is the relative ranking of each of these aspects? – Implicit rules for ranking calculations established after the calculations are completed • May lead to sense of unfairness where relative rankings may impact future contract work – Quality of the experimental data can be a key factor • A more orderly and well defined procedure is highly desirable 2 Making the World Safer
Metrics • Provide an unbiased mathematical basis for comparing calculations with experimental data or analytic solutions • Emphasis on ACTD applications • Recommended metrics to be evaluated against early JOLT experimental data • Refinements to be made as appropriate 3 Making the World Safer
Some Definitions • Normalization – scaling of the result to achieve some objective. This could include: – Range Normalization • Limit the metric values to a fixed, predetermined range – Amplitude Normalization • Normalize the difference between calculated and measured values • Weighting – 1) Multipliers to assign different importance to different parts of the calculation, e.g., the waveform – 2) Multipliers to assign different quality to different experimental data 4 Making the World Safer
Validation Metric Objectives • Permit an assessment of the state of the art of each of the theoretical predictions of interest to the ACTD customer • Provide a clear and unbiased means of comparing different calculations – Ground rules established in advance of comparison • Reproduce the results of truly expert judgment comparisons – A validation metric should not be a replacement for expert judgment • Define and quantify the expert judgment process – What is important – What is the relative importance of different aspects • Include consideration of experimental data uncertainties if appropriate • Provide an audit trail for the assessment process 5 Making the World Safer
What Aspects of the Calculations are Important? • Emphasis should be on what is important for the customer – Avoid concentrating on interesting nuances • Key Issues – Peak environments (velocity, displacement, strain) – Tunnel damage (rubble depth, rubble velocity, material damage) • Significant issues – Wave shape near the peak – Significant TOA (shock, peak velocity, reliefs, reflections, etc.) • Unimportant Issues – Very low amplitude TOA resulting from numerics – Wave shape in the tail – High frequency content (unless peak accelerations are of interest) • Different problems may require different emphasis 6 Making the World Safer
Example of TOA Issues • Calculation “a” compares well with data although calculation “b” may be closer in many metrics • Difference in TOA is of no significance for predicting tunnel damage • While we probably want to favor a computation that gets a good approximation to the TOA, we propose to evaluate the TOA separately from the waveshape in making the comparison 7 Making the World Safer
Is a Single Metric Sufficient? • Single metric cannot capture complex aspects of importance to the customer – Database of tunnel damage insufficient for direct comparison – Must calculate environment and predict tunnel damage – Multiple aspects of environments must be considered to have confidence in the predictions • Must compare code results with what can be reliably measured • Important to tie into existing database – IGVN approach 8 Making the World Safer
Relative Importances – a proposal (scale of 0-10) • Key Issues – (10) Peak environments (velocity, displacement, strain) – (10) Tunnel damage (rubble depth, rubble velocity, material damage) • Significant issues – (5) Wave shape near the peak – (2) Significant TOA (shock, peak velocity, reliefs) • Unimportant Issues – (0) Very low amplitude TOA resulting from numerics – (1) Wave shape in the tail – (1) High frequency content (higher if peak accelerations are of interest) We propose to measure each component separately and combine them using these weights (or others to be decided). 9 Making the World Safer
For the waveshape metric we propose a weighted sum of squares N ∑ ⎡ ⎤ = • − 2 M w ( c m ) / Normalization ⎣ ⎦ i i i = i 0 m t c Where is the measured value of a function at time and is the i i i computed value at the corresponding time*. w Normalization is a divisor to be discussed later, and is a weight, also to i be discussed later. Important points about weights : 1. They depend on m only, not on c , so that same weights are used for comparisons of all computations 2. They will be chosen to give greater weight in regions of large m since those are consdiered more important to failure modes. * - the corresponding time is a time shifted to remove time of arrival differences so that we are measuring only the waveshape differences in this calculation. 10 Making the World Safer
What is the Rationale for the Sum of the Squares? If M (measurement) and C (computation) are data from same model, we expect to see close values more often than very different values: Probability of difference Difference d Much more Much less close IF each measurement contains the sum of many small errors due to various causes (rock variations, instrumentation, …) its error is Gaussian (“law of large numbers”). Probability looks like A. exp(- d 2 / σ 2 ) where σ is an estimate of how much difference, d , we might expect (the standard deviation). If we have many measurements, d i , each with its own σ i , joint probability is proportional to the product exp[-( d 1 / σ 1 ) 2 ] exp[-( d 2 / σ 2 ) 2 ] … exp[-( d n / σ n ) 2 ] = exp[- Σ ( d i / σ i ) 2 ] Thus, SS = Σ ( d i / σ i ) 2 is an appropriate combination of the various differences. Zero means “perfect” and infinity means “perfectly awful.” 11 Making the World Safer
Why the Sum of the Squares - continued IF we had statistical information about the distributions of experimental errors, and of data such as rock characteristics, I would argue that we should stay with a metric that ranges from 0 to infinity, normalizing it by the known/estimated standard deviations. This would give us information that could be used in Bayesian updating of probabilities of damage. Then, if you really want the range [0,1] you can take the exponential of its negative to get a probability. However, in this case, we do not have any statistical information, so I will propose a normalization in which a metric value of 0.4 means a 40% deviation from the waveform – that is, if the calculated waveform is exactly 60% of the measured, or exactly 140%, the metric will be 0.4 (and similarly for other values). 12 Making the World Safer
Weighting of Experimental Data • Better data should count more than poor data • Currently data quality determined by expert judgment – Usually done by the experimenter – Based on cleanness of the data, observed scatter, etc. • For now, consider weighting experimental data points to reflect data quality & quantity – Give each data used in a metric (i.e. the waveform from a gage, or the TOA data from a gage) a rating 0 ≤ Q with larger numbers meaning better quality/quanity of data • More post shot data analysis is desirable – e.g., better comparisons of data from different gages – Could improve quality and assessment of data 13 Making the World Safer
The Waveform Metric We compute the following: = (maximum measured) max( ) m m max i i p ⎛ ⎞ N m ∑ = − = = i ⎜ ⎟ Q ( t t ) with t t and t t + − + (Qual…ntity) i 1 i 1 0 1 M 1 M ⎝ m ⎠ = i 1 max p ⎛ ⎞ N m ∑ = − i ⎜ ⎟ 2 ( ) ( ) QS m t t + − QS (Q scaled factor) i i 1 i 1 ⎝ ⎠ m = i 1 max p ⎛ ⎞ N m ∑ = − − i 2 ⎜ ⎟ Metric value M W M ( m c ) ( t t ) / QS + − W i i i 1 i 1 ⎝ m ⎠ = i 1 max M W = 0 if waveforms agree (after TOA shift). M W = 1 if c = 0 or c = 2 m. For now, I recommend a value p = 1. This places more importance to larger m regions The sum is an approximation to twice the integral, so it is nearly time spacing independent. Q is the area under the graph of (| m |/m max )2 . If we had statistical knowledge of data quality, that should also be factored in. QS is chosen such that the metric for c = 0 is 1. 14 Making the World Safer
Combining multiple waveform metrics If we have a set of waveforms and have evaluated their metrics, M Wj , j = 1, 2, …, m and the corresponding Q j values, we compute the combined metric and Q value as: = ∑ m this means we have more data Q Q j as the basis for our metric = j 1 = ∑ m This is a weighted average 2 M Q M / Q W j Wj = j 1 Similar approach applies to combination of any metrics of same type 15 Making the World Safer
Recommend
More recommend
