Statistical Tests Amanda Stathopoulos amanda.stathopoulos@epfl.ch Transport and Mobility Laboratory, School of Architecture, Civil and Environmental Engineering, Ecole Polytechnique F´ ed´ erale de Lausanne Transport and Mobility Laboratory Decision-Aid Methodologies 1 / 65
Outline Introduction 1 Case study 2 Informal tests t-tests Likelihood ratio test 3 Test of generic attributes Test of taste variation Test of nonlinear specifications 4 Piecewise linear specification Power series Box-Cox Non nested hypotheses 5 Cox test Adjusted likelihood ratio index Further tests 6 Outlier analysis Market segments Appendix 7
Introduction Introduction Modeling Difficult to determine the most appropriate model specification A good fit does not imply a good model Formal testing is necessary, but not sufficient No clear-cut rules can be given Good modeling = good (subjective) judgment + good analysis Wilkinson (1999) “The grammar of graphics”. Springer ... some researchers who use statistical methods pay more attention to goodness of fit than to the meaning of the model... Statisticians must think about what the models mean, regardless of fit, or they will promulgate nonsense. Transport and Mobility Laboratory Decision-Aid Methodologies 2 / 65
Introduction Introduction Hypothesis testing Four steps step 1: State the hypotheses H 0 null hypothesis H 1 alternative hypothesis step 2: Set the criteria for a decision step 3: Compute a test statistic step 4: Make a decision Step 1: Analogy with a court trial H 0 : defendant is “presumed innocent until proved guilty” H 0 is accepted, unless the data argue strongly to the contrary Transport and Mobility Laboratory Decision-Aid Methodologies 3 / 65
Introduction Introduction Step 2: Criterion for a decision Court-room: criterion is to show guilt beyond reasonable doubt Implies defining the level of significance α Step 3: Test statistic Determine the likelihood of obtaining a sample outcome if the H 0 hypothesis were true How far we accept to be from the H 0 Step 4: Decide Decide if null is retained or rejected Gives the probability value (p-value of obtaining an outcome, given that the H 0 is true) Transport and Mobility Laboratory Decision-Aid Methodologies 4 / 65
Introduction Introduction Possible decision outcomes Accept H 0 Reject H 0 H 0 is true Correct (1- α ) Type I error (prob. α ) H 0 is false Type II error (prob. β ) Correct (1- β ) Relations For a given sample size N , there is a trade-off between α and β . Only way to reduce both types of error probabilities is to increase N . π = 1 − β is the power of the test, that is, the probability of correctly rejecting H 0 . Researcher directly controls Type I errors by fixing α Transport and Mobility Laboratory Decision-Aid Methodologies 5 / 65
Outline Introduction 1 Case study 2 Informal tests t-tests Likelihood ratio test 3 Test of generic attributes Test of taste variation Test of nonlinear specifications 4 Piecewise linear specification Power series Box-Cox Non nested hypotheses 5 Cox test Adjusted likelihood ratio index Further tests 6 Outlier analysis Market segments Appendix 7
Case study Summary of case-studies Netherlands mode choice Swissmetro Intercity travelers Travelers St. Gallen - Geneva Choice between train & car Choice between train, car & 228 respondents swissmetro Revealed preference data 441 respondents with self-reported trip characteristics Stated preference (swissmetro is a non-existing mag-lev train) Transport and Mobility Laboratory Decision-Aid Methodologies 6 / 65
Case study Informal tests Informal tests Sign of the coefficient Do the estimated parameters have the right sign? Example: Netherlands Mode Choice Case Robust Coeff. Asympt. Parameter estimate std. error t -stat p -value ASC car -0.798 0.275 -2.90 0.00 β cost -0.0499 0.0107 -4.67 0.00 β time -1.33 0.354 -3.75 0.00 Transport and Mobility Laboratory Decision-Aid Methodologies 7 / 65
Case study Informal tests Informal tests Value of trade-offs Are the trade-offs reasonable? How much are we ready to pay for a marginal improvement of the level-of-service? Example: reduction of travel time The increase in cost must be exactly compensated by the reduction of travel time β cost ( C + ∆ C ) + β time ( T − ∆ T ) + . . . = β cost C + β time T + . . . Therefore, ∆ T = β time ∆ C β cost Transport and Mobility Laboratory Decision-Aid Methodologies 8 / 65
Case study Informal tests Informal tests Value of trade-offs: example with Netherlands data In general: ∂ V /∂ x Trade-off: ∂ V /∂ x C 1 / Hour 1 / Guilder = Guilder Units: Hour Parameter Coeff. Guilders Euros CHF ASC car -0.798 15.97 7.25 11.21 β cost -0.0499 β time -1.33 26.55 12.05 18.64 (/Hour) Transport and Mobility Laboratory Decision-Aid Methodologies 9 / 65
Case study t-tests t -test Question Is the parameter θ significantly different from a given value θ ∗ ? H 0 : θ = θ ∗ H 1 : θ � = θ ∗ Statistic Under H 0 , if ˆ θ is normally distributed with known variance σ 2 ˆ θ − θ ∗ ∼ N (0 , 1) . σ Therefore ˆ θ − θ ∗ P ( − 1 . 96 ≤ ≤ 1 . 96) = 0 . 95 = 1 − 0 . 05 σ Transport and Mobility Laboratory Decision-Aid Methodologies 10 / 65
Case study t-tests t -test H 0 can be rejected at the 5% level ( α = 0 . 05) if � � � � ˆ θ − θ ∗ � � � � � ≥ 1 . 96 . � σ Comments If ˆ θ asymptotically normal If variance unknown A t test should be used with n degrees of freedom. When n ≥ 30, the Student t distribution is well approximated by a N (0 , 1) Transport and Mobility Laboratory Decision-Aid Methodologies 11 / 65
Case study t-tests t -test Swissmetro: model specification Car Train Swissmetro ASC car 1 0 0 ASC train 0 1 0 β cost cost cost cost β time time time time β headway 0 headway headway Transport and Mobility Laboratory Decision-Aid Methodologies 12 / 65
Case study t-tests t -test Swissmetro: coefficient estimates Robust Parameter Coeff. Asympt. number Description estimate std. error t -stat p -value 1 ASC car -0.262 0.0615 -4.26 0.00 2 ASC train -0.451 0.0932 -4.84 0.00 3 β cost -0.0108 0.000682 -15.90 0.00 4 β headway -0.00535 0.000983 -5.45 0.00 5 β time -0.0128 0.00104 -12.23 0.00 H 0 : β cost = 0: rejected at the 5% level H 0 : β headway = 0: rejected at the 5% level H 0 : β time = 0: rejected at the 5% level Transport and Mobility Laboratory Decision-Aid Methodologies 13 / 65
Case study t-tests t -test Comparing two coefficients H 0 : β 1 = β 2 . The t statistic is given by β 1 − � � β 2 � var( � β 1 − � β 2 ) var( � β 1 − � β 2 ) = var( � β 1 ) + var( � β 2 ) − 2 cov( � β 1 , � β 2 ) Transport and Mobility Laboratory Decision-Aid Methodologies 14 / 65
Case study t-tests t -test Comparing two coefficients Example: alternative specific or generic coefficients? Below alternative specific time Car Train Swissmetro ASC car 1 0 0 ASC train 0 1 0 β cost cost cost cost β time car time 0 0 β time train 0 time 0 β time Swissmetro 0 0 time β headway 0 headway headway Transport and Mobility Laboratory Decision-Aid Methodologies 15 / 65
Case study t-tests t -test Swissmetro: coefficient estimates (alternative specific time) Robust Parameter Coeff. Asympt. number Description estimate std. error t -stat p -value 1 ASC car -0.371 0.120 -3.08 0.00 2 ASC train 0.0429 0.121 0.36 0.72 3 β cost -0.0107 0.000669 -16.00 0.00 4 β headway -0.00532 0.000994 -5.35 0.00 5 β time car -0.0112 0.00109 -10.28 0.00 6 β time Swissmetro -0.0116 0.00182 -6.40 0.00 7 β time train -0.0156 0.00109 -14.29 0.00 Transport and Mobility Laboratory Decision-Aid Methodologies 16 / 65
Case study t-tests t -test Variance-covariance matrix Parameter 1 Parameter 2 Covariance Correlation t -stat β time car β time train 7.57e-07 0.634 4.70 β time car β time Swissmetro 1.38e-06 0.696 0.31 β time Swissmetro β time train 1.47e-06 0.740 3.19 H 0 : β time car = β time train var( � β t . car − � var( � β t . car ) + var( � β t . train ) − 2 cov( � β t . car , � β t . train ) = β t . train ) 1 . 188 × 10 − 6 + 3 . 312 × 10 − 06 − 2 × 7 . 570 × 10 − 07 = 8 . 622 × 10 − 07 = Transport and Mobility Laboratory Decision-Aid Methodologies 17 / 65
Case study t-tests t -test H 0 : β time car = β time train β t . car − � � β t . train = − 0 . 0112 − ( − 0 . 0156) � √ = 4 . 739 8 . 622 × 10 − 07 var( � β t . car − � β t . train ) We can reject the H 0 of parameter equality What about β time car = β time metro and β time metro = β time train ? Homework to calculate the t-ratios for these parameter differences! Transport and Mobility Laboratory Decision-Aid Methodologies 18 / 65
Outline Introduction 1 Case study 2 Informal tests t-tests Likelihood ratio test 3 Test of generic attributes Test of taste variation Test of nonlinear specifications 4 Piecewise linear specification Power series Box-Cox Non nested hypotheses 5 Cox test Adjusted likelihood ratio index Further tests 6 Outlier analysis Market segments Appendix 7
Recommend
More recommend
