Testing the rationality of categorical predictions Carlos Madeira - PowerPoint PPT Presentation

Testing the rationality of categorical predictions Carlos Madeira Central Bank of Chile May, 2017 Carlos Madeira (Central Bank of Chile) Rationality of categorical predictions May, 2017 1 / 18

Introduction Data on subjective expectations is now common, especially in education studies (Dominitz and Manski, 1996, Manski, 2004, Zafar, 2011, Attanasio and Kaufmann, 2014, Giustinelli, 2016). In the last 20 years studies have used more accurate measures of subjective expectations (Delavande and Rohwedder, 2008), including its central tendency (median, mode, mean) and uncertainty (Std, IQR, IDR). However, many datasets still use less formal or less precise measures of beliefs, including point predictions and qualitative statements (Manski, 2004). Some surveys are already quite long and exhaustive, but need some info on beliefs that may affect future actions/investments - see Michigan Survey of Consumers (MSC) or the ECB’s Household Finance and Consumption Survey (HFCS). In asset portfolio choice, studies show that retail investors are unable to use standard-errors and other uncertainty measures, but can rank risk choices (Hackethal and Inderst, 2011). Carlos Madeira () Rationality of categorical predictions May, 2017 2 / 18

This work I build a test of rationality for categorical predictions under the assumption that agents provide their subjective mode or a fixed-quantile. Manski (1990) already provides a test of rationality for binary outcomes using a moment inequality and assuming agents report the median (=mode) outcome. Das, Dominitz and Soest (1999) extend Manski’s analysis to the general case of multiple ordered categorical expectations, under a subjective mode or a fixed-quantile, showing it requires testing several moment inequalities. DDS (1999) do not provide a statistical test of whether these inequalities jointly hold. I use a multiple moment inequality statistic which is asymptotically distributed as a weighted chi-square distribution (Kudo, 1963, Wolak, 1987, Rosen 2008) when all the inequalities bind and rejects the null-hypothesis with a confidence level equal or smaller than θ . Carlos Madeira () Rationality of categorical predictions May, 2017 3 / 18

First: Test against a prediction rule using only observables i = g ( X i , ε p Let Y i be the actual outcome, P R i ) the respondents’ guess and P i = f ( X i ) an alternative prediction rule f ( . ) . Outcomes Y are ordered: 1 , ..., K . X i is observable and ε p i is private information. If respondents use the loss function L ( . ) then rationality demands: 1.1) H 0 : µ ≡ E [ L ( Y i , P i = f ( X i )) − L ( Y i , P R i ) | X i ] ≥ 0. The null hypothesis of rationality H 0 : µ = 0 vs. H 1 : µ < 0 is tested by: � 1 � N ∑ N i = 1 L ( Y i , P i = f ( X i )) − L ( Y i , P R 1.2) t = √ n i ) → N ( 0 , 1 ) , ˆ Std ( L ( Y i , P i = f ( X i )) − L ( Y i , P R i )) Valid under three assumptions. A.1) Y i , P R are independent across i i (which excludes aggregate shocks); A.2) the loss function L ( . ) is known both by the agents and the econometrician; A.3) the test is only valid against a specific prediction rule given by f ( X i ) . In a sense this simple test treats the categorical outcomes as numeric values. Carlos Madeira () Rationality of categorical predictions May, 2017 4 / 18

Test of a fixed-quantile rationality hypothesis A.1) plus less restrictive version of A.2), which is that outcomes are ordered. α -quantile rationality implies: 2.1) c 1 > 0, with c 1 ≡ α − Pr ( Y i ≤ k − 1 | X i , P R i = k ) , 2.2) c 2 ≥ 0, with c 2 ≡ ( 1 − α ) − Pr ( Y i ≥ k + 1 | X i , P R i = k ) . Let P j | k = Pr ( Y i = j | X i , P R i = k ) and its estimate P j | k = ∑ N i = 1 1 ( Y i = j , x i , P R i = k ) , where n k = ∑ N ˆ i = 1 1 ( x i , P R i = k ) . n k Statistics for testing c 1 > 0 and c 2 ≥ 0 separately are: 2.3) t 1 ( α − quantile) = √ n k ( α − ∑ k − 1 j = 1 ˆ P j | k − c 1 ) , 2.4) t 2 ( α − quantile) = √ n k (( 1 − α ) − ∑ K j = k + 1 ˆ P j | k − c 2 ) . t = [ t 1 , t 2 ] ∼ N ( 0 , Σ ∗ ) . A valid asymptotic test of H 0 : c 1 > 0 , c 2 ≥ 0 is provided by W = inf c [ t � ( ˆ Σ ∗ ) − 1 t st c ≥ 0 ] , with H 0 rejected if W > ¯ χ θ 1 . Carlos Madeira () Rationality of categorical predictions May, 2017 5 / 18

Test of a subjetive mode rationality The mode can be applied to purely qualitative outcomes. 3.1) λ j | k ≥ 0, with λ j | k ≡ Pr ( Y i = k | x i , P R i = k ) − Pr ( Y i = j | x i , P R i = k ) , ∀ j � = k , H 0 ) λ ( k ) = [ λ 1 | k , .., λ K | k ] ≥ 0 can be tested using: 3.2) t ( j | k ) = √ n k ( ˆ P k | k − ˆ P j | k − λ j | k ) , j � = k . t ( k ) ≡ [ t ( 1 | k ) , .., t ( K | k )] ∼ N ( 0 , Σ ∗∗ ) . A valid asymptotic test of H 0 : λ ( k ) ≥ 0 is provided by W ( k ) = inf λ ( k ) [ t ( k ) � ( ˆ Σ ∗∗ ) − 1 t ( k ) st λ ( k ) ≥ 0 ] , χ θ with H 0 rejected if W ( k ) > ¯ K − 1 . Carlos Madeira () Rationality of categorical predictions May, 2017 6 / 18

Multiple inequalities imply non-normal asymptotics χ θ ¯ K − 1 is a critical value of a weighted average of chi-square distributions of degree 0 to K − 1, with weights w ( K − 1 , h , Σ ∗∗ ) in Wolak (1987): χ θ K − 1 ) = ∑ K − 1 h = 0 w ( K − 1 , h , Σ ∗∗ ) Pr ( χ 2 χ θ Pr ( ¯ χ K − 1 > ¯ h > ¯ K − 1 ) . The asymptotic Type I error of this test is equal to or smaller than θ . W ( k ) is asymptotically distributed as ¯ χ K − 1 only when H 0 is valid and all the inequalities are binding. If H 0 is valid and only m inequalities of the vector λ ( k ) bind (with m < K − 1), then W ( k ) is asymptotically distributed as ¯ χ m and the test has a Type I error smaller than θ (Rosen, 2008). If H 0 is valid and no inequalities bind, then W ( k ) goes asymptotically to 0. Carlos Madeira () Rationality of categorical predictions May, 2017 7 / 18

Monte Carlo Study: Scenarios Scenarios to test whether the Mode or Median is rejected as the probability of each outcome given a forecast, Pr ( Y i = j | P R i = k ) P R Scenario i = k / Y i = j j =1 2 3 4 H0 valid? Yes/No: Binding constraints Mode Median 1 k = 1 0.50 0.35 0.10 0.05 Yes: 0 Yes: 1 2 2 0.25 0.25 0.25 0.25 Yes: 3 Yes: 1 3 3 0.05 0.20 0.25 0.50 No: 0 Yes: 1 4 4 0.05 0.15 0.40 0.40 Yes: 1 No: 0 Others: j = 1 2 3 4 5 k = 2 0.15 0.35 0.35 0.15 Yes: 1 Yes: 1 6 2 0.30 0.35 0.25 0.10 Yes: 0 Yes: 0 7 3 0.10 0.35 0.30 0.25 No: 0 Yes: 0 Carlos Madeira () Rationality of categorical predictions May, 2017 8 / 18

Type I error Mode: 10,000 MC samples Type I error Mode (%): θ ≤ 5% Multi-condition test Das-Dominitz-Soest test Scenario N=25 50 100 250 N=25 50 100 250 1 0.3% 0.1% 0.0% 0.0% 1.3% 0.4% 0.1% 0.0% 2 7.7% 6.1% 5.5% 5.1% 19.6% 17.9% 15.6% 15.0% 3 NA NA 4 2.7% 2.2% 2.2% 2.2% 8.2% 7.0% 6.7% 6.2% 5 2.7% 2.0% 1.8% 2.1% 9.2% 7.4% 6.9% 6.1% 6 1.9% 1.0% 0.5% 0.1% 6.0% 3.7% 1.9% 0.7% 7 NA NA Carlos Madeira () Rationality of categorical predictions May, 2017 9 / 18

Type I error Median: 10,000 MC samples Type I error Median (%): θ ≤ 5% Multi-condition test Scenario N=25 50 100 250 500 1 5.5% 6.0% 4.2% 5.6% 5.4% 2 5.8% 5.8% 4.5% 5.8% 4.8% 3 5.2% 6.0% 4.7% 6.1% 5.1% 4 NA 5 5.5% 5.8% 4.5% 5.7% 4.8% 6 0.1% 0.0% 0.0% 0.0% 0.0% 7 1.8% 1.1% 0.5% 0.1% 0.0% Multi-condition is only slightly better than the DDS test and only in samples lower than 25 observations. Carlos Madeira () Rationality of categorical predictions May, 2017 10 / 18

Application: The Beginnning School Study (BSS) Extensive panel data on expectations from the BSS (1982-2002). 838 families selected with children in first-grade in 1982 from 20 Baltimore Public Schools. On average 80% of the sample is observed in each year. Stratified selection (no expansion factors/survey weights given). Poor socioeconomic background of the BSS families. Expectations and outcomes of academic scores. Others: time studying, education and occupation as adults. "Aspirations" and "Best Guesses" are separate. Outcomes are reported directly from administrative sources (schools). Parents interviewed at home or by phone. Children in school. Parent (Fall) and Student (Fall, Spring) interviews done at the beginning of the academic quarter. Teacher interviews at the end of the year. Carlos Madeira () Rationality of categorical predictions May, 2017 11 / 18

Maternal age and education Table 2.1: Age and education level of students’ parents Age at birth mother Education mother 10-19 28.1% Grade 9 or Less 18.9% 20-25 40.4% Some High-school 19.7% 26-30 20.7% Graduate High-school/ GED 34.2% 31-35 8.1% Some College 17.0% 36-52 2.7% Finish 4-year College 6.1% Missing 10.5% Advanced Degree 4.1% Missing 6.6% Carlos Madeira () Rationality of categorical predictions May, 2017 12 / 18