Alecos Papadopoulos PhD Candidate (supervisor: Prof. Pl. Sakellaris) - PowerPoint PPT Presentation

Economics Research Workshop March 2017 Making sense of “nonsense probabilities”: binary choice models when the underlying error term has bounded support. Alecos Papadopoulos PhD Candidate (supervisor: Prof. Pl. Sakellaris) (N ΟΤΕ : this piece of research is not PhD-related) Athens University of Economics and Business School of Economic Sciences / Dpt of Economics papadopalex@aueb.gr https://alecospapadopoulos.wordpress.com/ 1 of 51

Intro & Motivation • Binary Choice, Linear Probability Model (LPM):           x β Pr , 0,1 y x E y x y i i i i i i Usually treated as • a linear approximation to the true data generating mechanism • a device to showcase the problems with OLS estimation in models with Limited Dependent variables • Prominent among these problems : we may obtain “nonsense probabilities”     ˆ ˆ   Pr y x 1, Pr y x 0 i i i i Alecos Papadopoulos Binary choice and nonsense probabilities (March 2017) 2 of 51

Intro & Motivation • Provide a proper statistical foundation for the LPM • It becomes the true/correct specification • “Nonsense probabilities” become a useful indication of the structure of the underlying population • For the LPM this foundation is : the error term in the underlying latent regression follows a Uniform distribution (Amemiya 1981, p. 1489). • The Uniform distribution has inherently a bounded support . • The bounded support alone is the reason for the appearance of “nonsense probabilities”. Alecos Papadopoulos Binary choice and nonsense probabilities (March 2017) 3 of 51

Intro & Motivation The General Binary Choice model with bounded error Latent regression     y g x u , i i i               u F u , u , , , 0, E u 0 x x i i i i 1 2 1 2 i i     0 u i 1          F u F          i 1  F u u ,         i i 1 2 F F  2 1     1 u i 2 Alecos Papadopoulos Binary choice and nonsense probabilities (March 2017) 4 of 51

Intro & Motivation     y I y 0 Define Then i i               Pr y 1 x Pr y 0 x Pr g x u 0 x i i i i i i i        Pr u g x x i i i             Pr y 1 1 Pr u g x x x i i i i i       u , Since the support is bounded, we have i 1 2                   g x Pr u g x x 1 Pr y 1 x 0 i 2 i i i i i                    g x Pr u g x x 0 Pr y 1 x 1 1 i i i i i i Alecos Papadopoulos Binary choice and nonsense probabilities (March 2017) 5 of 51

Intro & Motivation Complete model at the binary-variable level when the support of the underlying error term is bounded:        1 g x i 1                      Pr 1 y x 1 F g x g x i i i 1 i 2         0 g  x i 2 Is this a useful model for Econometrics? Alecos Papadopoulos Binary choice and nonsense probabilities (March 2017) 6 of 51

Intro & Motivation It reflects a population partitioned in three subgroups: • In the familiar “middle” one, the underlying disturbance affects and co-determines the binary choice as usual. No matter how strongly the systematic part points towards the one direction, the realization of the disturbance may even “overturn” the binary choice towards the other direction. • In the two “extreme ” subgroups the underlying disturbance does not affect the binary choice – only the systematic factors do . • This does not imply individuals with different preferences • But it does imply that for some individuals , their “core relevant” characteristics are so strongly realized, that nothing else matters for the binary choice. Alecos Papadopoulos Binary choice and nonsense probabilities (March 2017) 7 of 51

Intro & Motivation Is it plausible and useful? • It is a plausible real-world situation, a recognition that non-negligible extremes may exist. • It is useful for businesses which always look to segment the market to either tailor their products or increase efficiency in their marketing campaigns. • It is useful for policy , which for socio-political reasons attempts in many cases to take into account small but visible subsets of the population. Alecos Papadopoulos Binary choice and nonsense probabilities (March 2017) 8 of 51

Intro & Motivation NONSENSE PROBABILITIES and what do they tell us Assume that the complete model holds. Initially we can only estimate the incomplete specification           Pr y 1 x 1 F g x i i i   ˆ y   Suppose that for some observation we obtain Pr 1 x 1 i i Then, however imperfectly, the estimator indicates that           F g x F          1  i 1 F g x 1 1 1         i F F 2 1                   F F g x g x 1 i 1 i   y   Pr 1 x 1 But this is the condition to have i i Alecos Papadopoulos Binary choice and nonsense probabilities (March 2017) 9 of 51

Intro & Motivation The result depends solely on the very existence of a bounded support for the error term , and NOT on • whether the specification is linear • whether the distribution of the error term is symmetric around zero • whether it is symmetrically bounded/truncated around zero. So not something specific to the Uniform distributional assumption.   F  could e.g. be a Truncated Normal, or a Truncated Logistic distribution, or some non-symmetric distribution Alecos Papadopoulos Binary choice and nonsense probabilities (March 2017) 10 of 51

Intro & Motivation As regards estimated probabilities, the statistically justified action is to set the conditional probability equal to 1 or 0 , whenever we obtain “nonsense probabilities”. (has been proposed as an “ad hoc” correction in the LPM) The whole matter would end here… but by misspecifying the model the estimator becomes inconsistent. To see this we re-cast the model as a mixture. Alecos Papadopoulos Binary choice and nonsense probabilities (March 2017) 11 of 51

Intro & Motivation First, Scope restriction: Assume for the remainder   g x • is affine i            • (symmetric around 0) 1 F g x F g x i i      • (symmetrically truncated around zero) 1 2 MIXTURE MODEL FORMULATION                   I I g x , I I g x , E I p i r i , i 1, i i r i , r Then, true model can be written           1,      Pr y 1 x E y x p F g x 1 p I i i i i r i r i Alecos Papadopoulos Binary choice and nonsense probabilities (March 2017) 12 of 51

Intro & Motivation MIXTURE MODEL FORMULATION           1,      Pr y 1 x E y x p F g x 1 p I i i i i r i r i         Pr 1 y x F g x Specifying i i i         Pr y 1 x F g x or worse (untruncated) i i i leads to inconsistency. • This is due to the existence of the observations from the two extreme subgroups. • These obs are “irrelevant” as regards the estimation of the unknown coefficients. • We need to somehow separate and discard them, and use only the “relevant” ones. Alecos Papadopoulos Binary choice and nonsense probabilities (March 2017) 13 of 51

An MM algorithm to clean the sample A natural idea is to use what the estimator signals to us: • Start by (mis)specifying a model using a bounded/truncated distribution • Discard those observations for which we obtain “nonsense probabilities”. • Re-estimate • Discard any additional observations labeled “irrelevant” given the new estimates • etc. up until no new observations are discarded. …but we are using a misspecified model, and an inconsistent estimator… Can we trust the algorithm ? We can. If we use least-squares estimation. Alecos Papadopoulos Binary choice and nonsense probabilities (March 2017) 14 of 51