Estimating Recall the general mean-variance specification E( Y | x - PowerPoint PPT Presentation

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Estimating θ Recall the general mean-variance specification E( Y | x ) = f ( x , β ) , var( Y | x ) = σ 2 g ( β , θ , x ) 2 . We have, so far, considered θ as a known constant. In many cases, we instead need to estimate it. 1 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response One approach is to continue to use the Gaussian likelihood: n n { Y j − f ( x j , β ) } 2 log g ( β , θ , x j ) − 1 � � − n log σ − σ 2 g ( β , θ , x j ) 2 . 2 j =1 j =1 Differentiating w.r.t. θ leads to { Y j − f ( x j , β ) } 2 − σ 2 g ( β , θ , x j ) 2 � � n � ν θ ( β , θ , x j ) = 0 . σ 2 g ( β , θ , x j ) 2 j =1 2 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Here ν θ ( β , θ , x j ) = ∂ log g ( β , θ , x j ) = g θ ( β , θ , x j ) g ( β , θ , x j ) . ∂ θ For consistency with earlier estimating equations, rewrite as { Y j − f ( x j , β ) } 2 − σ 2 g ( β , θ , x j ) 2 � � n 2 σ 2 g ( β , θ , x j ) 2 ν θ ( β , θ , x j ) � 2 σ 4 g ( β , θ , x j ) 4 j =1 = 0 . 3 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Combining with the σ 2 equation derived earlier, { Y j − f ( x j , β ) } 2 − σ 2 g ( β , θ , x j ) 2 � � n � 2 σ 4 g ( β , θ , x j ) 4 j =1 2 σ g ( β , θ , x j ) 2 � � × = 0 . 2 σ 2 g ( β , θ , x j ) 2 ν θ ( β , θ , x j ) 4 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response The σ 2 and θ equations may be put in the standard form n � D T j V − 1 ( s j − m j ) = (( q +1) × 1) . 0 j j =1 We need s j = { Y j − f ( x j , β ) } 2 m j = σ 2 g ( β , θ , x j ) 2 � T 2 σ g ( β , θ , x j ) 2 � D j = 2 σ 2 g ( β , θ , x j ) 2 ν θ ( β , θ , x j ) V j = 2 σ 4 g ( β , θ , x j ) 4 . 5 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Since β is also unknown, we must also solve the corresponding equation n { Y j − f ( x j , β ) } f β ( x j , β ) � σ 2 g ( β , θ , x j ) 2 j =1 n � � { Y j − f ( x j , β ) } 2 � + − 1 ν β ( β , θ , x j ) = 0 . σ 2 g ( β , θ , x j ) 2 j =1 In principle, we must solve the ( q + 1) variance parameter equations and the p β equations jointly . 6 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Joint Estimating Equations Gaussian ML: 2 σ 2 g 2   � σ 2 g 2 f β j j ν β j � − 1 � n � 1 /σ 0 � Y j − f j � j � = 0 . 2 σ 4 g 4 ( Y j − f j ) 2 − σ 2 g 2  2 σ 2 g 2  0 0 j j j ν θ j j =1 GLS: � σ 2 g 2   f β j 0 n � − 1 � � 1 /σ 0 Y j − f j � � � j = 0 . ( Y j − f j ) 2 − σ 2 g 2  2 σ 2 g 2  2 σ 4 g 4 0 0 j j j ν θ j j =1 7 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Remarks Solve ( p + q + 1) estimating equations jointly . Approach in GLS is called pseudo-likelihood (PL) approach; that is, some parameters are estimated by ML while others are estimated by ad hoc but consistent methods. — GLS-PL When g ( · ) does not depend on β but does involve unknown θ , ν β ( β , θ , x j ) = 0 , and the normal theory ML and GLS equations coincide. 8 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Implementation Write estimating equations in this form, then solve using an idea similar to IRWLS n � D T j ( α ) V − 1 j ( α ) { s j ( α ) − m j ( α ) } = 0 j =1 where j ( α ) = ∂ D T ∂ α m j ( α ) . 9 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Two strategies Solve the entire set of parameters together:   β  . α = σ  θ Iterate between: � σ � Given α = , solve for β ; θ Given β , solve for α . 10 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Implementation Use Taylor series approximation, α ≈ α ∗ + � − 1 D T ( α ∗ ) V − 1 ( α ∗ ) D ( α ∗ ) � D T ( α ∗ ) V − 1 ( α ∗ ) { s ( α ∗ ) − m ( α ∗ ) } where V ( α ) = block diag { V 1 ( α ) , . . . , V n ( α ) } , and D T ( α ) = { D T 1 ( α ) , . . . , D T n ( α ) } 11 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Iterative update � − 1 � D T ( a ) V − 1 D T ( a ) V − 1 � � α ( a +1) = α ( a ) + ( a ) D ( a ) s ( a ) − m ( a ) ( a ) Remark: The ML or PL quadratic equations are more unstable than are the linear equations, and can be ill-behaved in practice. 12 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response A General Class of Estimators of θ Motivation Estimating equations for θ would be based on the residuals { Y j − f ( x j , β ) } . For squared residuals, the effect of “outlying” or “unusual” observations can be magnified. Other functions of the residuals might be considered. 13 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response 14 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Recall The standardized residual: ǫ j = Y j − f ( x j , β ) σ g ( β , θ , x j ) Box-Cox transformation: u λ − 1  λ � = 0  h ( u , λ ) = λ log | u | λ = 0 .  Key idea Consider forming estimating equations based on general power transformations of absolute residuals. 15 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Key assumption The appropriate moments of ǫ j are not dependent on x j and are constant for all j : � x j � | ǫ j | λ � � � | ǫ j | λ � E = E = constant ∀ j , � x j � | ǫ j | 2 λ � � � | ǫ j | 2 λ � E = E = constant ∀ j . Definitions Define η such that e λη = σ λ E ( | ǫ j | λ ) Identify | Y j − f ( x j , β ) | λ as the “response”. 16 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response It follows that � x j � | Y j − f ( x j , β ) | λ � � = σ λ g ( β , θ , x j ) λ E � | ǫ j | λ � = e λη g ( β , θ , x j ) λ E and | Y j − f ( x j , β ) | λ � � x j = σ 2 λ g ( β , θ , x j ) 2 λ var | ǫ j | λ � � � � var = σ 2 λ g ( β , θ , x j ) 2 λ � | ǫ j | λ � 2 � | ǫ j | 2 λ � � � E − E � σ 2 λ E � | ǫ j | 2 λ � − e 2 λη � g ( β , θ , x j ) 2 λ = 17 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Estimating equations � | Y j − f ( x j , β ) | λ − e λη g ( β , θ , x j ) λ n � � λ e λη g ( β , θ , x j ) λ τ θ ( β , θ , x j ) = 0 g ( β , θ , x j ) 2 λ j =1 defines a family of estimating equations for θ , indexed by λ . The quadratic PL equation is obtained when λ = 2. 18 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Extended quasi-likelihood Recall that quasi-likelihood (QL) was an attempt to give a “distributional” justification for GLS-type estimation of β in models in which the variance depends on β through the mean, but there are no additional, unknown, variance parameters θ (that is, θ is known). When θ is unknown, the idea here is to define a “scaled exponential family-like” “loglikelihood” to be used as a basis for joint estimation of β , σ , and θ . 19 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Recall The log quasi-likelihood function � µ ℓ QL ( β , σ ; y ) = 1 y − u g ( u ) 2 du . σ 2 y where µ = f ( x , β ). Estimate β and σ by maximizing n � ℓ QL ( β , σ ; y j ) . j =1 20 / 21 Estimating Variance Parameters

ST 762 Nonlinear Statistical Models for Univariate and Multivariate Response Define the log extended quasi-likelihood function � µ ℓ EQL ( β , σ, θ ; y ) = 1 g ( u ) 2 du − 1 y − u 2 πσ 2 g 2 ( y , θ ) � � 2 log σ 2 y where µ = f ( x , β ). Estimate β , σ , and θ by maximizing n � ℓ EQL ( β , σ, θ ; y j ) . j =1 21 / 21 Estimating Variance Parameters