Unbiased K-L estimator for the linear regression model

Unbiased K-L estimator with prior information

Hoerl and Kennard⁵ developed the ridge estimator to mitigate multicollinearity in the linear regression model. The ridge estimator of β with the biasing parameter k is:

The modified ridge technique was proposed with the addition of the prior information⁶. This is expressed as follows:

According to 16, the unbiased ridge estimator with the introduction of prior information J is given as

where J and ${\widehat{\beta }}_{OLS}$ are uncorrelated and J ̴ N(β, D) such that $D=(\frac{{\sigma }^2}{k})I_p$ and I_p is p × p identity matrix. J is estimated by $J=\frac{{\displaystyle {\sum }_{i=1}^p{\widehat{\beta }}_i}}{p}$.

Modified ridge-type method proposed by 3 is given as follows:

where A_kd = [S+k(1+d)]^–1S.

The unbiased modified ridge type estimator¹⁹ was developed and defined as follows:

where A_kd = [H+k(1+d)]^–1 H such that $D=\frac{{\sigma }^2}{k(1+d)}$. Consequently, $J~N\left(\beta ,\frac{{\sigma }^2}{k(1+d)}\right)$ for

k ˃ 0, 0<d<1.

Recently¹³, proposed the K-L estimator and found that this estimator generally outperform the ridge regression estimator. The K-L estimator of β is defined as:

where A_k = (H + kI)^-1 (H – kI)

This research proposes an unbiased K-L estimator following the convex method. The convex method is defined as:

where G is a p×p matrix and I is an identity matrix of p×p dimensions. Thus, the MSE of $\widehat{\beta }$ (G, J) is

Such that,

The value of G from (11) is G = D(σ²H^–1 + D)^–1. Accordingly, D = σ²(I – G)^-1GH^-1). We observed that the convex estimator β(G, J) is an unbiased estimator of β and possesses minimum MSE for optimal value of G. Consequently, the new unbiased estimator is defined as

where A_k = (H + kI)^–1(H – kI) and the value of $V=\frac{{\sigma }^2(S-kI)S^{-1}}{2k}$. Therefore, $J~N\left(\beta ,\frac{{\sigma }^2(S-kI)S^{-1}}{2k}\right)$ for k ˃ 0.

It can be expressed conveniently that ${\widehat{\beta }}_{UKL}(k,J)$ is unbiased for β The new estimator has properties defined as follows:

It follows from Equation (13) that the proposed estimator is unbiased. This classified the new estimator into the same class with the OLS estimator. The biasedness of the estimator is also zero. This is proved as follows:

Given that there exists an orthogonal matrix Q, such that $Q^{\prime }{X}^{\prime }XQ$ = Ε = diag (e₁,e₂,...,e_p) where e_i is the i^th eigenvalue of $X^{\prime }X$, E and Q are the matrices of eigenvalues and eigenvectors of $X^{\prime }X$ respectively. Equation (1) can be expressed canonically as:

where Z = XQ, $\alpha =Q^{\prime }\beta$ and $Z^{\prime }Z=E$ = Ε. For Equation (15), we get the following representations:

Lemma 1.1 Let N be an n×n positive definite matrix and α be some vector, then $N-\alpha {\alpha }^{\prime }\ge 0$ if and only if ${\alpha }^{\prime }{N}^{-1}\alpha \le 1$²⁰.

Lemma 1.2 Let ${\widehat{\alpha }}_i$ = C_iy i = 1, 2 be two linear estimators of α. Suppose that D = Cov(${\widehat{\alpha }}_1$) – Cov(${\widehat{\alpha }}_2$) > 0, where Cov(${\widehat{\alpha }}_i$), i = 1, 2 denotes the covariance matrix of ${\widehat{\alpha }}_i$ and bias(${\widehat{\alpha }}_i$) = b = (C_iX – I)α, i = 1, 2. Consequently,

if and only if $b_2^{\prime }{[{\sigma }^{2}D+b_1{b}_1^{\prime }]}^{-1}{b}_2<1$ where $MSEM({\widehat{\alpha }}_i)=Cov({\widehat{\alpha }}_i)+b_i{b}_i^{\prime }$²¹.

Theoretical comparison

${\widehat{\mathit{\alpha }}}_{\mathit{OLS}}$ and ${\widehat{\mathit{\alpha }}}_{\mathit{UKL}}\mathit{(}\mathit{k}\mathit{,}\mathit{J}\mathit{)}$

Theorem 1.1. ${\widehat{\alpha }}_{UKL}(k,J)$ is preferred to ${\widehat{\alpha }}_{OLS}$ by using the matrix mean square error as criteria for k > 0.

Proof

Recall that,

The difference between (23) and (24) is as follows:

Simplifying (25) further, we observed that E^–1 – (E + k)^–1 (E – k)Λ^–1 will be positive definite since 2k > 0 for k > 0.

${\widehat{\mathit{\alpha }}}_{\mathit{R}\mathit{R}}\mathit{(}\mathit{k}\mathit{)}$ and ${\widehat{\mathit{\alpha }}}_{\mathit{UKL}}\mathit{(}\mathit{k}\mathit{,}\mathit{J}\mathit{)}$

Theorem 3.2. ${\widehat{\alpha }}_{UKL}(k,J)$ is preferred to ${\widehat{\alpha }}_{RR}(k)$ by using the matrix mean square error as criteria for k > 0.

Proof

where B_k = (E + kI)^–1

The difference of Equation (24) and (26) is as follows:

Simplifying (27) further, we observed that E(E + k)^–2 – (E + k)^–1 (E – k)E^–1 will be positive definite since k² > 0.

${\widehat{\mathit{\alpha }}}_{\mathit{URR}}\mathit{(}\mathit{k}\mathit{)}$ and ${\widehat{\mathit{\alpha }}}_{\mathit{UKL}}\mathit{(}\mathit{k}\mathit{,}\mathit{J}\mathit{)}$

Theorem 3.3. ${\widehat{\alpha }}_{UKL}(k,J)$ is preferred to ${\widehat{\alpha }}_{URR}(k)$ by using the matrix mean square error as criteria for k > 0.

Proof

We observed that σ²(E + k)^–1 – σ²(E + k)^–1(E – k)E^–1 will be positive definite since k > 0.

${\widehat{\mathit{\alpha }}}_{\mathit{KL}}\mathit{(}\mathit{k}\mathit{)}$ and ${\widehat{\mathit{\alpha }}}_{\mathit{UKL}}\mathit{(}\mathit{k}\mathit{,}\mathit{J}\mathit{)}$

Theorem 3.4. ${\widehat{\alpha }}_{UKL}(k,J)$ is preferred to ${\widehat{\alpha }}_{KL}(k)$ by using the matrix mean square error as criteria for k > 0.

Proof

where E_k = (E + k)^–1. Consequently,

We observed that σ²(E – k)²E^–1(E + k)^–2 – σ²(E – k)E^–1(E + k)^–1 will be positive definite if k > 2Λ for k > 0.

Selection of the biasing parameters

In this study, we adopt the following biasing parameter for the ridge and the unbiased ridge estimators:

For the K-L estimator, we adopted:

For the proposed estimator, the following biasing parameters were examined:

Application: poultry waste data

The poultry waste data adopted in this study was found and analyzed in Qian et al.^24,25 and was also recently employed by Lukman et al.¹⁹. The study was aimed at modelling the high heating values of proximate-based model. The response variable is High Heating Values (HHV), while the independent variables are Fixed Carbon (FC), Volatile Matter (VM), and Ash (A). The linear regression model is:

where ε is the normally distributed random error term. In this study, the Jarque-Bera (JB) test was employed to know the distribution of the residual. The test statistic and its p-value are 0.6409 and 0.7258, respectively. The result shows that the residual in the model is normally distributed. We diagnosed if the model has the problem of multicollinearity. According to Lukman et al.¹⁴, the model suffered from the problem of multicollinearity because the variance inflation factors (VIF_FC =997.819, VIF_VM =2163.504, VIF_ASH =1533.782) are greater than ten (10). Also, there is evidence of multicollinearity with the use of the condition number (CN).

Following Lukman et al.^3,4, moderate level of multicollinearity is observed if the CN ranges are between 100 and 1000 but severe multicollinearity is encountered if CN is greater than 1000. For effective modelling, we considered some alternative estimators to the ordinary least squared estimator in this study. These include the ridge estimator, unbiased ridge estimator, K-L estimator and the unbiased K-L estimator. The estimators’ performance was examined using the mean square error. We also adopt the leave-one-out cross-validation to validate how well the estimators perform¹⁴. The performance of the estimator is assessed through the mean squared prediction error (MSPE). The estimator with the least MSE and MSPE is considered as the best. The result is available in Table 3.

From Table 3, the regression estimates of the following methods are the same: URR, UKL, and OLS as expected. They also possess a smaller mean squared error when compared with the OLSE. The estimators all exhibit the same regression coefficient signs. The proposed estimator UKL demonstrated the best performance in terms of the MSE and the MSPE. Although its performance is a function of the biasing parameter k.

Underlying data

Zenodo: Regression Model to Predict the Higher Heating Value of Poultry Waste from Proximate Analysis. http://doi.org/10.5281/zenodo.5078977²⁵.

This project contains the following underlying data:

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).