Ridge Estimation and its Modifications for Linear Regression with Deterministic or Stochastic Predictors

Abstract

A common problem in multiple regression analysis is having to engage in a bias variance trade-off in order to maximize the performance of a model. A number of methods have been developed to deal with this problem over the years with a variety of strengths and weaknesses. Of these approaches the ridge estimator is one of the most commonly used. This paper conducts an examination of the properties of the ridge estimator and several alternatives in both deterministic and stochastic environments. We find the ridge to be effective when the sample size is small relative to the number of predictors. However, we also identify a few cases where some of the alternative estimators can outperform the ridge estimator. Additionally, we provide examples of applications where these cases may be relevant.