Quantum mechanics

Statistics

Covariance matrix

Applied mathematics

Eigenvalues and eigenvectors

Estimation of covariance matrices

Geology

Regression analysis

Random matrix

Paleontology

Covariance

Asymptotic distribution

Physics

Regression

Mathematics

Estimator

Ridge

Matrix norm

Linear discriminant analysis

The Annals of Statistics

We provide a unified analysis of the predictive risk of ridge regression and regularized discriminant analysis in a dense random effects model.We work in a high-dimensional asymptotic regime where p, n → ∞ and p/n → γ > 0, and allow for arbitrary covariance among the features.For both methods, we provide an explicit and efficiently computable expression for the limiting predictive risk, which depends only on the spectrum of the feature-covariance matrix, the signal strength and the aspect ratio γ .Especially in the case of regularized discriminant analysis, we find that predictive accuracy has a nuanced dependence on the eigenvalue distribution of the covariance matrix, suggesting that analyses based on the operator norm of the covariance matrix may not be sharp.Our results also uncover an exact inverse relation between the limiting predictive risk and the limiting estimation risk in high-dimensional linear models.The analysis builds on recent advances in random matrix theory. Introduction. Suppose a statistician observes n training examplesdrawn independently from an unknown distribution D, and wants to find a rule for predicting y on future unlabeled draws x from D. In other words, the statistician seeks a function h : R p → Y, h(x) = g(w x) for which E D [ (y, h(x))] is small, where (•, •) is a loss function; in regression Y = R and is the squared error loss, while in classification Y = {0, 1} and is the 0-1 loss.Such prediction problems lie at the heart over several scientific and industrial endeavors in fields ranging from genetics [Wray, Goddard and Visscher (2007)] and computer vision [Russakovsky et al. (2014)] to Medicare resource allocation [Kleinberg et al. (2015)].When the number of features p is large, accurate prediction is not always possible.Thus, in order to guarantee good results, the statistician needs to invoke some "enabling hypothesis" that encodes domain-specific knowledge about the problem and guides model fitting.Popular options include the "sparsity hypothesis," that is, that there is a good predictive rule depending only on w x for some sparse weight vector w [Donoho et al. (1992), Hastie, Tibshirani and Wainwright

High-dimensional asymptotics of prediction: Ridge regression and classification