Random variable

Universality (dynamical systems)

Mathematics

Gaussian process

Multivariate random variable

Quantum mechanics

Central limit theorem

Eigenvalues and eigenvectors

Conjecture

Gaussian

Combinatorics

Applied mathematics

Statistics

Discrete mathematics

Physics

Equivalence (formal languages)

Random matrix

IEEE Transactions on Information Theory

We prove a universality theorem for learning with random features. Our result shows that, in terms of training and generalization errors, a random feature model with a nonlinear activation function is asymptotically equivalent to a surrogate linear Gaussian model with a matching covariance matrix. This settles a so-called Gaussian equivalence conjecture based on which several recent papers develop their results. Our method for proving the universality theorem builds on the classical Lindeberg approach. Major ingredients of the proof include a leave-one-out analysis for the optimization problem associated with the training process and a central limit theorem, obtained via Stein's method, for weakly correlated random variables.

Universality Laws for High-Dimensional Learning With Random Features