Aerospace engineering

Descent (aeronautics)

Cover (algebra)

Mathematics

Manifold (fluid mechanics)

Statistics

Mathematical optimization

Initialization

Descent direction

Randomness

Engineering

Gradient descent

Artificial intelligence

Coordinate descent

Programming language

Geometry

Saddle point

Saddle

Computer science

Applied mathematics

Artificial neural network

Mechanical engineering

Mathematical Programming

We establish that first-order methods avoid strict saddle points for almost all initializations. Our results apply to a wide variety of first-order methods, including (manifold) gradient descent, block coordinate descent, mirror descent and variants thereof. The connecting thread is that such algorithms can be studied from a dynamical systems perspective in which appropriate instantiations of the Stable Manifold Theorem allow for a global stability analysis. Thus, neither access to second-order derivative information nor randomness beyond initialization is necessary to provably avoid strict saddle points.

First-order methods almost always avoid strict saddle points