Bayesian probability

Combinatorics

Archaeology

Mathematics

Kernel (algebra)

Suite

History

Hyperparameter optimization

Operating system

Algorithm

Support vector machine

Mathematical optimization

Machine learning

Random search

Artificial intelligence

Set (abstract data type)

Programming language

Bayesian optimization

Computer science

Speedup

Hyperparameter

Journal of Machine Learning Research

Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration nonstochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.

Hyperband: a novel bandit-based approach to hyperparameter optimization