Machine translation

Contrast (vision)

BLEU

Computer security

Economics

Set (abstract data type)

Computer science

Natural language processing

Task (project management)

Management

Text generation

Operations management

Artificial intelligence

Programming language

Metric (unit)

Training set

Machine learning

Key (lock)

Text generation has made significant advances in the last few years. Yet, evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU and ROUGE) may correlate poorly with human judgment. We propose BLEURT, a learned evaluation metric for English based on BERT. BLEURT can model human judgment with a few thousand possibly biased training examples. A key aspect of our approach is a novel pre-training scheme that uses millions of synthetic examples to help the model generalize. BLEURT provides state-of-the-art results on the last three years of the WMT Metrics shared task and the WebNLG data set. In contrast to a vanilla BERT-based approach, it yields superior results even when the training data is scarce and out-of-distribution.

BLEURT: Learning Robust Metrics for Text Generation