Electrical engineering

Quadratic growth

Inference

ENCODE

Biology

Gene

Algorithm

Engineering

Language model

Voltage

Artificial intelligence

Chemistry

Sequence (biology)

Computer science

Theoretical computer science

Speech recognition

Genetics

Biochemistry

Transformer

arXiv (Cornell University)

Transformer models have obtained remarkable accomplishments in various NLP tasks. However, these models have efficiency issues on long sequences, as the complexity of their self-attention module scales quadratically with the sequence length. To remedy the limitation, we present Memformer, a novel language model that utilizes a single unified memory to encode and retrieve past information. It includes a new optimization scheme, Memory Replay Back-Propagation, which promotes long-range back-propagation through time with a significantly reduced memory requirement. Memformer achieves O(n) time complexity and O(1) space complexity in processing long sequences, meaning that the model can handle an infinite length sequence during inference. Our model is also compatible with other self-supervised tasks to further improve the performance on language modeling. Experimental results show that Memformer outperforms the previous long-range sequence models on WikiText-103, including Transformer-XL and Compressive Transformer.

Memformer: The Memory-Augmented Transformer