Inference

Block (permutation group theory)

Mathematics

Geodesy

World Wide Web

Machine learning

Language model

Benchmark (surveying)

Artificial intelligence

Composite material

Geography

Materials science

Natural language processing

Range (aeronautics)

Geometry

Paragraph

Computer science

Question answering

We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory consumption and training/inference time, which also enables attention heads to capture either short- or long-range contextual information. We conduct experiments on language model pre-training and several benchmark question answering datasets with various paragraph lengths. BlockBERT uses 18.7-36.1% less memory and 12.0-25.1% less time to learn the model. During testing, BlockBERT saves 27.8% inference time, while having comparable and sometimes better prediction accuracy, compared to an advanced BERT-based model, RoBERTa.

Blockwise Self-Attention for Long Document Understanding