Perplexity

Algorithm

Engineering

Language model

Recurrent neural network

Artificial intelligence

Natural language processing

Filler (materials)

Computer science

Chemical engineering

State (computer science)

Artificial neural network

RNN language models have achieved stateof-the-art perplexity results and have proven useful in a suite of NLP tasks, but it is as yet unclear what syntactic generalizations they learn.Here we investigate whether state-ofthe-art RNN language models represent longdistance filler-gap dependencies and constraints on them.Examining RNN behavior on experimentally controlled sentences designed to expose filler-gap dependencies, we show that RNNs can represent the relationship in multiple syntactic positions and over large spans of text.Furthermore, we show that RNNs learn a subset of the known restrictions on filler-gap dependencies, known as island constraints: RNNs show evidence for wh-islands, adjunct islands, and complex NP islands.These studies demonstrates that stateof-the-art RNN models are able to learn and generalize about empty syntactic positions.

What do RNN Language Models Learn about Filler–Gap Dependencies?