Algorithm

Residual

Feature learning

Theoretical computer science

Operating system

Computation

Telecommunications

Computer science

Visual arts

Modular design

Architecture

Latency (audio)

Mathematics

Politics

Political science

Leverage (statistics)

Block (permutation group theory)

Artificial intelligence

Geometry

Representation (politics)

Deep learning

Residual neural network

Convolutional neural network

The ability to learn richer network representations generally boosts the performance of deep learning models. To improve representation-learning in convolutional neural networks, we present a multi-branch architecture, which applies channel-wise attention across different network branches to leverage the complementary strengths of both feature-map attention and multi-path representation. Our proposed Split-Attention module provides a simple and modular computation block that can serve as a drop-in replacement for the popular residual block, while producing more diverse representations via cross-feature interactions. Adding a Split-Attention module into the architecture design space of RegNet-Y and FBNetV2 directly improves the performance of the resulting network. Replacing residual blocks with our Split-Attention module, we further design a new variant of the ResNet model, named ResNeSt, which outperforms EfficientNet in terms of the accuracy/latency trade-off.

ResNeSt: Split-Attention Networks