Memory management

Feature (linguistics)

Block (permutation group theory)

Mathematics

Operating system

Algorithm

Distributed computing

Computer engineering

Residual

Philosophy

Parallel computing

Geometry

Overhead (engineering)

Computer science

Linguistics

Computer hardware

Semiconductor memory

Runtime and memory consumption are two important aspects for efficient image super-resolution (EISR) models to be deployed on resource-constrained devices. Recent advances in EISR [16], [32] exploit distillation and aggregation strategies with plenty of channel split and concatenation operations to fully use limited hierarchical features. In contrast, sequential network operations avoid frequently accessing preceding states and extra nodes, and thus are beneficial to reducing the memory consumption and runtime overhead. Following this idea, we design our lightweight network backbone by mainly stacking multiple highly optimized convolution and activation layers and decreasing the usage of feature fusion. We propose a novel sequential attention branch, where every pixel is assigned an important factor according to local and global contexts, to enhance high-frequency details. In addition, we tailor the residual block for EISR and propose an enhanced residual block (ERB) to further accelerate the network inference. Finally, combining all the above techniques, we construct a fast and memory-efficient network (FMEN) and its small version FMEN-S, which runs 33% faster and reduces 74% memory consumption compared with the state-of-the-art EISR model: E-RFDN, the champion in [49]. Besides, FMEN-S achieves the lowest memory consumption and the second shortest runtime in NTIRE 2022 challenge on efficient super-resolution [28]. Code is available at https://github.com/NJU-Jet/FMEN.

Fast and Memory-Efficient Network Towards Efficient Image Super-Resolution