Distributed learning

Algorithm

Gossip

Gossip protocol

Bandwidth (computing)

Social psychology

Embedded system

Theoretical computer science

Convergence (economics)

Pedagogy

Economics

Scalability

Computer science

Distributed computing

Computer network

Database

Psychology

Bottleneck

Artificial intelligence

Economic growth

Machine learning

IEEE Transactions on Parallel and Distributed Systems

Recently, federated learning (FL) techniques have enabled multiple users to train machine learning models collaboratively without data sharing. However, existing FL algorithms suffer from the communication bottleneck due to network bandwidth pressure and/or low bandwidth utilization of the participating clients in both centralized and decentralized architectures. To deal with the communication problem while preserving the convergence performance, we introduce a communication-efficient decentralized FL framework GossipFL. In GossipFL, we 1) design a novel sparsification algorithm to enable that each client only needs to communicate with one peer with a highly sparsified model, and 2) propose a new and novel gossip matrix generation algorithm that can better utilize the bandwidth resources while preserving the convergence property. We also theoretically prove that GossipFL has convergence guarantees. We conduct experiments with three convolutional neural networks on two datasets (IID and non-IID) under two distributed environments (14 clients and 100 clients) to verify the effectiveness of GossipFL. Experimental results show that GossipFL takes less communication traffic for 38.5% and less communication time for <inline-formula><tex-math notation="LaTeX">$49.8$</tex-math></inline-formula> % than state-of-the-art solutions while achieving comparative model accuracy.

GossipFL: A Decentralized Federated Learning Framework With Sparsified and Adaptive Communication