Efficient Client Sampling with Compression in Heterogeneous Federated Learning

Abstract

Federated Learning (FL) has emerged as a promising decentralized machine learning (ML) paradigm where distributed clients collaboratively train models without sharing their private data. However, due to their limited resources and heterogeneous properties, only a small subset of clients can participate at a given time. Furthermore, the high dimensions of ML models incur a massive communication overhead which considerably slows down the convergence of FL. To address the aforementioned challenges, we propose FedHSC, a framework that considers both system and statistical heterogeneity. Specifically, at each communication round, the clients are sampled based on their data properties combined with the importance of their local learning update. After completing their local training, the selected clients share compressed updates with the server for aggregation. The compression rate is adjusted for each client to meet the communication delay requirement. Experimental results on CIFAR-10 show the efficiency of our approach and its robustness to Non-IID data.