A New Lossless Compression Paradigm for Federated Learning: A Quantile-Based Framework for Bandwidth Efficiency Without Accuracy Degradation

Abstract

Federated Learning (FL) is a machine learning technique that preserves data privacy and security by training models directly on decentralized edge network devices. This generates substantial communication overhead due to the repeated exchange of model updates across numerous edge network devices. Quantization has tackled this challenge by reducing communication overhead and computational costs by quantizing model updates. Although selecting the most suitable quantization level to balance communication efficiency and model accuracy is challenging, failing to achieve this balance results in excessive compression, leading to accuracy degradation due to the lossy nature of the quantization technique. This challenge was tackled in this paper via a Quantile-based lossless compression method named Pcodec, which implements lossless compression in the FL context. Pcodec is a Quantile-based lossless compression algorithm designed for numerical data that utilizes mode identification with delta encoding and binning, where binning groups similar values into entropy-coded bins and stores the exact offset within each bin, thus achieving high compression ratios and efficient processing speed. Using MNIST and CIFAR-10 datasets and models such as CNN and ResNet18, we demonstrate that Pcodec achieves up to 58.19% size reduction with no accuracy loss compared to standard quantization methods. The experiments showed that the proposed Quantile-based compression approach in FL reduces up to 2.81× the communication overhead between each server and edge network device while maintaining the accuracy. In comparison to quantization, the Quantile approach reduced the communication overhead by 2.74×, tackling the main challenge of FL context by reducing communication overhead with a remarkably high compression ratio while maintaining the model’s accuracy.