Split Averaging: Bridging the Heterogeneity Gap in Clients Data for Federated Learning

Federated Learning (FL) has gained significant prominence to overcome the issue of data silos in various domains. However, since its introduction FL has been confronted with the presence of Non-Independent and Identically Distributed (Non-IID) data, hindering its broad-scale adoption. In this paper, we present a novel method named Federated Split Averaging (FSA) to tackle the problem of Non-IID data. FSA solves the key challenge that classical FL fails to overcome, specifically accounting for real-world scenarios where data instances from certain classes are completely missing. Unlike conventional FL, where a cloud server blindly averages clients' model parameters, FSA classifies clients into strong and weak groups and aggregates their parameters separately. The spitted parameters are then used to compute dynamic penalty factors, which regularize clients' training and accelerate convergence. Experimental results on real-world datasets demonstrated that the proposed method can significantly improve model accuracy in handling Non-IID data, achieving up to 7.23% improvement as compared to other state-of-the-art solutions.