Data selection for efficient model update in federated learning
- 5 April 2022
- conference paper
- conference paper
- Published by Association for Computing Machinery (ACM) in Proceedings of the 2nd European Workshop on Machine Learning and Systems
Abstract
The Federated Learning (FL) workflow of training a centralized model with distributed data is growing in popularity. However, until recently, this was the realm of contributing clients with similar computing capability. The fast expanding IoT space and data being generated and processed at the edge are encouraging more effort into expanding federated learning to include heterogeneous systems. Previous approaches distribute light-weight models to clients are rely on knowledge transfer to distil the characteristic of local data in partitioned updates. However, their additional knowledge exchange transmitted through the network degrades the communication efficiency of FL. We propose to reduce the size of knowledge exchanged in these FL setups by clustering and selecting only the most representative bits of information from the clients. The partitioned global update adopted in our work splits the global deep neural network into a lower part for generic feature extraction and an upper part that is more sensitive to this selected client knowledge. Our experiments show that only 1.6% of the initially exchanged data can effectively transfer the characteristic of the client data to the global model in our FL approach, using split networks. These preliminary results evolve our understanding of federated learning by demonstrating efficient training using strategically selected training samples.Keywords
This publication has 5 references indexed in Scilit:
- Towards Mitigating Device Heterogeneity in Federated Learning via Adaptive Model QuantizationPublished by Association for Computing Machinery (ACM) ,2021
- Towards Federated Learning with Attention Transfer to Mitigate System and Data Heterogeneity of ClientsPublished by Association for Computing Machinery (ACM) ,2021
- A Hybrid Deep Learning Architecture for Privacy-Preserving Mobile AnalyticsIEEE Internet of Things Journal, 2020
- One-shot learning of object categoriesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2006
- Learning to Learn Using Gradient DescentLecture Notes in Computer Science, 2001