Hardware Resource Analysis in Distributed Training with Edge Devices
Open Access
- 26 December 2019
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 9 (1), 28
- https://doi.org/10.3390/electronics9010028
Abstract
When training a deep learning model with distributed training, the hardware resource utilization of each device depends on the model structure and the number of devices used for training. Distributed training has recently been applied to edge computing. Since edge devices have hardware resource limitations such as memory, there is a need for training methods that use hardware resources efficiently. Previous research focused on reducing training time by optimizing the synchronization process between edge devices or by compressing the models. In this paper, we monitored hardware resource usage based on the number of layers and the batch size of the model during distributed training with edge devices. We analyzed memory usage and training time variability as the batch size and number of layers increased. Experimental results demonstrated that, the larger the batch size, the fewer synchronizations between devices, resulting in less accurate training. In the shallow model, training time increased as the number of devices used for training increased because the synchronization between devices took more time than the computation time of training. This paper finds that efficient use of hardware resources for distributed training requires selecting devices in the context of model complexity and that fewer layers and smaller batches are required for efficient hardware use.Keywords
Funding Information
- Institute for Information and Communications Technology Promotion (1711073912)
This publication has 21 references indexed in Scilit:
- Intelligent Buildings of the Future: Cyberaware, Deep Learning Powered, and Human InteractingIEEE Industrial Electronics Magazine, 2016
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural NetworksPublished by Springer Science and Business Media LLC ,2016
- Edge Computing: Vision and ChallengesIEEE Internet of Things Journal, 2016
- Toward software defined smart homeIEEE Communications Magazine, 2016
- GeePSPublished by Association for Computing Machinery (ACM) ,2016
- A risk analysis of a smart home automation systemFuture Generation Computer Systems, 2016
- SINGAPublished by Association for Computing Machinery (ACM) ,2015
- CaffePublished by Association for Computing Machinery (ACM) ,2014
- Association of eating speed and energy intake of main meals with overweight in Chinese pre-school childrenPublic Health Nutrition, 2013
- Artificial neural networks (the multilayer perceptron)—a review of applications in the atmospheric sciencesAtmospheric Environment, 1998