Reduced-Order Modeling of Deep Neural Networks
- 1 May 2021
- journal article
- research article
- Published by Pleiades Publishing Ltd in Computational Mathematics and Mathematical Physics
- Vol. 61 (5), 774-785
- https://doi.org/10.1134/s0965542521050109
Abstract
We introduce a new method for speeding up the inference of deep neural networks. It is somewhat inspired by the reduced-order modeling techniques for dynamical systems. The cornerstone of the proposed method is the maximum volume algorithm. We demonstrate efficiency on neural networks pre-trained on different datasets. We show that in many practical cases it is possible to replace convolutional layers with much smaller fully-connected layers with a relatively small drop in accuracy.Keywords
This publication has 17 references indexed in Scilit:
- Rectangular maximum-volume submatrices and their applicationsLinear Algebra and its Applications, 2018
- Efficient Rectangular Maximal-Volume Algorithm for Rating Elicitation in Collaborative FilteringPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2016
- Deep Residual Learning for Image RecognitionPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2016
- Wide Residual NetworksPublished by British Machine Vision Association and Society for Pattern Recognition ,2016
- Accelerating Very Deep Convolutional Networks for Classification and DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Understanding and Exploiting Spatial Properties of System Failures on Extreme-Scale HPC SystemsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2015
- Speeding up Convolutional Neural Networks with Low Rank ExpansionsPublished by British Machine Vision Association and Society for Pattern Recognition ,2014
- Computational Advertising: Techniques for Targeting Relevant AdsFoundations and Trends® in Theoretical Computer Science, 2014
- Nonlinear Model Reduction via Discrete Empirical InterpolationSIAM Journal on Scientific Computing, 2010
- Model compressionPublished by Association for Computing Machinery (ACM) ,2006