An architecture for the deployment of statistical models for the big data era

Abstract

Statistical models are commonly fit to bulk datasets, and they are applied in quasi real-time to previously unseen data. Challenges lie not only in fitting these models to data, but also in keeping track of their development and deployment process. It is common practice to re-engineer data pre-processing functions that were created during model development in order to build a version for deployment that works on streams of data. This approach is error-prone and inefficient. In this paper, we present our Model Deployment and Execution Framework (MDEF), to tackle these challenges in response to the volume, velocity, and variety of big data.