Stela: Enabling Stream Processing Systems to Scale-in and Scale-out On-demand
- 1 April 2016
- conference paper
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
Abstract
The era of big data has led to the emergence of new real-time distributed stream processing engines like Apache Storm. We present Stela (STream processing ELAsticity), a stream processing system that supports scale-out and scale-in operations in an on-demand manner, i.e., when the user requests such a scaling operation. Stela meets two goals: 1) it optimizes post-scaling throughput, and 2) it minimizes interruption to the ongoing computation while the scaling operation is being carried out. We have integrated Stela into Apache Storm. We present experimental results using micro-benchmark Storm applications, as well as production applications from industry (Yahoo! Inc. and IBM). Our experiments show that compared to Apache Storm's default scheduler, Stela's scale-out operation achieves throughput that is 21-120% higher, and interruption time that is significantly smaller. Stela's scale-in operation chooses the right set of servers to remove and achieves 2X-5X higher throughput than Storm's default strategy.Keywords
This publication has 15 references indexed in Scilit:
- Online parameter optimization for elastic data stream processingPublished by Association for Computing Machinery (ACM) ,2015
- Elastic Scaling for Data Stream ProcessingIEEE Transactions on Parallel and Distributed Systems, 2013
- Adaptive online scheduling in stormPublished by Association for Computing Machinery (ACM) ,2013
- Integrating scale out and fault tolerance in stream processing using operator state managementPublished by Association for Computing Machinery (ACM) ,2013
- StreamCloud: An Elastic and Scalable Data Streaming SystemIEEE Transactions on Parallel and Distributed Systems, 2012
- A view of cloud computingCommunications of the ACM, 2010
- SPADEPublished by Association for Computing Machinery (ACM) ,2008
- DryadACM SIGOPS Operating Systems Review, 2007
- Design, implementation, and evaluation of the linear road bnchmark on the stream processing corePublished by Association for Computing Machinery (ACM) ,2006
- AuroraPublished by Association for Computing Machinery (ACM) ,2003