Adaptive online scheduling in storm
- 29 June 2013
- conference paper
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 207-218
- https://doi.org/10.1145/2488222.2488267
Abstract
Today we are witnessing a dramatic shift toward a data-driven economy, where the ability to efficiently and timely analyze huge amounts of data marks the difference between industrial success stories and catastrophic failures. In this scenario Storm, an open source distributed realtime computation system, represents a disruptive technology that is quickly gaining the favor of big players like Twitter and Groupon. A Storm application is modeled as a topology, i.e. a graph where nodes are operators and edges represent data flows among such operators. A key aspect in tuning Storm performance lies in the strategy used to deploy a topology, i.e. how Storm schedules the execution of each topology component on the available computing infrastructure. In this paper we propose two advanced generic schedulers for Storm that provide improved performance for a wide range of application topologies. The first scheduler works offline by analyzing the topology structure and adapting the deployment to it; the second scheduler enhance the previous approach by continuously monitoring system performance and rescheduling the deployment at run-time to improve overall performance. Experimental results show that these algorithms can produce schedules that achieve significantly better performances compared to those produced by Storm's default scheduler. Copyright © 2013 ACMKeywords
This publication has 19 references indexed in Scilit:
- Input data organization for batch processing in time window based computationsPublished by Association for Computing Machinery (ACM) ,2013
- Active Replication at (Almost) No CostPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2011
- A model for continuous query latencies in data streamsPublished by Association for Computing Machinery (ACM) ,2011
- Low-Overhead Fault Tolerance for High-Throughput Data Processing SystemsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2011
- SODA: An Optimizing Scheduler for Large-Scale Stream-Based Distributed Computer SystemsLecture Notes in Computer Science, 2008
- MapReduceCommunications of the ACM, 2008
- Flexible Multi-Threaded Scheduling for Continuous Queries over Data StreamsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2007
- Dynamic Load Distribution in the Borealis Stream ProcessorPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2005
- An improved duplication strategy for scheduling precedence constrained graphs in multiprocessor systemsIEEE Transactions on Parallel and Distributed Systems, 2003
- Dynamic critical-path scheduling: an effective technique for allocating task graphs to multiprocessorsIEEE Transactions on Parallel and Distributed Systems, 1996