Evaluation of Rate-Based Adaptivity in Asynchronous Data Stream Joins

Abstract

Continuous query systems are an intuitive way for users to access streaming data in large-scale scientific applications containing many hundreds of streams. A challenge in these systems is to join streams in such a way that memory is conserved. Storing events that could not possibly participate in a join any longer wastes memory and limits scalability of the query processing system. This paper reports an experiment we conducted to validate an algorithm we developed for adaptive rate, adjustable join windows. We posit that a rate-based strategy can result in memory savings, can be sufficiently responsive to rapid changes in stream rates, and can execute with suitably low overhead. Based on the results, we conclude that the algorithm adds between 0.007% and 2.6% overhead, with significant gains in memory utilization possible depending on the particular workload.