Joshua S. Auerbach, Martin Hirzel, et al.
Proceedings of the ACM on Programming Languages
This article addresses the profitability problem associated with auto-parallelization of general-purpose distributed data stream processing applications. Auto-parallelization involves locating regions in the application's data flow graph that can be replicated at run-time to apply data partitioning, in order to achieve scale. In order to make auto-parallelization effective in practice, the profitability question needs to be answered: How many parallel channels provide the best throughput? The answer to this question changes depending on the workload dynamics and resource availability at run-time. In this article, we propose an elastic auto-parallelization solution that can dynamically adjust the number of channels used to achieve high throughput without unnecessarily wasting resources. Most importantly, our solution can handle partitioned stateful operators via run-time state migration, which is fully transparent to the application developers. We provide an implementation and evaluation of the system on an industrial-strength data stream processing platform to validate our solution. © 1990-2012 IEEE.
Joshua S. Auerbach, Martin Hirzel, et al.
Proceedings of the ACM on Programming Languages
Giuliano Losa, Vibhore Kumar, et al.
DEBS 2012
Byeongcheol Lee, Martin Hirzel, et al.
Software - Practice and Experience
Scott Schneider, Martin Hirzel, et al.
DEBS 2013