Stream processing performance for blue Gene/P supercomputer

Abstract

Stream processing systems are designed to support ap- plications that use real time data. Examples of streaming applications include security agencies processing data from communications media, battlefield management systems for military operations, consumer fraud detection based on on- line transactions, and automated trading based on finan- cial market data. Many stream processing applications are faced with the challenge of increasingly large volumes of data and the requirement to deliver low-latency responses predicated by analysis of that data. In this paper, we assess the applicability of the Blue Gene architecture for stream computing applications. This work is part of a larger effort to demonstrate the efficacy of using a Blue Gene for stream- ing applications. Blue Gene supercomputers provide a high-bandwidth low-latency network connecting a set of I/O and compute nodes. We examine Blue Gene's suitability for stream com- puting applications by assessing its messaging capability for typical stream computing messaging workloads. In par- ticular, this paper presents results from micro-benchmarks we used to evaluate the raw performance of Blue Gene/P (Blue Gene/P) supercomputer under loads produced by high volumes of streaming data. We measure the perfor- mance of data streams that originate outside the supercom- puter, are directed through the I/O nodes to the compute nodes and then terminate outside. Our performance experi- ments demonstrate that the Blue Gene/P hardware delivers low-latency and high-throughput capability in a manner us- able by streaming applications. © 2008 IEEE.