Securing Kafka with Encryption-at-Rest

Abstract

Over the last decade Kafka has become a key component of Big Data processing pipelines. Kafka is increasingly used as a store of data, not just as a means of transferring it from one location to another. For cloud providers and enterprises this means that Kafka must conform to the same security and compliance requirements as conventional data storage systems such as relational databases. One important requirement is encryption-at-rest which is not currently supported by Kafka. We present an analysis of different ways to implement encryption in Kafka and then describe the first complete system for implementing encryption-at-rest at the granularity of a Kafka topic at scale. We demonstrate the challenges in implementing encryption policy, key distribution, key rotation and data re-encryption in Kafka, using our working implementation for illustration.