Software-defined object storage in multi-tenant environments

Abstract

Object stores are becoming pervasive due to their scalability and simplicityto manage data growth. Their rampant adoption, however, contrasts with their scant flexibility to support multi-tenancy. Very often, this results in a deficient adaptation of the system to the heterogeneous tenants’ demands and the multiple applications sharing the same object storage infrastructure. In this manuscript, we present Crystal, the first Software-Defined Storage (SDS) architecture whose core objective is to efficiently support multi-tenancy in object stores. Abide by the principles of software-defined systems, Crystal separates the storage's control logic from the underlying storage nodes, proxies and caches, promoting the (logical) centralization of storage control and the ability to program the storage layer. For storage control, Crystal enables high-level policies at the tenant, container, and object granularities. It also translates these policies into a set of distributed controllers that can orchestrate computations at the storage layer based on real-time workload information. We demonstrate Crystal through three use cases on top of OpenStack Swift: One that proves its storage automation capabilities; one that exploits the pushdown of lambda functions on data streams to improve analytics data ingestion; and another that differentiates IO bandwidth in a multi-tenant scenario. We show that Crystal is an extensible platform to deploy new SDS services for object stores with small overhead.