Global Object Store

All major cloud services, including Amazon S3, Google Cloud Storage, Azure Blob Storage, and of course IBM Cloud Object Storage deploy object stores, repositories for storing objects (such as documents, folders, and business objects) and the metadata defining an object's classes and properties. In general, object stores are typically regional in character. An app writing to an object store must specify the cloud and region to which it is writing and reading from, because different clouds have different APIs and remote access is subject to increased latency and potential costs.

A global object store seeks to relieve these limitations by serving as a thin layer on top of multiple object stores from different clouds and/or different regions.

There are multiple benefits to using a global object store. Usability is improved through the use of a unified and simplified access, using a single protocol (the S3 “core” API), and users don’t need to remember where the objects are or choose where to write them to. The global object store can also improve performance by using local object stores, which allow you to either write locally, create local replicas, or migrate data to where it is being used. The technology also allows users to cut network costs by reducing traffic.

The open source project is a collaboration with a team from the University of California Berkeley’s Sky Computing Lab to implement a global object store. A prototype of the global object store can be seen here.

Current design choices and features of our global object store include using a metadata DB to locate objects; writing “locally” to achieve the best write latencies; creating a replicated cache copy upon remote reads for repeated reads; a novel self-adapting cache eviction policy aimed at optimizing overall costs; and policies and APIs for pre-fetching and managing placement policies.