Evolving databases for new-gen big data applications

Abstract

The rising popularity of large-scale real-time analytics applications (real-time inventory/pricing, mobile apps that give you suggestions, fraud detection, risk analysis, etc.) emphasize the need for distributed data management systems that can handle fast transactions and analytics concurrently. Efficient processing of transactional and analytical requests, however, require different optimizations and architectural decisions in a system. This paper presents the Wildfire system, which targets Hybrid Transactional and Analytical Processing (HTAP). Wildfire leverages the Spark ecosystem to enable large-scale data processing with different types of complex analytical requests, and columnar data processing to enable fast transactions and analytics concurrently.