Deadline and QoS aware data warehouse

Abstract

A data warehouse infrastructure needs to support the requirement of (day time) ad hoc query response time and (night time) batch workload completion time. The following tasks need to be finished in a batch window: (1) Apply one day's delta data to the base tables; (2) refresh MQTs (Materialized Query Tables) for ad hoc queries and batch workloads; (3) run batch queries. Tools are available to optimize each step; however, many factors need to be considered for improving the overall performance of a data warehouse (i.e. meeting batch window deadline and ad hoc query response time). We have prototyped a Data Warehouse Operation Advisor to systematically study each component contributing to the batch window problem, and then perform global optimization to achieve desired results.