BI-Bench : A Comprehensive Benchmark Dataset and Unsupervised Evaluation for BI Systems

A comprehensive benchmark is crucial for evaluating automated Business Intelligence (BI) systems and their real-world effectiveness. We propose a holistic, end-to-end framework that assesses BI systems based on the quality, relevance, and depth of insights. It categorizes queries into descriptive, diagnostic, predictive, and prescriptive types, aligning with practical BI needs. Our fully automated approach enables custom benchmark generation tailored to specific datasets. Additionally, we introduce an automated evaluation mechanism that removes reliance on strict ground truth, ensuring scalable and adaptable assessments. By addressing key limitations, our user-centered framework offers a flexible and robust methodology for advancing next-generation BI systems.