Experiment workbench: A co-agent for assisting data scientists

Abstract

The analysis of large volumes of data is a field of study with ever increasing relevance. Data scientists is the moniker given for those in charge of extracting knowledge from Big Data. Big data is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making. The exploration done by data scientists relies heavily on the practitioner experience. These activities are hard to plan and can change during execution - a type of process named Knowledge Intensive Processes (KiP). The knowledge about how a data scientist performs her tasks could be invaluable for her and for the enterprise she works. This work proposes Experiment Workbench (EW), a system that assists data scientists in performing their tasks by learning how a data scientist works in-situ and being a co-agent during task execution. It learns through capturing user actions and using process mining techniques to discover the process the user executes. Then, when the user or her colleagues work in the learned process, EW suggests actions and/or presents existing results according to what it learned towards speed up and improve user wok. This paper presents the foundation for EW development (e.g., the main concepts, its components, how it works) and discuss the challenges EW is going to address.