Towards Automating the AI Operations Lifecycle
Matthew Arnold, Jeffrey Boston, et al.
MLSys 2020
Information Extraction (IE) in IT is an important foundational task that is needed for many AIOps applications. A major challenge of IE in IT is that we often do have not have sufficient labelled data for training machine learning algorithms since acquiring labels is labor-intensive and costly. In this paper, we propose to leverage Large Language Models (LLMs) to address this challenge of low resources and study two data augmentation strategies, i.e., using LLMs to generate pseudo labels and generate synthetic data. We use multiple IE tasks and datasets, including a new Semantic Troubleshooting-Segment Extraction task and Named Entity Recognition, to evaluate the benefits of LLMs. Our experiment results suggest that data augmentation using LLMs, specifically, using SeqMix model that combines active labeling with synthetic data samples generated in the embedding-vector space, is a promising approach IT domain IE. Our study also shows that although data augmentation and direct labeling with the state-of-the-art, ChatGPT model achieves a high performance on general domain IE, there is a need to adapt it for IE from IT text data. Moreover, our initial exploration of two label weighting and selection strategies (confidence and consistency-based) suggests that they could be used to improve data augmentation with ChatGPT for IT domain IE. Finally, we also suggest directions for future research on the new STSE task, including developing better evaluation metrics.
Matthew Arnold, Jeffrey Boston, et al.
MLSys 2020
Genady Ya. Grabarnik, Filippo Poltronieri, et al.
CASCON 2023
Saurabh Pujar, Luca Buratti, et al.
DAC 2023
Jasmina Bogojeska, Ioana Giurgiu, et al.
Interfaces