Hammad Ayyubi, Rahul Lokesh, et al.
ACL 2023
Human-annotated labels and explanations are critical for training explainable NLP models. However, unlike human-annotated labels whose quality is easier to calibrate (e.g., with a majority vote), human-crafted free-form explanations can be quite subjective. Before blindly using them as ground truth to train ML models, a vital question needs to be asked: How do we evaluate a human-annotated explanation's quality? In this paper, we build on the view that the quality of a human-annotated explanation can be measured based on its helpfulness (or impairment) to the ML models' performance for the desired NLP tasks for which the annotations were collected. In comparison to the commonly used Simulatability score, we define a new metric that can take into consideration of the helpfulness of an explanation for model performance at both fine-tuning and inference. With the help of a unified dataset format, we evaluated the proposed metric on five datasets (e.g., e-SNLI) against two model architectures (T5 and BART), and the results show that our proposed metric can objectively evaluate the quality of human-annotated explanations, while Simulatability falls short.
Hammad Ayyubi, Rahul Lokesh, et al.
ACL 2023
Natalia Martinez Gil, Kanthi Sarpatwar, et al.
NeurIPS 2023
Jatin Arora, Youngja Park
ACL 2023
Daniel Karl I. Weidele, Hendrik Strobelt, et al.
SysML 2019