Historical Methods for AI Evaluations, Assessments, and Audits

In this paper, we present historical analysis as a critical method for integration within AI evaluation, assessments, and audit protocols. While existing scholarly traditions at FAccT have cultivated an appreciation for the value of historical insights, historical analysis has yet to be proposed and implemented within evaluation efforts in an intentionally programmatic manner. We situate this absence in the political and economic context of a growing AI auditing industry, which has incentivized the progressive narrowing of assessment methodologies and, as such, the terrain of discoverability. Drawing on our previous experiences designing and conducting historical analysis for an impact assessment on AI-enabled virtual agents, we detail one possible framework for integrating historical analysis alongside other evaluation methodologies. Reflecting on the utility of historical methods against the goals of AI assessments, we highlight key opportunities where historical surveys can help point to a wider range of risks and impacts, contributing to the harm discovery phase of evaluation protocols. Additionally, we argue that insights from historical analysis can help substantiate findings and present them in their appropriate social, cultural, political, and economic context.