As AI systems increasingly mediate access to information, we need better ways to understand how they behave in practice. In this talk, I use election-related information as my main case study to ask what a rigorous audit of an AI system should look like. I will take a broad view of auditing, one that requires (1) deciding what behaviors to probe, (2) developing the statistical tools to interpret what we observe and (3) developing the experimental tools to make audits more scalable and adaptive. To that end, I will first discuss how large-scale, repeated probing can reveal systematic patterns in model responses, drawing on my team’s audit of the election-related information provided by leading language models during the 2024 U.S. presidential election. I will then discuss our new statistical methods for drawing reliable conclusions from model outputs when high-quality human evaluation is limited. Finally, I will highlight my lab’s work on how AI agents can enable more adaptive and interactive forms of auditing. Taken together, these efforts reflect my broader view of AI auditing as an ecosystem spanning measurement, statistical inference, and experimental design.

Zoom:https://upenn.zoom.us/j/97645937545

Meeting ID: 976 4593 7545