Hugging Face’s Policy Chief: Navigating the Challenges of Embedded AI Evaluators
Artificial intelligence labs have built their credibility on model benchmarking, a supposedly reliable system for evaluating and comparing different AI products.
However, the evaluation techniques are complex and still evolving, as noted by Irene Solaiman, chief policy officer at Hugging Face, during a recent TechCrunch Disrupt event. For the industry to place confidence in these evaluations, a more consistent scientific approach is needed.
Given this context, can we really trust the current tests available?
“Trust is a loaded term,” Solaiman explained. “A major factor in trusting results is whether those results can be reproduced. Do you have sufficient access to the model itself? This is where open weights play a crucial role,” she added.
The challenge of reproducibility in model evaluations has long been a contentious issue, remaining a significant area of discussion within the field.
Beyond the validity of individual tests, there is also the important question of how one test measures up against others.
“These evaluations are a developing science, and we lack meaningful ways to compare them with one another,” Solaiman remarked.
“They tend to lose their effectiveness relatively quickly, and we don’t have a robust ecosystem that we really need,” she added. “There is a growing demand for a cohesive evaluator system, but the criteria for determining effective evaluators across various tasks needs significant standardization.”
The concept of embedded evaluators, where impartial third-party reviewers are integrated within AI organizations, has gained traction in recent months. This comes as AI labs face a surge in safety concerns stemming from their technologies. Hugging Face has notably been involved in one such incident.
Rather than waiting for government regulations to address potential safety issues, labs are seeking third-party evaluations. Nonetheless, there are lingering doubts about the neutrality and effectiveness of these evaluators. For instance, Anthropic faced criticism after hiring Accenture for its evaluation, as many questioned the consulting firm’s qualifications for the role.
Currently, the focus of evaluations is shifting from testing just the model to assessing the entire ecosystem surrounding it, which includes the tools used to interact with the model. Specifically, the performance of a model can be significantly influenced by the harness, or interface. As Solaiman noted, a less powerful model paired with a superior harness could outshine a cutting-edge model.
“Agentic evaluations are particularly challenging and costly,” Solaiman stated. “We have many tests for models, but not nearly as many for harnesses. We are in the process of creating better evaluations for the comprehensive agent system, as that’s where users derive real value,” she added.
In summary, the evaluation landscape is somewhat chaotic at present, and for benchmarking to remain a valuable tool for industry leaders, a standardized, cohesive approach across the industry may become essential.
