AI Evaluations Are Not Yet Automatically a Reliable Deciding Factor
Test results may appear objective, but reproducibility remains a problem, and comparisons across evaluation suites are still unclear. It is therefore not yet easy to conclude that one model is better than another for every task.
Measurement today looks beyond the model itself to include the harness and agent that users can access. A weaker model may perform better when paired with a more suitable harness, while evaluating agents remains difficult and expensive.
The industry therefore still needs to develop shared standards so that evaluation results become more consistent and more useful for real-world decision-making.
Why AI Evaluation Results Are Not Yet Fully Reliable
Irene Solaiman views benchmarking as a developing field, so it is still unclear how meaningfully the results from different evaluation suites can be compared. Some tests also become ineffective within a short period, meaning reliability cannot depend on scores alone.
The key question is whether the results can be reproduced. Evaluators need sufficient access to the model, which is why open weights play an important role: they make it possible to inspect the evaluation process in greater detail. If results cannot be reproduced, even an apparently precise score may not be enough to confirm that one model is better than another.
Problems That Arise When Test Scores Stand in for Reality
Different evaluation suites may measure different things, making it difficult to compare scores across suites. Although the numbers may appear to show which model is better, they may not fully reflect real-world usage.
The problem is that some test suites become ineffective within a short period. Without shared standards, these scores risk causing people to make decisions based on an incomplete picture.
From Measuring Models to Measuring the Entire System
Evaluation no longer looks only at the model itself. It must also consider the harness, or the tools users employ to access the model, as well as the agents that work with the system. The harness directly affects performance.
As a result, a weaker model may outperform a leading model when paired with a better harness. Evaluating agents, meanwhile, remains difficult and expensive, so methods for measuring the entire system need to be developed more clearly.
What Questions Does Each Layer of Evaluation Answer?
Measurement does not end with the model itself, because real users experience the combined impact of the tools and agents working with it.
| Factor | Model evaluation | Harness evaluation | Holistic agent evaluation |
|---|---|---|---|
| What is measured | Model capabilities | Tools used to access the model | The entire system through which users receive value |
| Status according to sources | More evaluations are available | There are still relatively few evaluations | Methods for better evaluation are being developed |
| Challenges | Results must be made comparable | Directly affects performance | Difficult and expensive to evaluate |
This perspective shows that a model benchmark alone may not fully reflect real-world usage.
When Evaluators Are Embedded in AI Organizations
An embedded evaluator is an external observer placed inside an AI organization to examine model safety issues. The idea gained attention after incidents that raised concerns about the dangers of the technology, with Hugging Face being one of the organizations previously targeted.
Many labs view this approach as an alternative to government oversight. However, the key question is whether evaluators can be sufficiently impartial and capable. When Anthropic hired Accenture as its first evaluator, it drew criticism that a consulting firm might not be particularly well suited to the task.
Impartiality Does Not Guarantee Quality
Placing external evaluators inside an AI organization may appear impartial, but impartiality alone does not answer how well the evaluation will be conducted. Each task requires clearly defined qualifications and standards.
Anthropic’s decision to hire Accenture as its first evaluator therefore prompted criticism over whether the consulting firm was suitably qualified for the work. This reflects the fact that embedded evaluators still need common standards for both capabilities and working methods. Simply being labeled a third party is not enough.
What Kind of Evaluation System Would Be Truly Useful?
A useful evaluation system must produce reproducible test results, and evaluators must have the necessary level of access to the model, particularly when open weights are available so that results can be properly verified.
Real-world usage requires looking beyond the model itself, including the harness or tools used to access it, because a weaker model may perform better with a suitable harness. Agents should also be evaluated holistically, across the entire system through which users receive actual value, rather than by looking only at the model’s score.
Options Being Debated by the Industry
The AI industry is weighing independent evaluators embedded within organizations, government oversight, and evaluation suites created by labs. Each option still has its own obstacles.
| Factor | Third parties within organizations | Government | Lab-created evaluation suites |
|---|---|---|---|
| Strengths | Examine work from within | Play an oversight role | Test models directly |
| Limitations | Impartiality and qualifications remain in question | No proactive oversight of the kind mentioned by the sources yet | Evaluation methods remain complex and difficult to compare with other suites |
| Key issue | More standards are needed | Still an alternative promoted by labs | Results must be reproducible and model access must be sufficient |
Strengths and Weaknesses of Embedded Evaluators
Having external evaluators inside an AI organization may help detect and reduce model safety risks more closely, especially as evaluation expands to examine the entire system surrounding the model rather than the model alone.
However, the limitations remain clear: standards are not yet unified, evaluator impartiality remains a concern, and there are questions about whether the companies taking on this work are suitably qualified. Agentic evals are also difficult and expensive because they require evaluating the model, harness, and agent together.
Pros
- +Help examine safety risks within AI organizations
- +Can evaluate both the model and the surrounding system
Cons
- −Evaluator standards and impartiality remain unclear
- −Agentic evals are difficult and expensive
Costs That Do Not Appear in Test Scores
Costs do not end with running the test, because agentic evals are both difficult and expensive. The more the evaluation must examine the system surrounding the model, the greater the burden becomes.
Another problem is that the evaluator ecosystem is not yet strong enough. Different evaluation approaches remain difficult to compare, and the standards defining what makes an evaluator qualified still need to be developed more clearly.
Organizations therefore pay not only for testing, but also need to invest in evaluation methods that are consistent, reproducible, and usable across the industry.
What the Industry Must Do Before Relying on Evaluation Scores
The industry must move beyond simply creating more benchmarks and toward evaluation approaches based on shared principles. Results should be reproducible, and evaluators must have sufficient access to the model for verification.
What matters is not only which model has the higher score, but also whether different evaluations can be compared meaningfully. This includes examining the model, harness, and agent, because system performance depends on the components surrounding the model as well. Until standards become clearer, evaluation scores should not yet be treated as the final answer.