Transparency about Third-Party AI Evaluations

Independent evaluations are becoming increasingly central to AI governance, but relying on a third party to carry out an evaluation does not on its own ensure quality or impartiality. When third parties publish evaluation results intending to provide independent accountability about a particular system, they should be able to demonstrate that they were able to exercise independent judgment, had sufficient access to the systems they assessed, and that they acted transparently. Doing so will both increase trust in their work and to advance the independent evaluation ecosystem overall. This includes disclosing at least:

1. The methods used, and whether the evaluator controlled them. To build trust and facilitate independent review, evaluators should use state-of-the-art methods, and they should be transparent about those methods and how they were chosen. Evaluations that seek to validate an AI developer’s own chosen methods have value, but evaluators can provide more robust results when they have freedom to define the evaluation’s scope and methods.
2. Whether the evaluator had editorial control. The results of an evaluation cannot be treated as genuinely independent if AI providers can edit or suppress findings because they are negative. Providers’ feedback can be valuable, and results should be subject to vigorous disagreement where necessary, but an independent evaluation’s findings should represent the authentic views of the evaluator, without fear of retaliation.
3. The amount of access and time the evaluator had with the system. Evaluators cannot conduct a trustworthy evaluation without sufficient access to the systems and information in question, for a reasonable amount of time. While there is a legitimate need for AI providers and evaluators to safeguard intellectual property and user privacy, some system characteristics cannot be reliably evaluated without access to system versions and internals not available to the general public.
4. Whether the evaluator had conflicts of interest. Third-party evaluations are not independent, nor will they be externally trusted, if they are unduly influenced by financial or organizational ties to AI providers. Evaluators should be transparent about such relationships, including whether they receive compensation or other resources from providers, and should take concrete steps to limit conflicts of interest, such as recusing conflicted staff from evaluations.

As AI systems become increasingly capable and widely deployed, there is a growing need for trustworthy, independent evaluations of their capabilities and risks. Third-party evaluators should therefore be transparent about the conditions they are working under, such as by implementing frameworks like the AI Evaluator Forum's minimum conditions standard and related efforts. Governments, consumers, insurance providers, and the public at large should also demand transparency about independent evaluations so that they can be interpreted accurately and fulfill their increasingly central role in AI governance.