Minimum Operating Conditions for Independent Third Party AI Evaluations

Third-party evaluations come in many forms, but when they are supposed to provide a genuinely independent and trustworthy assessment of an AI system's capabilities or risks, they should be carried out under conditions that help ensure their independence, access, and transparency. We've developed a standard (AEF-1) to fill this gap.

Organizations across the AI sector recognize the need for greater transparency into the operating conditions of third-party evaluations, and signed a public letter urging developers and evaluators to share details on the independence and access enjoyed by evaluators, as well as any conflicts of interest they may have.

Members of the AI Evaluator Forum collaborated with key partners from across the AI ecosystem to create AEF-1, Minimum Operating Conditions for Independent Third Party AI Evaluations, a voluntary standard that evaluators can use to demonstrate how they achieved a baseline set of operating conditions for the independence, access, and transparency of a particular evaluation. See the full standard below.

To add the checklist for AEF-1 to your report, see here for a .docx template, with a LaTeX template coming soon.