Capability · Framework — evals

Inspect AI

Inspect AI is the evaluation framework the UK AI Safety Institute (AISI) uses internally to assess frontier models. It's dataset-centric (think HELM, BIG-Bench), with clean abstractions for 'solvers' (how the model answers), 'scorers' (how you grade), and support for tool-use, agentic, and multi-turn evals. It's become a standard in AI-safety and research circles, with good support for parallelism and reproducibility.

Framework facts

Category: evals
Language: Python
License: MIT
Repository: https://github.com/UKGovernmentBEIS/inspect_ai

Install

pip install inspect-ai

Quickstart

from inspect_ai import Task, task, eval
from inspect_ai.dataset import Sample
from inspect_ai.scorer import match
from inspect_ai.solver import generate

@task
def capital_quiz():
    return Task(
        dataset=[Sample(input='Capital of France?', target='Paris')],
        solver=generate(),
        scorer=match()
    )

eval(capital_quiz(), model='anthropic/claude-opus-4-7')

Alternatives

DeepEval — more product-oriented
lm-evaluation-harness — academic benchmarks
Ragas — RAG-specific
Braintrust — commercial platform

Frequently asked questions

Is Inspect AI for research only?

It's biased toward research use cases (benchmarks, safety evals, capability measurements) but works equally well for product-facing evals. If your evaluations involve datasets, solvers, and careful scoring, Inspect AI is one of the cleanest frameworks available.

Why adopt this over in-house scripts?

You get structured logs, reproducible runs, parallelism, tool-use support, and a scorer library for free. Once evals become part of your release process, a framework saves weeks of ad-hoc plumbing.

Sources

Inspect AI — docs — accessed 2026-04-20
Inspect AI on GitHub — accessed 2026-04-20