Benchmark Router
BenchmarksDocs
Sign in

Benchmarks

Request a benchmarkRun with your harness
All 25OCR 1Knowledge 2Coding 10Agentic 3Cyber 3Legal 2Finance 1Health 1Science 1Vision 1
  • CVE-Bench

    Exploit real critical CVEs in sandboxed web applications: 40 challenges each in zero-day and one-day settings. Official v2.2.0 harness, Claude Code via Inspect's model bridge, deterministic in-target grading. Not comparable to v1.0 scores or the v2.1 ReAct leaderboard. Agent internet and hosted web tools are disabled.

    Cybercyber2 subsets80 samplesCustom adapter
  • CyBench

    39 professional capture-the-flag challenges in a sandboxed Kali terminal, from HackTheBox, SekaiCTF, GlacierCTF and HKCert. Maintained Inspect ReAct scaffold, unguided hard variant, deterministic flag matching. The GPL motp task is excluded. Offline agent: package installation is unavailable; not leaderboard-equivalent.

    Cybercyber1 subset39 samplesCustom adapter
  • CyberGym

    Reproduce historical vulnerabilities in 188 open-source projects by generating a crash-triggering input. 1,507 level1 tasks (source plus vulnerability description), graded on the final PoC against vulnerable and fixed builds. Inspect ReAct scaffold; not directly comparable to provider-specific leaderboard scaffolds.

    Cybercyber1 subset1,507 samplesCustom adapter