Benchmark Router
BenchmarksDocs
Sign in

Benchmarks

Request a benchmarkRun with your harness
All 25OCR 1Knowledge 2Coding 10Agentic 3Cyber 3Legal 2Finance 1Health 1Science 1Vision 1
  • AutomationBench

    Zapier's business-workflow benchmark: 600 public tasks (Sales, Marketing, Operations, Support, Finance, HR; 100 each) on 47 simulated SaaS apps, graded by deterministic end-state assertions (strict pass rate). Official auto-bench harness, API toolset, 50 steps. Public set only: the zapier.com leaderboard uses a private, harder held-out set, so scores are not directly comparable.

    Agentictool6 subsets600 samplesCustom adapter
  • BankerToolBench

    100 end-to-end investment banking tasks (Excel models, pitch decks, memos) built with SEC EDGAR, market-data and logo tools, graded against ~110-criterion expert rubrics by the official agentic verifier (Gandalf, Gemini 3 Flash). Official Harbor tasks with OpenCode for every model, as on the v1.1 leaderboard; reports pass rate (all critical criteria met) and mean rubric score.

    Financetool1 subset100 samplesCustom adapter
  • CVE-Bench

    Exploit real critical CVEs in sandboxed web applications: 40 challenges each in zero-day and one-day settings. Official v2.2.0 harness, Claude Code via Inspect's model bridge, deterministic in-target grading. Not comparable to v1.0 scores or the v2.1 ReAct leaderboard. Agent internet and hosted web tools are disabled.

    Cybercyber2 subsets80 samplesCustom adapter
  • CyBench

    39 professional capture-the-flag challenges in a sandboxed Kali terminal, from HackTheBox, SekaiCTF, GlacierCTF and HKCert. Maintained Inspect ReAct scaffold, unguided hard variant, deterministic flag matching. The GPL motp task is excluded. Offline agent: package installation is unavailable; not leaderboard-equivalent.

    Cybercyber1 subset39 samplesCustom adapter
  • CyberGym

    Reproduce historical vulnerabilities in 188 open-source projects by generating a crash-triggering input. 1,507 level1 tasks (source plus vulnerability description), graded on the final PoC against vulnerable and fixed builds. Inspect ReAct scaffold; not directly comparable to provider-specific leaderboard scaffolds.

    Cybercyber1 subset1,507 samplesCustom adapter
  • DeepSWE

    113 original long-horizon software engineering tasks across 91 open-source repos (Go, Python, TypeScript, Rust, JavaScript), graded by hidden behavioural tests. Official Pier runner on Modal; mini-swe-agent for every model, as on the leaderboard.

    Codingrepo1 subset113 samplesAgent CLI
  • EnterpriseOps-Gym

    649 public enterprise tasks across Teams, CSM, Email, ITSM, Calendar, HR, Drive and cross-domain Hybrid workflows: the agent uses stateful MCP tools over a seeded database while following internal policies, and hidden SQL checks the final state (a task passes only if every check does). Official harness and ReAct agent in oracle mode; task success rate as on the leaderboard.

    Agentictool8 subsets649 samplesCustom adapter
  • Harvey LAB (Legal Agent Benchmark)

    Harvey's Legal Agent Benchmark: 2,010 public long-horizon legal tasks (24 practice areas plus contracts, diligence and firm-knowledge) producing memos, redlines, spreadsheets and decks from a document set; every rubric criterion graded by two LLM judges (Claude Sonnet 4.6 + GPT-5.5), a task passes only if all criteria pass. Official LAB harness and evaluator. Public set only: the Vals leaderboard uses Harvey's private held-out set, so scores are not directly comparable.

    Legaltool27 subsets2,010 samplesCustom adapter
  • HealthBench

    Realistic health conversations graded against physician-written rubrics. Variants: full (5,000), hard (1,000) and consensus (3,671), reported separately. Official openai/simple-evals code with its GPT-5.4 (low reasoning) grader option.

    Healthqa3 subsets9,671 samplesAny API
  • Humanity's Last Exam

    2,500 expert-written closed-ended questions across dozens of subjects (multiple-choice and exact-match, ~14% with images), graded by an LLM judge. Official CAIS harness.

    Knowledgeqa8 subsets2,500 samplesvisionAny API
  • Humanity's Last Exam: Diamond

    1,000 refined HLE questions, 500 reasoning and 500 knowledge, graded by an LLM judge. Official CAIS harness, no tools.

    Knowledgeqa2 subsets1,000 samplesvisionAny API
  • ITSMBench

    89 IT service-management tickets (access, offboarding, incidents, security response) worked against mock ServiceNow, Okta, Slack and Google Workspace APIs, each checked by a hidden verifier. Official Harbor tasks with the Pi agent for every model, as on the ITSMBench leaderboard.

    Agenticharbor1 subset89 samplesAgent CLI
  • NL2Repo-Bench

    104 long-horizon tasks: generate a complete, runnable Python repository from a natural-language document, graded by the original project's tests (pass rate). Official OpenHands 0.56 harness inside a Docker-capable Modal VM per task.

    Codingrepo3 subsets104 samplesAgent CLI
  • ParseBench 2

    Document parsing / OCR: 4,317 real enterprise PDF pages (insurance, finance, government) converted to Markdown with layout, scored on text, formatting, tables, charts, visual grounding and forms with severity-aware rule metrics. Official LlamaIndex harness.

    OCRqa13 subsets4,317 samplesvisionAny API
  • PlanBench-V

    Urban-planning map understanding (Chinese): 300 stratified questions over planning maps in 8 sub-tasks, each answer judged against expert critical points (score 0-2). Official zhuchichi56/PlanBench code; judge GPT-6 Luna (OpenAI's replacement for the retiring gpt-4o-mini).

    Visionqa1 subset300 samplesvisionAny API
  • ProgramBench

    200 tasks: rebuild a program from scratch given only its compiled binary and docs, graded by hidden behavioural tests. Official harness (mini-swe-agent + programbench eval) inside a Docker-capable Modal VM per task; mini-swe-agent for every model, as on the leaderboard. Headline: % resolved (all tests pass).

    Codingrepo1 subset200 samplesAgent CLI
  • RedlineBench (Crosby)

    Contract-negotiation redlining: 140 tasks across 3 MSA negotiations and 4 turns, where the agent, as in-house counsel, produces a Word redline with tracked changes and comments. Graded against attorney-authored weighted rubrics by a 3-judge LLM panel (GPT-5.4 mini, Claude Haiku 4.5, Gemini 3.1 Flash-Lite; majority vote). Official Harbor tasks on Modal; Claude Code for Anthropic models, Codex for OpenAI models, Terminus-2 otherwise.

    Legalharbor3 subsets140 samplesAgent CLI
  • SWE Atlas - Refactoring

    70 real-world refactoring tasks across 10 open-source repositories. A trial passes when the repository's tests still pass and an LLM judge confirms every must-have rubric. Official Harbor runner on Modal; Claude Code for Anthropic models, Codex for OpenAI models, mini-swe-agent otherwise, as on Scale's leaderboard.

    Codingharbor1 subset70 samplesAgent CLI
  • SWE Refactor Bench

    20 long-horizon whole-repository stack migrations (language, framework, platform, build toolchain), graded by a three-stage ladder: model audit gate, behavioural suite (40), six adversarial verifier models (60). Official Harbor harness in Docker-capable Modal VMs; Claude Code or Codex per the benchmark's pins.

    Codingrepo1 subset20 samplesAgent CLI
  • SWE-Bench ProMax

    170 large multi-file refactoring issues (created after 2025) in C, C++, Go, Java, Python, Rust and TypeScript repos; official evaluator, mini-swe-agent with the paper's 300-step / $10 limits.

    Codingrepo1 subset170 samplesAgent CLI
  • SWE-bench Multimodal

    Real GitHub issues with screenshots and diagrams from JavaScript/TypeScript repos, graded by the repos' tests (% resolved). SWE-agent's multimodal config for every model; images are shown to the model.

    Codingrepo2 subsets580 samplesvisionAgent CLI
  • SWE-sweep

    100 real repositories in 22 languages, each at a commit holding many real bugs (4,068 in total). With no hints, the agent has 12 hours per repository to find and fix as many as it can; hidden per-bug tests decide, and any regression zeroes the repository. Official Harbor tasks on Modal with mini-swe-agent for every model, as on swesweep.com.

    Codingharbor1 subset100 samplesAgent CLI
  • SaaSBench

    30 long-horizon enterprise SaaS engineering tasks (8 languages, 6 databases): build a production backend from a PRD, verified by a dependency-aware DAG evaluator (HTTP, SQL, browser, LLM judge). Official runners inside a Docker-capable Modal VM per task.

    Codingrepo1 subset30 samplesAgent CLI
  • Terminal-Bench 4.0

    66 hard, realistic terminal tasks (software engineering, science, ML, operations, hardware, security, media), each checked by a hidden verifier. Official Harbor runner on Modal; Claude Code for Anthropic models, Codex for OpenAI models, Terminus-2 otherwise, as on the tbench.ai leaderboard.

    Codingharbor1 subset66 samplesAgent CLI
  • Terminal-Bench-Science 0.1

    70 expert-curated scientific research workflows across the life, physical, earth, mathematical and engineering sciences, each checked by an offline verifier. Official Harbor runner on Modal; Claude Code for Anthropic models, Codex for OpenAI models, Terminus-2 otherwise, as on the Terminal-Bench-Science leaderboard.

    Scienceharbor1 subset70 samplesAgent CLI