API docs

Everything in the web app is available over HTTP. The CLI uses the same API.

Base URLhttps://benchmarkrouter.newmeasure.ai/api/v1

Quickstart

  1. 1 Create a key on the API keys page. It starts with br_ and is shown once.

    export BROUTER_API_KEY=br_...
  2. 2 Price a run, then submit it.

    curl -s https://benchmarkrouter.newmeasure.ai/api/v1/estimate \
      -H "Authorization: Bearer $BROUTER_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"benchmark_id": "hle", "model_id": "anthropic/claude-opus-5-5", "reasoning": "high"}'
    
    curl -s https://benchmarkrouter.newmeasure.ai/api/v1/runs \
      -H "Authorization: Bearer $BROUTER_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"benchmark_id": "hle", "model_id": "anthropic/claude-opus-5-5", "reasoning": "high"}'
  3. 3 Poll the run until status is succeeded or failed, then fetch the download links.

    curl -s https://benchmarkrouter.newmeasure.ai/api/v1/runs/$RUN_ID -H "Authorization: Bearer $BROUTER_API_KEY"
    curl -s https://benchmarkrouter.newmeasure.ai/api/v1/runs/$RUN_ID/download -H "Authorization: Bearer $BROUTER_API_KEY"

Benchmark and model ids come from GET /benchmarks and GET /models, or the Benchmarks page.

Authentication

Send your key as a bearer token on every request except GET /benchmarks and GET /models. A key acts as your account, so keep it secret; revoke it on the API keys page if it leaks.

Authorization: Bearer br_...

Conventions

Format
JSON request and response bodies.
Money
Amounts are credits, as decimal numbers in *_credits fields. 1 credit = $1. A run costs 10% of the model API cost; runs of your own harness are billed at the full API cost.
Ids
Runs are UUIDs. Benchmarks and models use the string ids returned by the catalog endpoints.

Errors

Non-2xx responses have the body {"error": "message"}.

StatusMeaning
400Invalid request: a missing or bad field, an unknown benchmark or model.
401Missing, malformed or revoked API key.
402Not enough credit to start a run or download results.
404Not found, or it belongs to someone else.
409The resource isn't in the right state yet, e.g. downloading a run that hasn't succeeded.
429Rate limited: too many runs or requests today.

Catalog

GET/benchmarks

No auth

Benchmarks you can run. options_schema is the JSON Schema for the benchmark's options; subsets are the subsets the benchmark itself defines. Agent benchmarks run on Harbor have a harness option: its oneOf lists the agent harnesses you can pick (for example "openhands"). Leave it out to use the benchmark's pinned harness.

Example
curl -s https://benchmarkrouter.newmeasure.ai/api/v1/benchmarks
Response
{
  "benchmarks": [{
    "id": "hle", "name": "...", "description": "...", "family": "...", "category": "...",
    "version": "...", "options_schema": { ... }, "subsets": [{ "id": "...", ... }],
    "requires_vision": false, "judge_model_id": "..." | null
  }]
}

GET/models

No auth

Models you can run against. Pass one of reasoning_levels as reasoning; it defaults to default_reasoning.

Example
curl -s https://benchmarkrouter.newmeasure.ai/api/v1/models
Response
{
  "models": [{
    "id": "anthropic/claude-opus-5-5", "display_name": "...", "provider": "anthropic",
    "reasoning_levels": ["low", "medium", "high"], "default_reasoning": "medium",
    "supports_vision": true, "input_usd_per_mtok": 5, "output_usd_per_mtok": 25
  }]
}

Runs

POST/estimate

API key

Price a run without submitting it. Each unit is one subset, priced from earlier runs (history) or token counts (tokens).

Request body
{
  "benchmark_id": "hle",
  "model_id": "anthropic/claude-opus-5-5",
  "reasoning": "high",          // optional
  "options": { ... },           // optional, validated against options_schema
  "subsets": ["..."]            // optional, default all
}
Example
curl -s -X POST https://benchmarkrouter.newmeasure.ai/api/v1/estimate \
  -H "Authorization: Bearer $BROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"benchmark_id": "hle", "model_id": "anthropic/claude-opus-5-5", "reasoning": "high"}'
Response
{
  "estimate_credits": 12.34,
  "units": [{ "subset": "...", "price_credits": 12.34, "basis": "history" }]
}

Errors · 400 invalid config, unknown benchmark or model, or a vision benchmark on a model without vision.

POST/runs

API key

Submit a run. Same body as POST /estimate.

Example
curl -s -X POST https://benchmarkrouter.newmeasure.ai/api/v1/runs \
  -H "Authorization: Bearer $BROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"benchmark_id": "hle", "model_id": "anthropic/claude-opus-5-5", "reasoning": "high"}'
Response
201 { "run": { ...see GET /runs/{id} } }

Errors · 400 invalid config · 402 balance below 10.00 credits, or the estimate exceeds your available credits (balance minus the estimates of your pending runs) · 429 daily run limit reached.

GET/runs

API key

Your runs, newest first. Same fields as a single run, without units and scores.

Example
curl -s https://benchmarkrouter.newmeasure.ai/api/v1/runs \
  -H "Authorization: Bearer $BROUTER_API_KEY"
Response
{ "runs": [{ ... }] }

GET/runs/{id}

API key

One run. status is pending, succeeded or failed. metrics and scores (the benchmark's official score) are filled once the run has succeeded. Poll this until the status is final.

Example
curl -s https://benchmarkrouter.newmeasure.ai/api/v1/runs/$RUN_ID \
  -H "Authorization: Bearer $BROUTER_API_KEY"
Response
{
  "run": {
    "id": "uuid", "status": "succeeded",
    "benchmark_id": "hle", "benchmark_version": "...", "model_id": "...", "reasoning": "high",
    "options": { ... }, "subsets": ["..."],
    "estimate_credits": 12.34, "charge_credits": 11.98,
    "metrics": { ... } | null, "error": null,
    "created_at": "...", "finished_at": "...",
    "units": [{ "subset": "...", "status": "succeeded", "n_samples": 500, "price_credits": 11.98, "metrics": { ... } }],
    "scores": [{ "label": "Accuracy", "subset": null, "value": 41.2 }]
  }
}

Errors · 404 run not found (or not yours).

Signed download links, valid for one hour: results.json (scores, one row per task, how the run was produced under provenance, and the SHA-256 of its other files under files; subset ""), tasks.csv (one row per trial) and manifest.json (every file with its SHA-256), then per subset the benchmark's own logs as logs.tar.gz and every trajectory as one line of trajectories.jsonl.gz.

Example
curl -s https://benchmarkrouter.newmeasure.ai/api/v1/runs/$RUN_ID/download \
  -H "Authorization: Bearer $BROUTER_API_KEY"
Response
{
  "files": [
    { "subset": "", "name": "results.json", "url": "https://..." },
    { "subset": "", "name": "tasks.csv", "url": "https://..." },
    { "subset": "", "name": "manifest.json", "url": "https://..." },
    { "subset": "...", "name": "logs.tar.gz", "url": "https://..." },
    { "subset": "...", "name": "trajectories.jsonl.gz", "url": "https://..." }
  ]
}

Errors · 402 balance is negative · 404 run not found · 409 run hasn't succeeded.

Billing

GET/balance

API key

Your credit balance, the estimates reserved by your pending runs, what is available for new runs, and the latest ledger entries (purchases, charges and adjustments).

Example
curl -s https://benchmarkrouter.newmeasure.ai/api/v1/balance \
  -H "Authorization: Bearer $BROUTER_API_KEY"
Response
{
  "balance_credits": 87.66,
  "reserved_credits": 12.34,
  "available_credits": 75.32,
  "ledger": [{ "id": "...", "amount_credits": -11.98, "kind": "charge", "ref": "...", "note": "...", "created_at": "..." }]
}

Start a Stripe Checkout to buy credits, between 10 and 10,000 credits (1 credit = $1). Open the returned URL in a browser.

Request body
{ "credits": 100 }
Example
curl -s -X POST https://benchmarkrouter.newmeasure.ai/api/v1/credits/checkout \
  -H "Authorization: Bearer $BROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "credits": 100 }'
Response
{ "url": "https://checkout.stripe.com/..." }

Requests

Ask for a benchmark we don't have yet. GET on the same path lists your requests.

Request body
{ "name": "SWE-bench Pro", "url": "https://...", "notes": "..." }   // only name is required
Example
curl -s -X POST https://benchmarkrouter.newmeasure.ai/api/v1/benchmark-requests \
  -H "Authorization: Bearer $BROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "SWE-bench Pro"}'
Response
201 { "request": { ... } }

Errors · 429 more than 20 requests in a day.

CLI

The brouter command wraps the same API. Install it from PyPI (Python 3.10+):

uv tool install benchmark-router-cli     # or: pipx install benchmark-router-cli / pip install benchmark-router-cli
uv tool upgrade benchmark-router-cli     # later, to update

Log in with an API key (or set BROUTER_API_KEY; set BROUTER_API_URL to point at another host).

brouter login
brouter benchmarks
brouter models
brouter estimate hle -m anthropic/claude-opus-5-5 -r high
brouter run hle -m anthropic/claude-opus-5-5 -r high -o key=value -s <subset> --wait
brouter harnesses terminal_bench
brouter run terminal_bench -m openai/gpt-5.6-sol --harness openhands --wait
brouter runs
brouter status <run-id> --wait
brouter download <run-id> -d ./results
brouter export <run-id> <run-id> -d ./export   # several runs: combined tasks.csv + manifest.json
brouter balance --ledger
brouter buy 100
brouter task <run-id>

brouter benchmark-request create "GPQA" --url https://...
brouter benchmark-request list
brouter harness request --name Acme --access api -b hle --details-file harness.md
brouter harness list
brouter comparisons estimate <harness-id> -m anthropic/claude-opus-5-5
brouter comparisons create <harness-id> -m anthropic/claude-opus-5-5 -m <model-id>
brouter comparisons list
brouter comparisons publish <comparison-id>
brouter comparisons domain <comparison-id> benchmarks.yourcompany.com
brouter comparisons unpublish <comparison-id>

A run can also come from a YAML file with benchmark, model, reasoning, options and subsets:

# run.yaml
benchmark: hle
model: anthropic/claude-opus-5-5
reasoning: high

# then
brouter run -c run.yaml --wait