Benchmarks
AutomationBench
Zapier's business-workflow benchmark: 600 public tasks (Sales, Marketing, Operations, Support, Finance, HR; 100 each) on 47 simulated SaaS apps, graded by deterministic end-state assertions (strict pass rate). Official auto-bench harness, API toolset, 50 steps. Public set only: the zapier.com leaderboard uses a private, harder held-out set, so scores are not directly comparable.
Agentictool6 subsets600 samplesCustom adapterEnterpriseOps-Gym
649 public enterprise tasks across Teams, CSM, Email, ITSM, Calendar, HR, Drive and cross-domain Hybrid workflows: the agent uses stateful MCP tools over a seeded database while following internal policies, and hidden SQL checks the final state (a task passes only if every check does). Official harness and ReAct agent in oracle mode; task success rate as on the leaderboard.
Agentictool8 subsets649 samplesCustom adapterITSMBench
89 IT service-management tickets (access, offboarding, incidents, security response) worked against mock ServiceNow, Okta, Slack and Google Workspace APIs, each checked by a hidden verifier. Official Harbor tasks with the Pi agent for every model, as on the ITSMBench leaderboard.
Agenticharbor1 subset89 samplesAgent CLI