Benchmarks
Humanity's Last Exam
2,500 expert-written closed-ended questions across dozens of subjects (multiple-choice and exact-match, ~14% with images), graded by an LLM judge. Official CAIS harness.
Knowledgeqa8 subsets2,500 samplesvisionAny APIHumanity's Last Exam: Diamond
1,000 refined HLE questions, 500 reasoning and 500 knowledge, graded by an LLM judge. Official CAIS harness, no tools.
Knowledgeqa2 subsets1,000 samplesvisionAny API