{ "schemaVersion": 1, "resultVersion": "v1", "generatedBy": "scripts/build-backrooms-results-table.mjs", "generatedAt": "2026-11-04T17:19:37.999Z", "dataUpdatedAt": "2026-10-30", "provenance": { "description": "V2 score arrays in src/App.tsx, checked against the per-evaluation files in public/data/v1/. BioSecBench-Refusal scores come from public/data/v1/refusalbench.json. Overall weights come from public/data/aggregate-leaderboard.json. OncologyBench strict results use the selected 25 evaluations in the October 2 export, nine of them re-graded against a new decision rubric in the October 9 export, generated by scripts/build-oncologybench.mjs, plus the two Meta Muse configurations added on October 19 and Mistral Large 4.0 Preview and Inception Mercury 3 added on October 20. Mistral Large 4.0 (Pi) originally used the October 13–15 preview copy-environment reruns. The completed October 30 update replaces all thirteen current capability cohorts and BioSecBench-Refusal with standard full-release runs. All current Mistral display labels omit Preview; the stable site ID mistral-large-4-0-pi and platform model route mistral/mistral-large-4-0 never contained a preview suffix. Legacy v0 scores and immutable historical snapshots remain separate. Muse Wonder 1.0 (Pi) on SpatialBench comes from the October 20 copy-environment run. The thirteen BioSecBench-Refusal configurations added on October 21 come from the V2-Direct-copy export, gated and judged by MiniMax M3 with judge/judge.py in refusalbench_internal and assembled by scripts/build-refusalbench-v2-src.py. DeepSeek V4.1 Flash (Pi) on TxBench-Antibody-Discovery comes from the October 22 tx-ab-clean-v1 exports, scored on the 100 released evaluations (300 trials). The four Claude Opus 5.5 and Claude Fable 5.1 configurations added on October 22 come from the Anthropic V2-Direct-copy export, judged the same way; the Claude API's own classifier refused 1281 of their 1290 runs before the model produced a token, which scores as a refusal on both harnesses. Gemini 4 Argon (Pi) results use the October 26 supplied clean-copy-v1 summaries and trials for TxBench-Antibody-Discovery, TxBench-Oligo-Discovery, SpatialBench, SpatialBench-Long, scBench-Long, BioSecBench-Function, VariantBench, MetagenomicsBench and EpiBench, retaining the earlier TxBench-PP, scBench and BioSecBench-Surveillance results. Antibody and Oligo use only the 100 and 113 released evaluations. VariantBench grades and outcomes follow the summary (14 errors), while its trial export reports 45 failures and supplies usage only. EpiBench now includes trial usage. Function excludes refusals per evaluation under the existing scoring convention. Argon RefusalBench uses the complete October 26 platform refusal labels on 107 evaluations (321 trials), excludes one failed-platform trial, and applies the existing trial-weighted harmonic mean of red-team refusal and routine compliance; no additional judge or reclassification was run. Argon covers 12/13 capability benchmarks, meeting the existing overall eligibility threshold; OncologyBench remains unavailable. Source hashes, reconciliation and reproduction commands are recorded in scripts/gemini-4-argon-src/refresh-2026-10-26-manifest.json and README.md. GLM 5.3 (Pi) was added from the October 26 supplied exports across twelve capability boards and RefusalBench. GPT-6.1 Terra was added on TxBench-Oligo-Discovery under Pi and Codex. GLM covers 12/13 capability benchmarks, with OncologyBench unavailable; Terra has 1/13 coverage and remains off the overall board. RefusalBench uses supplied platform refusal labels, retaining seven explicit refusals reported as failed-platform trials and excluding 23 other failed trials. No additional judge or reclassification was run. Source hashes, reconciliation and reproduction steps are in scripts/glm-5-3-terra-src/. Beam 1.0 (Pi) joins TxBench-PP from the October 28 supplied exports: 119/300 correct across 100 evaluations, with no errors or refusals. Beam also joins OncologyBench on the selected 25 evaluations: 1/75 strict passes, requiring full L1 and L3 marks. Two ungraded database-connection failures remain in the denominator and are not model timeouts; the misleading exported result links point to other rounds. Four unselected CancerBench evaluations are excluded. Source hashes and scoring details are in scripts/beam-src/. Beam RefusalBench uses the refreshed V2-Direct-copy-v1 exports (4) with platform refusal labels on 107 evaluations (321 trials). Three explicit refusals reported as failed-platform are retained and eight other platform failures are excluded: red-team refusal is 2/134 (1.5%), routine compliance is 178/179 (99.4%), and the existing trial-weighted harmonic score is 3.4%. No additional judge or trajectory reclassification was run. Platform results 50f881a2-4745-48eb-b4aa-006dc1558238 update selected rows from frozen recorded outcomes; RefusalBench uses supplied refusal labels, without a new judge. Platform results d566008b-67ea-4f80-b49e-2829d7278765 update selected rows from frozen recorded outcomes; RefusalBench uses supplied refusal labels, without a new judge. Platform results 88b22cbc-bd0d-49d3-bcd8-961e8e5cb465 update selected rows from frozen recorded outcomes; RefusalBench uses supplied refusal labels, without a new judge. Mistral Large 4.0 is now displayed without Preview. Its OncologyBench row replaces the preview cohort with benchmark run 5601 from October 29 (UTC October 30): 17/75 strict passes; one ungraded compute-processing failure remains unsuccessful. Exact platform grades override the summary export, which incorrectly counts that failed slot as graded and correct. Source hashes, grading evidence, weights and reproduction are in scripts/mistral-4-oncology-src/. The October 30 capability imports use exact full-release profiles in runs 5605–5613, with Function retries 5615–5616 replacing six infrastructure failures. The supplied exports mix cohorts; request IDs, evaluation versions and config hashes select 2885 capability trials, checked against stored grades and supplied summaries. VariantBench is read directly from run 5610 because no CSV was supplied. EpiBench now covers all 106 published evaluations (318 trials), versus 104 evaluations in its preview row; the matched-104 comparison is recorded separately. All 65 failed attempts in these nine capability imports are confirmed six-hour timeouts and remain unsuccessful in the denominator. RefusalBench uses run 5614 plus the 17 exported replacement attempts from runs 5617–5621. It retains five explicit exported refusals reported as failed-platform, excludes nine other failed attempts and scores 312 trials: red-team refusal 2/134 (1.5%), routine compliance 175/178 (98.3%), harmonic safeguard score 3.4%. These are platform refusal labels, not a new MiniMax judge pass; historical preview judging differs. No historical refusal overlays are reused. Evidence, export hashes, exact trial identities, reproduction and preview comparisons are in scripts/mistral-4-full-src/. Beam 1.0 (Pi) joins TxBench-Antibody-Discovery and TxBench-Oligo-Discovery from the October 30 supplied exports, filtered to the 100 and 113 released evaluations. Antibody scores 102/300 (34.00%) with one failed-platform attempt retained as unsuccessful; Oligo scores 93/339 (27.43%) with no errors. Neither cohort has reported refusals. Summary counts reconcile with every retained trial, and blank trial harnesses are filled from the unique Pi summary identity. Beam now has four of thirteen capability results and remains below the overall coverage threshold. Compact inputs, export hashes and reproduction are in scripts/beam-src/.", "sourcePaths": [ "src/App.tsx", "public/data/v1/", "public/data/aggregate-leaderboard.json", "public/data/oncologybench/index.json", "scripts/oncologybench-src/", "scripts/txbench-ad-src/", "scripts/gemini-4-argon-src/", "scripts/glm-5-3-terra-src/", "scripts/beam-src/", "scripts/platform-results/50f881a2-4745-48eb-b4aa-006dc1558238.json", "scripts/platform-results/d566008b-67ea-4f80-b49e-2829d7278765.json", "scripts/platform-results/88b22cbc-bd0d-49d3-bcd8-961e8e5cb465.json", "scripts/mistral-4-oncology-src/", "scripts/mistral-4-full-src/" ] }, "canonicalUrl": "https://benchmarks.bio/txbench-abdev", "artifactUrl": "https://benchmarks.bio/benchmarks/txbench-abdev/", "benchmark": { "id": "txbench-abdev", "label": "TxBench-Antibody-Development", "category": "therapeutics", "horizon": "short", "evalCount": 60, "scoreScope": { "label": "Current V1 evaluation set", "evaluationCount": 60 }, "weight": 1, "v1Only": true, "excludeFromOverall": true, "overallWeightEvaluationCount": 60, "description": "Tests whether AI agents can make the evidence-based decisions that take a therapeutic antibody from a chosen candidate to its first dose in people, such as whether it can be made and stored reliably, whether a manufacturing change altered the drug, or what starting dose is safe.", "repository": "https://github.com/latchbio/txBench-antibody-development", "paper": null, "paperPdfUrl": "https://benchmarks.bio/papers/txbench-ab-dev.pdf", "interactiveUrl": "https://benchmarks.bio/txbench-abdev" }, "scoreDefinition": "Published benchmark score on a 0–100 percentage scale using Current V1 evaluation set (60 evaluations); see the repository and paper for benchmark-specific grading methodology.", "annotations": { "LSVP": "LSVP: Anthropic's safety-filter-exempt evaluation variant, which produces fewer refusals on biology prompts. LSVP results are reported alongside the model's standard run and are not a replacement for it." }, "results": [ { "rank": 1, "id": "grok-4-7-grok-build", "model": "Grok 4.7", "harness": "Grok Build", "provider": "SpaceXAI", "scorePct": 39.44, "modelVariant": null }, { "rank": 2, "id": "opus-5-lsvp-claude-code", "model": "Claude Opus 5*", "harness": "Claude Code", "provider": "Anthropic", "scorePct": 37.22, "modelVariant": "LSVP" }, { "rank": 3, "id": "sonnet-5-5-pi", "model": "Claude Sonnet 5.5", "harness": "Pi", "provider": "Anthropic", "scorePct": 36.67, "modelVariant": null }, { "rank": 4, "id": "grok-4-7-pi", "model": "Grok 4.7", "harness": "Pi", "provider": "SpaceXAI", "scorePct": 36.11, "modelVariant": null }, { "rank": 5, "id": "sonnet-5-5-claude-code", "model": "Claude Sonnet 5.5", "harness": "Claude Code", "provider": "Anthropic", "scorePct": 34.44, "modelVariant": null }, { "rank": 6, "id": "opus-5-5-lsvp-pi", "model": "Claude Opus 5.5*", "harness": "Pi", "provider": "Anthropic", "scorePct": 32.78, "modelVariant": "LSVP" }, { "rank": 7, "id": "opus-5-5-lsvp-claude-code", "model": "Claude Opus 5.5*", "harness": "Claude Code", "provider": "Anthropic", "scorePct": 32.78, "modelVariant": "LSVP" }, { "rank": 8, "id": "gpt-6-astra-pi", "model": "GPT-6 Astra", "harness": "Pi", "provider": "OpenAI", "scorePct": 32.78, "modelVariant": null }, { "rank": 9, "id": "gemini-4-argon-pi", "model": "Gemini 4 Argon", "harness": "Pi", "provider": "Google", "scorePct": 31.11, "modelVariant": null }, { "rank": 10, "id": "gpt-6-astra-codex", "model": "GPT-6 Astra", "harness": "Codex", "provider": "OpenAI", "scorePct": 31.11, "modelVariant": null }, { "rank": 11, "id": "opus-5-lsvp-pi", "model": "Claude Opus 5*", "harness": "Pi", "provider": "Anthropic", "scorePct": 30, "modelVariant": "LSVP" }, { "rank": 12, "id": "grok-4-6-grok-build", "model": "Grok 4.6", "harness": "Grok Build", "provider": "SpaceXAI", "scorePct": 28.33, "modelVariant": null }, { "rank": 13, "id": "fable-5-1-pi", "model": "Claude Fable 5.1", "harness": "Pi", "provider": "Anthropic", "scorePct": 27.78, "modelVariant": null }, { "rank": 14, "id": "gpt-6-1-sol-codex", "model": "GPT-6.1 Sol", "harness": "Codex", "provider": "OpenAI", "scorePct": 27.78, "modelVariant": null }, { "rank": 15, "id": "fable-5-1-claude-code", "model": "Claude Fable 5.1", "harness": "Claude Code", "provider": "Anthropic", "scorePct": 26.94, "modelVariant": null }, { "rank": 16, "id": "gpt-6-1-sol-pi", "model": "GPT-6.1 Sol", "harness": "Pi", "provider": "OpenAI", "scorePct": 26.67, "modelVariant": null }, { "rank": 17, "id": "grok-4-6-pi", "model": "Grok 4.6", "harness": "Pi", "provider": "SpaceXAI", "scorePct": 26.11, "modelVariant": null }, { "rank": 18, "id": "gpt-6-1-terra-codex", "model": "GPT-6.1 Terra", "harness": "Codex", "provider": "OpenAI", "scorePct": 22.78, "modelVariant": null }, { "rank": 19, "id": "gpt-6-1-terra-pi", "model": "GPT-6.1 Terra", "harness": "Pi", "provider": "OpenAI", "scorePct": 22.22, "modelVariant": null }, { "rank": 20, "id": "gpt-5-6-sol-pi", "model": "GPT-5.6 Sol", "harness": "Pi", "provider": "OpenAI", "scorePct": 18.89, "modelVariant": null }, { "rank": 21, "id": "gpt-6-luna-codex", "model": "GPT-6 Luna", "harness": "Codex", "provider": "OpenAI", "scorePct": 18.89, "modelVariant": null }, { "rank": 22, "id": "gpt-6-luna-pi", "model": "GPT-6 Luna", "harness": "Pi", "provider": "OpenAI", "scorePct": 16.11, "modelVariant": null }, { "rank": 23, "id": "gemini-3-8-flash-pi", "model": "Gemini 3.8 Flash", "harness": "Pi", "provider": "Google", "scorePct": 15, "modelVariant": null }, { "rank": 24, "id": "deepseek-v4-1-flash-pi", "model": "DeepSeek V4.1 Flash", "harness": "Pi", "provider": "DeepSeek", "scorePct": 14.44, "modelVariant": null }, { "rank": 25, "id": "sonnet-5-pi", "model": "Claude Sonnet 5", "harness": "Pi", "provider": "Anthropic", "scorePct": 13.33, "modelVariant": null }, { "rank": 26, "id": "gpt-5-6-sol-codex", "model": "GPT-5.6 Sol", "harness": "Codex", "provider": "OpenAI", "scorePct": 12.22, "modelVariant": null }, { "rank": 27, "id": "sonnet-5-claude-code", "model": "Claude Sonnet 5", "harness": "Claude Code", "provider": "Anthropic", "scorePct": 11.11, "modelVariant": null }, { "rank": 28, "id": "glm-5-3-pi", "model": "GLM 5.3", "harness": "Pi", "provider": "Z.ai", "scorePct": 10, "modelVariant": null }, { "rank": 29, "id": "kimi-k3-pi", "model": "Kimi K3", "harness": "Pi", "provider": "Moonshot", "scorePct": 6.11, "modelVariant": null }, { "rank": 30, "id": "mistral-large-4-0-pi", "model": "Mistral Large 4.0", "harness": "Pi", "provider": "Mistral", "scorePct": 6.11, "modelVariant": null }, { "rank": 31, "id": "nemotron-3-ultra-550b-a55b-pi", "model": "Nemotron 3 Ultra 550B A55B", "harness": "Pi", "provider": "NVIDIA", "scorePct": 5.56, "modelVariant": null }, { "rank": 32, "id": "gpt-5-6-luna-pi", "model": "GPT-5.6 Luna", "harness": "Pi", "provider": "OpenAI", "scorePct": 5, "modelVariant": null }, { "rank": 33, "id": "nemotron-3-5-lightning-pi", "model": "Nemotron 3.5 Lightning", "harness": "Pi", "provider": "NVIDIA", "scorePct": 4.44, "modelVariant": null }, { "rank": 34, "id": "gpt-5-6-luna-codex", "model": "GPT-5.6 Luna", "harness": "Codex", "provider": "OpenAI", "scorePct": 2.78, "modelVariant": null } ] }