文章摘要
ARC Prize官网的排行榜页面,展示了ARC-AGI基准测试的验证排行榜、社区排行榜以及多个竞赛的排名情况。
文章总结
好的,这是根据您的要求,对原文主要内容进行的中文重述,已保留关键细节并删减了与主题无关的导航、页脚等元素。
ARC-AGI-3 排行榜
ARC-AGI 基准测试已从早期版本(ARC-AGI-1 和 2)的被动流体智力测量,演进到 ARC-AGI-3,该版本挑战 AI 智能体在新型交互环境中进行实时适应。
排行榜上的散点图展示了每项任务成本与性能之间的关键关系——这是衡量效率的重要指标。真正的智能不仅在于解决问题,更在于以最少的资源高效地解决问题。
数据解读
- 推理系统趋势线:代表同一模型在不同推理水平下的表现,用连线连接。这些趋势线展示了增加推理时间如何影响性能,通常随着思考时间的增加而呈现渐近行为。
- 基础大语言模型:代表标准语言模型(如 GPT-4.5 和 Claude 3.7)的单次推理结果,不具备扩展推理能力。这些点展示了模型未经额外推理增强的原始性能。
- Kaggle 系统:展示了来自 Kaggle 挑战赛的竞赛级提交方案,这些方案在严格的计算约束下运行(120 个评估任务,50 美元计算预算)。它们代表了为 ARC Prize 专门构建的高效方法。
排行榜细分
| AI 系统 | 作者 | 日期 | 系统类型 | ARC-AGI-1 | ARC-AGI-2 | ARC-AGI-3 | 每任务成本 | 总成本 (V3) | 代码/论文 | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | Claude Opus 5 (High) | Anthropic | 2026-07-24 | CoT | 97.5% | 88.3% | 30.2% | $1.45 | $20.7K | 📄 | | GPT-5.6 Sol (Max) | OpenAI | 2026-07-09 | CoT | 96.5% | 92.5% | 7.8% | $1.44 | $25.1K | — | | GPT-5.6 Terra (Max) | OpenAI | 2026-07-09 | CoT | 96.5% | 83.9% | 0.8% | $1.09 | $7.9K | — | | GPT-5.6 Luna (Max) | OpenAI | 2026-07-09 | CoT | 88.0% | 59.5% | 0.2% | $0.670 | $3.2K | — | | Claude Opus 4.8 (High) | Anthropic | 2026-06-01 | CoT | 92.0% | 72.1% | 1.5% | $2.74 | $10.0K | — | | GPT-5.5 (xHigh) | OpenAI | 2026-04-22 | CoT | 95.0% | 85.0% | N/A | $1.87 | N/A | — | | Claude 4.7 (Max) | Anthropic | 2026-04-16 | CoT | 92.0% | 75.8% | N/A | $7.43 | N/A | — | | GPT-5.4 (xHigh) | OpenAI | 2026-03-04 | CoT | 93.7% | 74.0% | N/A | $1.52 | N/A | — | | Gemini 3.1 Pro (Preview) | Google | 2026-02-19 | CoT | 98.0% | 77.1% | N/A | $0.962 | N/A | 📄 | | Gemini 3 Deep Think (2/26) | Google | 2026-02-12 | CoT | 96.0% | 84.6% | N/A | $13.62 | N/A | — | | GPT-5.2 (Refine.) | Johan Land | 2026-02-03 | Refinement | 94.5% | 72.9% | N/A | $38.99 | N/A | 📄💻 | | GPT-5.2 (xHigh) | OpenAI | 2025-12-11 | CoT | 86.2% | 52.9% | N/A | $1.90 | N/A | — | | Opus 4.5 (Thinking, 64K) | Anthropic | 2025-11-24 | CoT | 80.0% | 37.6% | N/A | $2.40 | N/A | — | | GPT-5.1 (Thinking, High) | OpenAI | 2025-11-13 | CoT | 72.8% | 17.6% | N/A | $1.17 | N/A | — | | GPT-5 Pro | OpenAI | 2025-10-06 | CoT | 70.2% | 18.3% | N/A | $7.14 | N/A | 📄 | | Claude Sonnet 4.5 (Thinking 32K) | Anthropic | 2025-09-29 | CoT | 63.7% | 13.6% | N/A | $0.759 | N/A | 📄 | | GPT-5 (High) | OpenAI | 2025-08-07 | CoT | 65.7% | 9.9% | N/A | $0.730 | N/A | — | | Grok 4 (Refine.) | J. Berman | 2025-08-07 | Refinement | 79.6% | 29.4% | N/A | $30.40 | N/A | 💻 | | Grok 4 (Thinking) | xAI | 2025-07-09 | CoT | 66.7% | 16.0% | N/A | $2.17 | N/A | 📄 | | o3-Pro (High) | OpenAI | 2025-06-10 | CoT + Synthesis | 59.3% | 4.9% | N/A | $7.55 | N/A | 📄 | | o3 (High) | OpenAI | 2025-04-16 | CoT | 60.8% | 6.5% | N/A | $0.834 | N/A | — | | o4-mini (High) | OpenAI | 2025-04-16 | CoT | 58.7% | 6.1% | N/A | $0.856 | N/A | — | | Claude 3.7 (16K) | Anthropic | 2025-02-24 | CoT | 28.6% | 0.7% | N/A | $0.510 | N/A | 💻 | | Gemini 2.5 Pro (Preview) | Google | 2025-02-19 | CoT | 33.0% | 3.8% | N/A | $0.813 | N/A | 📄 | | o3-mini (High) | OpenAI | 2025-01-31 | CoT | 34.5% | 3.0% | N/A | $0.547 | N/A | — | | o3 (Preview, Low) ¹ | OpenAI | 2024-12-20 | CoT + Synthesis | 75.7% | 4.0% | N/A | $200.00 | N/A | 📄 | | ARChitects | ARC Prize 2024 | 2024-11-03 | Custom | 56.0% | 2.5% | N/A | $0.200 | N/A | 📄💻 | | NVARC | ARC Prize 2025 | 2024-11-03 | Custom | N/A | 27.6% | N/A | $0.200 | N/A | 📄💻 | | Human Panel | Human | N/A | N/A | 98.0% | 100.0% | N/A | $17.00 | N/A | — |
注释
- 仅显示运行成本低于 10,000 美元的系统。
- 对于无法生成完整测试输出的模型,剩余任务将被标记为错误。
- 标记为“预览”的结果是非官方的,可能基于不完整的测试。
- ¹ ARC-AGI-2 分数基于部分测试结果和 o1-pro 定价估算。
- ² 临时成本估算基于 Gemini 3 Pro 定价。模型将在发布后重新测试。
评论总结
根据评论内容,主要观点和论据总结如下:
1. 对Opus 5在ARC-AGI-3上表现突出的质疑(认可度较高) - 多数评论认为Opus 5的显著提升可能是“benchmaxxing”(针对基准测试优化),而非通用智能的进步。 - 关键引用: - “Given the large outsized jump solely in the ARC-AGI-3 score, it would suggest that the model didn't become significantly more intelligent overall, but significantly better at solving those specific problems.”(kypro) - “Appears to be benchmaxxing”(KaoruAoiShiho,附链接)
2. 基准测试的可信度与局限性(认可度中等) - 评论指出基准测试容易被操纵,且与真实世界实用性脱节。 - 关键引用: - “I think it's way too easy to be deceptive with these benchmarks now... All you need is a naughty little markdown document that provides explicit instructions regarding how to solve the new puzzle variant”(bob1029) - “solving ARC-AGI and being useful turned out to be two different problems”(luciana1u)
3. 对模型实际表现的失望(认可度中等) - 部分用户反映,尽管模型在基准测试中表现优异,但实际使用体验并未显著提升。 - 关键引用: - “Why Anthropic models are always leapfrogging these benchmarks, but in real life work I do feel like after 3 weeks I am back to Claude Opus 4.5?”(throwaw12)
4. 成本与可及性讨论(认可度较低) - 评论提到模型运行成本高昂(如$20,000),并质疑排行榜中部分模型是否真的符合成本限制。 - 关键引用: - “cost 20k ???? man those are like software engineer from third world country”(tonyhart7) - “Only systems which required less than $10,000 to run are shown... Am I lost or are their many models on this ranking (Opus 5 included) that clear this?”(tudelo)
5. 对Fable模型缺席的遗憾(认可度较低) - 有评论认为Fable模型代表了某种“上限”,其缺席影响了后续模型评价。 - 关键引用: - “Why is Fable not on here? I wish Fable hadn't come out because it's taking the wind out of every release”(dyauspitr)
总结:评论整体对Opus 5在ARC-AGI-3上的表现持怀疑态度,认为其可能是针对基准测试的优化而非通用智能提升;同时指出基准测试易被操纵、与实际应用脱节,并关注成本与模型可及性问题。