● Covered by 1 source · 1 reportMedium impact
Schema Harness Achieves ~99% Efficiency on ARC-AGI-3 Public Set
Key points
- Schema achieved ~99% Relative Human Action Efficiency on ARC-AGI-3 public set.
- Models actuate tasks by forming hypotheses and revising based on observations.
- Performance improved significantly from 0.51% to 13.33% on semi-private and public sets.
Introduction to Schema
Schema is a new harness developed for frontier models, allowing them to interact with game environments in a way akin to physicists. The framework focuses on transforming abstract game mechanics into executable programs, enabling agents to test and validate their hypotheses against the reality of the game environment.
Understanding ARC-AGI-3
ARC-AGI-3 presents a unique challenge where AI agents must operate without explicit game rules or objectives. Agents receive minimal input, consisting only of a grid representation and possible actions. This environment necessitates the formation of sophisticated models to hypothesize how actions affect the state of the game.
Significant Achievements
Recent performance metrics reveal that frontier models improved their Relative Human Action Efficiency (RHAE) significantly, from 0.51% to 13.33% on the public set after several iterations. This demonstrates an upward trajectory in AI capability, though gaps remain relative to human performance benchmarks.
Implications for AI Learning
The success of Schema highlights a pivotal shift in AI training methodologies, underscoring the importance of how models utilize their learning frameworks. By formalizing the process of state grounding and mechanism discovery, it offers insights into enhancing unsupervised learning capabilities in AI.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Primary sources
arXiv 2402.12275Reporting from
Schema, a new harness for frontier models, achieves approximately 99% efficiency on the ARC-AGI-3 public set. This milestone indicates significant advancements in AI agents' ability to understand and navigate complex environments without explicit rules, enhancing their capability to learn and adapt in unsupervised situations.