● Covered by 1 source · 1 reportMedium impact1 neutral
Surge AI Releases DAYJOB: Tax Benchmark for Evaluating AI Agents on Professional Tax Assignments
Key points
- Surge AI launched DAYJOB: Tax benchmark.
- Evaluates AI agents on 50 professional tax assignments.
- Tests specialized expertise and professional judgment.
- Claude Sonnet 5.5 leads with 10.8% pass rate.
Introduction of DAYJOB: Tax
Surge AI has released DAYJOB: Tax, the latest addition to its DAYJOB benchmark family. This new benchmark focuses on evaluating AI agents' performance in professional tax assignments. It aims to test AI capabilities beyond simple instruction following, requiring specialized expertise and complex decision-making in tax preparation, compliance, and advisory services.
Benchmark Design and Evaluation
The DAYJOB: Tax benchmark comprises 50 distinct environments covering eight areas of tax practice. AI agents are presented with client requests and supporting records, then tasked with producing deliverables such as tax returns, workpapers, technical reviews, or client advice. The evaluation criteria include tax knowledge, calculations, retrieval, instruction following, and crucially, professional judgment, such as recognizing consequential issues not explicitly requested by the client.
Example Scenario and Performance Metrics
An example scenario involved a client seeking advice on condo sale timing for tax benefits. While one AI agent (Astra) failed to identify a critical deadline that could affect eligibility for a tax exclusion, another (Opus) consistently identified it. Performance is measured by a pass@1 rate at a 95% rubric threshold, assessing calculations, supporting evidence, consequential issues, and the completeness of the deliverable.
Initial Leaderboard Results
Initial results from the DAYJOB: Tax leaderboard show Claude Sonnet 5.5 leading with a 10.8% pass rate. Claude Fable 5.1 follows with 9.6%, and Claude Opus 5.5 achieved a 9.2% pass rate. These figures indicate the current state of AI performance in handling complex professional tax tasks.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Primary sources
GitHub column-tax/tax-calc-benchReporting from
Surge AI introduced DAYJOB: Tax, a new benchmark to evaluate AI agents on 50 professional tax assignments. This benchmark assesses AI's ability to handle complex tax scenarios requiring specialized expertise and professional judgment, extending Surge AI's DAYJOB family of benchmarks.