AI Models Struggle with StarCraft: Brood War, Codex Astra Leads Despite Beginner-Level Play

AI Performance in Brood War

An experiment was conducted to evaluate various AI models' ability to play StarCraft: Brood War. The results showed that none of the tested models could play beyond a beginner level. Codex Astra emerged as the leading model, consistently outperforming all others in the benchmark.

Codex Astra's Strategy and Limitations

Codex Astra's most effective strategy involved early game disruption, such as sending a Probe to attack enemy workers or buildings. This tactic was surprisingly effective against opposing agents, which often spent considerable time processing the threat. However, Codex Astra demonstrated weaknesses in sustained production, often delaying technology upgrades, trickling units into battle, and poorly managing worker units.

Codex also frequently created separate subagents for economy, production, and army control that lacked effective communication. This led to uncoordinated attacks where individual units were sent into battle without waiting for a larger, planned assault.

Grok Models' Challenges

Grok models, specifically Grok 4.6, exhibited significant issues with action execution. These models frequently produced long periods of reasoning with very few command batches. For instance, in one 43-minute game, Grok 4.6 logged over 11,000 reasoning tokens but issued only six command batches and failed to field an army.

Underlying AI Difficulties

Older AI models often treated the real-time strategy game as turn-based, leading to their defeat while they were processing. While newer models showed some improvement in recognizing the cost of thinking, some still fell into similar traps. The experiment highlights that current AI models are not yet capable of handling the complex, real-time strategic demands of Brood War effectively.