Token Efficiency of Programming Languages for LLM Agents Questioned

Token Efficiency Claims for LLMs

There is a recurring assertion that dynamic programming languages, or those with concise syntax, offer better token efficiency when used with Large Language Models (LLMs). This idea is frequently cited in discussions and even appears in AI-generated search summaries, suggesting that omitting explicit type declarations in dynamically typed languages makes code more compact and thus reduces token cost.

Initial Findings and Discrepancies

One widely referenced study indicated a significant token cost gap, with languages like Clojure and J demonstrating substantially lower token counts compared to static languages such as C, Rust, Go, and C++. For instance, C was found to be 2.6 times less token-efficient than Clojure, and J achieved nearly half of Clojure's token count for certain tasks.

Critique of Evaluation Methodology

A critical issue with these initial experiments is their reliance on trivial problems, often sourced from platforms like Rosetta Code. Problems solvable in as few as 70 to 109 tokens are not representative of the 'real work' involved in more complex programming tasks. Performance metrics derived from such simple problems may not accurately reflect token efficiency in practical applications, where the bulk of the work extends beyond basic output generation.

Generalization Concerns

The concern is that results from trivial tasks do not generalize to more substantial problems. Previous evaluations have shown that methods appearing highly effective on simple tasks can yield very different outcomes when applied to slightly more complex scenarios. This suggests that the claimed token efficiency gains of dynamic languages might diminish or disappear when evaluated against problems requiring more extensive logic and code.