Benchmark Examines AI Development Costs and Performance Across JavaScript Frameworks

Addressing 'Vibes-Based' Development

The web development community often makes technology decisions based on subjective feelings rather than empirical evidence. Claims about framework speed or tool effectiveness are frequently made without supporting data. This trend has been noted by industry figures who criticize the lack of rigorous evaluation in adopting new technologies.

AI Agents and Framework Selection

In the context of AI-assisted development, where agents generate code, similar 'vibes-based' claims are emerging. Developers suggest certain frameworks are 'better for agents' or result in 'faster' code with 'fewer mistakes,' often without presenting benchmarks. This observation prompted the creation of a new benchmark to provide concrete data.

Benchmark Design and Methodology

A benchmark was developed to assess AI agent performance across different JavaScript frameworks. The methodology involved instructing AI agents to build a small, non-trivial Pokémon application using frameworks like Next.js, Nuxt, and SvelteKit. The application is a content-based site with search and filter functionalities, similar to a Wikipedia page.

The output from the AI agents was tested against a specification conformance suite to ensure accuracy and prevent 'cheating.' Key metrics for evaluation included the development time taken by the agents, the number of tokens spent, the resulting lines of code, and the performance and accessibility scores obtained from Lighthouse audits.

Study Scope and Limitations

The benchmark focuses on a specific type of application: a content-heavy site with limited interactivity. This design choice means the results may not directly translate to highly interactive applications like design tools or email clients. However, the chosen application provides a clear and understandable use case for evaluating AI agent performance in web development.