Experiment Investigates AI Models' Performance on 'Pelican-on-Bicycle' SVG Generation

Background of the Benchmark

Simon Willison's prompt, 'Generate an SVG of a pelican riding a bicycle,' has become a well-known informal benchmark for evaluating major large language model releases. This benchmark often generates significant discussion on platforms like Hacker News regarding its usefulness and the possibility of AI labs optimizing their models specifically for this prompt, a phenomenon termed 'pelicanmaxxing'.

Experiment Design and Models Tested

To investigate 'pelicanmaxxing,' an experiment was conducted using a grid of 48 prompts, combining 8 animals (pelican, flamingo, heron, otter, raccoon, antelope, whale, cat) with 6 vehicles (bicycle, unicycle, skateboard, scooter, plane, boat). The phrasing for each prompt mirrored Willison's original.

Seven frontier models were tested via OpenRouter: GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2, and DeepSeek V4 Pro. Three samples were generated per prompt at a temperature of 1.0, resulting in 1,008 SVGs.

Evaluation Process

Each generated SVG underwent a three-stage evaluation pipeline. First, SVGs were rendered to PNGs, with regeneration attempts recorded for invalid or unrenderable outputs (11 retries out of 1,008 generations).

Next, GPT-5.6 Luna acted as an LLM judge, scoring each image on a 1-5 scale for the animal, the vehicle, and the coherence of the action. An average of these three ratings formed the overall 'judge score' for each image. The code for this experiment is publicly available on GitHub.