TIME serves AI crawlers a distinct, ad-embedded markdown version of its website

Dual Website Delivery for Humans and AI

TIME magazine has implemented a system that serves two distinct versions of its website content. Human users accessing the site receive the standard, full-featured HTML page, including design, images, and scripts. In contrast, identified AI crawlers are provided with a simplified, markdown-formatted version of the same content.

User-Agent Detection and Content Differentiation

The differentiation in content delivery is based on the User-Agent header sent by the client. By analyzing this header, TIME determines whether the request originates from a human browser (like Chrome or Safari) or an AI bot (such as ClaudeBot, PerplexityBot, or OAI-SearchBot). Some OpenAI bots, like GPTBot and ChatGPT-User, are blocked entirely, indicating a selective policy for AI access.

Embedded Ads and Token-Based Monetization

The markdown versions delivered to AI crawlers include embedded advertisements, facilitated by the ad-tech vendor Mobian. These ads are placed within the content where no human reader would encounter them. Each request from an AI bot for the markdown content generates a unique ad impression, identified by a UUID and counted in "tokens" rather than traditional pageviews, suggesting a new model for monetizing AI consumption of content.

Implications for Content Publishers and AI

This approach by TIME represents an evolving strategy for publishers to monetize content accessed by AI models. By serving ads directly into the data consumed by AI, publishers can potentially generate revenue from AI interactions that do not involve human eyeballs. The use of "tokens" as a metric for ad impressions also highlights a shift in how digital content consumption might be measured and valued in the age of AI.