Adding "Do not guess" instruction reduces AI model hallucination in web extraction by 70%

Impact of Explicit Instructions on AI Accuracy

A recent test evaluated the performance of AI models in web extraction tasks, specifically focusing on their tendency to invent information not present on a page. The study revealed that providing a clear instruction, "Use null for any field whose value is not on the page. Do not guess," significantly improved accuracy.

Across 16 models, the rate of made-up fields dropped from 70.7% when the instruction was absent to 20.2% when it was included. This demonstrates that explicit negative constraints can be highly effective in mitigating AI hallucination in specific contexts.

Testing Methodology and Results

The test involved 42 pairs of web pages across 7 types, designed to identify whether an extractor would invent missing fields. Each pair included one page with an answer and one without, along with decoy information. Models were scored on pages where fields were missing.

For example, on a page with a decoy "Was $493.00" but no current price, all 16 models without the instruction incorrectly identified $493 as the price. With the instruction, only one model made this error. Firecrawl, another tested service, made up 24 of 36 missing fields, copying decoy information, which was higher than most models even without the explicit instruction.

The Role of a Cheap Checker Model

The study also explored the effectiveness of using a secondary, cheaper AI model to verify extracted values. A GPT-6 Luna model was used to check values returned by the contestants.

This checker model successfully identified 20 of Firecrawl's 24 made-up values without rejecting any correct ones. The cost for checking all 126 unique returned page-and-value pairs was $0.0049, suggesting a cost-effective method for improving the reliability of AI-driven data extraction.