Airbnb researchers have released CASTLE, a new framework designed to overcome the cold-start problem in natural language search. This issue arises when there is insufficient real search traffic to train models effectively, particularly for new features or platforms. CASTLE uses large language models (LLMs) and de-identified booking session data to generate synthetic search queries and corresponding relevance labels.
Unlike prior approaches such as InPars and Promptagator, which treat query generation and relevance labeling as separate steps, CASTLE integrates these tasks. This joint generation process aims to reduce errors and minimize discrepancies between synthetic and real-user query distributions. The framework constructs relevance labels during the generation process itself, rather than through a separate evaluation step.
CASTLE utilizes contrastive listing pairs derived from de-identified booking sessions. When a user books a property after viewing multiple options, the chosen listing becomes a positive example, and an unbooked but viewed property serves as a negative example. An LLM is then prompted to generate a query that specifically matches the positive listing while excluding the negative one. This method inherently establishes relevance labels.
The framework incorporates three query-generation strategies: template-based prompts, few-shot prompts using approximately 500 seed queries from user research, and attribute-grounded prompts that focus on distinguishing listing characteristics. A variety mechanism is also included to encourage diverse query lengths and phrasing. For listings without sufficient booking history, CASTLE employs a Virtual Judge, an LLM-based evaluator, to assign relevance labels. A variant, CASTLE-R, is intended for production environments with existing search traffic.
In evaluations, CASTLE demonstrated a Kullback–Leibler (KL) divergence of 1.01 when compared to real-user query distributions. This figure is significantly lower than the 9.33 achieved by the best alternative method, indicating a closer resemblance to actual user behavior.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's briefNew every morning, and the back catalogue is archived by date.
Airbnb researchers developed CASTLE, a framework that uses de-identified booking sessions and large language models (LLMs) to generate synthetic search queries and relevance labels. This system addresses the cold-start problem in natural language search by enabling model training before sufficient real user data is available. CASTLE jointly generates queries and relevance labels, differing from previous methods that separate these tasks, and achieved a KL divergence of 1.01 against real-user query distributions.