Airbnb Introduces CASTLE for Training Natural Language Search Models
Airbnb introduced CASTLE, using de-identified booking sessions and LLMs to generate synthetic search queries that achieved a 1.01 KL divergence on real queries.
By Dillip Chowdary • Nov 16, 2026 • Source: InfoQ

At a glance
- Product
- CASTLE
- Developer
- Airbnb researchers
- Paper release
- Nov 16, 2026
- Baseline comparison
- Promptagator
- Query KL divergence
- 1.01
- Attribute KL divergence
- 0.08
In this post
Airbnb researchers introduced CASTLE, a framework designed to resolve the cold-start problem when training natural language search models without sufficient real-world search traffic. As detailed in InfoQ's report, the system extracts de-identified booking sessions and pairs them with large language models to simultaneously generate realistic synthetic search queries and matching relevance labels.
This analysis examines how CASTLE structures synthetic data generation, how it replaces two-stage labeling pipelines, and how it measures up against existing retrieval baselines. The system is built for search engineers, machine learning practitioners, and retrieval specialists who need to bootstrap semantic search across specialized inventories before production log data accumulates.
01Cold start hurdles in natural language search
Natural language search allows travel guests to describe detailed property preferences in conversational terms, including phrases like romantic cabin near skiing or pet-friendly beach house that sleeps eight. This shifts discovery away from rigid structured filters toward unstructured text intent.
Building effective search systems for conversational queries requires overcoming two foundational data collection obstacles. Machine learning teams must generate queries that mirror real user phrasing while accurately determining which specific inventory listings satisfy each query.
Earlier frameworks like InPars and Promptagator split query generation and relevance labeling into two disconnected stages. Those systems generate queries from documents and then run secondary model-based evaluation passes, a pipeline design that introduces compounding judgment errors and creates distribution drift away from real human queries.
02Contrastive listing pairs and generation strategies
CASTLE removes the secondary labeling pass by deriving contrastive listing pairs directly from de-identified booking histories. When a traveler books a stay after browsing alternative listings, the purchased accommodation serves as a positive example while a viewed but unbooked property provides the negative example.
An underlying large language model receives prompts to generate search queries that match the chosen property while deliberately excluding the rejected alternative. Because relevance labels emerge directly from this contrastive pairing process, the framework removes the need for downstream scoring models.
The framework integrates three complementary query-generation methods alongside a variety mechanism designed to diversify query length and phrasing:
- Template-based prompts construct queries using structured phrasing formats.
- Few-shot prompts guide generation using approximately 500 seed queries gathered from targeted user research.
- Attribute-grounded prompts focus query text on the distinguishing physical characteristics of individual listings.
Advertisement
Tech Pulse Daily
Get tomorrow's pulse first
Join engineers who read Tech Pulse before stand-up. Free, weekday mornings.
03Synthetic baselines and distribution divergence
Airbnb researchers measured query fidelity by computing the Kullback–Leibler divergence between synthetic outputs and actual production query distributions. CASTLE achieved a divergence score of 1.01 against real-user query distributions, compared with 9.33 recorded by the strongest baseline framework, Promptagator.
| Framework | Query KL Divergence | Attribute-Type KL Divergence |
|---|---|---|
| Promptagator | 9.33 | Not reported |
| Survey Seed Queries | Not reported | 0.09 |
| CASTLE | 1.01 | 0.08 |
This represents a 9.2-fold reduction in distribution divergence across full queries. When measured at the specific attribute-type level, CASTLE recorded a KL divergence of 0.08, slightly outperforming the 0.09 score achieved by manually curated survey seed queries.
04Virtual evaluation and production traffic variants
For accommodations without historical booking sessions, such as newly published property listings, the researchers built an alternative evaluation pipeline called the Virtual Judge. This evaluator uses a large language model to inspect and assign relevance scores to arbitrary query-listing pairs that lack organic engagement signals.
To transition models into active commercial use, the team also developed CASTLE-R. This operational variant takes over model fine-tuning once live user search traffic starts accumulating inside production environments.
An independent human evaluation verified the system by having three subject matter experts audit 200 sampled triplets. Evaluator agreement with CASTLE's automated relevance assignments ranged between 91% and 93% across the three reviewers.
05Evaluation split debate and data leakage guardrails
Following publication, the validation approach generated technical debate regarding whether training and evaluation splits kept template prompts and property IDs strictly disjoint. AI engineer Mark Tsai noted on LinkedIn that testing untouched real queries against isolated splits helps verify whether models learn general intent-attribute matches rather than memorizing familiar templates or inventory IDs.
Author Wendy Ran Wei confirmed that the team evaluated the model on untouched real queries while enforcing leakage guardrails during evaluation set preparation. She noted that the published paper does not include a controlled benchmark comparing random splits directly against stricter disjoint splits, identifying that comparison as a direction for future research.
Note: The complete research methodology, experimental configurations, and architectural descriptions are published on arXiv.
Frequently asked questions
What problem does Airbnb's CASTLE framework solve?
CASTLE resolves the cold-start problem in natural language search by generating synthetic search queries and relevance labels before real search traffic exists.
How does CASTLE establish relevance without a separate labeling step?
It prompts an LLM to write a query that matches a booked property from a de-identified session while excluding an unbooked alternative listing viewed during the same visit.
How did CASTLE compare against the Promptagator baseline?
CASTLE achieved a Kullback–Leibler divergence of 1.01 against real-user query distributions, delivering a 9.2-fold reduction compared to Promptagator's 9.33 score.
What is CASTLE-R?
CASTLE-R is a framework variant designed for production environments where live user search traffic has started to accumulate.
Sources
Author
Dillip Chowdary
Writes Tech Bytes coverage of AI, engineering, and the tools that actually ship. Editor of Tech Pulse Daily.
Related on Tech Bytes
Advertisement