Query Decomposition Does Not Solve Context Starvation in LLM Applications

The Problem with Query Decomposition

The article demonstrates that query decomposition, a common practice in LLM applications, does not inherently solve the problem of "context starvation." This issue arises when a multi-part query is broken down, and while relevant passages for each sub-query are successfully retrieved, not all of them make it into the limited context window provided to the LLM.

Illustrative Example

Using an example of a chat widget for GitLab's documentation, a five-part user query was processed. Despite the pipeline successfully retrieving all five correct passages corresponding to the sub-queries, only one passage was included in the final 2,000-token context. This indicates that four out of five sub-intents were starved of context, even though their evidence was found.

Defining Context Starvation

Context starvation is defined as a sub-intent receiving no allocation in the final packed context, regardless of whether its evidence was successfully retrieved. This definition focuses on allocation rather than retrieval, as current recall metrics might report perfect system performance even when sub-intents are starved due to allocation failures.

Distinction from Other Failure Modes

The article distinguishes context starvation from "semantic dilution" and "context poisoning." Semantic dilution occurs at retrieval time when a complex query's embedding is too generalized, preventing relevant evidence from being found. Query decomposition helps address semantic dilution, but it does not resolve context starvation, which happens later in the pipeline during context packing.