DeepAmbigQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness

Apple Machine Learning Research introduced DeepAmbigQA, a benchmark built around ambiguous multi-hop questions for measuring how completely large language models answer open-domain queries. The work targets LLMs that use integrated search tools, which already look strong on many QA tasks yet still fail to return a full answer set when a question is both ambiguous and multi-hop.

The motivating example is “Which actor from the film Heat won at least one Academy Award?” Solving it requires two linked steps: first, telling apart multiple films that share the title Heat; second, walking a large cast list, gathering evidence for each actor, and combining that evidence into a complete set of winners. DeepAmbigQA is designed so those two challenges appear together in the same item, instead of in separate evaluations.

Existing open-domain QA benchmarks rarely stress ambiguity and multi-hop evidence integration at once. Models can look capable when a question has a single clear referent or a short evidence path, then drop entities or stop early when titles collide and the answer is a set rather than one fact. DeepAmbigQA treats answer completeness as the core score, not just whether any correct name appears.

For engineers shipping search-augmented QA, that gap is operational. Incomplete answer sets show up as partial product answers, missed entities in research agents, and false confidence when the model resolves the wrong film or stops after the first award-winning actor. Builders who only track single-answer accuracy will understate risk on real multi-entity questions.

In market and competitive terms, open-domain QA with tool use is already a mainline capability race. DeepAmbigQA shifts attention from “can the model find something” to “did it enumerate the full set under ambiguity.” That is a harder, more product-relevant bar than classic single-hop or disambiguation-only suites.

What to watch next is whether model and retrieval stacks report DeepAmbigQA-style completeness alongside standard accuracy, and whether search-tool pipelines add explicit disambiguation and set-aggregation steps before they claim they handle complex open-domain questions.