Intent Alignment Under Ambiguity

Earlier work has repeatedly found the same uncomfortable truth that models overgeneralize when summarizing.¹ Retrieval does not guarantee faithful synthesis,² and longer reasoning can improve some tasks while making others slower or worse.³ In this post we focus on intent alignment under ambiguity, a failure mode that we deem particularly interesting for agents and coding assistants.

Hallucination benchmarks ask whether a model knows what it is justified in claiming. For an assistant or coding agent, there is an equally important question if it knows what is justified in assuming the user wants. We tested the recently released Meta’s Muse Glimmer 30B and Qwen3.8 27B, together with several comparison models, and combined reading comprehension, hallucination and summarization tests, and a new intent-alignment suite.

The two recent models were clearly the most interesting systems in our hardware class of 64GB unified memory on Apple silicon.

The main result is that reasoning mode changes the behavioral profile of these models in very different ways, while the primary weakness, summarization fidelity, remains strikingly stable.⁴

Intent alignment

Most model benchmarks assume that the task is well specified and only needs execution. Typical users, however, do not communicate that way as requirements tend to be incomplete, references are contextual, scope implicit, and people routinely expect an assistant to infer obvious details while asking about consequential ones.

A useful agent therefore needs needs calibrated initiative.

• act when the intent is clear
• infer when the context strongly supports one interpretation
• ask when different interpretations would materially change the outcome
• ask only for the information that is actually needed
• do not broaden the scope of the task
• make a decision when the user has explicitly delegated it

On the other hand, a system, like an engineer, that asks a clarification question about everything is also failing by being too cautious. A benchmark needs to include negative controls specifically to prevent that strategy too.

The goal of the benchmark is to move the epistemic boundary from facts to actions. Our hallucination test asks whether the evidence even licenses an answer, and the new intent benchmark asks whether the available context licenses an interpretation.

The characteristic failure is when a model makes an unsupported assumption and then executes competently on that assumption. The result may be a technically correct answer to the wrong question. I.e. unspecified intent → hidden assumption → competent execution → wrong outcome.

Muse Glimmer and Qwen3.8 27b differences

The aggregate intent scores hide substantial variations and needs to be broken down by category. The benchmark includes multiple forms of ambiguity, missing information, and scope/target control. An overview of selected categories is shown below. The full benchmark is available in the nullbench/2026-08-18-intent-alignment-under-ambiguity.

Glimmer is notably better at several forms of practical intent judgment. It handles multiple ambiguities well, performs strongly on operational boundaries, and is relatively good at asking for the missing information that matters.

Its weak repository-target result in no-think mode is also worth noticing. Turning thinking on dramatically improves that category, from 25 to 64, even though Glimmer’s overall result becomes worse.

This is why an overall intent score alone is inadequate. Thinking does not simply make Glimmer better or worse. It redistributes where it is careful.

Qwen3.8 shows that thinking substantially improves its behavior on multiple ambiguities, DevOps boundaries and evidence-backed inference, while repository target control remains weak and minimal-information acquisition actually declines.

This broadly agrees with recent independent work on ambiguous coding requirements. ClarifyCodeBench finds that strong code generation does not imply strong clarification ability, and that increasing reasoning compute improves code correctness much more than it improves ambiguity detection.⁵ UnderSpecBench stated that in underspecified DevOps tasks, agents tend not merely to fail but to guess, producing wrong-target and over-scope actions in a large fraction of runs.⁶ Those are close to the failures we are trying to isolate here.

Hallucination and Intent Alignment relationship

Our prior hallucination benchmark is fundamentally an epistemic-control test. It includes false premises, fabricated entities, private or unavailable information, unsupported exactness, partially knowable questions, ambiguity, subjective-consensus traps and other situations where fluent continuation is not necessarily justified. The new intent alignment benchmark extends that logic.

The common problem is unlicensed completion. Language models are trained to continue incomplete structures as that ability is useful almost everywhere. The problem in engineering is often that some gaps are supposed to remain gaps. A model has to distinguish between information that can legitimately be inferred and information that must be supplied externally. AbstentionBench reaches a similar conclusion from the factual side. Across twenty frontier models, scaling alone did little to solve unanswerable and underspecified questions, while reasoning fine-tuning degraded abstention performance by 24% on average.⁷ Intent alignment appears to be the agentic version of the same problem.

Qwen3.8 uses thinking productively

Qwen3.8’s result is the strongest case in this run for useful test-time computation. Turning thinking on, raises the overall result by 15.6 points and is observed to be generally beneficial. The gain is concentrated in the areas where additional reasoning appears genuinely useful.

A median response of 7.4 seconds on our test machine is still quite practical for a deliberate local assistant. Even the p95 remained around twenty seconds. This gives Qwen3.8 an unusual operating point. The no-think model is very fast, while the thinking model can spend additional compute and obtain a large improvement without routinely disappearing into multi-minute reasoning. The gain is not universal, however, as its summarization score barely changes. That is difficult to explain as insufficient reasoning capacity.

Glimmer demonstrates the opposite reasoning trade-off

Muse Glimmer reaches essentially the same top-level performance without needing the same reasoning strategy.

The no-think configuration is therefore the clearly better general operating point. Thinking triples the median response time and more than triples the p95 while slightly lowering overall quality. This is consistent with our earlier finding that reasoning mode is not a universal quality switch.³ It helps when the task contains structure that further reasoning can legitimately exploit. It can hurt when the uncertainty is irreducible. If a question lacks enough information, reasoning cannot manufacture that information. It can only generate increasingly plausible assumptions about what the missing information might have been.

That creates the distinction:

• reducible uncertainty: more reasoning may resolve it
• irreducible uncertainty: more reasoning may merely rationalize a guess

Glimmer seems unusually good at making some of those stopping decisions.

The summarization collapse remains

In our tests, reading comprehension asks whether the model correctly understands what the source says, while summarization asks whether it can restate that source without changing its meaning, scope, uncertainty, or balance. A model can therefore score highly on comprehension and correctly identifying the main idea, details, chronology, or implied conclusion, and still fail summarization by dropping caveats, broadening a population, turning association into causation, omitting null results, or making the source sound more certain than it was. The distinction is that comprehension measures understanding, whereas summarization measures preservation under transformation.

This part is almost unchanged from our earlier work. We previously wrote about structural overgeneralization in model summaries and later saw the same problem in tool-assisted research: models can retrieve good evidence and still alter its meaning during synthesis.^{1 2} The new models do not solve this issue.

All four configurations are poor. That result is interesting because reading comprehension is high. Qwen3.8 thinking scores almost 89 on comprehension while averaging 31.5 on summarization. Glimmer no-think is at 86 versus 30.8. The models appear to possess the information required for a faithful summary and then systematically transform it. We should note that this is often subtle and can be in the forms of a qualifier disappearing, a population restriction being omitted, an observational result becoming more causal, a null finding losing salience, or a cautious conclusion becoming a cleaner headline.

Peters and Chin-Yee found the same pattern in 4,900 scientific summaries. Most tested systems overgeneralized beyond their source material, with some prominent models doing so in 26–73% of cases; model summaries were nearly five times as likely as human-written science summaries to broaden the conclusions.⁸ ([DOI] 1 )

This remains one of the most insteresting signals in the benchmark because it rules out the simple explanation that the models did not fail because they could not read the source, but they failed while transforming something they had already understood.

More time did not rescue the pathological thinkers

Our original benchmark had a five-minute hard cutoff. If a model failed to return a final answer, the item received zero. Five minutes already seemed generous for these tasks, but it left open the possibility that some thinking configurations were being artificially suppressed. With more time the results barely changed, as the Qwen3.6 27B thinking example shows:

The longer timeout did allow the model to continue beyond five minutes. It simply did not improve the measured result.⁹ Benchmark discussions often treat test-time compute as if it were free, while it often makes models impractical.

The two models we would actually keep

Once latency and reasoning behavior are included, our practical comparison crystallizes into a simple table of strengths and weaknesses.

For this hardware class, the two configurations worth focusing on are therefore Glimmer no-think and Qwen3.8 think. They reach nearly the same overall result through different mechanisms. Glimmer appears to have a relatively strong immediate stopping rule. Qwen3.8 benefits substantially from additional deliberation but can still keep that deliberation inside an acceptable latency envelope.

Use cases and consequences

The practical risk increases as model output becomes more operational.

From understanding to boundary preservation

The combined test now separates four capabilities that ordinary leaderboards tend to collapse:

The results suggest these capabilities do not scale together automatically. A model can read extremely well and summarize badly. It can become much better at hallucination control with reasoning while barely improving source preservation. It can detect false premises but still choose an unsupported repository target. It can spend several additional minutes reasoning and produce no better answer. That is the part of the current model landscape we think deserves more attention.

Conclusion

Our previous work focused on hallucination, overgeneralization and synthesis fidelity. The intent benchmark extends the same idea into agent behavior. The common question is whether the model knows when its next inference is justified.

For local agents in the ~ 30B class, Muse Glimmer and Qwen3.8 show that this is already a practical problem. Both are capable enough to run useful workflows on commodity machines, and both also show that strong comprehension and strong reasoning do not guarantee faithful boundary preservation. Qwen3.8 demonstrates that additional reasoning can buy a large amount of useful epistemic judgment at reasonable latency. Glimmer demonstrates that a strong model can sometimes do better by stopping earlier. But neither solved summarization fidelity.

The important benchmark question for agents becomes whether the model knows which parts of the task they are entitled to complete without asking. Such is the boundary between useful autonomy and competent overreach.