Fast LLM Judging with No-Think Mode

Our previous benchmark runs showed that recent open-model gains often look less like clean jumps in intelligence and more like redistributions of behavior. Fine-tuning, abliteration, long-context extensions, sampler choices, and dataset curation shift where a model spends its behavioral budget. These changes can cut refusal noise, improve ergonomics, sharpen compliance, or make a model feel more direct, but they do not automatically raise the ceiling on hard reasoning.¹ Across the board, new results keep pointing back to the evaluation harness itself as one of the biggest levers.

The question in this run was more operational, can a local evaluation harness use fast no-think judges without losing the useful signal of a multi-model judge panel?

And our short answer is yes, with caveats. For compliance-style judging—JSON scoring, exact labels, refusal classification, A/B comparison, and short bounded outputs—it appears to be a better default than letting every judge enter a long reasoning phase. Under strict-output tasks and a one-minute target-answer cutoff, no-think mode often improves measured quality because it prevents models from spending the whole budget on hidden or semi-hidden treasoning before producing the actual answer.

On compliance type judge benchmarks, see the difference between “think”:

vs no-think:

“no-think”

“No-think” means the serving stack is instructed not to emit or allocate a separate reasoning phase before the final answer. In modern open-weight deployments this can mean several different things: suppressing <think>...</think> blocks, setting a reasoning budget to zero, disabling a chat-template flag such as enable_thinking, or passing a backend-specific request field such as think: false.

“no-think” is not a portable setting. For Qwen3, /think and /no_think are documented as soft switches when thinking mode is otherwise enabled, but Qwen’s own documentation says that when enable_thinking=False, those soft switches are not valid and the model should not produce think content or a <think> block.² Qwen3.5’s model card says it does not officially support the Qwen3 /think and /nothink soft switch, and that direct non-thinking responses require API/template parameters instead.³

The practical result is that prompt text like /no_think cannot not be treated as a reliable systems control. It may work for one model family, fail silently for another, and behave differently depending on the chat template.

Runtime controls: Ollama and llama.cpp

Ollama’s native API includes a think field for thinking models.⁴ In practice, we send think: false as a top-level chat request field rather than burying it inside sampler options. This matches real-world issue reports where Ollama’s chat API honored top-level think=False, while incorrectly placing think under options caused thinking tokens to consume the output budget and leave the visible response empty.⁵ For Ollama, no-think belongs in the request body as a first-class field.

llama.cpp is more conditional. The server exposes reasoning-related controls and chat-template keyword arguments, but behavior depends on the model, template, and version.⁶ For Qwen3.5-style models, upstream and user reports show that enable_thinking:false has sometimes failed to suppress thinking in specific llama.cpp builds or model/template combinations.⁷ That is why the benchmark result should distinguish between the config requested no-think and the runtime actually changed behavior. A filename tag or config flag proves intent; latency, raw-response length, visible <think> content, and token counts prove effect.

This difference showed up in the results as some llama.cpp GGUF models changed dramatically under no-think mode, whereas others were nearly identical.

Benchmark shape

The benchmark was compliance-heavy. It stressed exact format following, JSON-only answers, short labels such as A, B, GOOD, FALSE, and refusal-classification labels such as ATTEMPT, PARTIAL, and REFUSAL. It also included tone-compliance, sensitive-content boundary cases, ideological symmetry checks, factual controversy prompts, and calibration prompts.

It is not a benchmark of long-form theorem proving, multi-step coding, or open-ended planning but merely testing if a model can produce the right bounded output under tight constraints. For that task distribution, long visible or hidden reasoning is often a liability. It consumes tokens, increases latency, creates more opportunities for schema drift, and can cause a model to miss a cutoff before emitting the final answer.

NVIDIA’s Nemotron documentation describes reasoning as useful for complex multi-step questions, logical deduction, technical or mathematical problem-solving, and cases where accuracy is more important than response speed; the same documentation notes that reasoning increases latency because of additional reasoning tokens.⁸ Friendli’s Nemotron 3 documentation states the same tradeoff more directly: enable reasoning for complex or open-ended quality, disable it when fast deterministic responses are preferred.⁹. Thinking mode is a quality-speed tradeoff, but the direction of the quality effect depends on the task. For strict compliance tasks, thinking can reduce measured quality.

Target no-think results

The target-model runs showed three behavior classes.

The first class barely changed. Models such as Qwen3-Coder-Next, Qwen3-Coder-30B, Ministral, Granite tiny , and Gemma3 were effectively stable across thinking and no-thinking target runs. In those cases, either no meaningful reasoning mode was active, the no-think switch was not honored, the model was already producing short direct answers, or the benchmark’s stripping and parsing path hid the difference.

The second class improved dramatically. Qwen3.5 4B, Qwen3.5-4B, and Nemotron3-Nano showed large score and throughput gains under no-think mode. These models were not merely slower with thinking enabled but they were worse for this task too.

The third class got much faster but lost a little score. Gemma4 E4B variants showed this pattern. That is the actual expected tradeoff. Suppressing reasoning improves throughput but removes some useful deliberation. For tasks involving subtle tone, refusal boundaries, or controversy framing, that lost deliberation can matter.

The surprising part was the size of the improvement for some small thinking-capable models. The naïve assumption would be that thinking mode is slower but smarter. These runs refute that simplified view. A more accurate statement would be that thinking mode can help hard reasoning, but for bounded compliance tasks under a cutoff time, it can actively harm the measured result.

One-minute cutoff time

A model that eventually reaches the right answer after a long reasoning trace can still score poorly if it does not emit the final answer in time. A model that would be “smarter” with extended reasoning may be worse operationally if the harness needs a compact answer now.

This is especially important for local inference. Decode time is often dominated by generated tokens, not prompt ingestion. If a thinking model burns hundreds or thousands of tokens before the visible answer, the final answer may be delayed or truncated. This is why no-think mode can improve both speed and score: it reallocates the generation budget from hidden deliberation to the visible deliverable.

This does not mean hidden reasoning is useless but that the task must justify it. A compliance judge asked to emit one integer or a small JSON object is not doing the same job as a model solving a hard mathematical proof.

Judge no-think results

The judge-only rerun used the same cached target outputs but changed the judge panel to faster no-thinking judges. This test isolated judge behavior from target generation. The results showed that no-thinking judges are viable for this benchmark family, but they also showed that the new panel is not numerically interchangeable with the old one.

Scores rose broadly. Qwen3-Coder-Next rose from 67.25 to 74.0. Qwen3-Coder-Next Ollama rose from 67.25 to 71.5. Qwen3.5 4B Ollama rose from 64.375 to 68.5. Gemma4 E4B Q4 GGUF rose from 58.625 to 67.25. Granite tiny GGUF rose from 56.0 to 60.5. Olmo rose from 50.0 to 52.75.

That means the new panel is more permissive, or at least differently calibrated. The most visible changes were in consistency, refusal-gate sanity, and sensitive-content categories. This is not necessarily bad. The old panel may have been too harsh, too noisy, or too format-sensitive in certain places. But it means raw scores from the old and new panels should not be mixed in one leaderboard.

The correct interpretation is that a new panel creates a new benchmark version. It can be better operationally—faster, more stable, less noisy—but it changes the scale.

Final judge-panel recommendation

The first candidate panel was:

• Qwen3-Coder-30B-A3B-Instruct, Ministral-3-8B-Instruct, and Nemotron3-Nano-4B.

After the judge-only rerun, the better quality-oriented panel appears to be:

• Qwen3-Coder-30B-A3B-Instruct, Ministral-3-8B-Instruct, and Gemma-4-E4B-it-UD-Q4_K_XL.

The reason for the swap is that Nemotron3-Nano is attractive as a cheap exact-output judge, but Gemma4 E4B appears stronger as a target proxy under the new panel. It scored higher overall and appeared better on tone, refusal, and sensitive-content dimensions. That does not prove Gemma4 is a better judge in every sense, because target performance is only an indirect proxy for judge quality. But given the available evidence, Gemma4 is the better third judge if quality matters more than resource cost.

Nemotron3-Nano remains useful. It is small, fast, and strong on exact-output behavior. It is still a good choice for a resource-optimized panel, a first-pass filter, or a fourth low-cost judge. But it looks weaker as the default third member of a quality-oriented panel.

The recommended split is therefore:

For quality:

• Qwen3-Coder-30B-A3B-Instruct, Ministral-3-8B-Instruct, Gemma-4-E4B-it-UD-Q4_K_XL.

For resource efficiency:

• Qwen3-Coder-30B-A3B-Instruct, Ministral-3-8B-Instruct, Nemotron3-Nano-4B.

All judges should default to no-think mode for this benchmark type. The sampler should remain deterministic. Grammar constraints should be used wherever the task expects JSON, an integer, or a fixed label.

Why not Granite?

Granite tiny remains attractive on speed, but its judge profile is weaker. It underperformed on calibration and refusal sanity relative to the alternatives. A very fast weak judge can still be useful as a cheap format or structure check, but it should not be one of the three core panel members if the goal is fair scoring across subtle compliance categories. It is better suited as an efficiency-biased auxiliary judge than as a balanced panel member.

Why not use only one strong judge?

A single strong judge is cheaper and easier to reason about, but it bakes in one model family’s blind spots. A three-judge panel reduces correlated error, especially when the models come from different families and have different failure modes. In this setup, Qwen provides a strong coding/reasoning anchor, Ministral adds a compact non-Qwen perspective, and Gemma or Nemotron adds a third lineage with different stylistic and compliance tendencies. The tradeoff is obviously cost. Three judges multiply inference time, so judge speed matters. This is why no-think mode is so important for self-hosted commodity type models: it makes multi-judge scoring feasible without giving every judge an open-ended reasoning budget.

Quantization and backend differences

Ollama and llama.cpp variants should not be treated as exact apples-to-apples comparisons. As we noted in prior posts, even when model names are similar, backend differences can include quantization level, chat template, tokenizer handling, default sampler behavior, model revision, context handling, and reasoning-control implementation.

A heavier quantization can reduce memory use and improve speed while hurting quality. A different template can change whether no-think mode is honored. An Ollama renderer may handle thinking fields differently from a llama.cpp Jinja template. These differences plausibly explain why some Ollama and GGUF pairs moved together while others diverged. The reliable indicators if no-think worked are response time, generated-token count, raw response length, visible think tags, empty-after-strip failures, malformed JSON failures, and timeout rate.

What this run confirms

The run confirms four things.

1. no-think mode is a good default for strict compliance targets under a cutoff. It often improves throughput and can improve score.
2. no-think mode is viable for judges when the judge task is bounded: JSON scoring, integer scoring, exact labels, refusal classification, or A/B comparison.
3. no-think behavior is runtime- and template-dependent. Ollama’s native control is cleaner; llama.cpp requires more validation per GGUF/template/build.
4. the judge panel should be versioned. Changing judges changes the score scale. The new panel may be better, but it is not backward-compatible with the old panel.

Claims that were corrected

The first corrected claim is that /no_think can be used consistently. Qwen3 supports it as a soft switch only in specific conditions, and Qwen3.5 explicitly says the soft switch is not officially supported. Runtime parameters are more reliable than prompt strings.²³

The second corrected claim is that no-think necessarily means degraded intelligence. No-think may degrade hard reasoning, but it can improve strict compliance because the model stops spending budget on reasoning tokens and starts producing the requested bounded output.

The third corrected claim is that llama.cpp no-think can be assumed to work uniformly. It can work, and the benchmark showed cases where it clearly did, but upstream issues and the benchmark’s own mixed results show that model/template/build details matter.⁷

Practical conclusion

For this benchmark family, the strongest operating point is a fast, deterministic, no-thinking judge panel. The default quality panel should be

• Qwen3-Coder-30B-A3B-Instruct
• Ministral-3-8B-Instruct
• Gemma-4-E4B-it

A more resource-optimized panel chould swap Gemma4 for Nemotron3-Nano. A key methodological point is that evaluation harnesses are now part of model performance. A model’s apparent capability depends on sampler choices, templates, reasoning controls, output parsers, cutoffs, judge selection, and panel calibration. The benchmark does not only measure models, but models inside a serving and scoring system. When the task is strict, short, and cheaply verifiable, no-think mode is not a downgrade. In our tests, it is often the correct execution mode.