LAUNCH ETA: 2026 July

Searching Is Not Knowing

We’re evaluating current-news research as a demonstration that language-model capability is not a single scalar. A model may know how to invoke a search tool yet be unable to distinguish a wire report from an obscure aggregator. It may retrieve the correct document and then silently change “will occur within 30 days” into “has occurred”. It may find several mutually inconsistent updates and compress them into whichever headline best fits a coherent narrative. It may produce an elegant military style or morning briefing whose classification markings, probability bands, and operational language conceal the fact that the underlying research process was weak.

In this test, we maintain that the ~ 30B parameter class is likely to remain a practical upper bound for broadly available commodity-hardware LLM deployment for the next several years, although advances in memory capacity, pricing, and model efficiency could shift that threshold over time. For this reason, we evaluate several open-weight or locally deployable models in a live-news research and synthesis task, and we analyze the results through the lens of our prior “summarization collapse” framework1.

  • Qwen3.6-35B-A3B
  • Qwen3.6-27B
  • Gemma 4 31B
  • Gemma 4 26B-A4B
  • Gemma 4 12B running through MLX
  • Qwen3-Coder-30B-A3B
  • GLM-4.7-Flash-30B
  • GPT-OSS-20B

Our initial request was intentionally broad:

Give a brief about the last few days of the Iran war.

Each model then received a more explicit research request as a follow up prompt:

Find out more details, determine what is actually happening now, project the next few days and explain why, using a short military-style briefing.

We used the MCP enabled OpenCode harness with web-search and page-fetching tools available. The comparison therefore evaluates deployed systems rather than isolated model weights. The observable behavior reflects the model, system prompt, search interface, result ranking, tool schema, parser, context management, inference backend, quantization and output limits. We should note that tool-enabled model performance is a property of the entire execution stack.

Our experiment surfaced three broad classes of failure:

  • Research failures: the system did not search, searched too little, selected weak sources or anchored on stale material.
  • Synthesis-fidelity failures: the system started with relevant evidence but changed its scope, modality, uncertainty, salience or temporal meaning while compressing it.
  • Generative failures: the system fabricated facts, introduced unsupported analysis, retained corrected falsehoods or lost structural control of its output.

The strongest system was Qwen3.6-35B-A3B, but it was also not sufficiently reliable to publish without verification. Our central conclusion was that current-news analysis is a temporal evidence-management problem that requires reasoning over retrieved information, and has structural limitations with current architectures that are not necessarily resolved by scaling or improved retrieval.

A factual stress test

The underlying news situation was particularly useful because its state changed repeatedly within a narrow time window.

Contemporaneous Reuters reporting established that:

  • The United States and Iran had signed a 14-point interim memorandum on June 17, 2026.
  • The text was transmitted to the U.S. Congress on June 18.
  • The memorandum created a 60-day negotiation period for a final settlement.
  • The United States committed to lifting its naval blockade on Iranian ports within 30 days.
  • Iran committed to allowing free commercial transit through the Strait of Hormuz for 60 days.
  • A planned Swiss negotiation session was disrupted or publicly called off on June 19.
  • Switzerland confirmed on June 20 that confidential U.S.-Iran discussions were nevertheless continuing at Bürgenstock.
  • Israeli strikes killed at least 16 people in Lebanon shortly after a renewed ceasefire was due to take effect.2 3 4

This required more than ordinary fact retrieval. A reliable system had to construct an event-state transition:

Agreement reached
→ memorandum signed
→ implementation talks scheduled
→ public session disrupted
→ confidential discussions continue

The difference between those states is analytically important, as “Talks were called off” may have been accurate at one point in the sequence. “Talks continue confidentially” was accurate at a later point. A summary that selects either statement without its temporal context is not necessarily hallucinating, but it is still wrong as a representation of the current state.

The same problem applied to the naval blockade:

Commitment to remove blockade within 30 days
≠ authorization to begin removal
≠ partial operational drawdown
≠ blockade fully removed

Where several models collapsed these distinctions.

A Detailed Scorecard

Our scorecard should make synthesis fidelity explicit. Each system is rated from 1 to 10 across eight dimensions:

  • Tool control: recognition that current information requires retrieval and effective invocation of tools
  • Research coverage: breadth and persistence of investigation
  • Source judgment: authority, recency, independence and relevance of selected evidence
  • Temporal reconstruction: ability to reconcile updates into an event sequence
  • Factual control: resistance to fabrication and unsupported specificity
  • Synthesis fidelity: preservation of scope, modality, uncertainty, salience and framing
  • Forecast calibration: distinction between evidence and prediction
  • Output integrity: structure, completeness and usability

The overall score is calculated as:

10% Tool control
10% Research coverage
15% Source judgment
15% Temporal reconstruction
15% Factual control
15% Synthesis fidelity
10% Forecast calibration
10% Output integrity
ModelToolsCoverageSourcesTemporalFactsSynthesisForecastsOutputOverall
Qwen3.6-35B-A3B9976765971/100
Qwen3.6-27B9864554859/100
Gemma 4 31B7465656858/100
Gemma 4 26B-A4B3455656752/100
Gemma 4 12B MLX7354545749/100
Qwen3-Coder-30B-A3B8323223836/100
GLM-4.7-Flash-30B8422322331/100
GPT-OSS-20B2332111522/100

These scores are comparative judgments based on the observed run per system. They are not statistical estimates of general model quality. The closeness of Qwen3.6-27B and Gemma 4 31B was interesting as they failed in different ways:

  • Qwen27 conducted substantially more research but overextended its evidence.
  • Gemma31 exercised better stylistic restraint but did too little investigation.

Whichever approach is preferable depends on the subtask.

An explanatory distinction: hallucination vs synthesis distortion

The term “hallucination” is often used too broadly to explain most of these failures. Some outputs contained direct fabrications, while others began from real evidence but altered it during compression.

Direct fabrication

A fabricated claim is absent from the admissible evidence. Examples from the transcripts include:

  • GPT-OSS-20B inventing secret U.S.-Iran-Saudi talks before searching
  • GPT-OSS-20B inventing increased GCC naval patrols
  • Qwen3.6-27B locating a major diplomatic signing “in Tehran”
  • Qwen3-Coder producing a detailed aircraft-loss inventory from an inadequately verified source

Synthesis distortion

A synthesis distortion changes the meaning of evidence that was actually retrieved. Common transformations included:

“will remove within 30 days”
→ “has removed”

“interim memorandum”
→ “peace settlement”

“scheduled public session canceled, confidential talks continue”
→ “talks postponed indefinitely”

“ceasefire announced, lethal exchanges continue”
→ “de-escalation phase”

“conditional private-investment mechanism”
→ “$300 billion reconstruction grant”

The latter class is more difficult to detect because each sentence resembles something present in the source material. The model is not inventing an entirely new event but is changing tense, modality, scope, attribution or salience until the resulting statement no longer means the same thing.

Our previous work on summarization collapse

Our “summarization collapse” analysis 1 provides a useful behavioral taxonomy for these distortions and the benchmark has evolved substantially since the earlier test reports.

The framework separates four recurring defects:

  • Overgeneralization: extending a claim beyond its stated time, population, location or conditions
  • Hedge loss: replacing qualifications such as “may,” “suggests,” “planned” or “conditional” with certainty
  • Omission bias: removing null findings, adverse effects, counterevidence or inconvenient constraints
  • Framing distortion: changing the substantive interpretation through stronger or more emotionally loaded language5

Our previous benchmark operationalizes those ideas by checking whether summaries preserve details such as sample size, geography, time window, null results, adverse effects and hedging. It penalizes causal upgrades, prescriptive additions and omission of countervailing evidence.1 Those checks map well onto the current-news experiment.

Nullmirror categoryNews-analysis equivalentObserved example
OvergeneralizationScope or temporal expansionAn interim framework became a permanent settlement
Hedge lossModality collapseA 30-day obligation became a completed blockade removal
Omission biasSalience distortion“Fragile calm” summaries omitted lethal Lebanon fighting
Framing distortionNarrative relabelingA negotiation window became a “landmark peace agreement”
Causal upgradeUnsupported motiveA maritime measure was declared a bargaining tactic without evidence
Prescriptive additionUnrequested operational adviceGPT-OSS added military-readiness recommendations

We argued that these failures are structural, because instruction-tuned models are rewarded for fluent, confident and compact answers, while faithful qualifications make prose appear less decisive.

That interpretation is supported by a larger study by Uwe Peters and Benjamin Chin-Yee, who compared 4,900 model-generated scientific summaries with source texts and human-written research digests. Most tested models broadened conclusions beyond their source constraints, with reported overgeneralization rates ranging from 26% to 73% for several prominent systems. Model summaries were nearly five times more likely than human summaries to contain broad generalizations. Explicit instructions to be accurate did not consistently eliminate the problem.6 While the scientific-summary task is not identical to live-news synthesis, the transformation pressure can be thought of as the same:

qualified evidence
→ compact representation
→ fluent declarative narrative

The model has to decide what to preserve. Details that constrain a conclusion are often precisely the details most likely to be discarded:

  • The action has not happened yet.
  • The agreement is interim.
  • The source is attributing the claim to one side.
  • A later report partially supersedes the earlier report.
  • The ceasefire is nominal rather than operational.
  • The downside evidence contradicts the main narrative.

This is why we decided that synthesis fidelity deserves its own score and factual accuracy alone is insufficient. A summary can contain no entirely invented sentence and still be operationally misleading.

What our previous work explains—and what it does not

Our previous work explains part of the observed behavior, but it should not become a universal explanation. We can explain

  • Why commitments became completed actions
  • Why interim arrangements became final agreements
  • Why caveats disappeared
  • Why null or contradictory evidence was omitted
  • Why diplomatic framing dominated battlefield evidence
  • Why the requested “military briefing” style increased categorical language
  • Why concise models appeared careful while omitting significant information
  • Why aggregate model rankings concealed task-specific failure profiles

However, observed failures are further:

  • Selection of an old or low-authority source
  • Fabrication before retrieval
  • Persistence of a false claim after correction
  • Arbitrary, unsupported numeric forecasts
  • Failure to invalidate prior conversational state

These occur at different stages of the pipeline, if we consider a useful systems model as:

Retrieval trigger
→ Query generation
→ Source admission
→ Claim extraction
→ Temporal reconciliation
→ Synthesis
→ Forecasting
→ Presentation

Our previous work on summarization framework primarily explains defects in the synthesis stage. The “retrievability is not discovery” argument addresses source admission and visibility, our “short horizons, fragile state, orchestration first” argument addresses state management across the full workflow.7 8

Retrieval is not source evaluation

Our previous work on “retrievability is not discovery” is also applicable7. Search and RAG systems rank documents based on signals such as lexical relevance, embeddings, metadata and source indexes. Those signals do not inherently measure:

  • Truth
  • Authority
  • Investigative independence
  • Corroboration
  • Recency relative to the event
  • Whether a newer report supersedes the document
  • Whether a source is reporting original information or copying another source7

The Iran experiment exposed this repeatedly as GLM-4.7-Flash found detailed June 2–3 battlefield material and used it to answer a June 20 current-state question. The source was detailed and semantically relevant, but temporally obsolete for the requested task.

Qwen3-Coder found a page containing a complete-looking equipment-loss inventory that was entirely unrelated to the current conflict. The structured precision made it attractive to summarize, but the model did not establish that the source was authoritative or independently corroborated or even relevant.

The retrieval layer had succeeded in that it returned relevant text, but the research system failed because it treated retrieval rank as evidentiary admission. A credible news pipeline needs a separate source-admission function:

admissibility =
    authority
  × temporal relevance
  × directness
  × independence
  × corroboration
  × claim-specific competence

So a source can be topically relevant and still receive an admissibility score close to zero.

RAG reduces hallucination but does not eliminate it

The assumption that web search grounds a model sufficiently is also common, but contradicted by research on retrieval-augmented generation. RAGTruth contains roughly 18,000 retrieval-augmented outputs with span-level annotations for unsupported or contradictory content. Its premise is that models continue to produce claims that go beyond or conflict with retrieved evidence even when relevant material is present.9 That is exactly what occurred here too.

  • Qwen35 retrieving the agreement timetable but announced complete implementation
  • Qwen27 retrieving the diplomatic framework but invented a Tehran signing
  • Qwen-Coder retrieving one unrelated equipment loss table and treating its contents as confirmed
  • GPT-OSS searching only after hints but retained claims from its original fully fabricated answerss

Retrieval changes the model’s evidence environment, but does not impose an evidence boundary. Without an explicit claim-support gate, the final generator can freely interpolate between:

  • Retrieved facts
  • Parametric memory
  • Narrative priors
  • Earlier assistant messages
  • Generic geopolitical patterns
  • Genre conventions
  • Unsupported causal interpretation

The resulting answer may look grounded because citations or source names are present, even when the individual claims are not entailed by those sources.

Model-by-model findings

Qwen3.6-35B-A3B

Qwen35 was the strongest research system in the comparison.

It recognized the temporal nature of the request, searched without being prompted, expanded the investigation into multiple operational domains and opened enough sources to produce a genuinely useful strategic picture.

From the two simple prompts above, its final answer covered:

  • U.S.-Iran negotiations
  • The Lebanon front
  • Strait of Hormuz implementation
  • Nuclear issues
  • Sanctions
  • Domestic political friction
  • Short-horizon scenarios
  • Indicators to monitor

No other model matched its combination of breadth and organization.

The principal weakness was evidence reconciliation. It simultaneously reported that talks were “postponed indefinitely” and that senior envoys were moving toward Switzerland. A later Reuters update confirmed that confidential discussions were continuing. The model had enough information to detect uncertainty but compressed the evidence into a categorical status.

Its blockade summary exhibited modality collapse. A commitment to lift a blockade within a defined period became a statement that the blockade had already been lifted. The forecast section also used probability labels without a calibration framework.

Qwen’s official documentation emphasizes long context, agentic operation and model-specific reasoning/tool parsers. This supports the observed capability but also reinforces the deployment caveat: tool behavior depends on the surrounding inference stack.10

Possible role: primary investigator and analytical synthesizer, with mandatory downstream verification.

Qwen3.6-27B

Qwen27 displayed similarly strong tool initiative and search coverage.

It independently investigated:

  • Ceasefire terms
  • Lebanon
  • Hormuz traffic
  • Nuclear negotiations
  • Sanctions
  • Political criticism
  • Agreement weaknesses

Its research behavior was substantially stronger than Gemma31’s, but its factual restraint was substantially weaker.

The standout hallucination was that Trump and Pezeshkian signed the memorandum “in Tehran”. That location was not supported by the reference reporting. The model also supplied numerical forecast bands such as “70%+” and “30–70%” without explaining reference, calibration or confidence. Those numbers were not derived from the retrieved evidence and were not supported by any explicit probabilistic model. Those numbers provided the appearance of precision without the substance of probabilistic forecasting.

A community-built local research system reported strong results using Qwen3.6-27B inside an iterative search scaffold, but the project itself cautions that configuration-level benchmark results do not necessarily predict performance on arbitrary research topics.11

Possible role: high-recall research worker that generates queries, sources and candidate claims. It should not control final confidence or publication.

Gemma 4 31B

Gemma31 searched proactively but performed little visible follow-up. Its strength was rather output control and the response was compact, well structured and less prone to sprawling analytical invention than the Qwen outputs. Its apparent restraint was partly an omission artifact as the briefing:

  • Misstated the agreement date
  • Treated an interim memorandum as a completed peace agreement
  • Said the nuclear issue had been excluded rather than deferred and unresolved
  • Omitted continuing lethal activity in Lebanon
  • Introduced the phrase “DEFCON equivalent” arbitrarily without an operational basis

“DEFCON equivalent” is a useful example of style-induced epistemic inflation. It adds military register without adding measurable information. Google documents native function calling for Gemma 4 and specifies state-handling requirements for tool calls and thinking traces. These implementation requirements make it plausible that tool behavior can vary materially across wrappers and backends.12

Possible role: constrained editorial compression from a closed, verified evidence set.

Gemma 4 26B-A4B

Gemma26 initially declined to answer because the event was outside its internal knowledge. That is preferable to fabrication, but it is still a failure for a system that visibly had web-search tools. The user had to actively hint on tool usage and internet search. After retrieval, the answer was plausible but shallow. It identified the broad diplomatic and regional dimensions without reconstructing the agreement details or current Swiss negotiation state. This is primarily a tool-initiation failure, rather than a summarization failure.

A production agent should not rely entirely on the model to infer when retrieval is mandatory. Temporal predicates such as latest, current, ongoing, today, right now, and last few days should be treated as explicit retrieval triggers.

Gemma 4 12B

Gemma12 searched proactively and produced a readable briefing at a smaller deployment scale. Its main distortion was premature phase classification as it called the situation as “de-escalation phase” and predicted a significant reduction in kinetic activity despite ongoing lethal exchanges in Lebanon at the time of writing. The model appears to have treated the diplomatic framework as the dominant signal and downweighted contradictory operational evidence.

This is a synthesis-fidelity failure involving both framing and salience:

Diplomatic document receives high narrative weight
Operational contradiction receives low narrative weight
→ "de-escalation" becomes the summary frame

We should note that MLX is an Apple-silicon machine-learning framework and runtime ecosystem, not a single model configuration. Different Gemma conversions may use different quantization schemes, templates and precision levels.13

Possible role: lower-cost local summarization of already verified material.

Qwen3-Coder-30B-A3B

Qwen3-Coder is the most revealing specialist failure.

The model:

  • Recognized the need to search
  • Used tools correctly
  • Followed the requested format
  • Produced clean, concise output
  • Maintained structural coherence

It then claimed “73 aircraft destroyed, 29 naval losses”. The response expanded this into a detailed inventory that included:

  • 17 U.S. aircraft
  • Ten F-7s
  • Eight F-14s
  • Seven C-130s
  • Six Il-76s
  • Four F-15s

The evidence trail was one fetched military aggregation page. The model did not establish independent corroboration or relevance. This is unverified inventory transduction:

Structured table found on web page
→ table assumed authoritative
→ every row converted into confirmed fact that fits the request
→ aggregate totals presented as intelligence

The sequence also showed post-hoc evidence laundering. The model generated its briefing before fetching the detailed page, then repeated the briefing with a precise loss table appended. Retrieval made the narrative look more researched without causing the model to rebuild or challenge it. Qwen3-Coder is officially optimized for agentic coding and software workflows.14 Its behavior demonstrates that agentic competence does not transfer uniformly across domains. Choosing files, commands and code edits is not the same as judging journalistic evidence.

GLM-4.7-Flash-30B

GLM searched immediately but selected poor evidence for the task. It centered a June 20 briefing on operational reports from June 2–3 and classified negotiations as deadlocked despite current reporting that confidential talks continued.

This was a compound failure:

  • Source-admission failure
  • Stale-source anchoring
  • Temporal-state collapse
  • Synthesis distortion

The final response also degraded structurally:

  • Duplicate sections
  • Empty or malformed tables
  • Unresolved placeholders
  • Stray Markdown delimiters
  • Incomplete paragraphs

The phrase: “cruise missiles deployed in Iraqi waters” was notable because its unusual geographic specificity should have triggered additional verification. GLM’s official model card reports considerably stronger agentic and browsing benchmark performance than this transcript would suggest, including a reported BrowseComp score of 42.8 and τ²-Bench score of 79.5.15 The divergence strengthens the system-level interpretation. A model may perform well under its vendor’s validated harness and poorly under another tool, parser or runtime stack.

GPT-OSS-20B

GPT-OSS-20B produced the most severe failures. It answered the first current-news request without even searching and invented:

  • Iranian missile launches
  • Israeli strikes on Iranian-backed militias in Syria and Iraq
  • Secret U.S.-Iran-Saudi meetings
  • Increased GCC naval patrols
  • Humanitarian preparations

It then stated that its information only extended to 2024. After the user challenged the fabrication and hinted at tools, the model searched but failed to invalidate the earlier claims. Several reappeared in the later briefing. This is hallucination persistence caused by contaminated conversational state.

The appropriate recovery process would have been:

User challenges answer
→ mark every unsupported prior claim invalid
→ exclude prior assistant content from evidence
→ retrieve current sources
→ rebuild answer from admitted claims only

Instead, the system effectively performed:

Preserve original narrative
→ add search results
→ revise selected details

OpenAI’s own model card reports that, without browsing, GPT-OSS-20B achieved 6.7% accuracy and a 91.4% hallucination rate on SimpleQA, together with 15.5% accuracy and a 53.2% hallucination rate on PersonQA. The card notes that smaller models possess less world knowledge and tend to hallucinate more, while browsing can reduce the problem.16 The observed behavior is therefore consistent with the model’s documented factual limitations.

The hallucination and distortion cabinet

Not every item below is a direct hallucination. The distinction is the point.

ModelOutputClassificationWhy it matters
Qwen35Talks “postponed indefinitely” while envoys were moving toward SwitzerlandTemporal collapseMultiple valid updates were compressed into the wrong current state
Qwen27Agreement signed “in Tehran”FabricationA salient diplomatic location was invented
Gemma31Forces at a “DEFCON equivalent” readiness levelPseudo-technical framingMilitary vocabulary substituted for measurable evidence
Gemma12Conflict entered a “Verification Window” and de-escalation phaseFraming and omission distortionDiplomatic framing suppressed contradictory battlefield activity
Qwen-CoderExactly 102 losses, including 17 U.S. aircraftSource launderingExtraordinary precision was inherited from one inadequately vetted page
GLM“Cruise missiles deployed in Iraqi waters”Unsupported specificityA geographically unusual claim was accepted without corroboration
GPT-OSSSecret U.S.-Iran-Saudi meetingsDirect fabricationThe event was generated before retrieval
GPT-OSSGCC naval patrols and border-town injuriesHallucination persistenceUnsupported claims survived after the user corrected the model

The “helpful chatbot” persona and briefing-style authority

Previous benchmarks found that models frequently revert to a shared helpful, verbose and agreeable assistant persona even when given negative style constraints. We argued that much apparent stylistic flexibility is a brittle learned pattern rather than a fully controllable faculty.17

The military-briefing request prompt activated a different learned genre:

  • Classification labels
  • Threat conditions
  • Operational sectors
  • Scenario trees
  • Readiness language
  • Probability bands
  • “Bottom line” conclusions

Genre imitation is not epistemically neutral.

Compare:

The ceasefire may fail if fighting continues.

with:

THREAT CONDITION: ELEVATED
HIGH PROBABILITY OF CEASEFIRE COLLAPSE

The second formulation appears to reflect institutional analysis. It may contain no additional evidence. Style alone can therefore amplify error in two ways:

  • It increases user trust in unsupported claims.
  • It encourages the model to fill expected genre slots, even when no evidence supports them.

A military briefing may “expect”:

  • An adversary objective
  • A force posture
  • A confidence estimate
  • A likely course of action
  • A dangerous course of action
  • Indicators and warnings

If the research process did not establish those elements, the model may manufacture plausible entries to complete the pattern. This is genre-completion pressure.

Why aggregate leaderboards are insufficient

Our broader LLM fingerprints argument has been that aggregate rankings conceal operationally unacceptable weaknesses. A model can lead an overall benchmark while failing a specific subtask such as hedging fidelity, style control, controversial-content handling or latency.17 This is consistent with the eight-model experiment.

Qwen3-Coder is an especially clear example:

  • Strong tool posture
  • Strong output structure
  • Strong coding specialization
  • Extremely weak evidence judgment on the news task

Gemma31 showed the inverse profile:

  • Limited research depth
  • Better restraint and compression
  • Stronger suitability for downstream editing

GLM’s official browsing results did not predict its observed deployment behavior.

GPT-OSS can perform strongly on several reasoning tasks while remaining hazardous for unsupported factual answering.

A single overall model score therefore answers the wrong question.

The operational questions to us remain which model, under which runtime, should perform which stage of this workflow, with which externally enforced constraints?

Research on model routing reaches a similar conclusion. Routing surveys formalize systems that select models according to task requirements, cost, latency, privacy and quality rather than sending every query to a single generalist.18 The present results support routing on epistemic role, not only cost.

Possible model routing example

Workflow stagePreferred systemConstraint
Temporal query classificationDeterministic policy or small classifierCurrent-event triggers must force retrieval
Broad query decompositionQwen3.6-35BDo not allow it to publish directly
High-recall source discoveryQwen3.6-27BTreat all discoveries as candidates
Exact document extractionStructured parser or constrained model callPreserve tense, modality, attribution and dates
Source-quality assessmentRules plus independent verifierDo not infer authority from search rank
Temporal reconciliationExternal event ledgerSort by event time and update time
Contradiction detectionSeparate verifier modelOriginal synthesizer should not grade itself
Scenario generationQwen3.6-35BRequire explicit triggers and counterevidence
Executive compressionGemma 4 31BClosed evidence set; no new facts
Low-cost local rewriteGemma 4 12BVerified source packet only
Coding-agent workQwen3-CoderKeep within its specialization
Current factual work with GPT-OSSOnly behind hard retrieval gatesAbstain when retrieval fails
GLM researchRetest under validated serving stackCurrent run is not production-acceptable

The narrow difference between Qwen27 and Gemma31 illustrates why routing can improve overall performance. Qwen27 is a better investigator, while Gemma31 is a better editor. Using either for both stages sacrifices one of those strengths.

Methodological limitations

This comparison should not be mistaken for a universal model benchmark.

It used:

  • One geopolitical topic
  • One simple primary prompt sequence
  • One observed run per model
  • One product environment
  • Unspecified inference settings for several systems
  • Potentially different quantizations and context limits
  • Search results that changed over time

Our previous findings also require qualification in that we have used fixed prompts, controlled decoding and panels of open models acting as judges. Its results describe performance relative to its prompts, rubrics, hardware, backend and judge panel. Changing the panel changes the metric.19 We have also extended and refined earlier benchmark contents with results. Our prior source-bounded summarization tests are not identical to live, multi-document news research.

The Peters and Chin-Yee study provides stronger independent evidence that overgeneralization during summarization is widespread, but it concerns scientific texts rather than geopolitical reporting.6 RAGTruth establishes that retrieval does not eliminate unsupported generation, but it does not determine which of the eight models is best.9 BrowseComp evaluates difficult fact retrieval, not extended current-state synthesis. Deep Research Bench goes further by measuring long agent traces, hallucination, tool use and forgetting, but no benchmark fully captures live geopolitical reconstruction.20 21 The exact scores in this article should therefore be treated as a structured interpretation of the transcripts, not population estimates.

Conclusions

The eight-system comparison supports several engineering conclusions.

Qwen3.6-35B-A3B is the best primary researcher in this group

It demonstrated the strongest combination of:

  • Automatic retrieval
  • Search persistence
  • Topic decomposition
  • Cross-domain coverage
  • Useful analytical organization

It also made consequential errors in temporal reconciliation, implementation status and uncertainty calibration. It is a capable investigator, but not a source of record.

Qwen3.6-27B is useful for high-recall research

It retrieves aggressively and identifies relevant dimensions. Its tendency toward unsupported specificity, invented locations and false numeric precision makes it unsuitable for unsupervised publication.

Gemma 4 31B is the strongest downstream editor

It did not investigate deeply enough to lead the research process. Given a verified evidence packet, its restraint and compact output make it a plausible compression model.

Gemma 4 26B and 12B require external workflow control

The 26B system did not initiate retrieval reliably. The 12B system searched, but simplified the situation too aggressively. Both are better treated as bounded components.

Qwen3-Coder demonstrates domain-specific agent competence

It could operate the tools and produce structured output. It could not apply adequate evidentiary standards to military-loss reporting. Coding-agent skill is not general research-agent skill.

GLM’s tested deployment was unreliable

Its official benchmark profile suggests more capability than our transcripts showed. That is why checkpoint, parser, quantization and runtime must likely be evaluated together.

GPT-OSS-20B requires hard factual gates

Its initial fabrication and failure to purge false claims after correction are disqualifying for autonomous current-news work.

We can explain the synthesis layer

Our categories, overgeneralization, hedge loss, omission bias and framing distortion—describe many of the errors more accurately than hallucination.

Broader conclusions also still fit:

  • Retrieval visibility is not source quality.
  • Summarization is lossy and systematically biased toward cleaner narratives.
  • Aggregate leaderboards hide task-specific collapse.
  • Style imitation can amplify false authority.
  • A single generalist is the wrong production architecture.
  • Reliability emerges from routing, external state and verification.

The most important design principle is therefore to treat every model output as an untrusted transformation of evidence, not as evidence itself. A serious current-news research system should maintain a source ledger, claim graph, temporal event state, contradiction set, fidelity score and correction history outside the language model. The language model should search, extract, analyze or edit within a bounded stage, but it should never be the database, the verifier and the publisher at the same time.


  1. nullmirror. “LLM Fingerprints v1.3: GLM-4 Judiciary, Summarization Collapse.” September 27, 2025. ↩︎ ↩︎ ↩︎

  2. Reuters. “White House sends text of interim US-Iran agreement to US Congress.” June 18, 2026. ↩︎

  3. Reuters. “Switzerland says US-Iran talks continue at Bürgenstock, declines to identify participants.” June 20, 2026. ↩︎

  4. Reuters. “Israeli strikes kill 16 in Lebanon after truce, prospect of US-Iran talks revived.” June 20, 2026. Updated related report: “Israeli strikes kill at least 20 in Lebanon hours after ceasefire.”  ↩︎

  5. nullmirror. “Structural Overgeneralization in LLM Summarization.” August 10, 2025. ↩︎

  6. Uwe Peters and Benjamin Chin-Yee. “Generalization Bias in Large Language Model Summarization of Scientific Research.” Royal Society Open Science, 2025. arXiv version · PubMed Central full text  ↩︎ ↩︎

  7. nullmirror. “Retrievability Is Not Discovery.” September 5, 2025. ↩︎ ↩︎ ↩︎

  8. nullmirror. “Short Model Horizons Revisited.” November 21, 2025. ↩︎

  9. Cheng Niu et al. “RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models.” ACL 2024. arXiv version  ↩︎ ↩︎

  10. Qwen. “Qwen3.6-35B-A3B” official model card.  ↩︎

  11. LearningCircuit. “Local Deep Research” repository and benchmark documentation.  ↩︎

  12. Google AI for Developers:

     ↩︎
  13. Apple Machine Learning Research:

     ↩︎
  14. Qwen. “Qwen3-Coder-30B-A3B-Instruct” official model card.  ↩︎

  15. Z.AI. “GLM-4.7-Flash” official model card. Z.AI GLM-4.7 documentation  ↩︎

  16. OpenAI. “gpt-oss-120b & gpt-oss-20b Model Card.” 2025. ↩︎

  17. nullmirror. “LLM Fingerprints v1.4: The Cost of Quality, and Routing Decides Winners.” October 29, 2025. ↩︎ ↩︎

  18. Clovis Varangot-Reille et al. “Doing More with Less—Implementing Routing Strategies in Large Language Model-Based Systems: An Extended Survey.” 2025. ↩︎

  19. nullmirror. “nullbench: Judge Panel and Methodology.” August 24, 2025. ↩︎

  20. OpenAI. “BrowseComp: A Benchmark for Browsing Agents.” 2025. Research paper  ↩︎

  21. Nikos I. Bosse et al. “Deep Research Bench: Evaluating AI Web Research Agents.” 2025. ↩︎