Evaluating Local Models as VSCode Agents

We evaluated several open weight models in the ~30B-parameter class as local agents in the VSCode harness across two classes of repository work:

1. architecture and deployment planning, where the model had to inspect multiple executables and design a production integration plan
2. incident investigation, where the model had to correlate logs with configuration, networking layers, concurrency, and reconnection code before suggesting changes.

We deem single-prompt benchmarks as viable for comparison as the prompt delegates a bounded objective to an agent loop. Both tasks were single answer, multi-step, and required reasoning across multiple files. The harness provided a tool interface for searching, reading, and executing code. The tests measure whether a model can select the right evidence, reason across multiple layers, and produce a correct plan without introducing unsupported assumptions and avoid premature implementation.

Our main result was that repository-agent quality depended more on investigation strategy and epistemic discipline. A key finding is that Qwen3.6-27B stands out for initial investigation and planning → Qwen3.6-35B for focused code verification → Gemma4-31B for operational review.

Evaluation method

Every model received the same repository access and tool interface. Answers were judged on six dimensions.

Results

Overall ranking

What differentiated the models

1. Repository exploration was the strongest predictor

The best answers expanded beyond the files named in the prompt. They inspected:

• entry points;
• configuration loading;
• state and data paths;
• build scripts;
• package layout;
• signal handling;
• retry loops;
• callback ownership;
• embedded configuration;
• adjacent lifecycle code.

This produced materially better answers because repository-level questions rarely have a correct solution inside one file.

Weak models often began recommending after one or two reads, or after a failed tool call. Their outputs sounded reasonable but depended on invented assumptions.

A useful approximation is that effective repository-agent quality is the product of multiple factors, such as reasoning ability, tool reliability, exploration strategy, context retention, and factual calibration.

A failure in any factor can dominate nominal model capability.

2. Correct lifecycle classification was critical

One test required understanding which executables were persistent processes. A model incorrectly classified one as a one-shot command and therefore excluded it from the deployment design.

The mistake invalidated half of the proposed architecture. The general lesson is that an agent should establish these facts before planning:

• which execution paths block;
• which processes fork or remain foregrounded;
• which component owns child processes;
• which signals trigger reload or shutdown;
• which resources must survive transient connection loss.

Models that infer lifecycle from naming conventions such as cli, daemon, or server are unreliable unless they trace command dispatch and blocking paths.

3. Broad investigators were useful but sometimes overconfident

Qwen3.6-27B consistently gathered the most evidence and produced the most useful synthesis. It correctly connected implementation details to deployment and runtime behavior.

Its recurring weakness was converting plausible explanations into asserted root causes. For example, a closed network connection may be consistent with:

• a network interruption;
• a local proxy restart;
• application teardown;
• remote closure;
• timeout behavior;
• transport replacement.

Without lower-layer logs or a direct code path, one of these should not be declared the root cause.

Repository agents should classify claims explicitly:

• Confirmed from code
• Observed at runtime
• Likely interpretation
• Alternative explanations
• Additional evidence required

This preserves the value of aggressive investigation without overstating certainty.

4. Code verification and operational review were distinct skills

Qwen3.6-35B was strong at examining goroutine and state-machine code paths, identifying cancellation and retry logic, and tracing timeout propagation. It was less reliable when making broad architectural classifications.

Gemma4-31B showed the opposite profile. It was strong at identifying service identity, configuration paths, logging, privileges, deployment scope, and recovery time. It was less consistent at tracing code paths deeply enough to confirm root causes. However, it sometimes produced recommendations without completing the requested source inspection and made a protocol-layer identification error.

This supports a division of labor with current open weight models in this class:

• code verifier: concurrency, state machines, retries, timeout propagation;
• operational reviewer: permissions, packaging, logs, deployment, recovery objectives.

One model should not automatically be trusted for both.

5. Specialized coding models favored implementation over design

Qwen3-Coder-30B readily produced concrete service definitions and implementation structures. The output was readable, but it moved to code before resolving key architectural questions such as stable installation paths, ownership, restart semantics, and upgrade behavior. It also introduced unsupported configuration details and platform mistakes.

This reflects a common failure mode in coding models: optimizing for artifact production rather than determining whether the artifact represents the correct architecture.

Such models are useful after the plan is approved, especially for unit files and package scripts. They are less useful for initial investigation and design, which should be more cautious and evidence-driven.

6. Networking analysis required strict layer separation

The incident investigation in our test involved multiple independent layers:

1. application-facing socket;
2. local proxy connection;
3. anonymizing-network stream;
4. signaling channel;
5. ICE transport;
6. peer connection;
7. data channel;
8. forwarded application stream.

Several models conflated failures across these layers. Typical errors included:

• treating a destination port as a local control port;
• assuming signaling failure meant the established transport was already dead;
• treating a warning about relay fallback as the cause of a successful direct connection;
• interpreting temporal proximity as proof of causality.

A robust investigation should create separate state timelines for each layer. A failure at one layer may immediately terminate dependent layers, be recoverable independently, have no effect on an already-established layer, or merely correlate with a later failure. The investigation should not assume one of these without evidence.

This is where source-backed state-machine analysis outperformed generic network troubleshooting.

Model-family information and observed behavior

The harness labels may refer to aliases, quantizations, or provider-specific checkpoints. Public family information therefore explains tendencies only loosely.

Qwen

The public Qwen3 family includes dense and mixture-of-experts models, including a 30B model with roughly 3B active parameters. Qwen describes the family as improved in reasoning, coding, instruction following, and tool use.¹ Qwen3-Coder is separately positioned for agentic coding and long-context repository work.²

This broadly matches the observed strengths:

• strong repository exploration;
• effective search across multiple files;
• useful code-path reasoning;
• good synthesis of implementation details.

The tests also exposed risks not captured by headline coding benchmarks:

• premature architectural conclusions;
• causal overreach;
• weak distinction between hypotheses and confirmed behavior.

Gemma

Google describes Gemma 4 as an open model family available in several sizes, including 31B and a 26B-A4B mixture-of-experts variant, with up to 256K context and multimodal support.³

The tested Gemma models produced clear, organized operational analysis. They were particularly effective at identifying production concerns and communicating them compactly. They were less consistent at using repository tools deeply enough to establish low-level behavior.

This suggests strength in synthesis and review, but not sufficient evidence to treat them as the primary code investigator in this harness.

GLM-4.7-Flash

GLM-4.7-Flash is published as a 30B-A3B mixture-of-experts model intended to balance agentic performance with lightweight deployment. Its model card reports strong coding and agent benchmarks.⁴

Its poor result here was primarily an agent-execution failure:

• some malformed and some ineffective tool usage;
• little meaningful source inspection;
• generic recommendations;
• platform-semantic errors.

This demonstrates that benchmark scores do not directly predict performance under a different tool scaffold.

GPT-OSS-20B

OpenAI describes GPT-OSS-20B as an open-weight reasoning model designed for efficient local deployment, configurable reasoning effort, instruction following, and tool use.⁵

In these tests it produced broad checklists but weak repository grounding. It also introduced unsupported configuration details and platform mistakes.

A plausible explanation is that the model was optimized for efficient general reasoning rather than deep repository exploration under this particular harness. That remains an inference: quantization, reasoning settings, context handling, and tool integration may have been equally important.

Why benchmark rankings did not transfer directly

Published software-engineering benchmarks usually evaluate a model under a specific scaffold, prompt, inference configuration, and issue set. This evaluation specifically required:

• deciding what to inspect;
• understanding deployment systems;
• reasoning across multiple runtime layers;
• respecting “investigate first” constraints;
• avoiding unsupported claims;
• communicating uncertainty.

A model can therefore score well on patch-oriented benchmarks and still perform poorly as an open-ended repository consultant.

We should note that the harness itself is a factor. The VSCode agent interface is a specific tool scaffold with its own constraints and affordances. A model’s performance may vary under different scaffolds, prompts, or operational contexts. Results should be treated as properties of that complete system.

Prompting practices that improved rigor

The following constraints would prevent most observed errors:

Before planning, identify every persistent execution path and cite where it blocks.

Separate observed evidence, confirmed code behavior, inference, and unknowns.

Do not identify a root cause unless alternative explanations have been excluded.

Trace each runtime layer independently and explain how failure propagates between them.

List evidence that contradicts your preferred explanation.

Do not produce implementation until paths, ownership, lifecycle, and upgrade behavior are resolved.

Conclusion

The tests did not identify a universally best model but complementary roles.

• Qwen3.6-27B was the strongest general repository agent.
• Qwen3.6-35B was the strongest focused code verifier.
• Gemma4-31B was the strongest operational reviewer.
• Qwen3-Coder-30B was useful after architecture was settled.
• GLM-4.7-Flash-30B and GPT-OSS-20B were not reliable enough in this harness for authoritative repository work.

A viable pipeline could be structured as follows:

investigate and plan
    Qwen3.6-27B

verify code mechanics
    Qwen3.6-35B

review production implications
    Gemma4-31B

generate implementation
    Qwen3-Coder-30B or Qwen3.6-27B

validate deterministically
    compilers, linters, race detectors, package checks,
    fault injection, and integration tests

The key lesson is again that engineering depends much less on producing code quickly, much more on gathering the correct evidence, preserving abstraction boundaries, and knowing which conclusions remain uncertain.