_nullmirror
RSS
LAUNCH ETA: 2026 July

Local Web Research with SearXNG and Crawl4AI

• 5 min read

Local language models are useful for coding and analysis, but they have a limitation in that they cannot access current information unless an external tool provides it. Running a model through Ollama does not provide web access. The agent harness—VS Code, OpenCode, or another MCP-capable client—provides the orchestration layer that can however expose tools, execute them, and return their results to the model. In the spirit of self-hosting, this article describes a simple local research stack that combines SearXNG and Crawl4AI to provide current web search and page retrieval for local language models.

Architecture

VS Code / OpenCode
        │
        │ MCP
        ▼
    search-mcp
      ├── web_search → SearXNG
      └── fetch_page → Crawl4AI
                            │
                            ▼
                     Ollama model context

The components have separated responsibilities:

  • Ollama runs the language model locally.
  • VSCode or OpenCode is the agent harness and orchestrator.
  • search-mcp is a thin custom service that exposes structured research tools over MCP.
  • SearXNG performs metasearch across public search engines.
  • Crawl4AI fetches selected pages and converts them into Markdown.

The model could talk directly to SearXNG or Crawl4AI via curl, but using the MCP server provides a structured interface and ensures consistent integration. The harness executes those calls through the MCP server and inserts the returned data into the model context.

The exposed tools

Our MCP server provides two operations:

web_search(query)
fetch_page(url)

web_search calls the local SearXNG instance and returns a compact result set containing titles, URLs, snippets, and source metadata.

fetch_page sends a selected URL to Crawl4AI and returns extracted Markdown.

Search results are useful for discovery, but snippets are incomplete and often misleading. Fetching the actual page gives the model enough source material to produce a grounded answer.

A typical execution flow is:

User prompt
    ↓
Model decides current information is required
    ↓
web_search("Go MCP SDK latest release")
    ↓
Search results returned to model
    ↓
Model selects relevant primary source
    ↓
fetch_page("https://...")
    ↓
Markdown returned to model
    ↓
Model answers using retrieved content

How MCP exposes the tools

MCP is the protocol boundary between the agent harness and the research service. When VSCode or OpenCode connects to the MCP endpoint, it performs tool discovery. The server advertises the available tool names, descriptions, and input schemas. The harness then includes those schemas in the model session. A tool-capable model can emit a request such as:

{
  "name": "web_search",
  "arguments": {
    "query": "latest Go MCP SDK documentation"
  }
}

The harness intercepts this request instead of treating it as normal text. It calls the MCP server, receives the result, and appends that result to the conversation context. The model then continues reasoning with the retrieved data. The agent harness therefore owns the tool loop:

model → tool request → harness → MCP server → result → harness → model

Ollama only needs to support structured tool calling. It does not need to know how SearXNG, Crawl4AI, HTTP, or Docker work.

HTTP MCP

HTTP allows multiple clients to share one MCP service.

VS Code ──────┐
              ├── http://127.0.0.1:8081/mcp
OpenCode ─────┘

For this use case, HTTP is preferable when VS Code and OpenCode run on the same machine. It centralizes service configuration, logging, timeouts, result limits, and authentication if later required. It also allows multiple clients to share one SearXNG and Crawl4AI instance. Ensure that the endpoint binds only to loopback, so it remains local.

Why this is useful with Ollama

Local models are usually limited by three things:

  1. Their training data may be old.
  2. They cannot verify current claims.
  3. They cannot inspect documentation or web pages unless that content is supplied explicitly.

This stack addresses all three.

It is particularly useful for current package documentation, release notes, and changelogs. A model can reason over the retrieved Markdown in the same way it reasons over local source files. The quality of the final answer still depends on the model, but the factual input is no longer limited to its static training corpus.

Local execution and privacy

The model, MCP server, SearXNG instance, and Crawl4AI instance all run locally. The result is a locally controlled system:

  • prompts stay on the machine
  • model inference stays on the machine
  • tool orchestration stays on the machine
  • no hosted AI provider receives the conversation
  • zero commercial API keys are required

It is not network-isolated. SearXNG still sends queries to public search engines, and Crawl4AI connects to requested websites. Those services can observe the requests they receive. But we achieve local inference and local orchestration, with outbound network access only for search and page retrieval.

No API keys with SearXNG

SearXNG can use public search interfaces such as:

DuckDuckGo
Google
Bing
Startpage
Qwant
Yahoo
Yandex
Yep
Wikipedia
Wikidata
arXiv
Crossref

This avoids API key management, per-query billing, and provider-specific SDKs. It also avoids dependency on one search vendor. SearXNG normalizes results from multiple sources behind one local HTTP API. The tradeoff is that public engines may rate-limit, block, or change behavior. SearXNG reduces coupling but cannot eliminate upstream instability.

Config

This configuration loads every built-in SearXNG engine and category, then explicitly enables the keyless engines that SearXNG otherwise marks disabled:

use_default_settings: true

general:
  debug: false
  instance_name: "Local SearXNG"

server:
  port: 8080
  bind_address: "127.0.0.1"
  secret_key: "< random 32-byte hex string >"
  limiter: false
  image_proxy: true

search:
  default_lang: "en"
  safe_search: 0
  formats:
    - html
    - json

outgoing:
  request_timeout: 6.0
  max_request_timeout: 15.0
  pool_connections: 100
  pool_maxsize: 20

engines:
  - name: bing
    disabled: false

  - name: qwant
    disabled: false

  - name: yahoo
    disabled: false

  - name: yandex
    disabled: false

  - name: yep
    disabled: false

  - name: crossref
    disabled: false

plugins:
  searx.plugins.oa_doi_rewrite.SXNGPlugin:
    active: true

  • use_default_settings: true inherits SearXNG’s complete upstream engine list, including general, news, image, science, IT, and social categories. ( docs.searxng.org )

  • disabled: false overrides are required because loading an engine does not necessarily enable it. These six were shown as disabled by a running instance despite not requiring API keys. Engine overrides merge with defaults by exact name. ( docs.searxng.org )

  • Engines that genuinely require credentials remain unavailable until configured; these overrides do not supply or bypass API keys.

  • oa_doi_rewrite redirects DOI links toward available open-access copies. It is built in but inactive by default, so active: true is required. ( docs.searxng.org )

This structure automatically picks up newly added default engines after SearXNG upgrades while preserving the explicit enablement of the six engines above.