Local Vision-Language OCR Benchmark

For heterogeneous PDFs, a modern robust approach combines page rendering and vision model extraction, especially for scans, formulas, charts, tables, and diagrams. This benchmark evaluates open-weight vision-language models for local document OCR and semantic page reconstruction. The goal is to identify a practical production configuration, not merely the most capable model in isolation.

The workload covers visually rendered document pages containing ordinary text, tables, mathematical notation, charts, and diagrams. A separate manga OCR test checks a harder class of page: vertical Japanese, speech bubbles, nonlinear reading order, dark backgrounds, glare, and page curvature. Each model is evaluated on whether it can convert those pages into useful markdown-like output while preserving the important structure and content. The benchmark therefore measures both extraction quality and operational practicality: completion rate, average latency, failure modes, and suitability for local inference.

All tests were run on an Apple Mac mini with M4 Pro and 64 GB unified memory. The benchmark results should therefore be interpreted as practical local-inference results on a strong but still consumer-class machine, rather than datacenter GPU results.

Background

Traditional OCR engines are still strong for exact character recognition, but they do not natively understand page semantics. A vision-language model can, in principle, read the visible text, reconstruct tables, interpret diagrams, preserve formulas, and emit structured markdown in a single pass. That makes VLMs attractive for document conversion pipelines where layout and meaning matter as much as raw transcription.

The models tested here represent several current open-weight approaches. Qwen2.5-VL is designed for dynamic-resolution visual processing and document understanding, MiniCPM-V emphasizes high visual-token efficiency, Qwen3-VL represents a larger reasoning-oriented VLM family, and Granite Vision is positioned as a compact document-understanding model.¹²³⁴

Test setup

The benchmark uses two categories of document pages. The first category represents general OCR: normal text, tables, definitions, and structured document content. The second category stresses mathematical notation, formulas, bounds, and diagrams.

Each page is rendered at three resolutions: 150 DPI, 200 DPI, and 300 DPI. The resulting image sizes were:

The models evaluated were:

The output was judged on completion, text fidelity, table reconstruction, math correctness, diagram interpretation, and latency.

For the manga test, the main criteria were exact Japanese transcription, bubble ordering, and whether the model could avoid hallucinating dialogue from panel context.

Main recommendation

Generally, we use 150 DPI + qwen2.5vl:7b as the default.

This configuration completed all 12 benchmark pages, averaged about 35 seconds per page, and stayed close to the best models on text, tables, and math. Its visible errors were mostly small detail failures, such as reading NAMPS as NAM, and its diagram descriptions were not as rich as Qwen3-VL (see appendix). Those are real limitations, but they are acceptable for a default path because the model is much faster than the higher-correctness alternatives.

For maximum correctness, use 150 DPI + qwen3-vl:30b. It produced the strongest math and diagram results, but averaged about 73 seconds per page, making it roughly twice as slow as qwen2.5vl:7b. It is better treated as a high-accuracy fallback than as the normal production model.

glm-ocr is interesting in that it is much faster than qwen2.5vl:7b and strong on general text and tables, but it made math errors and omitted diagram structure. It is best treated as a fast general-OCR mode, not a high-assurance default.

DPI result

The best default resolution in this benchmark was 150 DPI.

This is counterintuitive only if we assume that higher DPI should always improve OCR. For VLMs, higher DPI also increases the visual-processing burden. These models convert images into visual tokens, and larger images can increase latency, memory pressure, and timeout risk. Qwen2.5-VL explicitly uses dynamic-resolution processing, meaning image size affects how the model represents visual input rather than being a neutral preprocessing detail.¹

The 150 DPI pages were already large enough for the strongest models to read most content correctly. Moving to 200 DPI increased the pixel count by roughly 78% over 150 DPI, but did not provide enough additional detail to improve extraction quality. In this benchmark, that made 200 DPI the worst middle ground: more expensive than 150 DPI, but not meaningfully more accurate.

300 DPI remains useful, but mostly as a retry mode. It is expensive, but it can help with tiny text, dense formulas, and weaker models. For the strongest models, however, it did not improve accuracy enough to justify using it by default.

This does not mean 200 DPI is universally bad. The defensible conclusion is narrower: for this workload, model set, prompt style, and machine, 200 DPI was an inefficient middle point. It likely increased visual-token and attention cost without crossing a meaningful readability threshold.

Why 200 DPI likely performed poorly

At 150 DPI, the page is about 1.92 MP, which is already close to the image budget where MiniCPM-V advertises high efficiency: about 1.8M pixels represented using only 640 visual tokens.² At 200 DPI, the page grows to 3.41 MP, which is substantially larger but often not visually richer enough to fix the hard OCR cases. At 300 DPI, the page is much more expensive, but at least it can expose genuinely smaller details.

So the practical tradeoff is:

150 DPI:
  Low cost.
  Enough detail for strong models.
  Best production default.

200 DPI:
  Much higher cost than 150 DPI.
  Not enough extra detail to fix difficult pages.
  More failures and timeouts in this benchmark.

300 DPI:
  High cost.
  Sometimes useful for small text and dense formulas.
  Best reserved for retries.

Note that this is merely our empirical pipeline result and not universally applicable.

Model result

• qwen2.5vl:7b at 150 DPI is the best speed/accuracy balance. It is not the absolute best model, but it is the best production model from this benchmark because it combines full completion, good text extraction, good table reconstruction, acceptable math extraction, and manageable latency.
• glm-ocr at 150 DPI is the fastest useful general OCR candidate. It completed all pages, averaged about 10.7 seconds per page, and performed well on normal text and tables. It should not replace qwen2.5vl:7b as the default because it was weaker on math and diagrams.
• glm-ocr at 200 DPI is a possible table-completeness mode. It captured an additional table section that its 150 and 300 DPI runs omitted, but average latency rose to about 27.8 seconds per page. It is worth considering for table-heavy documents, not as the universal default.
• qwen3-vl:30b at 150 DPI is the best correctness model. It captured formulas, bounds, and diagram structure most reliably. Its cost is substantial: roughly twice the latency of the recommended default.
• qwen2.5vl:3b at 300 DPI is a credible lower-cost option. It completed all pages, averaged about 28 seconds per page, and was surprisingly capable on general text and tables. It is weaker on diagrams and math, and 150 DPI is not safe for math-heavy pages because it produced a Gaussian exponent sign error.
• minicpm-v:8b is fast and completed all pages at 200 and 300 DPI, but it is not trustworthy for math. It hallucinated or corrupted several equations. That makes it useful for fast approximate OCR, but unsafe for technical documents where formula fidelity matters.
• gemma4:e4b was fast, especially at 300 DPI, but correctness was too uneven. It had table-value drift and formula drift, so it should not be used as the production extraction model for this workload.
• qwen3-vl:32b-instruct produced high-quality outputs when it completed, but it was far too slow, averaging roughly 400 seconds per page, and it missed some pages. It is not a practical sweet spot on this machine.
• granite3.2-vision should be disqualified for this benchmark despite its document-oriented positioning. Its outputs were often malformed or table-like in the wrong way, making quality unreliable for this pipeline.
• deepseek-ocr should be disqualified. The timing stats reported successful, fast completion, but the generated markdown files were effectively blank. This is a pipeline/model failure with Ollama rather than a usable OCR result.

General text and tables

For general OCR and tables, the best practical default remains qwen2.5vl:7b, with qwen3-vl:30b reserved for high-accuracy fallback. glm-ocr is the best fast mode for general OCR and tables, especially when throughput matters more than math fidelity.

• qwen2.5vl:7b handled most tables well, including the major standards tables and definitions table. It still made small but real errors, so exact extraction should not be assumed without validation. Compared with the rest of the field, however, it provided the best mix of quality, completion, and runtime.
• qwen2.5vl:3b was stronger than expected on tables. In some cases it preserved row labels better than the 7B model. Its weakness is robustness: it is less reliable on figures, diagrams, and math, so it should be considered a lower-cost option rather than the main model.
• glm-ocr was particularly strong on table-heavy general pages. At 200 DPI, it recovered an additional table section that its 150 and 300 DPI runs omitted. This makes it worth considering as a fast table extraction candidate, but not as the single production model because its math output was less reliable.

Math and diagrams

For math and diagrams, the best pure-accuracy model is qwen3-vl:30b.

It handled formulas, Q-function definitions, bounds, and diagram structure most reliably. The tradeoff is latency: the improvement is real, but not cheap.

qwen2.5vl:7b is good enough for many math pages, especially as a default first pass, but it is less descriptive on diagrams and may drop equation numbering. That makes it acceptable for production when paired with a fallback strategy, but not ideal as the sole high-assurance math extractor.

minicpm-v:8b and gemma4:e4b should not be trusted for math in this benchmark. They were fast, but they produced wrong integrals, wrong bounds, or normalized formulas where the original used the general Gaussian form. For technical documents, silent formula corruption is worse than slow inference.

glm-ocr should also be treated cautiously for math. It produced readable equation output, but made a significant substitution-bound error in the Gaussian/Q-function derivation and did not preserve diagram structure as well as Qwen3-VL. This makes it acceptable for general OCR, but not for high-assurance technical extraction.

Comic and manga OCR test

The manga pages were substantially harder than the textbook benchmark. The difficulty was not just character recognition. The harder parts were reading order, vertical Japanese, small slanted bubbles, black-background panels, page curvature near the gutter, glare, and dialogue that continues across panels.

The local models mostly failed this test:

• qwen2.5vl:7b was the best local model. It captured some important text on the first spread but failed the second spread almost completely.
• qwen2.5vl:3b was similar but noisier, with more character substitutions and partial fragments.
• gemma4:e4b produced text on both images but rewrote or hallucinated too much to be reliable.
• minicpm-v:8b mostly produced panel descriptions and invented dialogue rather than OCR.
• qwen3-vl:30b, qwen3-vl:8b, and qwen3-vl:4b produced blank outputs in this run, so those were pipeline/model failures for this task.
• granite3.2-vision again produced unusable malformed output.

Document OCR quality does not seem to predict manga OCR quality. A model can perform well on tables and formulas while failing on vertical Japanese speech bubbles. In contrast to the document benchmark, larger or slower models did not help on manga OCR. The manga test also showed that latency was not predictive of usefulness. Among the local models that produced nonblank manga output:

Recommended production policy

Use a two-tier pipeline.

The default path should be:

qwen2.5vl:7b @ 150 DPI

This gives the best observed production balance. It is accurate enough for general documents, strong on tables, acceptable on math, and much faster than the larger Qwen3-VL models.

An optional fast path is:

glm-ocr @ 150 DPI

The fallback path should be:

qwen3-vl:30b @ 150 DPI

Use this when the page contains dense equations, important diagrams, unclear chart labels, or when the default model produces low-confidence or incomplete output.

The DPI retry policy should be:

150 DPI first
300 DPI retry only when needed
avoid 200 DPI by default

300 DPI should be reserved for failed pages, tiny text, formula-heavy pages, or pages where the 150 DPI output appears suspicious. It should not be the normal path because it did not improve the best models enough in this benchmark.

Final decision

For one model and one DPI, choose:

qwen2.5vl:7b @ 150 DPI

For a production-grade two-tier setup, use:

Default:   qwen2.5vl:7b @ 150 DPI
Fast mode: glm-ocr @ 150 DPI for general text/tables
Fallback:  qwen3-vl:30b @ 150 DPI for math/diagrams/high confidence
Retry:     300 DPI for failures or high-detail pages

This is the best balance from the benchmark: fast enough to be usable on a local Apple Mac mini, accurate enough for general document conversion, and backed by a clear escalation path for math, diagrams, and low-confidence pages.

For manga/comic OCR, this benchmark does not currently identify a strong local VLM-only solution. The conclusion is to use the document OCR pipeline for documents, and use a specialized OCR-plus-VLM pipeline for comics.

Appendix

Top examples from the benchmark illustrate the range of model behavior, including fluent-but-wrong math, table corruption, semantic reversals, diagram omissions, and failed OCR-specialized runs.

1. Gaussian PDF sign error — math page 1

Correct form:

p(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{(x-m)^2}{2\sigma^2}}

Good: qwen3-vl:30b, qwen2.5vl:7b, gemma4:e4b

Bad: qwen2.5vl:3b @ 150 dpi drops the minus sign:

e^{\frac{(x-m)^2}{2\sigma^2}}

That is a severe semantic error: it turns a decaying Gaussian into an exploding function. This is a strong example showing why smaller models fail on nuanced mathematical content.

2. Q-function derivation hallucination — math page 2

qwen3-vl:30b and qwen2.5vl:7b preserve the derivation well:

Q(z)=\int_z^\infty \frac{1}{\sqrt{2\pi}}e^{-y^2/2}dy
Q(-z)=1-Q(z)

minicpm-v:8b badly rewrites the math:

Q(z)=\int_{-\infty}^{z} e^{-x^2/2} dx
Q(-z) = -Q(z)

It also corrupts the substitution and bounds. This is the best example of a model producing fluent but mathematically unsafe OCR.

In the Gaussian/Q-function derivation, GLM produced a bound like:

\int_{x_0-m}^{\infty}

where the correct normalized bound should be:

\int_{(x_0-m)/\sigma}^{\infty}

GLM is therefore also not useful for high-assurance formula extraction.

3. Table 1.1 row/column corruption — general page 5

qwen3-vl:30b gets the North America standards table cleanly, including:

NAMPS | Cellular | 1992 | FDMA | 824-894 MHz | FM | 10 kHz
USDC  | Cellular | 1991 | TDMA | 824-894 MHz | π/4-DQPSK | 30 kHz

qwen2.5vl:7b is mostly good but has a small OCR miss:

NAM | Cellular | 1992 ...

minicpm-v:8b has multiple table-value errors:

USDC ... π/4-QPSK
GSC ... FSJK
DCS-1900 ... GSM
PACS ... TDMA/FDMA, GHZ ... π/4-QPSK | DQPSK

gemma4:e4b changes actual data:

CDPD ... TDMA ... QPSK
FLEX ... FSK ... 12.5 kHz

4. Full duplex vs half duplex meaning reversed — general page 8

Correct:

Full Duplex Systems: simultaneous two-way communication, typically on two different channels.
Half Duplex Systems: same radio channel; user can only transmit or receive at a given time.

qwen3-vl:30b and qwen2.5vl:7b get this right.

gemma4:e4b reverses the concepts:

Full Duplex Systems: ... using the same radio channel ... user can either transmit or receive.
Half Duplex Systems: ... using two separate channels ...

This changes the technical meaning and shows that semantic interpretation is a factor.

5. Diagram handling / visual interpretation — math page 1

qwen3-vl:30b tries to preserve the figure structure with an ASCII sketch and labels m, x0, x.

qwen2.5vl:7b extracts the caption and formula but effectively omits the diagram.

gemma4:e4b gives a prose description:

[Diagram representing the Gaussian curve and shaded area]

minicpm-v:8b invents a Markdown image link:

![Figure D.1](https://latex.codecogs.com/...)

This is the best example for diagrams: Qwen3 is better when you care about visual structure; Qwen2.5 7B is enough if captions/text are sufficient.

6. Manga OCR: small vertical text and reading order

The manga stress test exposed interesting failures, a text box contained:

高度210km
ここはもう立派に
宇宙空間だ

qwen2.5vl:7b and qwen2.5vl:3b recovered 高度210km, while even a test with ChatGPT 5.5 misread it as 高度知能. This shows that exact small-text recognition can still beat semantic inference.

But the same models failed the second spread almost completely. Important dialogue was mostly omitted or corrupted by the local models. This makes a manga test useful precisely because it separates OCR transcription from page understanding. The models that were good enough for textbook tables were not good enough for comic speech-bubble extraction.

7. Manga hallucinated dialogue and meaning

gemma4:e4b is a useful example of why this task cannot be judged by whether the output looks like manga dialogue. It produced fluent-looking Japanese fragments, but changed the scene meaning.

The opening location caption should read:

高度210km
ここはもう立派に
宇宙空間だ

gemma4:e4b rewrote it as:

敵機追尾。
ここにはもう災害だ。
全艦隊が…

This changes a neutral setting caption: “Altitude 210 km; this is already outer space” into an invented military/emergency situation involving enemy tracking, disaster, and a fleet.

Another bubble should describe equipment capacity:

俺もユーリも
マシンの積載限界
とっくに超えてるし

gemma4:e4b changed the key term:

俺もユリーも、
マシンの戦艦限界
とっくに超えてるし

積載限界 means load/carrying capacity limit. 戦艦限界 is a different and essentially nonsensical phrase in context, pulling the line toward a battleship or warship interpretation. That makes it unsuitable even for vocabulary extraction when the vocabulary is technical or scene-specific.

The second spread showed the same issue. A line about low-orbit debris:

近頃はこんな低軌道にも
危険なくらい粗大ゴミが
ゴロゴロしてるんだ

was transformed into nonsensical text about a left-resistance device:

こちらは左抵抗装置にも
応度くだらない相手にみなが
ロゴロしてるんだ

The model preserved the rough visual impression of a technical space scene, but not the text. For manga OCR, gemma4:e4b may appear to be doing well, but is noisy and hallucinates a totally different narrative.