Embedding Models on Affordable Cloud VMs and Apple Silicon

Embedding model inference converts input into fixed-size vectors for similarity/search, while LLM inference generates or scores text token by token. Embedding inference is much cheaper because it does a single forward pass to produce a vector, while LLM inference repeatedly runs the model to generate each output token. It is used in e.g. a vector database pipeline, where embeddings are the ingestion primitive (documents, chunks, updates) and the query primitive (search, rerank candidates, analytics). That makes the operational question about what hardware and runtime sustain predictable embedding throughput under latency constraints.

This post reports on a benchmark that compares CPU-only embedding generation from small cloud VMs (DigitalOcean) to Apple Silicon, using standard OpenAI-compatible /v1/embeddings surface. Our focus is deliberately practical. End-to-end request latency, throughput (requests/sec), and stability under concurrency and batching, measured at the API boundary rather than via token-level microbenchmarks.

Apple Silicon, and in particularly our M4 test machine, operates in a different performance class than 1–2 vCPU cloud instances for this workload. Separately, runtime overhead (Ollama vs a minimal llama.cpp server) can dominate outcomes on constrained VMs, sometimes exceeding the performance differences between CPU tiers.

What exactly is being benchmarked?

Embedding generation is a forward pass through a transformer encoder: tokenization, repeated attention/MLP blocks, and a pooling/projection head to a fixed-size vector. There is no autoregressive decode loop. The computation is therefore dominated by dense linear algebra (GEMM-heavy), vectorized kernels (SIMD-sensitive), cache behavior, and memory bandwidth. Put differently: this is CPU quality and runtime efficiency dominated rather than IO (disk/network) bound.

For each target environment (Ollama or llama.cpp on a particular machine type), our setup provisions a low cost cloud VM, loads a specific embedding model, sends a standardized suite of embedding requests with varied input sizes and batch shapes, and measures end-to-end timing.

The hardware tier question on DigitalOcean

Embedding inference is strongly CPU-bound. DigitalOcean documents that Regular droplets come from a mixed/older CPU pool, while Premium droplets (the -amd / -intel SKUs) are guaranteed to run on one of their latest two CPU generations and use NVMe SSDs.¹ For CPU-bound inference, “Premium vs Regular” is a factor that spawned our curiosity for evaluation.

Upfront, in our measurements, Premium AMD consistently beat Premium Intel, and, unsurprisingly, both beat Regular. The magnitude is large enough that small savings can even be the wrong tradeoff for embeddings, because we pay it back in throughput, tail latency, and operational headroom.

The model question: bge-m3 vs the E5 family

The benchmark data below covers bge-m3 and a multilingual E5 variant. Even when we hold hardware constant, these models have different engineering profiles; it is useful to understand why.

bge-m3 is explicitly designed for multi-mode retrieval: a single checkpoint can produce dense embeddings, sparse lexical weights, and multi-vector (ColBERT-style) representations, which is a direct enabler for hybrid retrieval pipelines without stitching together separate models or a separate BM25-like component.^{2 3} Practically, it also supports long inputs (up to 8192 tokens) and emits 1024-dimensional vectors.² The combination of longer context, larger vector dimensionality, and larger parameter count tends to increase compute and memory demands versus smaller dense-only encoders.

By contrast, multilingual-e5-small is a compact, widely used dense embedding model. It emits 384-dimensional vectors and is commonly treated as a good multilingual default when we want speed and small vectors.⁴ Its usage also comes with a gotcha that becomes operationally important: semantic search quality depends on prefix discipline (query: vs passage:), and dropping the prefixes can noticeably degrade retrieval behavior.⁵ Its maximum input length is typically treated as 512 tokens in hosted documentation.⁶

A useful mental model is that bge-m3 buys retrieval flexibility (hybrid signals, longer context, multi-vector options) at higher inference cost, while small E5 variants buy cheap dense vectors at the cost of truncation and reduced retrieval expressivity. What the benchmark shows, however, is that on small cloud CPUs the runtime and CPU class can swamp these model-level considerations.

Methodology and Workload

All measurements here are from API-level embedding requests against /v1/embeddings. The harness runs multiple cases per environment (short, medium, long, and batched/mixed patterns), performs warmups, then executes timed runs.

From the benchmark output we supplied:

• runs per case: 10
• total requests per benchmark: 50
• concurrency: varies (1 or 2 in the data)

Raw results table

To avoid cherry-picking, the table below is constructed directly from the rows we provided. It includes both Apple Silicon and DigitalOcean environments, shows the runtime, model, concurrency, success rate, throughput, and the headline latency distribution (p50/p95).

Table 1 — End-to-end embedding benchmark results (raw-summary rows)

The headline comparisons are immediate:

1. Apple Silicon is unsurprisingly in a different throughput class. The Mac mini run reports 5.86 req/s at p50 ≈ 159 ms for bge-m3, while the best cloud configuration in our dataset (Premium AMD 2vCPU + llama.cpp) reaches 0.76 req/s at p50 ≈ 683 ms. That is roughly an order-of-magnitude throughput gap and a multi-x median latency gap.

2. Regular droplets are not merely slower but also have pathological tails under Ollama. The bge-m3 + Ollama run on DO Regular is 0.05 req/s with p95 ≈ 75 seconds, which changes how we must design timeouts, retries, and queue backpressure.

3. Runtime choice dominates on small CPUs. On multiple droplets, switching from Ollama to llama.cpp moves throughput by 5-8× for the same model class, and collapses p95 by large factors (though with the success-rate caveat visible in our data).

Interpreting the two big effects: CPU tier and runtime overhead

CPU tier: Premium AMD > Premium Intel > Regular

Across the 1vCPU cases, Premium AMD consistently outperforms Premium Intel for both bge-m3 and E5-large under llama.cpp, and both Premium tiers substantially outperform Regular. This aligns with DigitalOcean’s own plan description: Premium droplets are pinned to newer CPU generations, whereas Regular can land on older silicon.¹ With embedding inference being dominated by vectorized math and memory behavior, per-core quality matters far more than “number of cores” at the low end.

The implication is that if we are cost-optimizing CPU-only embeddings on small DigitalOcean droplets, start from Premium AMD as a default and treat Regular as a last resort for lightly loaded dev/test.

Runtime overhead: llama.cpp vs Ollama

The raw data show a consistent pattern that on droplets, llama.cpp throughput is dramatically higher than Ollama for the same nominal workload, even when both are ultimately driving similar kernels underneath. The plausible technical causes include (a) more direct control over batching and thread scheduling, (b) reduced service-layer overhead, (c) differences in memory allocation and copy behavior around request handling, and (d) model packaging/serving decisions (e.g., how input is tokenized and how outputs are marshaled). The benchmark is not attempting to attribute the gap to a single internal mechanism; it is recording the operational reality that in constrained CPU environments, server overhead is not amortized away but can become the bottleneck.

Scaling: 2 vCPU is not 2×

The dataset includes a comparison between Premium AMD 1vCPU and 2vCPU under llama.cpp at concurrency 1:

• 1vCPU: 0.62 req/s, p50 ~799 ms
• 2vCPU: 0.76 req/s, p50 ~683 ms

That is a ~20–25% uplift, not a doubling. This is expected when a workload is bounded by shared resources that do not scale with vCPU count in a linear way: memory bandwidth, cache contention, and hypervisor scheduling effects. Embedding inference in particular mixes compute-heavy layers with memory-traffic-heavy layers; adding a second vCPU helps, but quickly runs into the next shared constraint.

It would be a mistake is to treat embeddings like an embarrassingly parallel web workload where throughput scales linearly with cores on the same machine. The benchmark data argues strongly against that assumption for small VMs with the same memory constraints.

Concurrency and batching experiments

we also supplied three Premium AMD 2vCPU runs at concurrency 2 with different batching choices. The key results:

Concurrency=2, ubatch=512, batch=2048 (bge-m3, llama.cpp):

• 0.73 req/s, p50 ~933 ms

Compared to concurrency=1 on the same class of machine:

• 0.76 req/s, p50 ~683 ms

Throughput does not increase; latency worsens. This is consistent with CPU saturation: concurrency becomes a queue that inflates response times rather than a lever that increases utilization. Put plainly, the instance is already “busy enough” at concurrency 1.

A more extreme configuration (ubatch=2048, batch=2048) shows an even stronger cautionary tale:

Concurrency=2, ubatch=2048, batch=2048 (bge-m3, llama.cpp):

• 0.21 req/s, p50 ~6265 ms

This is a collapse in throughput paired with a large latency inflation. On a 2GB machine, aggressive batching pushes the system into a regime where memory pressure and kernel inefficiencies dominate. We see that embedding inference is memory hungry. Even if the model fits, a batching strategy can move us from compute-bound to memory-bound to thrash-bound.

Interestingly, in that same run set, multilingual-e5 under llama.cpp at concurrency 2 reports 0.70 req/s with p50 ~994 ms, i.e., it remains in the reasonable range where bge-m3 collapsed. That difference is consistent with model footprint and per-request compute, but the overarching point remains: on small machines, batching and concurrency knobs can easily make things worse, and the best settings are hardware- and model-dependent.

Apple Silicon is obviously far ahead

Apple’s high-performance cores deliver high single-thread and per-watt throughput, supported by strong memory bandwidth and a system design tuned for sustained vectorized compute. Embedding inference rewards exactly those properties. When we compare that against a small cloud VM where our vCPU is a slice of shared server silicon (often with lower sustained clocks, weaker per-core performance, and noisy neighbors), we should expect large gaps.

The most actionable interpretation is that a single high-performance local embedding worker can replace multiple tiny cloud instances for steady-state embedding throughput, especially when we include the runtime overhead costs and the tail latency stability costs.

Practical implications for vector store operators

The benchmark data supports a few concrete engineering conclusions.

1. 1vCPU droplets can be a precarious baseline for embeddings. Even when the median latency looks merely slow, the tails can be huge (especially with Ollama), and the system can saturate before we can scale it with concurrency.
2. if we must run embeddings on DigitalOcean, Premium AMD is a rational default tier. The Regular tier is both slower and more prone to extreme tail behavior under the same serving stack.¹
3. on constrained VMs, llama.cpp is the efficiency play, not a marginal optimization. In our results it is frequently the difference between unusable and barely viable, and sometimes the difference between barely viable and operationally acceptable, with the important caveat that the llama.cpp runs here show a lower recorded success rate and therefore require investigation (timeouts, error modes, or harness thresholds) before one should declare victory.
4. embeddings capacity planning should be framed around saturation points, not around theoretical scaling. The concurrency experiments show a pattern typical of CPU-bound inference: once the machine is saturated, we can trade throughput for latency (or vice versa), but we do not get more total work done.

Aarchitecture that matches the data

A design that fits these results suggests separating the public-facing service responsibilities from the embedding compute responsibilities. The cloud side handles what clouds are good at: API ingress, auth, rate limits, queues, retries, and observability. The embedding worker side runs on hardware that is predictably good at dense math (Apple Silicon in this dataset), exposes an internal endpoint, and maintains model residency in memory for sustained throughput, aka, call home for embedding inference. If our workload is steady (ingestion pipelines, reindex jobs, continuous updates), this can be economically and operationally superior to running many tiny cloud instances. If our workload is bursty or geographically distributed, cloud still makes sense, but we should treat the minimum viable droplet as Premium AMD, multiple vCPUs, enough RAM to avoid batching collapse, and assume that runtime choice is a first-order decision.

Takeaways

The empirical story is:

• embedding inference is a CPU- and memory-behavior problem;
• cloud tiering (Premium vs Regular) is decisive for CPU-bound inference on DigitalOcean;
• runtime overhead can dominate on small machines;
• scaling cores and increasing concurrency are not free multipliers at low resource levels;
• Apple Silicon delivers an order-of-magnitude class improvement in embedding throughput relative to small cloud VMs.

If our system’s critical path depends on embeddings, as most vector-store-backed systems do, then treating embeddings as a generic web workload is likely a poor design choice.