For most production semantic search, classification and RAG indexing workloads, a modern server CPU is enough to serve embedding models — no GPU required. Small encoder checkpoints such as MiniLM-class models run comfortably in the tens to hundreds of embeddings per second per core on CPU once exported to a compiled runtime, and the embedding step is rarely the latency bottleneck. The cases where a GPU (or a managed embedding API) genuinely pays for itself are narrow: very large batch backfills, high-dimensional large checkpoints, or strict single-digit-millisecond p99 budgets that you have actually measured. This article gives you a decision framework, a reproducible benchmark harness, and the tuning knobs that move CPU throughput the most.
When CPU Inference Wins
Embedding models are encoder transformers, and the workloads that use them split cleanly into two shapes. Online query embedding is tiny: one short text, batch size one, latency dominated by tokenization and framework overhead rather than matrix math. Offline corpus embedding is big but patient: millions of documents, throughput-oriented, schedulable at night. Both shapes fit CPU well.
A useful rule of thumb: if your corpus refreshes incrementally (thousands of documents per day, not millions per hour) and your query path targets sub-50 ms, CPU serving covers the embedding tier while your GPUs stay busy on generation. This matters when you co-locate an embedding service next to an LLM: keeping the small encoder on CPU removes VRAM contention and frees the GPU for the model that actually needs it. Before you commit to a latency budget for the whole feature, work through a latency budget worksheet so the embedding share of the budget is explicit rather than assumed.
Backends and Quantization Choices
The sentence-transformers library supports three inference backends, and the official documentation recommends a clear decision path for CPU: if a small accuracy loss is acceptable, use OpenVINO with int8 quantization; if not, use OpenVINO on Intel CPUs and plain ONNX Runtime elsewhere. The ONNX backend is selected with backend="onnx", and optimization levels up to O3 (operator fusions, fast GELU approximation) are exported once with export_optimized_onnx_model(). One export-time trap worth knowing: the ONNX export converts only the Transformer component, so if you run the graph outside sentence-transformers you must apply pooling and normalization yourself.
Quantization applies at two independent layers, and conflating them is a common mistake. Model quantization (int8 weights via OpenVINO/Optimum) speeds up compute. Embedding quantization shrinks the vectors you store and search: converting float32 embeddings to binary reduces memory and storage by 32x, and the documented binary-plus-rescore pipeline (binary top-k retrieval, then float32 rescoring) preserves up to roughly 96% of retrieval performance while using 32x less memory. The reference setup in the docs serves 41 million Wikipedia texts from about 5.2 GB of RAM for the binary index plus an int8 index on disk — versus roughly 200 GB for plain float32. Scale that ratio down and a multi-million-document corpus fits in a modest VM.
A Reproducible Benchmark Harness
Below is a harness you can adapt and run on your own hardware. It is an example procedure, not a measured case study — record your own numbers before making capacity decisions.
- Pin versions: sentence-transformers, optimum and onnxruntime versions in a lockfile, because backend behavior changes between releases.
- Load the same checkpoint three ways: PyTorch CPU, ONNX (
backend="onnx",provider="CPUExecutionProvider"), and OpenVINO int8 where Intel hardware applies. - Fix inputs: identical texts, identical truncation (e.g. 256 tokens), fixed padding so runs are comparable.
- Warm up each backend on representative inputs before timing; first-call latency includes graph initialization and misleads naive benchmarks.
- Measure both shapes that matter: single-text latency (p50/p99) and batched throughput (texts/second at batch 32 and 256).
- Check quality, not just speed: compare cosine similarity of quantized vs float32 embeddings on a sample and verify retrieval overlap on a golden query set.
- Record environment: CPU model, core count, RAM, thread settings, library versions, date.
Thread Tuning That Moves Numbers
Most CPU embedding services are quietly throttled by thread configuration, not by the model. ONNX Runtime documents the knobs explicitly. By default, intra_op_num_threads = 0, which means the session uses one thread per physical core — a 6-core machine with 12 hyperthreads gets 6 intra-op threads — and default spinning is enabled, which trades CPU cycles for lower latency. When you host several embedding sessions on one box, disable spinning (session.intra_op.allow_spinning = 0) to stop idle worker loops burning cores that other services need. If you set the thread count explicitly, note that ONNX Runtime then skips its default per-core affinity, so pin threads yourself when co-tenancy matters.
A starting configuration for a sidecar embedding service on an 8-core shared VM:
intra_op_num_threads = 4— half the physical cores, leaving headroom for the app and vector store.- Spinning disabled on both pools to keep idle CPU near zero.
- Batch online queries (micro-batching 10–50 ms windows) so per-request overhead amortizes across the batch.
- Export the model once at optimization level O3 and ship the artifact; do not re-export at pod start.
Decision Checklist and Limits
Run through this checklist before choosing managed embedding APIs over self-hosted CPU inference:
| Signal | CPU self-hosting is enough when… | Move to GPU/API when… |
|---|---|---|
| Query rate | < a few hundred embeddings/s | Sustained thousands per second |
| Corpus size | Incremental refresh, millions of docs with binary+int8 indices | Full re-embed of very large corpora on tight deadlines |
| Latency budget | Embedding share ≥ 10 ms, p99 measured | Sub-5 ms p99 mandated end to end |
| Model size | MiniLM/BGE-small/base class encoders | Multi-GB embedding models (e.g. 8B-class decoders) |
| Co-tenancy | CPU cores available next to app tier | Every core contested by latency-critical services |
Be honest about the limits. The published backend benchmarks that show large speedups from compiled CPU runtimes use tuned batches and specific hardware; your single-request latency will differ, and the docs themselves warn that results must be retested per model and workload. Embedding agreement checks are not retrieval-quality evaluations — a quantized model can produce near-identical vectors and still shift your top-k results. Treat vendor benchmark charts as hypotheses and your own harness as the evidence. Once embeddings are served, log what actually leaves the service; the same hygiene you apply to LLM logging applies here, and an observability logging policy defines what to keep and what to redact. If you later expose the embedding tier behind a shared gateway, rate limiting and quota design should be settled before the first internal team hammers it.
Sources
- Sentence Transformers — Speeding up Inference (backends, optimization levels, backend recommendations), consulted 2026-10-16.
- Sentence Transformers — Embedding Quantization (binary/int8 quantization and rescoring), consulted 2026-10-16.
- ONNX Runtime — Thread management (intra-op threads, spinning, affinity), consulted 2026-10-16.