When building RAG or semantic search, embeddings are often the first component teams outsource: send text to a hosted API, get vectors back, and hand them to a vector database. It is quick to start, but as data volume grows, compliance tightens, or bills become hard to predict, teams begin looking for alternatives.

There are three common routes: a hosted embedding API, self-hosting open-weight models, and using the built-in vectorization of a vector database. None is universally better; the differences come down to five things: dimensions and language, latency, privacy, cost, and operations.

This article breaks down those five dimensions and provides a scenario selection table and a migration checklist. For specific models, dimensions, and pricing, refer to each vendor's official documentation and our model pricing.

Note: OpenAI, Azure, Cohere, Pinecone, Qdrant, Milvus, Weaviate, BAAI, and Qwen are trademarks of their respective owners. This article compares technical approaches only and does not imply any partnership or authorization. Model specifications and API limits may change between versions; refer to official documentation.

1. Distinguish the Three Routes

Hosted embedding API. This includes direct model-vendor access, cloud-provider model services, and OpenAI-compatible gateways. The advantages are no operations burden, pay-as-you-go billing, and quick switching. The disadvantages are that text leaves your network and cost grows with call volume.

Open-weight models. Examples include BGE-M3, Qwen3-Embedding, EmbeddingGemma, multilingual-e5, and the jina family. Weights can be downloaded and deployed on your own servers or private cloud. The advantage is that data stays inside your network; the disadvantages are that you must provision inference resources and handle versions, scaling, and monitoring yourself.

Built-in vectorization in a vector database. Pinecone's integrated embedding, Weaviate's vectorizer, Qdrant's FastEmbed, and Milvus's embedding functions fall into this category. You write text directly, and the database or a model it calls generates vectors, saving a layer of glue code. But built-in vectorization does not necessarily mean local inference; many implementations still call an external model service. pgvector only stores and retrieves vectors; it does not generate them.

2. Dimensions: Higher Is Not Always Better

Dimensions determine representational capacity and directly affect storage and retrieval cost. Common examples: OpenAI's text-embedding-3-small defaults to 1,536 dimensions and 3-large to 3,072; BGE-M3 uses 1,024; Qwen3-Embedding's 0.6B, 4B, and 8B models correspond to 1,024, 2,560, and 4,096 dimensions respectively; EmbeddingGemma defaults to 768 and can be truncated to 512, 256, or 128. These are examples only; refer to official documentation.

Storage can be estimated roughly: number of vectors × dimensions × bytes per float. One million 1,024-dimensional float32 vectors take about 4 GB of raw data; 3,072 dimensions take about 12 GB, before indexes, replicas, and metadata.

# Estimate raw vector footprint, excluding indexes, replicas, and metadata
chunks = 1_000_000
for dim in (768, 1024, 1536, 3072):
    gb = chunks * dim * 4 / 1024**3
    print(f"{dim} dims: {gb:.1f} GB")

The key point is that dimensions and model are bound together. Vectors from different models or different dimensions settings cannot be mixed in one index; after changing models you must recompute everything, not pad with zeros or truncate to fit the length.

3. Language Coverage: Test Chinese Separately

Many models claim support for hundreds of languages, but "supported" does not mean it performs well on your corpus. Chinese, mixed Chinese-English, Russian, product model numbers, abbreviations, and typos all affect recall.

When selecting a model, do not rely on leaderboards alone. Run a small evaluation with real user questions, real document chunks, and hard cases (paraphrases, cross-language queries, numbered clauses). Track at least recall@k and MRR, then check whether the ranking matches intuition. These models all emphasize multilingual capability, but your own data should have the final say.

Tokenization efficiency also matters. Under the same 8,192-token limit, Chinese, English, and code fit different numbers of characters. Chunking with a token margin is safer.

4. Latency and Throughput: Watch p95, Not the Average

Hosted API latency consists of network round trips, queueing, and inference. Cross-region calls, network jitter, and peak-hour queues can push p95 well above p50. Batch requests spread cost but increase single-response time, making them suitable for offline ingestion.

Self-hosted latency mainly comes from inference queueing and batching. Small models can run on CPU but have limited throughput; large models need GPUs, along with consideration of VRAM, concurrency, and cold starts. Without an operations team, this cost is often underestimated.

Built-in database vectorization adds another link to the call chain: both writes and queries wait for the model to return. It saves code complexity, not necessarily latency. Measure ingestion throughput and query p95 separately.

5. Privacy and Compliance: Where Does the Text Go?

A hosted API means text leaves your network. Confirm whether data is used for training, how long it is retained, which region it is stored in, and whether enterprise data agreements are supported. For personal information, contracts, and financial data, complete a compliance review first.

Open-source self-hosting keeps text inside your VPC, but that does not automatically mean compliance: model licenses, logs, backups, and access control are all your responsibility. With built-in database vectorization, check whose model is called underneath and which region requests are sent to.

Vectors themselves can also leak information. Embeddings are not encryption; attackers can perform similarity inference. Treat permission labels, tenant isolation, and access logs as seriously as the source text.

6. Cost Structure: Unit Price Is Only One Item

When comparing options, do not compare only the price per million tokens. Full cost includes at least:

  • Ingestion tokens: initial vectorization and incremental updates.
  • Query tokens: every question must be embedded again.
  • Recompute and migration: full-library reruns when changing models or chunking rules, plus dual writes, traffic switching, and rollback.
  • Storage and memory: vectors, indexes, replicas, and metadata.
  • Inference and operations: self-hosted GPU, CPU, power, deployment, upgrades, and monitoring.

Recompute cost is the most commonly underestimated. A full rerun can last hours or even days. For specific unit prices, refer to our model pricing; prices change with vendors and versions, so this article does not hard-code numbers.

7. Choose a Route by Scenario

Scenario Preferred route Why Watch out for
Small team validating quickly Hosted API No operations, pay as you go Record model name, dimensions, and chunking version
Data cannot leave the network Open-source self-hosting Text stays in your VPC Assess GPU, licensing, and backups
Already using a vector database at modest scale Built-in vectorization One less layer of glue code Confirm the underlying model and request region
Massive offline ingestion, cost-sensitive Self-hosting or batch hosted Batching spreads cost Run recall evaluation before full ingestion
Multilingual support, cross-border e-commerce Multilingual hosted model or BGE-M3-class model Broad language coverage Validate less-common languages with real data
Latency-sensitive online queries Hosted API deployed nearby, or a small local model Fewer network round trips Watch p95, not just the average
Dense plus sparse hybrid retrieval Open-source models with multiple outputs Several vector types per inference Indexing and scoring are more complex

This table is only a starting point. The real choice depends on existing infrastructure: if you already use a cloud, look at its model service first; if you already have a GPU cluster, the marginal cost of self-hosting is lower; if you already use a vector database, built-in vectorization can save engineering time.

8. Migration Checklist

  1. Inventory the current state: model name, dimensions, distance metric, chunking rules, index parameters, and daily call volume.
  2. Build an evaluation set: a few hundred real questions and correct documents, run as a fixed regression.
  3. Pick two or three candidates; do not change too many variables at once.
  4. Dual-write a small share of traffic: write old and new vectors, query the new index, and record recall differences.
  5. Validate dimensions, normalization, and distance metric; confirm that cosine, dot product, and index settings are consistent.
  6. Create a new collection for a full recompute; do not mix vectors of different dimensions in the old index.
  7. Shift traffic by percentage, keep the old index and a rollback switch, and observe for at least one business cycle.
  8. Archive the model version, dimensions, chunking version, and recompute date in the data dictionary.

The most common migration failure is not poor quality but inconsistent dimensions, normalization, or distance metrics. Align the engineering details first, then discuss quality.

Frequently Asked Questions

Can open-source models fully replace an embedding API?

In most scenarios, yes, especially for teams with sensitive data or stable call volume. But you must maintain inference services, run model evaluations, and handle version upgrades. During rapid validation, a hosted API is easier.

What is the difference between built-in database vectorization and calling an API directly?

Built-in vectorization folds vector generation into the database write path, requiring less code, but underneath it often still calls an external model service. It does not automatically solve privacy and cost issues; before signing, confirm model source, region, and billing.

Must I recompute everything when changing embedding models?

Yes. Vector spaces of different models are incompatible, and mixing them makes retrieval unreliable. The same dimensions do not mean they can be mixed. Run a small-scale evaluation before a full recompute.

Do higher dimensions always improve quality?

Not necessarily. Higher dimensions can improve representational capacity but also increase storage and retrieval cost. Many models support Matryoshka truncation, allowing a trade-off between quality and cost, but re-evaluate after truncation.

What kind of GPU does self-hosted embedding need?

It depends on model size, batch size, latency target, and concurrency. Small models can serve low-QPS scenarios on CPU; large models usually need GPUs and batching. Stress-test with real traffic first, then decide on specifications.

Next Steps

If you want to validate quality with a hosted setup first, create a test key at ZSQH AI Gateway, run ingestion and queries through the OpenAI-compatible interface, and then decide whether to self-host. Model availability and pricing are listed on the model pricing page.

Related reading: