For teams in mainland China, the hard part of adopting an Embedding API is usually not the SDK. What actually consumes time is whether the network path is stable, how to pay and get invoices, and what to do when a batch vectorization job fails halfway through hundreds of thousands of texts. Embedding requests are short, highly concurrent, and large in total volume, so the selection logic differs from that of chat APIs.

This article compares three common routes — direct vendor access, a self-hosted proxy, and a domestic AI gateway — and provides reliability design for batch jobs, failover boundaries, and a pre-launch checklist. For specific models, dimensions, and billing, refer to model pricing.

Note: OpenAI and other model and service names are trademarks of their respective owners, and this article has no affiliation with those vendors. Cross-border networking, data export, and payment settlement must comply with applicable laws and regulations as well as platform terms. Consult legal and compliance staff before implementation.

1. The three routes at a glance

Direct vendor access calls the model vendor's endpoint directly. It has the shortest path and fast synchronization of documentation and features. It suits teams that already have cross-border networking, overseas payment channels, and reimbursement processes. The cost is that network quality, USD billing, exchange rates, and invoices are all your responsibility.

A self-hosted proxy deploys a reverse proxy or gateway on compliant servers to centralize authentication, logging, retries, and rate limiting. It offers strong control, but you also take on servers, certificates, upgrades, and on-call duty. If you build one only to save effort, operational costs often exceed expectations.

A domestic AI gateway provides an OpenAI-compatible endpoint with RMB settlement, usually including a console, usage statistics, and invoices. Integration cost is low and multiple models can be unified behind one entry point. You need to verify the operating entity, data terms, quota rules, and incident response.

Dimension Direct vendor access Self-hosted proxy Domestic AI gateway
Network and access Depends on cross-border links; must optimize yourself Route can be chosen; must operate yourself Domestic entry point, usually usable out of the box
Payment and invoices USD settlement; longer reimbursement chain Depends on account and server entity RMB settlement; invoices available
Batch throughput Constrained by account limits Can queue, but still limited upstream Depends on gateway quotas and scheduling
Data boundary Assess export and terms yourself You are responsible for data location Review provider terms and qualifications
Operational cost Medium, mainly network and account High: servers, certificates, monitoring Low; centralized console management
Model coverage Mainly a single vendor Expandable yourself Usually covers multiple vendors

No single route fits every team. If you use one model and networking and payment are smooth, direct access is least work. If you need a fixed egress and auditing, choose a self-hosted proxy. If you lack operations staff but need multiple models on one bill, evaluate a domestic gateway.

2. Network testing: Embedding is about throughput, not just latency

Chat APIs care about time to first token. Embedding cares about batch throughput and stability. A single request may contain dozens to hundreds of texts, and the response body is hundreds or thousands of floating-point numbers. Link problems show up as timeouts, truncation, and 5xx errors.

Do not test with a single "hello." Use document chunks of realistic length and assemble requests at production batch size. Observe at least four metrics: per-batch time, P50/P95 latency, error rate, and behavior during evening peak. Also confirm the proxy or gateway's request body size limit, connection reuse, and timeout settings. When a large request is truncated by an intermediary, the error message is often far from intuitive.

Long jobs must be resumable. Number inputs by content hash, persist each successful batch, and rerun only the missing portion after a failure instead of rebuilding the entire corpus.

3. Payment, contracts, and invoices

The common sticking point for teams in China is the paying entity. Personal card top-ups, third-party payments, and inconsistent reimbursement documents often prevent finance from booking the expense. Before choosing a route, clarify: who signs the contract, what settlement currency is used, what invoices can be issued, whether prepaid credit expires, and what the refund rules are.

For usage-based billing, also confirm the billing basis: input tokens or request count, whether failed requests are counted, whether retries are billed again, and whether usage can be split by project or key. Put these into a procurement comparison table and test with the same data set to calculate the real cost. Actual fees follow model pricing and the contract.

4. Reliability design for batch jobs

Vectorization is usually offline batch processing. The worst case is failing at 80% with no way to resume. Design in this order:

  1. Generate a content hash for each text and use it as an idempotency key, so repeated submissions overwrite rather than duplicate entries.
  2. Estimate and split batches by tokens, while respecting three limits: items per request, length per item, and total tokens.
  3. Increase concurrency gradually and watch the 429 rate. Do not open maximum concurrency immediately.
  4. Apply exponential backoff with random jitter to 429 and 5xx responses, with a maximum retry count.
  5. Record successful item counts, token usage, and latency after each batch, and persist failed samples separately.
  6. Validate vector dimensions, empty vectors, and NaN before writing. Isolate abnormal data rather than writing it silently.
  7. After the full run, sample and compare to confirm counts, dimensions, and index statistics match.

The process is not complex, but it prevents the "looks finished but a batch is missing" problem.

5. Rate limits, retries, and failover

Embedding endpoints usually rate-limit by both request count and tokens. When a limit is hit, the API returns 429; when a gateway is overloaded, 5xx errors or timeouts may appear. Retry strategies should distinguish error types: 429 and transient 5xx errors can be retried with backoff, while retrying 401, 403, and parameter errors is pointless. Fix the configuration first.

import os
import time
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ZSQH_API_KEY"],
    base_url=os.environ["ZSQH_BASE_URL"],
    timeout=60.0,
    max_retries=0,
)

def embed_batch(texts, model, attempts=4):
    for i in range(attempts):
        try:
            resp = client.embeddings.create(model=model, input=texts)
            vectors = [item.embedding for item in resp.data]
            dims = {len(v) for v in vectors}
            if len(dims) != 1 or len(vectors) != len(texts):
                raise ValueError("dimension or count mismatch")
            return vectors, resp.usage
        except Exception as exc:
            if i == attempts - 1:
                raise
            wait = min(2 ** i, 30) + 0.5 * (i + 1)
            print(type(exc).__name__, "retry in", wait)
            time.sleep(wait)

vectors, usage = embed_batch(
    ["First piece of text", "Second piece of text"],
    os.environ["EMBEDDING_MODEL"],
)
print(len(vectors), len(vectors[0]), usage)

There is an easily overlooked failover boundary: vectors from different embedding models cannot be mixed. Even with the same dimensions, the vector spaces differ, and writing vectors from two models into the same index noticeably degrades retrieval. When switching upstream providers, first confirm whether it is the same model and version. If the model must change, build a new index, recompute the full set, compare quality, and then switch. Do not change the model name casually during an outage.

6. Pre-launch checklist

  • Dimensions: consistent with the index configuration, and batch response count matches input count.
  • Baseline: use a fixed set of test texts and compare cosine similarity against direct access results; they should usually be very close.
  • Usage: whether the usage in responses matches the backend bill. Verify batch requests batch by batch.
  • Rate limits: concurrency cap, 429 ratio, and post-retry success rate are acceptable.
  • Timeouts: whether the largest batch completes within the timeout during evening peak.
  • Data: whether inputs contain personal information, and whether the provider's retention, logging, and deletion policies are clear.
  • Exit: whether keys can be rotated, balances refunded, and old indexes and rollback paths retained.

If any item is unclear, run a small-traffic pilot first. Do not migrate the production index directly.

7. Choosing by constraints

First check whether networking and payment can remain stable long term, then whether you need fixed egress and auditing, and finally model coverage and billing capabilities. If you need a unified endpoint, RMB settlement, and multi-model comparison, create a test key at the ZSQH AI Gateway, run a small batch first, and scale gradually. Available models and billing methods are listed on model pricing.

FAQ

How does Embedding API access differ from chat API access?

Chat APIs focus more on time to first token and streaming stability. Embedding is batch-oriented with short responses and high total volume, so it focuses on throughput, per-request limits, and resumable failures. Include batch load testing and resumability in your design.

Is direct access to an Embedding API stable from mainland China?

It depends on the local egress, time of day, and account limits, so no general answer applies. Observe real batch sizes and real concurrency over time, record error rates and P95 latency, and compare against a gateway option. Do not draw conclusions from a single successful request.

What should I do if batch vectorization is interrupted?

Use content hashes as idempotency keys, persist by batch, and fill in only the missing part after a failure. Before rerunning, verify the count and dimensions already in the store so you do not duplicate writes or treat a half-batch as a complete result.

After switching gateways, can old vectors still be used?

If it is still the same model and version, they can usually continue to be used. If the model changes, the index must be rebuilt. Before migrating, compare cosine similarity between old and new vectors on fixed samples, and keep the old index as a fallback.

Can invoices and usage be split by project?

It depends on the provider. Before signing, confirm whether usage can be exported by project, key, or model, whether the billing basis includes failed and retried requests, and whether invoice contents meet procurement requirements.

Related reading: