An embedding API relay is often treated as a "change the endpoint and pay a little less" choice. But vectorization is a batch workload, and a migration can involve hundreds of thousands of texts. If a relay rate-limits, truncates, or quietly swaps models, the loss is not a single request but the trustworthiness of the entire index.

This article addresses four questions: what forms a relay can take, how to design batch requests, how to verify billing, and how to migrate and validate. For specific models and pricing, refer to our model pricing.

Note: OpenAI and other model and service names are trademarks of their respective owners. This article has no affiliation with the relevant vendors. The relay market changes quickly, and this article does not endorse any provider.

1. Relay, Proxy, and Gateway Are Not the Same Thing

  • Network proxy: addresses reachability only, forwarding HTTP or HTTPS requests; you still manage the key yourself.
  • Commercial relay: a provider forwards requests upstream, possibly with billing, quotas, logs, and invoices; some offerings share or resell keys, and stability depends on the operator.
  • Unified AI gateway: provides an OpenAI-compatible entry point, aggregates multiple models, issues independent keys, and includes a console and usage statistics.
  • Self-hosted proxy: you deploy your own reverse proxy or gateway; this offers the most control, but operations and upstream accounts are your responsibility.

Before signing, confirm which layer the provider offers, who owns the key, and where data passes through. If these concepts are unclear, every later comparison loses focus.

Dimension Direct vendor access Self-hosted proxy Commercial relay Unified AI gateway
Billing basis Vendor bill, clear rules Upstream bill plus your costs Depends on the operator; reconcile Unified in-platform bill
Batch throughput Limited by account quota Can queue, still limited upstream May use a shared pool; confirm Depends on gateway limits and scheduling
Key ownership You hold it You hold it May be shared or resold Usually an independent key
Data boundary You assess it You are responsible Review terms and retention Review terms and qualifications
Invoices and contracts Depends on the entity No third-party contract Depends on provider capability Usually invoiced
Operations cost Medium High Low to medium Low

2. Batch Requests: Stress-Test Before Discussing Price

Embedding throughput is determined by three variables: items per request, text length per item, and concurrency. Providers impose limits on items per request, total tokens, and request body size; exceeding them may return a parameter error, or an intermediate layer may truncate the request outright.

Stress-test with production data. Take a batch of real document chunks, assemble requests in small, medium, and large batches, and observe per-batch duration, P95 latency, the 429 rate, and error types. Bigger is not always better: with very large batches, a single failure is expensive to rerun, and timeouts are harder to control.

A recommended batching process:

  1. First measure the text-length distribution, and estimate by tokens rather than splitting by item count.
  2. Set a per-batch cap while leaving headroom for long texts.
  3. Start with low concurrency and increase each round, recording throughput and error rate.
  4. Stop increasing concurrency once target throughput is reached; do not consume all headroom.
  5. Persist each batch's input hash, token usage, and results for reconciliation and resuming.

3. Retries and Partial Failures

Embedding requests are read-only, so retries do not create duplicate writes, but they can amplify failures into billing. For 429 and transient 5xx errors, use exponential backoff with random jitter; for 401, 403, 404, and parameter errors, stop immediately and fix the key, endpoint, or request body first.

Handle partial failures separately. If 3 of 200 items in a batch are rejected for being too long, do not retry the entire batch indefinitely; extract the problematic texts, fix or truncate them, and submit them separately while the rest are stored normally. Use a content hash as an upsert key so reruns do not create duplicate records.

import hashlib
import os
import time
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["RELAY_API_KEY"],
    base_url=os.environ["RELAY_BASE_URL"],
    timeout=60.0,
    max_retries=0,
)
MODEL = os.environ["EMBEDDING_MODEL"]

def key_of(text):
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

def embed_with_retry(batch, attempts=4):
    for i in range(attempts):
        try:
            resp = client.embeddings.create(model=MODEL, input=batch)
            store = {
                key_of(batch[d.index]): d.embedding
                for d in resp.data
            }
            return store, resp.usage
        except Exception as exc:
            if i == attempts - 1:
                raise
            time.sleep(min(2 ** i, 30))

texts = ["document chunk one", "document chunk two", "document chunk three"]
store, usage = embed_with_retry(texts)
print(len(store), len(next(iter(store.values()))), usage)

4. Billing Transparency

Embeddings are usually billed by input tokens, with no output tokens, which differs from chat models. When reconciling, ask:

  • How tokens are counted, and whether the count matches the vendor's tokenizer.
  • Whether failed and retried requests are billed.
  • Whether batch discounts, caching, or tiered pricing exist.
  • Whether usage can be exported by project, key, and model.
  • Whether prepaid credit expires, and whether balances are refundable.
  • How invoice content, currency, and exchange rates are determined.

A "low multiplier" does not mean a low total cost. If a relay bills failed retries, or usage statistics do not match response usage, hidden losses can erase the low price. For specific prices and rules, refer to model pricing and the provider's bill; do not rely on verbal promises.

5. Key Risk and Data Boundaries

The most easily overlooked issue with relays is key ownership. Shared keys, resold quota, or credit of unknown origin can cause four problems: upstream bans interrupt your business too; shared multi-tenant capacity causes rate limits to affect each other; request content passes through a third party with opaque logging and retention; and when the terms of service prohibit resale, disputes lack a basis.

When choosing a provider, confirm at minimum the company entity, terms of service, privacy policy, data retention location, invoicing capability, and failure contact. Do not send text containing personal information, contract text, or secrets to a relay that cannot explain where data goes. For heavily regulated scenarios, prioritize direct access, private deployment, or services that can sign a data processing agreement.

6. Migration Checklist

When moving from direct access to a relay or gateway, proceed in this order:

  1. Inventory existing models, dimensions, batch parameters, and index structure; record baseline success rate and P95 latency.
  2. Run a fixed sample on direct access and save vectors, usage, and timing.
  3. Create a test key with budgets and alerts; do not reuse a production key.
  4. Change only base_url and api_key, and run a minimal request in staging.
  5. Call the relay with the same sample and compare dimensions, item counts, usage, and cosine similarity.
  6. Canary 5%–10% of batch jobs, observe for 3–7 days, and keep direct access as a fallback.
  7. After full migration, reconcile for a week and confirm bills, stored item counts, and failed-rerun records match.
  8. Keep at least one version of the old index and clean it up only after retrieval quality is stable.

If the relay serves a different model, old vectors cannot be mixed in directly. Build a parallel index, compare retrieval quality after a full recompute, and then decide whether to switch.

7. How to Verify a Relay

  • Vector consistency: send the same input through direct access and the relay, then compare dimensions and cosine similarity; if they differ noticeably, check first whether the model was swapped or truncated.
  • Usage consistency: reconcile response usage against the backend bill batch by batch, focusing on whether failures and retries are billed twice.
  • Latency and error rate: test with real batches during peak evening hours, recording P50/P95, 429, and 5xx rates.
  • Batch limits: confirm limits on items per request, total tokens, and request body size so production batches are not truncated.
  • Key isolation: confirm whether keys are independent, whether they can be rotated, and whether quota and rate limits are shared with other customers.
  • Contract and support: check the entity, terms, invoices, and ticket response; only a personal contact and a refusal to provide usage details are red flags.

The model field in a response is not sufficient evidence on its own, because it can be rewritten. Combined verification is more reliable than any single check.

8. When Not to Use a Relay

A relay may not be worthwhile when data is sensitive and retention policies cannot be reviewed, when direct vendor support is required, when call volume is very small, or when the team has no capacity for reconciliation. A relay also cannot bypass upstream rate limits or terms of service. If you only need occasional small-scale vectorization, direct access plus a simple script may be less trouble.

If you need a unified entry point and multi-model comparison, create a test key at ZSQH AI Gateway, validate vector consistency with a fixed sample first, and then migrate batch jobs; available models and billing are shown on the model pricing page.

Related reading:

Frequently Asked Questions

How is an embedding API relay different from a chat relay?

A chat relay focuses on streaming output and time to first token; an embedding relay focuses on batch throughput, per-request limits, resuming after failures, and vector consistency. For reconciliation, note that embeddings usually bill only input tokens.

Can vectors returned by a relay be mixed with direct-access vectors?

Mixing is advisable only with the same model and version. Different models have different vector spaces, and even the same dimensions cannot be mixed in one index. Before migration, compare cosine similarity using a fixed sample.

What should I do if half of a batch request fails?

First identify the failure type: rate limits and transient 5xx errors can be retried with backoff; overlong inputs or parameter errors require extracting the problematic text and handling it separately. Use a content hash as an idempotency key so reruns fill only missing items and avoid duplicate writes.

How is relay billing calculated?

Most bill by input tokens, but token counting, whether failures and retries are billed, and whether batch discounts exist vary by provider. Rely on the bill and usage details, and put these items in your comparison table before signing.

Does migrating to a relay require re-embedding?

With the same model and version, usually not; changing models requires rebuilding the index. Even when no recompute is needed, keep the old index and a rollback path, and run a consistency check on a fixed sample.