A Rerank API relay looks like nothing more than "forwarding requests upstream," but reranking occupies a critical position in the search pipeline. If it is a hundred milliseconds slow, users notice. If its billing unit is opaque, month-end reconciliation becomes a problem. If its model version changes quietly, ranking quality drifts with it.

This article covers common relay and proxy models, their latency impact, billing reconciliation, and migration steps, so a team can trade off "launch quickly" against "maintain for the long term." For specific models and prices, refer to each provider's documentation and the in-site model pricing.

Note: Cohere, Jina AI, and other names are trademarks of their respective owners. This article has no affiliation with the vendors mentioned. The relay market changes quickly; the forms described here are common in 2026 and do not constitute an endorsement of any provider.

1. Rerank relays are not chat relays

The main problems with a chat API relay are streaming output, context length, and concurrency. The main problems with a rerank relay are synchronous latency, document count, and billing definitions. A reranking request is usually short but extremely sensitive to p95. One request may carry dozens of documents, and cost grows with count multiplied by length. The response contains only indices and scores, so when something goes wrong it is hard to tell from the response alone which upstream served it.

Another easily overlooked point: reranking has no unified OpenAI-compatible standard. Cohere uses /v2/rerank and Jina uses /v1/rerank, and field names and score ranges may differ. If a relay performs protocol translation, confirm that the translated behavior matches the upstream, especially the truncation policy and the semantics of top_n.

2. Four common relay models

Model Control Source of latency Billing transparency Key ownership Suitable for
Commercial relay Low, dependent on provider Relay routing plus cross-border path Depends on call-level detail Provider holds upstream key Fast validation, small traffic
Self-built reverse proxy High, self-operated Server region plus upstream path Upstream bill directly visible Self-held Teams with operations capacity
Cloud regional gateway Medium, constrained by the cloud product Same-region calls usually shorter Cloud bill can be split Self or cloud account Already deployed on the same cloud
Unified AI gateway Medium-high, multiple upstreams Gateway region plus upstream routing Unified bill, verify multipliers Gateway provider or self-held Multiple models and projects

Running your own proxy does not eliminate risk: you still handle servers, certificates, retries, and rate limiting, and you carry compliance responsibility for the upstream account. A commercial relay removes operations work, not the responsibility for choosing well.

Start with an end-to-end budget: recall, rerank, generation, and network overhead. The reranking increment should stay within a few hundred milliseconds, or users will feel search "catch" for a moment. Jina's documentation gives a reference point: one hundred documents of about 256 tokens each with a 64-token query take roughly 150 milliseconds; a single 4096-token document can push this into seconds. Cohere recommends no more than 1,000 documents per request. A relay adds one more hop: RTT, queuing, and possibly protocol translation.

Rank latency reductions by cost-effectiveness: reduce the number of candidate documents, shorten each document, set top_n lower, place the relay node near the business region, and reuse connections. If the relay node and upstream are in different regions, cross-border RTT may exceed the model inference time itself. For search-as-you-type interfaces, move reranking to the second request or rerank only the top 20 to 50 candidates.

When measuring, distinguish "relay response time" from "end-to-end search time." A good number for the former does not imply a good number for the latter, because retries, DNS, TLS handshakes, and queuing can all hide behind an average.

4. Billing transparency: align three definitions

Rerank billing commonly uses one of three definitions: per request, per document or search unit, or per input token. The same business volume can produce very different bills under different definitions. Details to confirm include whether top_n affects billing, how truncated documents are billed, whether failed requests are billed, whether cache hits receive a discount, whether there is a minimum spend, and whether multipliers are included in the unit price.

The most effective approach is a reconciliation experiment: prepare a fixed batch of requests, send them directly and through the relay, and record call count, document count, token count, and actual charges. If the relay provides only a total balance and no per-call detail, later it will be hard to investigate why the bill rose this month. Enterprise buyers should also confirm the invoicing entity, invoice timing, and refund rules so that prepaid balances do not become a sunk cost.

5. Key and data risk

Commercial relays often use a shared upstream key. That makes onboarding fast, at the cost of losing control over upstream rate limits, model versions, and the risk of suspension; if the upstream changes policy, every tenant is affected at once. A self-built proxy, by contrast, makes you responsible for key leakage, abuse, and runaway billing.

Reranking requests carry queries and document passages, which may include customer records, contract clauses, or internal knowledge-base content. Before signing, ask clearly: are requests logged, how long are logs retained, are they used for training, in which region is data stored, and what is the deletion mechanism? For sensitive workloads, prefer a provider willing to sign data processing terms, or deploy the reranking model in your own environment.

Also be wary of vague phrases such as "partnered with the vendor." Whether a service is authorized or serves enterprise customers should be established from contracts and verifiable qualifications, not from a marketing page alone.

6. Migration checklist

  1. Inventory current calls: endpoint path, model, candidate count, timeout, and retry settings.
  2. Save a baseline: record ranking results, p95 latency, and error rate over a batch of real queries.
  3. Abstract the client: put the URL, key, and model name into configuration instead of scattering them through business code.
  4. Run side by side: send the same requests through the original service and the new relay, then compare rankings and quality metrics.
  5. Calibrate scores: score distributions differ between models, so do not carry over old thresholds.
  6. Roll out gradually: start with 5 to 10 percent of traffic and watch search metrics and billing.
  7. Verify fallback: confirm that timeouts, 429, and 5xx can revert to the original service or the original ranking order.
  8. Keep a rollback window: retain the old configuration and keys for at least one full billing cycle.

Step 5 is where migrations most often go wrong. A reranking score expresses only relative order within one candidate set and is not comparable across models. Model version, truncation length, and whether structured documents were passed all change the result.

7. How to verify a relay

Begin with static checks: endpoint path, authentication method, whether the model name can be pinned, whether the response contains index and relevance_score, and whether the score range matches the documentation. Then run dynamic tests: send identical payloads directly and through the relay, compare ranking differences, and record p50, p95, error rate, and 429 frequency. Finally, reconcile the bill to confirm that call detail lines up.

Below is a minimal latency test script. The important point is to use documents of realistic length rather than equal-length placeholder text:

import statistics, time, requests

URL = "https://<relay-host>/v1/rerank"
HEADERS = {"Authorization": "Bearer <key>"}
PAYLOAD = {
    "model": "<model>",
    "query": "What is the refund policy?",
    "documents": ["Realistic-length document passage goes here " + str(i) for i in range(50)],
    "top_n": 5,
}

latencies = []
for _ in range(30):
    start = time.perf_counter()
    resp = requests.post(URL, json=PAYLOAD, headers=HEADERS, timeout=3)
    resp.raise_for_status()
    latencies.append((time.perf_counter() - start) * 1000)

latencies.sort()
print("p50", statistics.median(latencies))
print("p95", latencies[int(len(latencies) * 0.95) - 1])

If the relay cannot provide a stable model version identifier or call logs, use it only for non-critical workloads. Production should retain a switchable backup path and be retested periodically. Teams preparing to connect multiple providers can start with the sign-up page to review the current endpoint list, then estimate budget using the model pricing page.

8. FAQ

How much slower is a relay than a direct connection?

It depends on node location and routing. A same-region relay may add only a few milliseconds; a cross-border hop can add tens to hundreds of milliseconds. Measure p95 with real requests rather than relying on a vendor's average.

Why does the relay bill not match the upstream bill?

The usual reason is a different billing definition: per request, per document, per token, or including multipliers, minimum spend, and failed requests. Before signing, ask the provider for the billing formula and a sample of call detail.

Is it normal for rankings to change after switching relays?

Yes. Model version, truncation length, structured-document handling, and score normalization can all differ. Compare on the same query set and recalibrate business thresholds.

What if the relay service shuts down?

Prepare two things in advance: abstract the client into switchable configuration with a direct or alternative-provider backup path, and avoid loading the entire budget onto a single relay.

Does a self-built reverse proxy count as a relay?

It is a relay in the technical sense. It gives you control over keys and logs, but you take on servers, certificates, rate limiting, compliance, and responsibility for the upstream account.