Rerank API is not on the generation path, yet it directly determines whether the first few search results are correct. It sends "a query plus a batch of candidate documents" to a reranking model and returns an order sorted by relevance. Because the call happens while a user is waiting, the real problem for teams in China is not "can it be called at all," but whether the network is stable, whether the bill can be reimbursed, whether latency is controllable, and whether the system degrades gracefully when something fails.
This article compares three paths — direct connection, commercial relay, and unified gateway — and provides a pre-launch verification checklist. Specific models, endpoint paths, and prices follow each provider's documentation and the in-site model pricing page.
Note: Cohere, Jina AI, and other names are trademarks of their owners; this article has no affiliation with the relevant vendors. It does not hard-code unit prices; consult each provider's current quote and contract before purchasing.
1. Why Rerank API is a separate access problem
Rerank differs from a chat API in three ways. First, it is a synchronous short request: a call may take only tens to hundreds of milliseconds, but the user is waiting for that result. Second, it is not part of the OpenAI Chat Completions standard; fields and paths are defined by each vendor — Cohere uses /v2/rerank, for example, while Jina uses /v1/rerank — so migration is not just a base URL change. Third, the billing unit may be requests, documents, tokens, or search units, making the bill structure more complex than input/output tokens.
Access planning in China therefore has to satisfy four things at once: a network path that stays connected over time, payment and invoicing that fit enterprise processes, latency within the search budget, and a plan for rate limits and failures. Checking only whether a response comes back is far from enough.
2. Choosing between direct, relay, and unified gateway
Common paths fall into four categories. Direct connection to an overseas vendor suits teams with international network access and foreign-currency payment capability. Direct connection to a domestic vendor is easier for network and invoicing, but models and protocols must be adapted again. A commercial relay bundles network and payment. A unified gateway emphasizes multi-vendor consolidation, key management, and one bill.
| Path | Network | Payment and invoicing | Latency control | Rate limits and failover | Suitable for |
|---|---|---|---|---|---|
| Overseas vendor, direct | Depends on international links; volatile at peak hours | Usually requires a foreign card; invoicing takes longer | Region can be chosen, but cross-border RTT is high | You handle 429 and retries | Teams with an overseas entity and operations capacity |
| Domestic vendor, direct | Usually stable within China | RMB settlement, convenient invoicing | Region selectable, short path | Constrained by vendor quota | Teams mainly serving domestic users |
| Commercial relay | Provider optimizes the route | RMB top-up; confirm invoicing | Depends on the relay's routing | Provider handles fallback, but ask about boundaries | Small teams that want to launch quickly |
| Unified gateway | Multiple routes on the gateway side | One bill, easier procurement | Depends on gateway deployment region | Multiple upstreams and fallback can be configured | Multiple models and projects in parallel |
Do not compare only unit prices. Put "who diagnoses a failure," "which entity issues the invoice," and "can per-call latency and billing details be provided" into the evaluation table; these often matter more than the points saved. If you plan to test unified access, you can first review the currently supported interfaces and models through the registration entry, then decide whether to use it in production.
3. Network and latency: start with the search budget
Reranking usually sits after recall and before generation. A workable budget is roughly: recall 20–80 ms, rerank P95 within 300 ms, generation 1–3 seconds. For search-as-you-type scenarios, reranking may need to become asynchronous or apply only to second-page results.
Latency is mainly affected by three variables: the number of candidate documents, the length of each document, and the total tokens of query plus documents. Jina's documentation gives a reference point: 100 documents of about 256 tokens each with a 64-token query take roughly 150 ms, but a single 4,096-token document can push that into seconds. Cohere's documentation recommends no more than 1,000 documents per request and truncates each document at 4,096 tokens by default. These numbers vary with model and load, but the pattern holds: reduce candidates first, truncate second, and only then consider a faster model.
In engineering, prioritize four actions: set top_n small and return only the items actually used; chunk or truncate long documents first; call from a region close to the business; and reuse connections to avoid a new handshake on every request. When measuring, look at P95 and P99, not just the average.
4. Payment, invoicing, and contracts
The common problems with direct overseas access are payment methods and reimbursement rules. Foreign cards, exchange rates, invoices from an overseas entity, and refund rules for prepaid balances all affect the finance workflow. Domestic vendors and most gateways can settle in RMB, but you should still confirm the invoice type, invoicing timing, whether top-ups can be invoiced, and whether refunds return via the original route.
Confirm five things in writing before signing a contract or placing an order: what the billing unit is, whether failed requests are billed, whether there is a minimum spend, how price changes are announced, and the data retention and logging policy. For reranking, requests carry business document excerpts that may contain customer information or internal material, so the data-processing terms cannot be assumed to be "the same as the chat interface."
5. Rate limits, timeouts, and failover
Rerank rate limits usually consider both RPM and TPM. Because cost grows with "document count × length," a single batch rerank can easily hit quota at peak. The client should set an explicit timeout, for example 800 ms to 1.5 seconds, retry at most once after a failure, and add random jitter to avoid a retry storm.
More important is the degradation path. When reranking fails, the system should fall back to the original recall order or switch to another provider, rather than failing the whole search. Below is a call example with a timeout; adjust the field names to the provider's documentation:
curl -sS --max-time 1.5 https://<your-endpoint>/v1/rerank \
-H "Authorization: Bearer $RERANK_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "<per provider documentation>",
"query": "What documents are needed to register a company in Russia",
"documents": ["Excerpt one", "Excerpt two", "Excerpt three"],
"top_n": 5,
"return_documents": false
}'
Note that paths and fields are not standardized across vendors; return_documents, score ranges, and structured-document support must all be checked one by one. Do not assume that swapping the base URL of a chat interface is enough to call reranking.
6. Pre-launch verification checklist
- Verify the protocol: confirm the request path, required fields, score range, and maximum document count.
- Run an offline comparison with real queries and 50–100 candidates, and record changes in ranking quality.
- Load-test in the target region and record P50, P95, P99, and error rates.
- Trigger 429 deliberately to verify that the backoff policy works.
- Simulate timeouts and 5xx responses, and confirm fallback to the original order or a backup provider.
- Reconcile a fixed workload by comparing call logs with the actual bill.
- Confirm the contracting entity, invoice type, refund rules, and data terms.
- Release to a small traffic slice and watch both search click-through rate and interface latency.
The point of this checklist is not to "test once," but to leave reproducible records. Reranking quality and latency are strongly tied to the business corpus; someone else's benchmark is only a reference.
7. FAQ
What is the difference between Rerank API and Embedding API?
Embedding turns text into vectors for coarse recall; Rerank scores "query–document" pairs directly for precise ordering. They are usually chained together and cannot replace each other.
Can a domestic vendor's Rerank directly replace an overseas model?
Interfaces and score distributions may both differ. After switching, recalibrate thresholds and compare rankings on the same query set; do not judge by whether the interface returns 200.
Is a dedicated line required to call an overseas API directly?
Not necessarily. First measure packet loss, P95, and timeout rates during the target hours. If the business is latency-sensitive, then evaluate a dedicated line, regional gateway, or relay; there is no need to start with the heaviest option.
How much slower does reranking make search?
It depends on the candidate count and document length. Keeping candidates to a few dozen and truncating each document, as described above, usually holds the increase to hundreds of milliseconds; long documents and large batches are noticeably slower.
Does every unified gateway support Rerank?
No. Rerank is not part of the OpenAI chat interface standard, so support depends on the gateway's interface list. Confirm protocol compatibility before discussing price and quota; you can check current capabilities after entering through the registration entry.
