Reranking sits at the last mile of the retrieval pipeline: first recall dozens to hundreds of candidates with vector or hybrid search, then score each query-document pair with a cross-encoder and move the genuinely useful passages to the front. It generates no text, yet it reliably fixes the "the answer is at position ten" problem.
There is no single correct way to implement reranking, however. A hosted API, an open-weight model, and a vector database's built-in capability can all work; they differ in quality, latency, language coverage, privacy, cost, and operations. This article separates the decision along six dimensions and closes with a scenario table and a migration checklist. For specific models and prices, refer to the in-site model pricing.
Note: Cohere, Jina, Voyage, Qwen, BGE, Weaviate, Milvus, Elasticsearch, and other names are trademarks of their respective owners and are used here for technical discussion only. Capabilities, quotas, and terms change; check the relevant vendor documentation before integrating.
1. First confirm that reranking is the bottleneck
Not every weak RAG result justifies adding a reranker. Look for three signals first. The correct passage is retrieved but often ranked below fifth place. After hybrid retrieval, the top few results are visibly noisy because keyword matching has overwhelmed semantic relevance. The generated answer cites the wrong passage, but manually substituting the correct one produces a right answer.
If the correct passage never appears in the recall set, reranking cannot help; fix chunking, embeddings, or recall depth first. If the candidate set contains only three or five documents, reranking usually adds little. Experience suggests that reranking pays off best when the candidate set holds roughly twenty to two hundred documents. Beyond a few hundred, latency and cost climb quickly.
2. What each of the three routes looks like
Hosted Rerank API. A vendor wraps the model and inference into an endpoint that typically accepts a query, documents, and top_n, then returns indices and scores ordered by relevance. The advantages are fast integration, generally broad multilingual coverage, and no GPU to manage. The drawbacks are that data leaves your network, you inherit the vendor's quotas and request format, and long-term cost grows with call volume.
Open-weight cross-encoder. You deploy the model on your own GPU or inference service. Common choices include the BGE reranker family, Qwen3-Reranker, and Jina Reranker. The advantages are that data stays inside your network, you can fine-tune, and you can pin a version. The drawbacks are GPU capacity, autoscaling, model upgrades, and maintaining an inference framework. Frameworks such as vLLM and TEI already expose reranking endpoints, which reduces some of the adaptation work.
Built-in reranking in a vector database. Some vector stores offer native or integrated reranking, such as Weaviate's rerank module, Elasticsearch's rerank retriever, and the rerank function in newer Milvus releases. Other products handle retrieval only and delegate reranking to an external service. The advantages are one fewer network hop and a single boundary for permissions and operations. The drawbacks are a restricted model choice and limited room for deep customization.
One important caveat: reranking has no unified OpenAI-compatible standard the way Chat Completions does. The Jina, Cohere, and Voyage families do not use identical fields. For example, top_n versus top_k, whether document text is returned, and the per-document truncation parameter all differ. Endpoints may be /v1/rerank or /v2/rerank. When switching vendors, protocol adaptation often consumes more time than the model itself.
3. How the six dimensions compare
Quality. Both hosted APIs and open-weight models may be cross-encoders; the differences lie more in training data, language distribution, and version. Do not rely on public leaderboards alone. Evaluate with your own business queries and candidate sets. If a built-in capability calls the same class of model, quality is usually close, but fewer parameters are adjustable.
Latency. Reranking time grows roughly with the number of candidates multiplied by document length. A hosted API adds a public-network round trip, which suits smaller candidate sets and stable concurrency. Self-hosting removes an external dependency but introduces GPU queuing. A built-in option eliminates the network hop yet may compete with retrieval for the same resources.
Language coverage. Treat Chinese, mixed Chinese-English, and lower-resource languages separately. Many models look similar on English benchmarks but diverge sharply on long Chinese documents and industry terminology. Prepare three evaluation sets: Chinese, English, and mixed.
Privacy. As soon as documents leave your network, assess data terms, transport encryption, log retention, and cross-border paths. Contracts, customer records, medical text, and financial text usually call for self-hosting or a private deployment.
Cost. Hosted pricing is metered per call or per token and scales roughly linearly with volume. Self-hosting means fixed GPU cost plus staff time, which is not necessarily cheaper at low volume. A built-in option may be included in a database plan or billed separately by inference volume. Include retries, offline evaluation, and peak-capacity expansion in the calculation.
Operations. Hosting shifts model upgrades and autoscaling to the vendor, but you still manage keys, quotas, and differences between providers. Self-hosting requires someone to watch GPUs and handle driver and framework upgrades. A built-in option merges two systems into one, which also concentrates the failure domain.
4. Choosing by scenario
| Scenario | Suggested route | Rationale | Watch out for |
|---|---|---|---|
| Fast validation, dozens of candidates | Hosted Rerank API | Usable the same day, no GPU needed | Confirm data may leave your network |
| Chinese-first, sensitive data | Self-hosted open weights | Private deployment, strong Chinese models available | Reserve GPU capacity and ops staff |
| Already using a vector database, simple needs | Built-in reranking | One less integration, unified permissions | Verify model and parameters are sufficient |
| High concurrency, latency-sensitive | Self-hosted or same-region built-in | Avoids public-network hops and queuing | Load-test, and do not look only at averages |
| Multilingual, lower-resource languages | Hosted API or a strong multilingual open model | Broader coverage with less effort | Validate with a lower-resource evaluation set |
| Very low volume, no GPU team | Hosted API | Lower fixed cost | Set budget and quota alerts |
This table is only a starting point. A single team can combine routes: use a hosted API as the production baseline, self-host open weights for offline evaluation, and let built-in reranking handle only low-risk workloads.
5. How to evaluate quality without relying on intuition
Prepare at least three things: a batch of real queries, a label identifying the correct document for each query, and the current recall set. Track Recall@k, MRR, or nDCG rather than judging by whether results "look more relevant." Export the same candidate set before and after reranking, inspect the top ten manually, and record the error type: missing recall, broken chunking, or the reranker pushing the correct passage down.
The Python below does one thing: check whether the correct document appears in the top k before and after reranking. It is vendor-neutral; the input is two ordered lists of indices.
def hit_rate(cases, k):
hits = 0
for gold, ranked in cases:
if gold in ranked[:k]:
hits += 1
return hits / len(cases)
# cases: [(index of the correct document, reranked index list), ...]
before = hit_rate(baseline_cases, 5)
after = hit_rate(reranked_cases, 5)
print(f"Recall@5: {before:.3f} -> {after:.3f}")
Use at least one hundred samples, covering different business lines and document lengths. Before launch, examine p95 latency and cost per call; do not let averages hide the long tail.
6. Migration checklist
When moving from one Rerank API to another, or from hosted to self-hosted, work in this order:
- Freeze an evaluation set and the current baseline; record Recall@k, MRR, p95 latency, and cost per call.
- Confirm the new endpoint path, authentication method, and request fields, especially top_n versus top_k, whether document text is returned, and truncation parameters.
- Align score semantics: is the score a zero-to-one relevance value or an unnormalized logit, and does sorting run ascending or descending? Do not reuse thresholds across them.
- Check the document count per request, tokens per document, request body size, and concurrency limit. Split requests on the client rather than waiting for an API error.
- Implement retries and fallbacks for timeouts, 429, and 5xx. On fallback, preserve the original recall order so the whole pipeline does not fail.
- Compare old and new results with shadow traffic for at least a week, recording ranking differences and answer-quality changes.
- Roll out gradually and define rollback thresholds, for example reverting when the error rate or p95 exceeds the baseline by a set percentage.
- Update monitoring and billing definitions, and confirm that retries, caching, and offline evaluation are not omitted.
FAQ
What is the difference between a Rerank API and an Embedding API?
An embedding model converts text into vectors for large-scale recall. A reranker reads the query and each document together, which is more precise but slower. The usual pattern is to recall dozens to hundreds of candidates and then rerank the top ones.
Are open-weight rerankers always cheaper than hosted APIs?
No. At low volume, fixed GPU cost and operations labor usually exceed pay-per-use pricing. Self-hosting becomes easier to amortize when volume is high, data is sensitive, or customization is required. Include peak-capacity expansion in the comparison.
Is built-in vector database reranking good enough?
It depends on the scenario. When the candidate set is modest and there are no special model requirements, the built-in capability saves an integration. If you need a specific language, domain fine-tuning, or finer parameter control, a dedicated reranking service is more flexible.
How do I choose a reranker for Chinese content?
First check whether the model explicitly supports Chinese or multiple languages, then evaluate it with your own Chinese queries and documents. Do not substitute English leaderboard rankings. Sample mixed-language, long-document, and industry-terminology cases separately.
Do score thresholds need recalibration after switching rerankers?
Usually yes. Score distributions differ: some models normalize to zero-to-one, others output logits. If your pipeline filters on a fixed threshold, recalibrate it or switch to rank position instead of absolute score.
Next steps
If you only want to get the pipeline running, create a test key on ZSQH AI Gateway, compare recall quality before and after reranking on the same query set, and estimate call cost from the model pricing page. Model availability and billing rules are governed by that page.
Related reading:
