Reranking is the second ordering step in a retrieval pipeline: a vector or keyword search first recalls a batch of candidates, then a reranking model scores each one and the most relevant items are placed in the language model's context. It solves the problem of "plenty was recalled, but the top of the list is wrong." It cannot solve the problem of "nothing relevant was recalled at all."

This article moves from endpoint and authentication to request and response, then covers top_n and candidate-set size, vector search integration, latency degradation, common errors, and evaluation metrics. Models and pricing follow the in-site model pricing page.

Note: Cohere, Jina AI, BAAI, and Alibaba Cloud are names owned by their respective rights holders and are used here for technical explanation only. ZSQH AI Gateway is not affiliated with these vendors; interfaces, models, and billing are governed by the in-site documentation.

1. Where reranking sits in the retrieval pipeline

A typical RAG retrieval flow has three stages: recall, rerank, and generation. Recall aims for speed and coverage, reranking aims for precision, and generation uses only the filtered context.

Stage Typical method Input Output Main metric
Recall Vector search, BM25, hybrid retrieval User query 30–100 candidates Recall@k
Rerank Rerank model Query + candidate set 5–10 items nDCG@10
Generation Large language model Reranked context Answer Correctness, faithfulness

A reranking model is usually a cross-encoder: it reads the query together with each document and outputs a relevance score. That is more accurate than vector similarity, and also slower. It belongs after recall, not in place of it.

2. Endpoint, authentication, and request parameters

The reranking endpoint on ZSQH AI Gateway is:

POST https://ai.zhongshangqihang.com/v1/rerank

Authentication uses Authorization: Bearer <key>, and the request body is JSON. The gateway also accepts the /rerank path. Key creation and security settings are covered in the companion tutorial.

Parameter Required Description
model Yes Rerank model name, subject to the in-site model list
query Yes User query; must not be empty
documents Yes Array of candidate documents, usually strings; object structures vary by upstream
top_n No Number of results to return; set it explicitly rather than relying on a default
return_documents No Whether to include document content in the results
max_chunk_per_doc No Maximum chunks per long document; support varies by upstream
overlap_tokens No Token overlap between chunks; confirm against upstream documentation

Most implementations accept an array of strings, while some platforms accept objects with a text field. Confirm the format with one minimal request before migrating a batch workload.

3. Request and response: Python and curl

Python example, recalling first and then reranking:

import os
import requests

api_key = os.environ["RERANK_API_KEY"]
base_url = "https://ai.zhongshangqihang.com/v1"

documents = [
    "The trial period is 14 days and can be extended before it ends.",
    "Bank transfer and platform escrow are both supported.",
    "Invoices are issued within 3 business days after payment.",
]

resp = requests.post(
    f"{base_url}/rerank",
    headers={"Authorization": f"Bearer {api_key}"},
    json={
        "model": "your-rerank-model-name",
        "query": "How long is the trial period?",
        "documents": documents,
        "top_n": 2,
        "return_documents": False,
    },
    timeout=20,
)
resp.raise_for_status()

for item in resp.json()["results"]:
    print(item["index"], item["relevance_score"], documents[item["index"]])

The same request with curl:

curl "https://ai.zhongshangqihang.com/v1/rerank" \
  -H "Authorization: Bearer $RERANK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-rerank-model-name",
    "query": "How long is the trial period?",
    "documents": ["The trial period is 14 days.", "Bank transfer is supported."],
    "top_n": 2
  }'

The results array is already sorted by relevance in descending order. Each item contains:

  • index: the position of the document in the original documents array, not its rank. Always use it to look up the source text.
  • relevance_score: the relevance score, usually normalized to between 0 and 1, but not a probability and not a linear difference.
  • document: returned only when return_documents is true, and its structure varies by upstream.
  • usage: token usage, useful for bill reconciliation and cost analysis.

Do not reuse relevance_score as a fixed threshold. When the model, language, or document type changes, the score distribution changes and the threshold must be recalibrated.

4. top_n and candidate-set size

top_n is the number of results returned; candidate-set size is the number of documents sent to the model. They are not the same thing. A common mistake is to set top_n very high while recalling only a dozen documents, leaving reranking with nothing to work with.

Candidate-set size Suitable scenario Relative latency Consideration
20–30 Small knowledge base, low-latency Q&A Low Low recall ceiling; answers are easily missed
50 A common starting point for most RAG Medium Run an evaluation before increasing or decreasing
100 Many documents, varied query phrasing Higher Watch per-document length and truncation
200+ High-value queries, offline evaluation High Cost and latency need separate assessment

Practical advice: start with a candidate set of 30–50 and set top_n to 5–10. A candidate set that is too small caps quality; one that is too large raises latency and cost and may inject noise into the context. Upstream services may cap the number of documents and per-document length, and overlong documents are truncated — possibly cutting off the key sentence.

Also note that some upstreams return only one result when top_n is omitted. Always pass top_n explicitly in production code; do not rely on default behavior.

Reranking is not an isolated step; the order of integration affects both quality and cost. A recommended flow:

  1. Rewrite the user query where necessary, resolving references and dropped subjects.
  2. Convert the query into a vector with an embedding model.
  3. Recall the top 30–50 with vector search, optionally running keyword or BM25 recall in parallel.
  4. Merge the two result sets and deduplicate by document ID.
  5. Apply permission and metadata filters before reranking, so unauthorized documents are never sent to the model.
  6. Call the Rerank API and take top_n of 5–10.
  7. Apply a score threshold or item cap, assemble the context, and pass it to the generation model.

A few points that are easy to overlook:

  • Filter before reranking: permissions, tenant, language, and version should be filtered at recall time or before reranking, not removed after generation.
  • Keep stable IDs: reranking returns indexes, so the application must map each index back to a document ID, title, and URL.
  • Rerank chunks, not documents: RAG usually reranks chunks rather than whole documents. Chunks that are too long dilute relevance; chunks that are too short lose context.
  • Use hybrid recall: proper nouns, model numbers, and identifiers are more reliable with keywords, while semantic questions are more reliable with vectors. Merging both before reranking usually beats either alone.

6. Latency, timeouts, and degradation

Reranking latency grows roughly with "number of documents × document length." One hundred short documents are usually fast; several hundred long documents become noticeably slow. Before launch, measure at least three points: P50 and P95 latency at candidate sets of 30, 50, and 100.

Engineering recommendations:

  • Give Rerank requests their own timeout; 10–30 seconds is a common starting point, then tighten it based on measurements.
  • Retry with backoff only for 429 and 5xx, and only a limited number of times. Do not retry 401, 403, or 422.
  • If reranking times out or fails, fall back to the vector recall order rather than failing the entire Q&A request.
  • Cache results for frequent, low-change queries. The cache key should include at least the query, document IDs, and model name.
  • Run vector recall and keyword recall in parallel, merge, then rerank, to reduce serial waiting.
  • The rerank interface returns the full result at once and does not support streaming; do not design the front end around a streaming contract.

If latency remains high, reduce the candidate set and document length first, then consider a smaller model. Moving straight to a larger model is not always worthwhile, especially when recall quality itself is low.

7. Common errors and troubleshooting

Status or symptom Common cause Action
400 / 422 Empty query, empty documents, or malformed JSON Check required fields and array types
401 Invalid or expired key, or a missing header Verify Authorization and key status
403 Key lacks model permission, or the account is restricted Check the model scope and account status
404 Wrong model name or endpoint path Verify /v1/rerank and the model list
429 Concurrency or quota has triggered rate limiting Back off and retry; check quota and rate limits
500 / 502 / 504 Upstream error or timeout Retry a limited number of times, then use the fallback path

The gateway validates query and documents and returns a parameter error immediately when either is empty. When troubleshooting, read the message field in the response body first, then compare it with the request log. Do not treat a 400 as a network problem and retry it repeatedly.

8. Evaluation: beyond "it feels more accurate"

Before launching reranking, prepare a small labeled set: 100–300 real queries, each labeled with 1–3 relevant documents. Without labeling capacity, manually inspect at least 50 queries and record both orderings.

Metric What it shows How to use it
Recall@50 Whether the correct answer is in the candidate set Judges the recall ceiling; fix recall first if it is low
nDCG@10 Ranking quality at the top Compare with and without reranking
MRR Position of the first relevant result Suited to question-answering queries
Answer correctness Whether the generated answer is right End-to-end validation
P95 latency How slow the slow requests are Sets candidate size and timeouts
Cost per query Retrieval cost per Q&A Estimate together with the pricing page

Keep variables fixed during evaluation: same queries, same recall results, same top_n, changing only whether reranking is enabled or which model is used. Then run a small live A/B test and watch correctness, no-answer rate, and latency. If quality drops after reranking, check first whether the candidate set missed the right document, whether documents were truncated, whether the model suits the language, and whether a threshold was copied from another scenario.

9. FAQ

No. Reranking can only reorder an existing candidate set; it cannot recover documents that were never recalled. Recall is responsible for coverage, and reranking is responsible for precision at the top.

What should top_n be set to?

When assembling context for a language model, 5–10 is a common starting point. If downstream business rules will filter further, 20–50 may be appropriate. Let evaluation decide.

Is a larger candidate set always better?

No. A larger candidate set raises the recall ceiling, but latency, cost, and noise rise with it. Start with 30–50 and use Recall@50 and nDCG@10 to decide whether to expand.

Why does quality get worse after adding reranking?

There are four common causes: the candidate set never recalled the correct document, chunks are too long or too fragmented, the model does not match the language, or a threshold was copied from another scenario. Check each one rather than simply swapping models.

Does the Rerank interface support streaming responses?

No. Reranking must see the entire candidate set before it can produce a complete ordering, so it typically returns at once. Treat it on the front end as an ordinary JSON request and set a reasonable timeout.