Reranking is one of the most underestimated cost items in a RAG pipeline. A single call looks inexpensive, but every user question runs one; once the candidate set grows, the bill scales with both query volume and document volume.
Vendors do not meter the same way: some charge per query, some per document, some per token. Comparing "how much does one call cost" across them is meaningless unless you convert to your own query volume, candidate-set size, and document length. This article breaks down billing structures, candidate sets, caching, and budget estimation without hard-coding unit prices; refer to the in-site model pricing page and vendor documentation for current figures.
Note: Cohere, Jina, Voyage, and BAAI are trademarks of their respective owners and are used here only to explain technical and billing structures. Billing units, limits, and prices may change; consult the official pages before purchasing.
1. What rerank does in the pipeline
A typical two-stage retrieval flow is:
User question
→ vector/keyword recall of 20–200 candidates
→ Rerank for precise ordering, keep the top 5–10
→ assemble context and pass it to the generation model
The input to reranking is "one query plus a batch of candidate documents"; the output is relevance scores and an ordering, with no generated text. It solves the problem where the correct passage was recalled but ranked tenth, so a simple cutoff would have discarded it.
Its cost position follows from that: one call per retrieval, sometimes several; every candidate passes through the model. Reranking is usually cheaper than upgrading to a larger generation model, but it is not free.
2. Three common billing models
| Billing model | Unit | What makes cost grow | Common example | Budget concern |
|---|---|---|---|---|
| Per query | One query with a certain number of documents | Number of queries; chunks of long documents also count | Cohere Rerank | Chunking rules multiply document counts |
| Per token | Query tokens and document tokens | Query length, candidate count, document length | Voyage, Jina, and others | Query tokens are multiplied by document count |
| Per document or per request | Each document or each request | Document count, request count | Some managed services | Confirm truncation and per-request limits |
| Self-hosted | GPU hours and operations | Concurrency, model size, utilization | Open-source BGE family | Fixed cost; watch utilization |
Cohere bills by search unit, which its pricing page defines as one query with up to 100 documents. Long documents are split into chunks, however, and each chunk counts toward the document total. "100 documents" therefore does not equal "100 billed documents"; knowledge bases with many long documents should be estimated on the post-chunking count.
Voyage bills per token with the formula: query tokens × number of documents + the sum of all document tokens. The longer the query and the larger the candidate set, the stronger the multiplier effect. The usage.total_tokens field in the response can be used for reconciliation. Its interface accepts up to 1,000 documents per request, and overlong input may be truncated by default; confirm the truncation rule in advance.
Jina also bills on total input tokens, counting both query and documents. Its documentation states that failed requests are not billed and offers automatic top-up at a balance threshold. Once the free allowance is exhausted, billing becomes paid; refer to the billing page for specifics.
Self-hosting an open-source reranking model such as the BGE family removes the per-call fee but adds GPU, bandwidth, and operations costs. At steady, large volume, self-hosting can be more controllable; at small or highly variable volume, fixed GPU cost often exceeds pay-as-you-go billing.
3. Sizing the candidate set
Rerank cost grows roughly with the number of candidate documents. Reranking all 200 recalled documents typically costs several times as much as reranking 50, and latency rises too.
| Candidate count | Relative cost | Latency | Suitable scenario |
|---|---|---|---|
| 20–30 | Low | Low | High-frequency support, latency-sensitive interfaces |
| Around 50 | Medium | Medium | A common starting point for general knowledge bases |
| 100 | Higher | Higher | Long documents, noisy recall |
| 200+ | High | High | Optimize recall first; not a recommended default |
Start with around 50 and use an evaluation set to compare Recall@k and answer correctness before deciding whether to add or remove candidates.
A common misconception: lowering top_n only reduces the number of returned items and usually does not reduce the number of billed documents. Savings must come from cuts before reranking, not after. Metadata filtering, near-duplicate removal, similarity thresholds, and restricting the document scope by business line are all more effective than post-hoc truncation.
4. Document length and chunking
Many rerank interfaces impose a token limit per document, and the excess is truncated. Truncated content does not participate in scoring and may cut away the key passage, making results worse.
Under token-based billing, longer documents cost more. Under document-based billing, a long document may be split into several billed documents. In both models, the chunking strategy affects quality and the bill at the same time.
- Send only the retrieved chunks to reranking, not entire documents.
- Align chunk size with the model limit, and do not lose titles or essential context.
- Deduplicate before indexing so near-identical chunks are not billed repeatedly.
- Record the token distribution per chunk and budget against P95 rather than the average.
5. Balancing latency and cost
Reranking adds latency, and the effect grows with the candidate set. The answer is not a single policy but a layered approach by scenario:
- High-frequency FAQ traffic can skip reranking and use a cache or keyword search directly.
- Latency-sensitive interfaces should cap the candidate count and apply metadata filters first.
- Offline jobs that care about quality and run at low concurrency can afford a larger candidate set.
- Set client timeouts and a retry cap. A retry after a timeout may bill the same query twice.
- Evaluate latency at P95 rather than the average.
6. Caching and deduplication
A rerank cache sits after retrieval and before generation. The cache key should include at least: the normalized query, document IDs and versions, the model name, and top_n. Missing any one of these can reuse a wrong result for another tenant.
- High-frequency questions: a TTL from hours to days is reasonable.
- Rapidly updated documents: a TTL of minutes, or active invalidation when a document changes.
- Permissions: the cache key must include tenant and permission scope; never allow cross-tenant hits.
- Failed results: do not cache them; short-TTL negative caching also requires care.
- Batch jobs: if the same document set is scored repeatedly on the same day, reuse the results directly.
In support and internal-policy Q&A, where repeated questions are common, caching usually yields more direct savings than switching to a cheaper model.
7. Estimating the budget
First convert "each query" into the target vendor's billing unit, then multiply by query volume:
Monthly cost ≈ daily queries × 30 × billing units per query × unit price
+ retry and peak redundancy
For token-based billing:
Tokens per query = query tokens × candidate document count + sum of all candidate document tokens
A reproducible estimation process:
- Pull average and peak daily query volume from logs.
- Sample 100–200 real queries and compute P50 and P95 for candidate count and document tokens.
- Convert each query into billing units under the vendor's model.
- Multiply by daily query volume and 30 days, then add 10%–30% for retries, growth, and peak redundancy.
- Multiply by the latest unit price from model pricing; do not use an old screenshot.
- Reconcile weekly after launch and correct the estimate against the actual bill.
# Cost estimate for token-based billing: fill the unit price from /pricing
QUERY_TOKENS = 60 # median from sampling
DOCS_PER_QUERY = 50
AVG_DOC_TOKENS = 180
UNIT_PRICE_PER_MILLION = 0.0 # placeholder; replace with the actual unit price
def cost_per_query(q_tokens, n_docs, doc_tokens, price):
total_tokens = q_tokens * n_docs + n_docs * doc_tokens
return total_tokens / 1_000_000 * price
per_query = cost_per_query(
QUERY_TOKENS, DOCS_PER_QUERY, AVG_DOC_TOKENS, UNIT_PRICE_PER_MILLION
)
print(f"About {per_query:.6f} per query")
print(f"About {per_query * 10_000:.2f} for 10,000 queries per day")
Unit prices, free allowances, and volume discounts can all change. The script should read the unit price from the current pricing page rather than hard-coding it in the repository.
8. Bill reconciliation and alerts
- Break billing down by key, project, and business line; do not mix all calls under one key.
- Record query count, candidate document count, tokens, model, and latency separately.
- Separate the cost of embedding, reranking, and generation; otherwise you cannot tell which stage to optimize.
- Watch for retries, empty queries, duplicate front-end submissions, and background reindexing jobs — common sources of bill spikes.
- Set daily budget and balance alerts, and notify the owner when the balance falls below a threshold.
Usage fields returned by the vendor can support reconciliation; refer to each vendor's documentation for field names and definitions.
FAQ
Is reranking more cost-effective than upgrading to a larger generation model?
In most cases, yes. Reranking processes only a few dozen candidates, so cost is controllable, and it directly improves the case where the correct passage sits lower in the list. But if recall never found the right material, reranking cannot rescue it; fix recall first.
Does lowering top_n save money?
Usually not. Billing is generally based on the candidate documents or tokens sent to the model, while top_n only affects how many items are returned to the generation model. To save money, reduce, deduplicate, and filter candidates before reranking.
Are more candidate documents always better?
Not necessarily. More candidates bring more noise, higher latency, and higher cost. Use an evaluation set to compare Recall@k and answer correctness and find the point of diminishing returns; a common starting point is around 50.
Should long documents be chunked first?
Yes. Most interfaces truncate at a token limit, and the excess is not scored. Document-based billing services may also split a long document into several billed documents. Preserve headings and context when chunking so the answer is not cut in half.
Is self-hosted reranking always cheaper?
Not necessarily. Self-hosting removes the per-call fee but adds GPU, deployment, scaling, and operations costs. With steady traffic and high utilization, self-hosting can be more controllable; with low or variable traffic, or a team without operations capacity, a pay-as-you-go API is simpler.
Next step
First measure candidate counts and document lengths on a batch of real queries, then use a small budget on ZSQH AI Gateway to run the full recall–rerank–generate pipeline and compare quality, latency, and cost with reranking on and off. Model availability and prices follow model pricing.
Related reading:
