Reranking is one of the highest-value steps in a RAG pipeline: it rescores only the few dozen candidates already recalled, yet it often lifts a useful passage from fifth place to first. Many teams want to validate the effect with a free tier first, but they need to understand what "free" actually removes and what it takes in exchange.

This article breaks down the forms of free quota, their limits, privacy boundaries, and quality checks, then explains which scenarios suit a free start and when to migrate to a paid plan. It deliberately avoids hard-coded quota figures; refer to the vendor's official pages for those.

Note: Cohere, Jina, BAAI, BGE, and other names belong to their respective owners. This article has no affiliation with or agency relationship to those organizations. Free quotas, trial terms, model licenses, and prices may change; rely on the vendor's official pages and model cards.

1. What "free" usually means

There are generally four kinds of "free reranking" on the market, with completely different cost structures:

  • Vendor trial quota. A hosted API offers a limited number of trial calls, usually with rate and concurrency limits. Some require a payment method, and the terms may restrict use to evaluation.
  • Platform promotional quota. A gateway or aggregation platform gives new users a small allowance. It is suitable for getting an endpoint working, but is still constrained by rate limits and data terms.
  • Open-source self-hosting. Weights can be downloaded at no charge, but you need a GPU or CPU, deployment work, and operations labor. Licenses must be checked individually; do not assume commercial use is permitted.
  • Lightweight self-built deployment. A smaller cross-encoder can run on CPU for low-concurrency scenarios. The barrier is low, but quality and throughput are usually weaker than larger models.

"Free" does not mean zero cost. It shifts cost into time, operations, privacy, or quality.

2. Common limits on free tiers

Free tiers almost never promise production-grade service. The following limits affect projects most:

Limit type Common form Project impact Mitigation
Call volume and rate Daily or per-minute caps, low concurrency Load tests and peak hours fail easily Add caching, queues, and client-side throttling
Candidates per request Only a limited number of documents per call Caps recall depth; long-tail content cannot enter Pre-rank to shrink candidates, then score in batches
Text length Long documents are truncated Long contracts and manuals lose key passages Chunk semantically, score segments, then merge
Data terms Free tiers may log requests or use them for service improvement Sensitive content should not be submitted directly Redact, use synthetic data, or self-host
Commercial license Trial quota limited to evaluation or barred from commercial use Licensing must change before launch Read terms early; do not run production on a trial
Stability and version No SLA; model version may change at any time Ranking drift makes regressions hard to diagnose Pin versions and keep a regression set

The most underestimated limits are candidates per request and text length. Poor results often mean the right candidates never entered the request or the key passage was truncated, not that the model is inadequate.

3. Open-weight rerankers: the real self-hosting bill

Open-weight models keep data in your own environment, but "free download" does not mean "zero cost." GPU memory, inference latency, model upgrades, monitoring, and incident handling all count. A low-concurrency internal tool can start on CPU; an online service usually needs GPUs and concurrency control.

If you only want to learn whether reranking is worth using, run a small comparison in five steps:

  1. Pull 100 to 300 queries from real logs, label relevance grades, and build an evaluation set.
  2. Freeze the candidate set using the top 20 to 50 results from production retrieval so every run is comparable.
  3. Run baselines first: the vector-similarity order without reranking, and the keyword or BM25 order.
  4. Then run candidate rerankers and record NDCG@10, MRR, Top-3 hit rate, and p95 latency.
  5. Check the model license and data path; confirm commercial use and log retention before deciding to launch.

Below is a minimal self-hosted example. The model name is illustrative only; the model card and license govern actual use:

from sentence_transformers import CrossEncoder

# Model name is illustrative; confirm language, length, and license on the model card
model = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

query = "What is the return policy?"
docs = [
    "Returns may be requested within seven days of delivery; the item must be in good condition.",
    "Shipping notes: remote areas may incur a surcharge.",
    "Membership points are issued after an order is completed.",
]

scores = model.predict([(query, d) for d in docs])
ranked = sorted(zip(scores, docs), reverse=True)
for score, doc in ranked:
    print(round(float(score), 4), doc[:40])

This code demonstrates ranking only. Production additionally needs timeouts, batch-size control, retries, and log redaction.

4. How to judge quality on a free plan

Free does not necessarily mean poor quality, and paid does not necessarily fit your corpus. Look at four things:

  • Language and domain: public leaderboards cannot replace your own evaluation. Test Chinese, lower-resource languages, and industry terminology separately.
  • Candidate quality: reranking can only reorder what was recalled. If the correct passage is absent, no model can recover it.
  • Score comparability: different models and versions use different score scales. Do not apply model A's threshold to model B; compare rankings only.
  • Long documents and mixed languages: these expose truncation and multilingual problems most readily, so the evaluation set must cover them deliberately.

A common mistake is testing a few demo questions, seeing the order look right, and shipping. Demo questions are too easy to reveal differences. Use real queries, real candidates, and graded labels.

5. Privacy and compliance boundaries

A reranking request contains the query and document text, and its information density may be higher than an ordinary chat. Once contracts, customer records, or code snippets are submitted to an external free tier, the risk includes not only disclosure but retention period, use for training, storage location, and deletion mechanisms.

The order should be: classify the data first, then choose the approach. Public material may use a hosted trial. Internal material should at least be covered by no-retention and no-training commitments. Sensitive data should preferably be private or covered by a data processing agreement. When using a gateway, check one more layer: whether request content is logged, how long logs are kept, and whether it is used for other purposes.

During the trial phase, use redacted or synthetic data to get the pipeline working. Do not connect the production knowledge base directly.

6. Scenarios that fit and do not fit

Free reranking is a reasonable fit for:

  • Proof of concept and internal demos.
  • Low-concurrency internal tools where occasional rate limiting is acceptable.
  • Offline batch evaluation that does not require real-time response.
  • Small knowledge bases with non-sensitive documents, where the team can deploy an open model itself.
  • Pre-purchase comparison, using one evaluation set to compare several options.

It is a poor fit for:

  • Customer-facing, high-concurrency services that require stable latency and failure commitments.
  • Sensitive data with no private deployment option, leaving only an external free tier.
  • Very long or numerous candidate documents, where per-request limits directly cap quality.
  • Enterprise scenarios with audit, retention, and permission-traceability requirements.
  • Long-term production where the team cannot operate GPUs and will eventually return to a hosted service.

7. Moving from free to paid

Consider migrating when rate limits become frequent, p95 latency is unstable, evaluation scores plateau, compliance review fails, or call volume keeps growing. There are usually three paths:

  • Move to a paid hosted API. Minimal code change and no operations burden; compare quota, rate limits, data terms, and invoicing.
  • Self-host open weights. Data stays controlled and the economics may improve at high volume, but include GPU, staffing, and reliability costs.
  • Use a unified gateway. Manage multiple models and keys through one entry point, which simplifies switching and metering; supported scope is governed by the in-site model list.

Whichever path you take, keep a fallback: when reranking times out, revert to vector similarity or keyword order, and record how often that happens.

FAQ

Can a free quota be used directly in production?

Most free tiers are positioned for evaluation and trials; their terms, rate limits, and stability are not designed to production standards. Before launch, confirm commercial licensing, SLA, and data handling, then choose paid hosting or private deployment.

Can open-weight rerankers be used commercially?

It depends on the specific model license. Some are permissive; others carry extra restrictions. Check the model card and LICENSE for each model before use, and do not assume commercial use merely because the weights are downloadable.

How large is the quality gap between free and paid reranking?

There is no universal answer. The difference comes from model size, language ability, input length, and candidate count. Comparing NDCG@10, Top-3 hit rate, and latency on the same evaluation set is more reliable than reading marketing pages.

Can reranking significantly improve RAG results?

Only if the correct passage was already recalled. Reranking cannot recover an answer that recall missed. Check Recall@k first, then judge the benefit; many problems actually lie in chunking and recall.

Should Chinese-language projects choose an open model or an overseas API?

Both are worth testing. Open models offer advantages in data control and cost; hosted services offer advantages in operational simplicity and stability. Add Chinese text, industry terminology, and long documents to the evaluation set, then decide based on data residency requirements and budget.

Next steps

To validate the rerank pipeline at low cost, sign up for ZSQH AI Gateway with a small budget, chain retrieval, reranking, and generation together, and then decide between hosted and self-hosted. Models and prices are governed by the model pricing page.

Related reading: