RAG Re-Ranking: Choose Useful Context for Your LLM
Retrieval often returns passages about the right topic without identifying the ones that answer the exact question. RAG re-ranking adds a second selection step: score the candidates against the question, then choose what enters the model’s context. This can make a document assistant more focused, provided the selection still retains the evidence the answer needs.
Figure: Authorized candidate passages pass through a relevance review. The selected context retains the main rule together with the exception needed to interpret it.
Give retrieval and re-ranking different jobs
A retrieval system searches a larger collection for plausible candidates. A re-ranker compares the smaller candidate set with the query in more detail. Some re-rankers use a cross-encoder that reads the query and passage together. The model and provider determine the actual scoring behavior and processing cost.
For a question about an expense-policy exception, ordinary topic matches may return the general policy, an old announcement and the current exception. The re-ranking step should help bring the relevant current passage forward. It cannot recover a document that the retriever never supplied, so assess retrieval coverage before tuning the second stage.
Use the explicit gateway API where it fits
DeepintShield documents a /v1/rerank endpoint that accepts a provider/model, query and documents. Documents can carry identifiers and metadata, and supported options include a maximum number of results and returning the matched text. Provider-specific options remain provider-dependent; confirm them for the selected route.
Keep the returned index or document identifier connected to the original authorized source. This allows the application to build citations and explain which evidence entered the answer. Avoid treating a relevance score as a universal probability of correctness, especially when switching models or comparing scores from different implementations.
Keep authorization and content checks separate
A highly relevant document may still be outside a user’s access or contain instructions that should not guide the agent. Enforce document permissions in the application or retriever before sending passages for downstream processing. Apply the configured RAG content checks to the relevant operation, then build context from the accepted content.
Re-ranking answers ‘how useful is this passage for this question?’ RAG security asks about content and source-policy suitability. Neither question establishes the other. This distinction becomes especially useful when a document combines a valid business answer with unrelated instructions embedded in its text.
Visual guide: RAG Re-Ranking: Choose Useful Context for Your LLM
Figure: RAG pipeline showing a question, bi-encoder retrieval, candidate passages, cross-encoder scoring, bounded selection and LLM context.
Evaluate questions that need several passages
Some answers rely on one paragraph; others require a definition, an exception and an effective-date note from different sources. Test both cases when choosing a result cap. An attractive reduction in context size is less useful if the resulting answer omits the qualifying condition that matters to the reader.
Measure retrieval quality, re-ranking cost, end-to-end delay and final answer support together. If you also compress selected passages, test the combined path: first removing whole passages and then shortening the remainder can change the evidence in different ways. Keep a full-context comparison set so later changes can be assessed against the same questions.
Create a retrieval failure that reranking can actually solve
Suppose an employee asks which expense receipts are needed for overseas travel. The retriever returns a current travel policy, an older policy, a domestic-expense FAQ and a paragraph about passport documents. Reranking can help prioritize relevance among these candidates, but it cannot retrieve a missing overseas-policy exception that never entered the candidate set. Record whether a failure originated in candidate retrieval, relevance ordering or answer generation before choosing the corrective action.
Build a small question set with known supporting passages and plausible distractors. Include a question needing two passages, such as an eligibility rule plus its exception. Test different retained-document counts against answer completeness instead of assuming the first-ranked passage is sufficient. DeepintShield exposes an explicit rerank request with the query, documents and optional result limit; the application still decides what candidates to submit and what returned content to place in model context. Reranking API
Keep identity and provenance attached to candidates
Authorize document access before constructing the candidate list. Preserve stable document or chunk identifiers through ranking so the selected text can be traced back to its source and version. If the provider returns indexes, map them to the exact submitted array rather than a separately sorted copy. A score associated with the wrong document can produce an answer that looks sourced while citing an unrelated record. Include this mapping in automated integration checks.
After ranking, apply the content-handling process required for retrieved material and consume the allowed, sanitized content. A high relevance score is not a document permission, a safety verdict or a freshness check. Keep those decisions distinct in traces and in the code that assembles the prompt. For an empty permitted result set, ask for clarification or route to a human-supported search rather than quietly returning the unauthorized or rejected candidates to fill the context window.
Measure relevance improvement against the extra work
Measure how often the required supporting passage survives the final selection, how often an exception is retained and whether the generated answer cites the correct source. Add rerank latency and provider cost to the full retrieval path. For simple queries, a good original ranking may already be sufficient; for ambiguous queries, the additional stage may help. Report these groups separately so an average gain does not hide a class of questions whose supporting detail is consistently removed.
Simulate rerank timeout, malformed results and an unavailable model. Choose whether the workflow may use the previously authorized original ranking or must stop, based on its answer-quality requirements. That fallback should be explicit and observable, not a silent behavior that makes every request appear reranked. If a backup reranker is configured, test its response mapping and retained evidence too. A second model can change the ranking distribution, so it needs the same evaluation questions rather than only a connectivity check.
For the expense example, inspect where the decisive overseas exception appears in the reranked list and whether it survives the final context budget. A higher relevance position is only useful if the application actually includes the passage needed for the answer. Record this evidence with the answer-level evaluation.
Frequently asked questions
Is a top-ranked passage automatically trusted?
No. Relevance, document access and content trust are separate checks. Keep source identifiers and permissions intact, and evaluate whether the selected passages actually support the final answer.
Should relevance scores from different rerankers be compared directly?
Not without validation. Different models and providers can produce differently scaled scores and ordering behavior. Use the supported documents and final answer quality as the comparison target. Calibrate any threshold on labeled examples for the chosen model, and retest that threshold when the model or candidate construction changes.
Documentation and further reading
Related guide: Prompt Compression with LLMLingua-2: Practical Limits
