Skip to content

Preserve retrieval candidates when evaluating rerankers - #1634

Open
Afloat16 wants to merge 1 commit into
FlagOpen:masterfrom
Afloat16:fix/preserve-reranking-candidates-20261112
Open

Afloat16 wants to merge 1 commit into
FlagOpen:masterfrom
Afloat16:fix/preserve-reranking-candidates-20261112

Conversation

@Afloat16

Copy link
Copy Markdown

Problem

EvalReranker replaces each query's candidate mapping in the supplied search_results dictionary with a truncated top-k mapping. Reusing the same retrieval results for another reranker or depth silently loses candidates: a top-1 run followed by top-3 can only rerank the one candidate left by the first call.

Changes

Build the top-k candidate mappings locally without modifying the caller's retrieval results. Ranking, truncation, identical-ID filtering, and score computation keep their existing order and behavior.

Validation

  • Eight native regression cases use a tiny BERT sequence classifier and local BERT vocabulary saved to disk, then run the actual BaseReranker inference path through EvalReranker.
  • Check the original result values and nested mappings, top-1/top-2/top-4 selections with identical-ID filtering on/off, and comparisons at increasing/decreasing reranking depths.
  • Exact final baseline: 7 failed, 1 passed. After the patch: 8 passed.
  • New cases plus existing Trainer/import checks: 16 passed, 1 skipped. The skipped check requires Transformers 5.x.
  • Tested with PyTorch 2.15.0+cpu, Transformers 4.57.1, FAISS CPU 1.16.0 and pytrec-eval-terrier 0.5.11. Tests skip cleanly when optional evaluation dependencies are absent; they were present for these runs. No remote model download or GPU execution was used.
  • Ruff and git diff --check pass.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant