Skip to content

Handle empty evaluation reranking candidates - #1632

Open
Afloat16 wants to merge 1 commit into
FlagOpen:masterfrom
Afloat16:fix/empty-reranking-candidates-20261112
Open

Afloat16 wants to merge 1 commit into
FlagOpen:masterfrom
Afloat16:fix/empty-reranking-candidates-20261112

Conversation

@Afloat16

Copy link
Copy Markdown

Problem

EvalReranker always calls its scoring model, including when no query-document pairs remain. An empty retrieval run, zero reranking depth, or identical-ID filtering that removes all candidates reaches AbsReranker.compute_score([]) and raises an IndexError before evaluation can continue.

Changes

If the final pair list is empty, return an empty score mapping for each input query. Mixed batches that still contain candidates continue through the normal scoring path.

Validation

  • Eight regression cases use a locally saved tiny BERT classifier and vocabulary through the actual BaseReranker and evaluation wrapper.
  • Cover no queries, one/multiple empty candidate lists, all candidates removed by identical-ID filtering, zero reranking depth, and mixed empty/nonempty query groups.
  • Exact final baseline: 6 failed, 2 passed. After the patch: 8 passed.
  • New cases plus existing Trainer/import tests: 16 passed, 1 skipped. The skipped check requires Transformers 5.x.
  • PyTorch 2.15.0+cpu, Transformers 4.57.1, FAISS CPU 1.16.0 and pytrec-eval-terrier 0.5.11; no remote checkpoint download or GPU run. Optional evaluation imports are skipped if their dependencies are absent; they were present for validation.
  • This patch only handles an empty candidate set at the evaluation-wrapper boundary. Input candidate mutation and filtering depth remain separate behavior.
  • Ruff and git diff --check pass.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant