Skip to content

Document AIME onboarding and retire the historical review blocker - #61

Open
crepuscular-drizzle wants to merge 1 commit into
mainfrom
docs/aime-onboarding
Open

crepuscular-drizzle wants to merge 1 commit into
mainfrom
docs/aime-onboarding

Conversation

@crepuscular-drizzle

Copy link
Copy Markdown
Collaborator

Summary

  • Add an AIME adapter README covering the pinned source, runtime dependencies, task splits, scoring, and validation commands.
  • Update dataset onboarding to point readers to public adapter documentation. Retire the AIME-specific review blocker and preserve the original review and diagnostic notes in Git history.
  • Record the 60-task official evaluation pool and the optional 37/23 hash split. The earlier diagnostic was unmeasured, and no side-by-side benchmark comparison with Harbor is on record.

Validation

  • python3 tools/check_dataset_blockers.py and python3 tools/datasets_doc.py --check passed. All 11 pinned historical run records remain intact.
  • 56 focused registry, evidence-retention, and AIME scorer tests passed.
  • 12 AIME QA contract tests passed with synthetic data. Synthetic split checks passed, and sandboxed oracle and no-op controls scored 1 and 0.
  • The dataset lint skipped the unprovisioned AIME release. No model evaluation was run.

Copilot AI balanced review requested due to automatic review settings November 14, 2026 11:31

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants