Skip to content

feat(qualification): introduce base-branch evaluator for review - #699

Open
wz-gsa wants to merge 3 commits into
mainfrom
feat/tier3-evaluator-bootstrap
Open

wz-gsa wants to merge 3 commits into
mainfrom
feat/tier3-evaluator-bootstrap

Conversation

@wz-gsa

@wz-gsa wz-gsa commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Introduce the reviewed Tier-3 evaluator separately so the Python/cross-platform
integration can use base-branch rules. The nine-file dependency closure is copied
unchanged from 05b4d90b67a4cd745cf023de97d42f1c92b8fc2f; CODEOWNERS covers both
qualification/ and its directly loaded resolver. Adds adoption notes, offline
tests, and classification in the existing Bash relevance inventory.

No workflow, ledger, release, or waiver is activated.

Related Issues

Prerequisite for #697. No issue closure requested.

Type of Change

  • Bug fix
  • New feature
  • Documentation update
  • Breaking change

Checklist

  • Read CONTRIBUTING.md
  • Conventional Commits
  • Tested in an isolated Linux VM
  • Documentation updated
  • Secret scanning passed

Test Results

  • python -B -m unittest discover -s qualification/tests -v: 6 passed.
    Covers missing/malformed ledger, expired waiver, empty evidence, candidate
    package shadowing, and real Git-backed qualification followed by surface-change
    and nonancestor rejection. The synthetic oracle is never executed.
  • Managed commit hooks: applicable secret/hygiene/Markdown/ShellCheck/full Bash
    suite and mode checks passed; documentation/test follow-up hooks passed.
  • Independent fixed-head QA/security: no remaining blockers at
    dffe9311d59e6617c15905aaab474f5c4acb6048; nine evaluator files verified
    byte-identical to the reviewed source revision.

Additional Notes

Security: evaluator startup requires trusted Python/Git, -B -s -P, a sole
absolute evaluator PYTHONPATH, candidate CWD, and clean environment. The later
workflow must implement those conditions and use trusted temporary report output.
Ledger authenticity and waiver approval remain human review controls.

Rollback: revert this introduction before dependent workflow activation.
This PR alone changes no runtime or release behavior. A missing ledger fails.

AI-assisted implementation, source review, isolated validation, and independent review.

Co-authored-by: OpenCode

Import the reviewed evaluator dependency closure, protect its resolver through CODEOWNERS, and add offline isolation and fail-closed checks. No ledger, release workflow or qualification waiver is activated.

Source: GSA-TTS/agentic-coding-quickstart at 05b4d90. AI-assisted.
Require python -P with a sole absolute PYTHONPATH entry and describe the bootstrap fixtures as temporary directories without Git ancestry coverage.

AI-assisted.
Exercise positive qualification followed by surface-change and nonancestor rejection. Specify trusted Python startup flags and the bootstrap-only command.

AI-assisted.
@wz-gsa
wz-gsa requested a review from a team as a code owner November 13, 2026 18:25

@mogul mogul left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES

AI-assisted adversarial review by OpenCode (GPT 6.1), including an independent review sub-agent, requested by @mogul, of #699 at dffe9311d59e6617c15905aaab474f5c4acb6048.

Blocking evaluator correctness

  1. P2 — Validate transcript attempts and verify the derived outcome before recording. qualification/ledger.py:430–474 checks canonical serialization, oracle commands and the first environment, but line 463 copies doc.outcome directly. It does not validate per-attempt verdicts/output hashes or reconcile the outcome with the reproduction rule in qualification/repro.py:38–71. Thus a canonical transcript declaring PASS with a FAIL/BLOCKED attempt can produce a PASS ledger entry; verify_evidence reconstructs through this same function and accepts the same contradiction. This is an internal-consistency defect, not a claim that hashes can prove a real run occurred. Validate the transcript version/fields, each attempt's identity/verdict/hash, and derive or check the final outcome against the reproduction rule. Add negatives for PASS paired with FAIL/BLOCKED, malformed attempts, and divergent binding identities. Evidence is static control-flow inspection, not an executed reproduction.

  2. P2 — Audit changed existing entries, not just new transcript hashes. qualification/gate.py:234–249 reduces base entries to their transcript hashes, and lines 255–267 select required evidence only for hashes absent at the base. Changing an existing entry's revision, timestamp or verdict while retaining its hash makes required empty; with no transcript blocks, the command exits 0 saying no entries were added. The field-for-field comparison in ledger.py:583–584 never runs. Compare complete base/candidate entries and require evidence for mutations, or enforce append-only immutable entries. Add a Git-backed base-relative regression that changes rev or recorded_at without changing the hash and omits evidence. This is a successful-audit correctness issue even though ledger authenticity remains a human control.

Blocking engineering requirements

  1. P2 — Include the durable decision record for this evaluator boundary and policy. Imported module documentation references ADR-0044m, but the exact-head tree has no such record or scoped replacement. Introducing the evaluator and directly loaded host resolver establishes new module boundaries and qualification rules; repository AGENTS.md and Coding Practices §11.1 require an ADR. Include a scoped introduction record explaining base-versus-candidate authority, trusted Python/Git/environment preconditions, evidence consistency, waivers, resolver dependency and unactivated integrations. Do not import historical claims of installed workflows or release gates as present behavior.

  2. P2 — Resolve new size-limit exceptions explicitly. py-acq/src/acq_py/hostexec.py has 856 lines and qualification/ledger.py 616, both newly added to this base. The PR supplies no explicit exception justification required by Coding Practices §13.3 / CONTRIBUTING.md. Split along justified boundaries or document the specific exceptions and why preserving the complete reviewed dependency closure is necessary. Prior review elsewhere and byte-identical import do not themselves supply the exception rationale.

Verification gap and advisory

The six new bootstrap tests are not invoked by hosted CI or the current one-command pre-commit check. Logs contain no qualification unittest execution, and the hook inventory has no qualification test hook. The local six-pass claim remains author-reported. Wire evaluator/regression tests into the standard verification path and retain observed output when addressing the findings; green Bash checks cannot establish correctness of these new Python rules. No live Tier-3 qualification is requested for this unactivated foundation.

qualification/BOOTSTRAP.md also lacks the Markdown frontmatter required by the universal documentation contract. Its trusted-startup and future-integration scope distinctions are otherwise useful.

What was verified

All ten reported check runs pass. Hosted CI checks out merge 0f520c81a0a68756006656e4b20b1226c48740bd; its parents are the exact reviewed head and base 36722391e2df15bf9cf5c8f0da70a69a6febc0b9. Pre-commit reports Gitleaks, hygiene, Markdown, ShellCheck and executable-bit checks passing; Linux Bats ran 1,279 cases with 7 skips, and the three Bash 3.2 shards total 1,279 cases with 5 skips, no observed failures. MSB startup probe tests report 27 passed. npm audit reports 3 low and 2 moderate findings, no high/critical; dependencies are unchanged.

Independently compared Git blobs: all nine imported evaluator/resolver files exactly match source 05b4d90b67a4cd745cf023de97d42f1c92b8fc2f. Parsed all ten added Python files as Python 3.11 syntax in memory without executing them. Added diff lines have no trailing whitespace; secret-marker inspection found a transcript-directory placeholder, not credential material. CODEOWNERS covers qualification and the directly loaded resolver.

No confirmed credential disclosure or candidate-execution vulnerability established under the documented startup preconditions. No candidate code/tests executed locally, dependencies installed, workspace files modified, live backends exercised, or release/waiver/ledger activated. Required branch-protection enforcement and reviewer-team membership were not verified. This AI-assisted review uses the requesting account; it is not independent human review or merge/production authorization.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants