Repository navigation
Conversation
Import the reviewed evaluator dependency closure, protect its resolver through CODEOWNERS, and add offline isolation and fail-closed checks. No ledger, release workflow or qualification waiver is activated. Source: GSA-TTS/agentic-coding-quickstart at 05b4d90. AI-assisted.
Require python -P with a sole absolute PYTHONPATH entry and describe the bootstrap fixtures as temporary directories without Git ancestry coverage. AI-assisted.
Exercise positive qualification followed by surface-change and nonancestor rejection. Specify trusted Python startup flags and the bootstrap-only command. AI-assisted.
mogul
left a comment
There was a problem hiding this comment.
Verdict: REQUEST CHANGES
AI-assisted adversarial review by OpenCode (GPT 6.1), including an independent review sub-agent, requested by @mogul, of #699 at dffe9311d59e6617c15905aaab474f5c4acb6048.
Blocking evaluator correctness
-
P2 — Validate transcript attempts and verify the derived outcome before recording.
qualification/ledger.py:430–474checks canonical serialization, oracle commands and the first environment, but line 463 copiesdoc.outcomedirectly. It does not validate per-attempt verdicts/output hashes or reconcile the outcome with the reproduction rule inqualification/repro.py:38–71. Thus a canonical transcript declaring PASS with a FAIL/BLOCKED attempt can produce a PASS ledger entry;verify_evidencereconstructs through this same function and accepts the same contradiction. This is an internal-consistency defect, not a claim that hashes can prove a real run occurred. Validate the transcript version/fields, each attempt's identity/verdict/hash, and derive or check the final outcome against the reproduction rule. Add negatives for PASS paired with FAIL/BLOCKED, malformed attempts, and divergent binding identities. Evidence is static control-flow inspection, not an executed reproduction. -
P2 — Audit changed existing entries, not just new transcript hashes.
qualification/gate.py:234–249reduces base entries to their transcript hashes, and lines 255–267 select required evidence only for hashes absent at the base. Changing an existing entry's revision, timestamp or verdict while retaining its hash makesrequiredempty; with no transcript blocks, the command exits 0 saying no entries were added. The field-for-field comparison inledger.py:583–584never runs. Compare complete base/candidate entries and require evidence for mutations, or enforce append-only immutable entries. Add a Git-backed base-relative regression that changesrevorrecorded_atwithout changing the hash and omits evidence. This is a successful-audit correctness issue even though ledger authenticity remains a human control.
Blocking engineering requirements
-
P2 — Include the durable decision record for this evaluator boundary and policy. Imported module documentation references ADR-0044m, but the exact-head tree has no such record or scoped replacement. Introducing the evaluator and directly loaded host resolver establishes new module boundaries and qualification rules; repository AGENTS.md and Coding Practices §11.1 require an ADR. Include a scoped introduction record explaining base-versus-candidate authority, trusted Python/Git/environment preconditions, evidence consistency, waivers, resolver dependency and unactivated integrations. Do not import historical claims of installed workflows or release gates as present behavior.
-
P2 — Resolve new size-limit exceptions explicitly.
py-acq/src/acq_py/hostexec.pyhas 856 lines andqualification/ledger.py616, both newly added to this base. The PR supplies no explicit exception justification required by Coding Practices §13.3 / CONTRIBUTING.md. Split along justified boundaries or document the specific exceptions and why preserving the complete reviewed dependency closure is necessary. Prior review elsewhere and byte-identical import do not themselves supply the exception rationale.
Verification gap and advisory
The six new bootstrap tests are not invoked by hosted CI or the current one-command pre-commit check. Logs contain no qualification unittest execution, and the hook inventory has no qualification test hook. The local six-pass claim remains author-reported. Wire evaluator/regression tests into the standard verification path and retain observed output when addressing the findings; green Bash checks cannot establish correctness of these new Python rules. No live Tier-3 qualification is requested for this unactivated foundation.
qualification/BOOTSTRAP.md also lacks the Markdown frontmatter required by the universal documentation contract. Its trusted-startup and future-integration scope distinctions are otherwise useful.
What was verified
All ten reported check runs pass. Hosted CI checks out merge 0f520c81a0a68756006656e4b20b1226c48740bd; its parents are the exact reviewed head and base 36722391e2df15bf9cf5c8f0da70a69a6febc0b9. Pre-commit reports Gitleaks, hygiene, Markdown, ShellCheck and executable-bit checks passing; Linux Bats ran 1,279 cases with 7 skips, and the three Bash 3.2 shards total 1,279 cases with 5 skips, no observed failures. MSB startup probe tests report 27 passed. npm audit reports 3 low and 2 moderate findings, no high/critical; dependencies are unchanged.
Independently compared Git blobs: all nine imported evaluator/resolver files exactly match source 05b4d90b67a4cd745cf023de97d42f1c92b8fc2f. Parsed all ten added Python files as Python 3.11 syntax in memory without executing them. Added diff lines have no trailing whitespace; secret-marker inspection found a transcript-directory placeholder, not credential material. CODEOWNERS covers qualification and the directly loaded resolver.
No confirmed credential disclosure or candidate-execution vulnerability established under the documented startup preconditions. No candidate code/tests executed locally, dependencies installed, workspace files modified, live backends exercised, or release/waiver/ledger activated. Required branch-protection enforcement and reviewer-team membership were not verified. This AI-assisted review uses the requesting account; it is not independent human review or merge/production authorization.
Summary
Introduce the reviewed Tier-3 evaluator separately so the Python/cross-platform
integration can use base-branch rules. The nine-file dependency closure is copied
unchanged from
05b4d90b67a4cd745cf023de97d42f1c92b8fc2f; CODEOWNERS covers bothqualification/and its directly loaded resolver. Adds adoption notes, offlinetests, and classification in the existing Bash relevance inventory.
No workflow, ledger, release, or waiver is activated.
Related Issues
Prerequisite for #697. No issue closure requested.
Type of Change
Checklist
Test Results
python -B -m unittest discover -s qualification/tests -v: 6 passed.Covers missing/malformed ledger, expired waiver, empty evidence, candidate
package shadowing, and real Git-backed qualification followed by surface-change
and nonancestor rejection. The synthetic oracle is never executed.
suite and mode checks passed; documentation/test follow-up hooks passed.
dffe9311d59e6617c15905aaab474f5c4acb6048; nine evaluator files verifiedbyte-identical to the reviewed source revision.
Additional Notes
Security: evaluator startup requires trusted Python/Git,
-B -s -P, a soleabsolute evaluator PYTHONPATH, candidate CWD, and clean environment. The later
workflow must implement those conditions and use trusted temporary report output.
Ledger authenticity and waiver approval remain human review controls.
Rollback: revert this introduction before dependent workflow activation.
This PR alone changes no runtime or release behavior. A missing ledger fails.
AI-assisted implementation, source review, isolated validation, and independent review.
Co-authored-by: OpenCode