Repository navigation
Conversation
Tier 0 was three positive evaluations and two negative, as constants. That is the right bar for a catalogue that starts with this harness and the wrong one for a catalogue that predates it: fifty skills written before these checks existed cannot produce two hundred and fifty prompts before their first green run, so the choice on offer was to run none of the checks at all. The folder, frontmatter and reference checks are worth having on day one, and they were being held hostage by the dataset requirement. --min-positive and --min-negative lower the bar and raise it again as the datasets land. Zero for both stops requiring a dataset; a skill that ships one anyway is still read against whatever the bar is, because "not required of every skill" is not "unread where present". Where that bar may be set is the point, so it is a flag and nothing else. No file in the repo under test, none inside a skill, and no environment variable: the bar a skill is measured against is not the skill author's to set, and a file beside the skill would let one pull request add a thin dataset and lower the number judging it, leaving a reviewer to notice the second change to understand the first. On the command line it lives in the workflow, which is a path a repo can put behind branch protection and code owners. Two tests assert that property rather than trusting it. The applied bar is printed at the top of every structural run, passing or failing, so a report says what it was holding skills to. Signed-off-by: John Lybeck <john.lybeck@amd.com>
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · a2cc6e5
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
DEGRADED REVIEW: single-context pass, no fan-out workers were spawned. Full review of the whole change (prw-base...HEAD, one commit).
No ticket named. The description matches the change's stated scope.
Blocking
- (new) YAML
#comments insideargs: >-folded scalars become literal arguments and break every structural run —.github/workflows/reusable.yml:301-310,.github/workflows/skill-evals.yml:369-377
In a block scalar,#lines are content, not comments. The action passesSKILLSCOPE_ARGSthroughshlex.split(bootstrap/launch.py:104), which does not strip comments. Splitting the parsed workflow value yields tokens'#','`!=','rather','than', … andpython3 -m skillscope structural '#'fails withskillscope: error: unrecognized arguments: #. The structural job in both workflows would fail on every run, including runs where nomin_*input is set. The comment's backticks and quotes risk further shlex errors. Nothing tests the workflow wiring, so the test suite cannot catch this.
Confidence 100 · logic · Fix: delete the#lines from bothargsblocks; put the explanation in a real YAML comment above theargs:key, or in the input'sdescription.
Non-blocking
- (new) The "applied bar is printed at the top of every structural run" behaviour has no test —
skillscope/cli.py:302-325
Both the bar-on and bar-off message branches are untested, as is the CLI-to-config wiring (_configure,_add_dataset_bar_arguments). The new tests callconfig.buildanddatasets.tier0_errorsdirectly; nothing runs--min-positivethrough argparse. The description promises this print as a feature.
Confidence 75 · mechanical · Fix: add a CLI-level test runningstructuralwith--min-positive 0 --min-negative 0and one without flags, asserting on the printed line.
Decisions for the author
- Default
0is accepted for one bar but not both — tradeoff
With only one bar at zero, a skill with no dataset still fails (test_one_bar_still_standing_still_requires_a_dataset); both at zero is the only "off" switch. Coherent and tested, but confirm it is intended.
Positive signals
- The bar is settable only via CLI flag, with no file or environment fallback, and tests assert that property rather than relying on a comment.
_barrejects negative and non-numeric values with a flag-named error, and an existing dataset is still read when the bar is off.stdoutis flushed before the stderr failures, so the bar line appears first in the CI log.
Deployment notes
- Two new optional inputs,
min_positiveandmin_negative, on both reusable workflows, and two new flags onstructural,routingandbehavioral. Defaults are unchanged (3 and 2). MIN_POSITIVE_CASESandMIN_NEGATIVE_CASESare removed fromdatasetsin favour ofconfig.DEFAULT_MIN_*; no other in-repo users found, but external importers would break.
What this covered
Every changed file read completely (7 files, prw-base...HEAD at a2cc6e5), plus the consumers of the changed symbols (bootstrap/launch.py, action.yml, tier0_errors callers), read from the prepared checkout. The one commit carries a Signed-off-by trailer (check_signoff.py exit 0).
Test verdicts: the new TestTheDatasetBarIsTheRepos tests pass. The full suite shows 10 failures and 68 errors, identical in count at prw-base, so environmental. No revert or single-branch mutation was run, so neither test verdict is established. The blocking finding was reproduced with shlex.split and the CLI against the actual workflow value, and re-checked by the driver against the head commit.
Did not run: fan-out workers, the history and prior-changes passes, the CI rollup read, Step 7 design workers; agent-instruction and code-comment passes were inline, not independent. Not reconciled against any discussion here.
Tier 0 is three positive evaluations and two negative, as constants in
datasets.py. That is the right bar for a catalogue that starts with this harness, and the wrong one for a catalogue that predates it. A repo adopting skillscope over fifty skills written before these checks existed cannot produce two hundred and fifty prompts before its first green run, so the choice on offer is to run none of the checks at all. The folder, frontmatter and reference checks are worth having on day one, and the dataset requirement was holding them hostage.--min-positiveand--min-negativeonstructural,routingandbehavioral, with matchingmin_positive/min_negativeinputs on bothreusable.ymlandskill-evals.ymlWhere the bar may be set
It is a flag and nothing else: no file in the repo under test, none inside a skill, and no environment variable, unlike
SKILLSCOPE_SKILLS.The bar a skill is measured against is not the skill author's to set. Were it a file beside the skill, one pull request could add a thin dataset and lower the number judging it, and a reviewer would have to notice the second change to understand the first. On the command line it lives in the workflow, which is a path a repo can put behind branch protection and code owners. An environment variable would be set by whoever administers a self-hosted runner, who is not the person reviewing the workflow diff.
TestTheDatasetBarIsTheReposasserts that property rather than leaving it to be assumed, so a later change that adds a file or an env fallback fails a test instead of quietly widening who can lower the bar.How to test
Boundaries worth poking: one bar at zero and the other above it still requires a dataset (only both at zero turns the requirement off), and a negative or non-integer value is refused in the preflight rather than part-way through a run.