Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions .github/workflows/selftest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,22 +24,27 @@ permissions:

jobs:
imports:
# One rule, not a linter. `F401` finds imports nothing uses, which is how
# Two rules, not a linter. `F401` finds imports nothing uses, which is how
# dead code announces itself: two rounds of review found functions whose
# last caller had been deleted, and in both cases the orphaned imports were
# sitting there saying so. Twelve of them had accumulated by the time
# anybody looked.
#
# `F841` is the same signal one scope down, and was added after a review
# found a local left behind by a deleted report section: the list was
# still being built on every run and nothing ever read it. `F401` cannot
# see that, and the whole suite passing cannot either.
#
# Deliberately narrow. Adopting a full style configuration is a change to
# every file and a decision about house style; this is the one check that
# pays for itself immediately and reformats nothing.
name: Unused imports
name: Unused imports and locals
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/ruff-action@v3
with:
args: check --select F401 skillscope tests tools
args: check --select F401,F841 skillscope tests tools

unit:
name: Unit tests (${{ matrix.os }}, Python ${{ matrix.python }})
Expand Down
18 changes: 15 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,14 @@ SPDX-License-Identifier: MIT

![Skillscope](assets/banner.png)

A test harness for AI agent skills. Install the harness, or point a workflow at your repo. Skillscope enables three main types of tests:
A test harness for AI agent skills. Install the harness, or point a workflow at your repo. Skillscope enables four main types of tests:

* **Structural**: Checks whether the skill is well structured (contains the expected files, a proper hierarchy, etc.).
* **Routing**: Checks whether the skill triggers when it should and stays quiet when it shouldn't. You may also point to more than one skill here to explore how skills interfere with each other's routing.
* **Behavioral**: Runs the agent end to end and uses an LLM judge to confirm the skill behaves correctly.
* **Staleness**: Runs the behavioral cases a second time with the skill withheld, and compares. Tells you whether the cases still need the skill, or whether the model has caught up and the suite is passing on its own.

All three read one dataset per skill, `<skill>/evals/evals.json`. How to write one is in [docs/authoring-evals.md](docs/authoring-evals.md).
All four read one dataset per skill, `<skill>/evals/evals.json`. How to write one is in [docs/authoring-evals.md](docs/authoring-evals.md).

> [!IMPORTANT]
> **Early Days**: Flags, workflow inputs, and report formats may evolve quickly as this repo takes shape.
Expand All @@ -37,6 +38,7 @@ skillscope structural # no agent, no tokens
skillscope structural --external # the same, plus checking external URLs
skillscope behavioral --skill my-skill # needs an authenticated `claude` CLI
skillscope routing --routing-room my-skill,its-neighbour # also needs a key in the environment
skillscope staleness --skill my-skill # the same key: its baseline has to be an empty room
```

`routing` needs `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN` set, and refuses
Expand All @@ -47,9 +49,18 @@ describe a room nobody asked for. `behavioral` installs one skill and grades
what the agent produced, so a stray skill is at worst noise there, and a plain
`claude login` is fine.

`staleness` is in `routing`'s camp, not `behavioral`'s, and asks the same
credential question on Claude Code. Its baseline half has to be a room with
**no** skills in it, so the runner's own are not noise there — they are the
measurement. Either set one of those variables, or use `--engine
claude-code`, whose guest has no user-level skills at all. It is Claude Code
only: unlike `routing`, which scales its refusal per harness, it refuses
any `--harness` other than `claude-code`, because there is no staleness leg
for another one yet.

## Run it from your CI

Simply add an `evals.json` file to your skill and a workflow that points to our reusable workflow. This will triger all graders to run directly from your repo.
Simply add an `evals.json` file to your skill and a workflow that points to our reusable workflow. This will trigger the three graders the workflow runs -- `structural`, `routing` and `behavioral` -- directly from your repo. `staleness` is not one of them: it runs both halves of every case and is meant to be asked for deliberately rather than on every push.

<img src="assets/reusable_workflow.png" alt="Skillscope" width="720">

Expand All @@ -72,6 +83,7 @@ You may also choose to customize or even disable some tests if you prefer it. Se
| `skillscope structural` | Skill folders, datasets, and markdown references. |
| `skillscope routing` | Which skill fires, with several installed together. |
| `skillscope behavioral` | What a skill does once it has fired. |
| `skillscope staleness` | Whether those cases still need the skill, and what the agent spends without it. |
| `skillscope select` | The CI plan for a change, as JSON. |
| `skillscope list-skills` | The skills that have a dataset, as JSON. |
| `skillscope template` | The dataset template a new skill starts from. |
Expand Down
8 changes: 5 additions & 3 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ description: >-

inputs:
command:
description: 'The skillscope subcommand: structural, routing, behavioral, select, list-skills, template.'
description: 'The skillscope subcommand: structural, routing, behavioral, staleness, select, list-skills, template.'
required: true
args:
description: "Further arguments for the subcommand, shell-quoted."
Expand Down Expand Up @@ -47,8 +47,10 @@ inputs:
default: "true"
install-claude:
description: >-
Install Node and the claude CLI. Needed by `routing` and `behavioral`,
pointless for `structural` and `select`.
Install Node and the claude CLI. Needed by `routing`, `behavioral` and
`staleness` -- staleness drives the behavioral leg for both of its
halves, so it needs the CLI exactly as much. Pointless for `structural`
and `select`.
required: false
default: "false"
node-version:
Expand Down
7 changes: 7 additions & 0 deletions docs/authoring-evals.md
Original file line number Diff line number Diff line change
Expand Up @@ -224,8 +224,15 @@ skillscope structural # your folder, prose, and da
skillscope behavioral --skill <your-skill> # your skill, end to end
skillscope routing --routing-room <your-skill>,<a-neighbour>
skillscope routing --only <case-id> --keep-logs logs # one case, keeping the transcript
skillscope staleness --skill <your-skill> # do your cases still need the skill?
```

`staleness` is the one aimed at this file's subject. It runs your behavioral
cases twice, once with the skill installed and once without, and reports which
of them passed either way. Those are cases that are not evidence about the
skill — either it has stopped mattering, or the case never tested it. It costs
two runs, so it belongs on a schedule rather than in the loop.

`structural` is instant and needs no agent. Everything else needs the `claude`
CLI authenticated, plus whatever your own cases need. Flags, what each check
asserts, the routing bar, and how to read a report are in
Expand Down
6 changes: 5 additions & 1 deletion docs/harnesses.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,4 +145,8 @@ Claude Code, and a harness other than it cannot yet use them:
each agent is defined once. Until it does, a harness other than Claude Code has no
sandboxed leg;
* **`.claude/skills`** as the path routing treats as a skills inspection, rather
than a harness's own `skills_dir`.
than a harness's own `skills_dir`;
* **`staleness`**: its baseline is only a measurement if the room is empty, and
the check that establishes that reads Claude Code's credentials and no other
harness's, so it refuses any `--harness` other than `claude-code` rather
than scaling the refusal per harness.
151 changes: 149 additions & 2 deletions docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ CLI flag, and every workflow input. For writing a skill's dataset, see
* [References](#references)
* [Routing, and the bar it is held to](#routing-and-the-bar-it-is-held-to)
* [Hardware a skill needs](#hardware-a-skill-needs)
* [Is the skill still earning its place?](#is-the-skill-still-earning-its-place)
* [In CI: one job](#in-ci-one-job)
* [In CI: the full pipeline](#in-ci-the-full-pipeline)
* [Versions](#versions)
Expand Down Expand Up @@ -249,6 +250,146 @@ Legs with a scoped environment run as a separate job, because a job's
credentials are fixed before its matrix expands. A repo that declares no scoped
environment gets one matrix, labels and all.

## Is the skill still earning its place?

A behavioral run says a skill's cases pass. It cannot say they pass *because
of* the skill, and those are different claims: an expectation any competent
agent meets is a measurement of the model with the skill's name on it.

`skillscope staleness` runs the same cases twice, once with the skill
installed and once with nothing installed, and reports which ones moved.

```bash
skillscope staleness --skill my-skill
```

On the default engine this needs `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN`
in the environment and refuses to start without one, the same way `routing`
does and for a stronger version of the same reason: the key is what lets the
run redirect the CLI away from the runner's own config dir, and the baseline
half has to be a room with no skills in it. Without it, every skill installed
on the machine joins the baseline, and a case one of them solves there is
reported as a case the skill under test was not needed for. A plain `claude
login` is enough for `behavioral` and is not enough here. `--engine
claude-code` needs no key for this, since its guest has no user-level skills
at all.

The report leads with what each half cost, because that is the cheaper
question and often the more useful answer. An agent holding the skill goes
straight to the work; an agent without it explores. On the first real run of
this command the graded half finished in 2m23s and the baseline was still
going half an hour later, with every case passing on both sides -- a skill
plainly earning its keep, which a verdict table alone reports as `no_signal`
throughout.

| Half | Wall time | Model calls | Tokens | Cost |
| --- | --- | --- | --- | --- |
| with the skill | 2m23s | 37 | 184220 | n/a |
| nothing installed | 28m00s+ | 214 | 1044901 | n/a |

A `+` marks a half the `--timeout` stopped: that row is a floor, not a
measurement. Cost reads `n/a` unless the provider supplies pricing, which a
gateway generally does not and which inspect's own registry supplies for no
model at all; `SKILLSCOPE_MODEL_PRICING` is the way in. **A missing cost is a
missing number and not a zero** -- dollars arrive with a run's final event,
which a half the bound stopped never reaches, so the half that spent *more*
is the likelier one to show nothing. Tokens and wall clock are the reliable
columns.

Then the verdicts. Three, and an error state:

| Verdict | Means |
| --- | --- |
| `earns_it` | passed with the skill, failed without it -- the case discriminates |
| `harmful` | failed with the skill, passed without it -- the skill steered the agent wrong |
| `no_signal` | came out the same both ways |
| `ungraded` | nothing could be concluded: every attempt errored on one side or both, a grader never answered, a case was missing from a side, or the baseline was not the empty room it claimed to be |

Three rather than more. Earlier versions split out whether the skill fired,
whether a case claimed anything positive at all, and whether a shared failure
was a broken case or a hard one. All three are still reported, as counts and
notes beside the verdict rather than as verdicts of their own. None of them
was a different answer to the question this command asks; each was one of
these answers with a caveat attached.

A verdict says what changed, not why. So the graded side also records whether
the agent *reached for* the skill, using the same detector the routing leg
uses: installing a skill does not make an agent use it, and "installed and the
outcome differed" is a weaker claim than "the skill fired and the outcome
differed". A case that moved without the skill firing is still `earns_it` --
the outcome did differ -- but it is counted and flagged above the case table,
so the headline share can be discounted by a reader who cares. Guard rails,
`unexpected_behavior`, are kept out of the verdict entirely, since a good one
holds whether or not the skill is installed; they are counted on their own,
and in both directions. A guard broken *with* the skill installed is the
serious finding. A guard the **baseline** broke and the installed side held
is the opposite one, and the report says so: the skill stopped the agent
doing the forbidden thing. Both sides can reach the same answer with only one
of them wrecking something on the way, and no verdict in the table can
express that. The first real run of this command found one unprompted, an
unaided agent attempting a `sudo` write on a case the skill kept read-only.

`no_signal` is the verdict worth acting on, and it is three findings wearing
one name. The pass counts and the advice column say which you are holding:

* **both sides passed** -- the agent already does this unaided, so the case is
not evidence about the skill. Either the skill has stopped mattering or the
case never tested it, and the second is more common, which makes this as
much a check on the dataset as on the skill.
* **both sides failed** -- a broken case or a hard one. Nothing here has been
shown to work well enough to then ask whether the skill is why, so this
points at the case rather than the skill.
* **a side disagreed with itself** -- only possible above one attempt. The
baseline managed it at least once, which is not the same finding as a case
both sides managed every time. Re-run before reading anything into it.

A case that claims nothing positive at all is `no_signal` too, since neither
side can meet a claim that was never made. The advice column says so rather
than calling it broken: its guard rails were still checked, and the case is
working exactly as written.

### Spending less on a run

Both halves of a comparison are agent passes, and the baseline is the slower
one: an agent holding the skill goes straight to the work, and an agent
without it explores. Three flags decide what a run costs.

| Flag | Default | Effect |
| --- | --- | --- |
| `--attempts N` | 1 | Run every case N times on each side. Costs N times as much. A side counts as having passed when **any** of its attempts did, so a higher N is a harder question rather than a steadier reading of the same one: `earns_it` at three attempts needs the baseline to have failed all three times, and a baseline that passes once in three takes the case to `no_signal`. Verdicts compare with verdicts from runs at the same N, not across different ones. What you gain is the counts printed beside each verdict: `3/3` against `0/3` is a result, `3/3` against `1/3` is a baseline that managed it once, and `2/3` against `1/3` is two sides that both disagreed with themselves. The advice column calls that last one out rather than letting it read as a dead case. |
| `--max-cases N` | every case | Grade only the first N cases of each selected skill, after `--only` and `--extended` have chosen them. Both sides run the same N, and the report says the run did not cover the whole suite. |
| `--concurrency N` | inspect's default | How many cases may be in flight at once. Where the wall clock goes on the baseline half. Raise it as far as the engine's sandboxes and the model's rate limits allow. |

Both halves write which side they were, whose skill they were about, and where
that skill was on disk into their inspect log metadata (`skillscope_baseline`,
`skillscope_skill`, `skillscope_skill_dir`).

`skillscope_skill_dir` is the load-bearing one. It is what lets the baseline be
judged empty: the check recognises a tool call that reached the skill by
comparing against the skill's absolute path, so a reader working that path out
for itself would be working out its own rather than the run's, and would report
a baseline the run had voided as a clean one. A log written without the key is
read as an unknown room rather than an empty one.

Budget more wall clock than a `behavioral` run of the same cases, and more
than double it. `--timeout` defaults to twice what the single-pass commands
use for that reason. Both passes share one `--timeout`, and the baseline is
reliably the slower half: the agent holding the skill goes straight to the
work, while the one without it explores. On a two-case skill the graded pass
took 2m23s and the baseline had not finished its second case half an hour
later. A bound that expires partway reports the unreached cases as `ungraded`,
after the graded half has already been paid for.

No verdict it reaches fails a run, for the reason the `claude-code` leg does
not gate either: the agent is nondeterministic on both sides, so a single pair
is a place to look rather than a verdict. An expired `--timeout` or a refused
run still exits non-zero, because those are statements about the run rather
than about a skill. [`tools/benchmark_engines.py`](../tools/benchmark_engines.py)
is the shape of the honest version of this comparison — it measures agreement
against how much one engine already disagrees with itself — and a staleness
number deserves the same scepticism. It also costs twice what `behavioral`
costs, which is why no workflow runs it for you.

## Which engine grades a run

Two choices decide what runs the cases, and they are separate: `--engine` is
Expand Down Expand Up @@ -304,8 +445,14 @@ routing leg at all unless one of the harness's credential variables is set
(see [harnesses.md](harnesses.md): `ANTHROPIC_API_KEY` or
`ANTHROPIC_AUTH_TOKEN` for Claude Code, `COPILOT_GITHUB_TOKEN`, `GH_TOKEN` or
`GITHUB_TOKEN` for Copilot CLI), which is what lets it redirect the CLI's
config dir away from the runner's own. Refusing beats being
quietly wrong about every case.
config dir away from the runner's own.

`staleness` is stricter, and not on the same terms. It has no leg for any
harness but Claude Code yet, so it refuses any `--harness` other than
`claude-code` whatever credentials are present, rather than scaling the
refusal per harness the way routing does. On Claude Code it then asks routing's question as well, because
there the empty room is not one input to the measurement, it is the
measurement. Refusing beats being quietly wrong about every case.

Both legs stop at the decision, by different routes. `claude-code`'s calls
cross inspect's bridge, so the one that reveals the decision is declined
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ authors = [{ name = "Advanced Micro Devices, Inc." }]
#
# It was an extra while `legacy` existed: that engine was standard library only,
# so a repo that had not migrated installed nothing new. `legacy` is gone, and
# keeping the extra would ship a tool whose two graded commands both fail on a
# keeping the extra would ship a tool whose graded commands all fail on a
# plain `pip install` -- which is what CI found, by running the suite without
# extras and getting `ModuleNotFoundError: inspect_ai` from the default engine.
#
Expand Down
Loading
Loading