Skip to content

test(rag): labelled set and harness for groundedness prompts vs hallucination_detection - #1762

Open
Charansripadi wants to merge 5 commits into
generative-computing:mainfrom
Charansripadi:test/1721-groundedness-eval-set
Open

Charansripadi wants to merge 5 commits into
generative-computing:mainfrom
Charansripadi:test/1721-groundedness-eval-set

Conversation

@Charansripadi

@Charansripadi Charansripadi commented Oct 24, 2026 •

Copy link
Copy Markdown

Pull Request

Issue

Part of #1721. This adds the labelled set and the harness; the per-arm results follow once #1718 has merged and the decision rule is agreed on #1721.

Description

Adds test/stdlib/requirements/groundedness_eval/, the measurement set and harness for deciding #1669.

  • labelled_set.jsonl: 128 hand-written items. Every fact is invented, so neither arm can pass a claim from the model's own knowledge, and the set has no third-party licence terms. It covers grounded (30), cross-document synthesis (20), genuinely partial (20), invented (30), needs-no-citation (10), mixed (10) and long responses of 20 to 80 sentences (8). Each sentence has a gold label in the adapter's own set (faithful, partial, unfaithful, NA). The responses are split exactly as the adapter input splits them (NLTK Punkt), and a unit test checks that for every item.
  • harness.py:
    • run validates the set with arm 1 (GroundednessRequirement as shipped) or arm 2 (find_citations for the citation output, verdict from flag_hallucinated_content with the rule in the issue), in either allow_partial_support mode, with repeats. Output is resumable JSONL. Each row records latency, the model, and each adapter's name, revision and binding type. Token totals come from generation.usage through a scoped plugin, and are stored only when every generation call in the validation reported usage. Only the adapter's own output errors count as hard failures; anything else stops the run.

    • compare refuses inputs that can't give a meaningful comparison (mixed arms, bindings, modes, models or adapter revisions, missing or duplicate rows, swapped files, an unfinished repeat). It then reports, per arm:

      • false fails and misses, separately;
      • partial-only misses in strict mode;
      • label use (gold vs adapter label);
      • hard failures;
      • stability across repeats.

      It then applies the decision rule, a plain 5-point margin, and reports the exact McNemar p-value and a paired bootstrap CI for information.

    • Backends other than local HF (e.g. Granite Switch) plug in with --backend-factory module:function.

  • README.md: set design, how to run, and the proposed decision rule (also posted on design(rag): measure hallucination_detection against the groundedness judgment prompts before swapping #1721 for agreement before any run).

Arm 1 is meant to run after #1718, so that an errored result counts as a hard failure rather than a "not grounded" verdict.

Testing

  • Tests added to the respective file if code was changed
  • New code has 100% coverage if code was added
  • Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

test_groundedness_eval.py has 307 unit tests, no model needed. They cover set integrity, the arm-2 verdict rule (including no-record and malformed records), McNemar and the CI, metrics, the decision rule, input checks, error handling, call recording, and run/resume. The run paths that call a real backend are not covered by unit tests; they need a GPU and the adapters.

Attribution

  • AI coding assistants used

Adding a new component, requirement, sampling strategy, or tool?

If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.

  • Component
  • Requirement
  • Sampling Strategy
  • Tool

NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing with this pull request.

…cination_detection

Adds the measurement set and harness for generative-computing#1721, ahead of the arms
themselves (which wait on generative-computing#1718 and generative-computing#1719):

- labelled_set.jsonl: 128 hand-written items with fictional facts, so
  neither arm can pass a claim from world knowledge. Covers grounded,
  cross-document synthesis, genuinely partial, invented, non-factual,
  mixed, and long (20-80 sentence) responses. Every response carries
  per-sentence gold labels in the adapter's own label set, split exactly
  as the adapter input splits it (NLTK Punkt), checked by a unit test.
- harness.py: runs arm 1 (GroundednessRequirement as shipped) or arm 2
  (find_citations for the output, verdict from flag_hallucinated_content
  with the rule in the issue) in either allow_partial_support mode, with
  repeats, per-validation latency and token counts, and resumable JSONL
  output. compare() reports false fails and misses separately, exact
  McNemar on disagreements, a paired bootstrap CI, hard failures, label
  use and stability, and applies a pre-registered decision rule.
- README.md: set design, how to run, and a proposed decision rule to be
  agreed on the issue before any run.

Refs generative-computing#1721

Signed-off-by: Sri Charan Sripadi <charansripadi@gmail.com>

@jakelorocco jakelorocco left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this issue!

I see Nigel opened this issue and has been working with you on it; so I will defer to him for now. Whatever the outcome, I don't think the code for determining that outcome should live in the main mellea codebase.

I do see the positives for saving harnesses like this; we should figure out where to save things like this so that we can reuse / repurpose them for future experiments.

@planetf1 planetf1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. The set design is careful: invented facts so neither arm can lean on the model's own knowledge, gold labels that line up with the verdicts, and the Punkt split pinned by a test. The unit tests are thorough, and the pure logic (McNemar, the verdict rule, resume) checks out.

Nothing here blocks merge. The inline comments are all about making the measurement trustworthy before the runs on #1721, plus a few nits.

- compare: reject files that mix arms, bindings, modes or models, hold
  the wrong arm, have missing, unknown or duplicate rows, stop partway
  through a repeat, or mix adapter revisions.
- errors: only ValueError and AdapterSchemaMismatchError from the adapter
  calls become hard failures, and the row records the exception type.
  Anything else stops the run and is retried on resume. Adapters are
  resolved outside the try, since a failed download can be a ValueError.
- provenance: record the model and each adapter's name, revision and
  binding type on every row. The model is part of the resume key.
- tokens: store totals only when every generation call reported usage,
  and record both call counts.
- rule: state the decision rule as the margin it is. Rename the rule to
  `margin` and pin that McNemar cannot fire inside it on this set.
- only catch ImportError around the plugin scope, and warn when it happens.
- pin that the sentences join back to the response with single spaces.
- README: commands that work under zsh.

Signed-off-by: Sri Charan Sripadi <charansripadi@gmail.com>
@Charansripadi

Copy link
Copy Markdown
Author

@planetf1 thanks for the review. All of it is fixed in 05357a9, as a new commit on top so your earlier comments still line up.

  • compare inputs. It now refuses a file that mixes arms, bindings, modes or models, holds the wrong arm, has missing, unknown or duplicate rows, or stops partway through a repeat. It also refuses two files with different models or a mix of adapter revisions. I ran your three bad inputs through the CLI and each one now errors. It runs once per binding, as the README says.
  • Errors. Only ValueError and AdapterSchemaMismatchError from the adapter calls become hard failures, and each row records the exception type. Anything else stops the run and writes no row, so resume retries that item. I also resolve both adapters outside the try, because resolve_adapter can raise ValueError and a failed download could too. That's a setup problem and shouldn't count against the adapter.
  • Provenance. Each row has the model and each adapter's name, revision and binding type, and the model is in the resume key.
  • Tokens. The plugin counts generation calls and calls with usage, and stores None for the totals unless they match. The README now says this depends on the call path.
  • Nits. ImportError only, the join assert, and README commands that work under zsh. The ImportError branch also warns, so a missing hooks extra doesn't look like a backend with no usage.

One choice to check. You offered two fixes for resume, and I took the second one. Recorded adapter errors stay done on resume. If they were retried, a flaky failure could keep rerunning until it passed and the 2% gate would never see it. Now that environment failures never get written, what's left in the file is the adapter's own. After an adapter change you write a new file. Say if you'd rather have it the other way.

Rule. It's now a plain margin rule (--rule margin), and the McNemar p is reported for information only. I checked your claim and it holds: within the margin the smallest two-sided p is 0.125 (lenient, 0 vs 4), so one-sided is 0.0625. A test pins it for n = 34, 74 and 94. I've put the updated proposal on #1721.

Testing. The folder's tests pass on this branch, with and without the hooks extra, and on current main with the folder copied over, since the branch is 19 commits behind. ruff and mypy are clean on the folder. Nothing has run against a real model yet, so the hooks are only tested by invoking them directly with test payloads, not from a real backend. The real runs still wait on #1718.

Where it lives. @jakelorocco asked whether this belongs in the main codebase, and said that's your call. I'm happy to move it wherever you prefer, or to keep only the set and the results here. Let me know and I'll do it in this PR.

@planetf1 planetf1 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 05357a9 on top of e9c8c72. Every point from the earlier review is fixed, and I've resolved those threads. Two new problems change what compare decides and need fixing before any run. They're contained to the harness, so I'm fine with this merging if the fixes follow before the runs. Both are small, and I've tested them (code and tests below).

Verified: 306 passed, 1 skipped (the cpex scoped-hooks test, which passes with --extra hooks); ruff and mypy clean. The new compare checks, token totals, narrowed hard-failure handling and provenance all behave as described, apart from the two problems below.

Fix before any run

  1. An errored validation on an invented-claim item is scored as a catch.

    When an arm can't produce a verdict (bad JSON, missing sentence records and so on), the row is written with errored: true and passed: None. The README says an errored row "counts as a fail", and on gold-pass items it does: the false-fail check is passed is not True (harness.py:618, :834), so None counts as a false fail. The miss check goes the other way. A miss is an invented-claim item that passed, tested as passed is True (harness.py:621, :836). None is not True, so an errored row on an invented-claim item is scored as if the arm had caught the invented claim.

    For a groundedness check this is backwards. A validation that errored did not detect anything, so it should not improve the miss rate. As it stands, an adapter that returns unusable output on hard invented-claim items looks better than one that answers them and gets them wrong.

    It is enough to flip the verdict. The miss pool is 34 items, so the margin allows arm 2 one extra miss. The 2% hard-failure gate allows 7 errored validations of 384, and at temperature 0 an item that errors once will usually error on every repeat, so 2 items can error and still pass the gate. Two misses hidden that way is 5.9 points, which is more than the margin. I checked it with synthetic rows:

    • Arm 2 misses 3 invented-claim items that arm 1 catches: miss rate 8.8%, go: false, which is correct.
    • Same arm, but 2 of those 3 items error instead of passing: miss rate 2.9%, hard failures 6 of 384 (under the gate), go: true.

    This is not just theoretical. The long items (40, 60 and 80 sentences) are where truncated or malformed adapter output is most likely, and half of them are invented-claim items.

    The fix is to treat an errored row as wrong in both pools: a false fail on a gold-pass item, and a miss on an invented-claim item. For a no-worse test that is the conservative reading, and it keeps both arms scored on the same items, so the comparison stays paired:

    # arm_metrics
    miss += int(o.passed is not False)
    ...
    pmiss += int(o.passed is not False)
    
    # compare.errors(), miss pool
    return [prim[i.id].passed is not False for i in pool]

    by_category has the same gap at harness.py:612 (wrong = passed != expected). wrong = o.errored or passed != expected should fix it, though I haven't tested that one. README:54 would also need to say what "counts as a fail" means on invented-claim items. These two tests fail on the current code and pass with the change:

    def test_errored_invented_claim_counts_as_a_miss():
        invented = [i.id for i in ITEMS if i.has_invented_claim]
        m = arm_metrics(ITEMS, _outcomes("adapter", errored_ids=invented[:2]))
        assert m.misses == 2
    
    
    def test_errors_cannot_hide_misses():
        invented = [i.id for i in ITEMS if i.has_invented_claim]
        second = _outcomes(
            "adapter", wrong_ids=invented[2:3], errored_ids=invented[:2], repeats=3
        )
        report = _compare(_outcomes("prompts", repeats=3), second)
        assert report["comparison"]["miss"]["arm2_rate"] == pytest.approx(3 / 34)
        assert report["go"] is False
  2. A run that stops between repeats is accepted as finished.

    run_arm runs every item at repeat 0, then every item at repeat 1, and so on (harness.py:443). The unfinished-run check in _check_inputs (harness.py:767) asks one question: do all items have the same set of repeats? That catches a run that stopped partway through a repeat, where some items have {0, 1} and others only {0}. It cannot catch a run that stopped at the end of a repeat, because then every item has the same set. Nothing in the file records how many repeats were asked for, so a --repeats 3 run killed after repeat 0 looks exactly like a finished --repeats 1 run.

    That matters for both numbers that use the extra repeats:

    • The hard-failure gate sees a third of the evidence. The threshold is still 2%, but it is measured over 128 validations instead of 384. Flaky adapter output, the kind that errors on some repeats and not others, gets one chance to show up instead of three: an item that errors one time in three is caught about 70% of the time over three repeats, and 33% over one. The pre-registered gate assumes three repeats.
    • Stability is silently empty. With one repeat, no item can change verdict, so unstable_items reports 0. That reads as "perfectly stable" when nothing was measured.

    Separately, compare never checks that the two files have the same number of repeats. Arm 1 with three repeats against arm 2 with one is accepted, so the two arms' hard-failure and stability figures sit side by side in the report without measuring the same thing. The primary verdicts are unaffected, since they use repeat 0 only.

    Recording the requested repeats on each row fixes both. Add repeats: int | None = None to Outcome, set it from run_arm's repeats argument, and check it in _check_inputs:

    requested = {o.repeats for o in outcomes}
    if None in requested:
        raise ValueError(f"{expected_arm}: rows do not record --repeats; rerun")
    wanted = max(r for r in requested if r is not None)
    if set(next(iter(repeats.values()))) != set(range(wanted)):
        raise ValueError(
            f"{expected_arm}: run asked for {wanted} repeats but has "
            f"{len(next(iter(repeats.values())))}; finish the run first"
        )

    Then check across the two files, after the per-file loop:

    if max(o.repeats or 0 for o in first) != max(o.repeats or 0 for o in second):
        raise ValueError("compare runs made with the same --repeats")

    Taking the largest requested value means that resuming a run with a higher --repeats (to extend it) still works. The _outcomes test helper needs to pass repeats=repeats to Outcome. These two tests cover the new checks:

    def test_compare_rejects_a_run_stopped_after_a_whole_repeat():
        stopped = [replace(o, repeats=3) for o in _outcomes("adapter")]
        with pytest.raises(ValueError, match="asked for 3 repeats"):
            _compare(_outcomes("prompts", repeats=3), stopped)
    
    
    def test_compare_rejects_files_with_different_repeats():
        with pytest.raises(ValueError, match="same --repeats"):
            _compare(_outcomes("prompts", repeats=3), _outcomes("adapter"))

    With both fixes and all four new tests, the suite gives 311 passed, and ruff and mypy are clean on the harness.

Also before the runs

  1. The revision check cannot see Granite Switch. The embedded path hard-codes revision=None (mellea/backends/adapters/_core.py:1028), so every row on that binding has revision: null and the mixed-revision check passes without checking anything. The only provenance left is --model, and with --backend-factory that is a label nobody checks against the factory. Please record the served model id or the Switch checkpoint in the row, or say in the README that revision pinning covers local-file only.
  2. A run without the hooks extra only warns (harness.py:315). The README documents this, but such a file has no adapter provenance and no token counts, and compare still accepts it. That should not be possible for the adapter arm of a pre-registered run. Either make the missing extra an error for --arm adapter, or have compare refuse an adapter file whose rows have no adapters.
  3. Hard failures by type. error_type is stored, but the report only gives a total. ValueError is broad: it covers bad JSON and schema errors, which is the intent, but also the input checks in mellea/stdlib/components/intrinsic/_util.py (for example the "context forwards no history" check at line 160), so a harness or input problem would count against the adapter. A Counter of error_type per arm would let a reader audit the 2% figure.

Smaller

  • Errored rows still carry the tokens of the calls that did run (say find_citations succeeded and flag_hallucinated_content raised), and arm_metrics averages over them, which lowers arm 2's cost. Averaging over rows that didn't error fixes it.
  • The usage in the harness.py module docstring still says python -m; the README uses uv run python -m.

Assessment of the decision rule is on #1721.

…eats

- An errored validation now counts as a miss on invented-claim items, not
  only as a false fail on gold-pass items, in arm_metrics, compare and
  by_category, so errors cannot hide misses.
- Rows record the requested --repeats; compare rejects a run that stopped
  between repeats, rows without --repeats, and two files with different
  --repeats.
- compare refuses an adapter file without adapter provenance (no hooks extra).
- Hard failures are reported by error_type; token means skip errored rows.
- README: errored scoring, revision pinning covers local-file only, repeats.
- Module docstring usage uses uv run.

Signed-off-by: Sri Charan Sripadi <charansripadi@gmail.com>
@Charansripadi

Copy link
Copy Markdown
Author

Thanks for the careful re-review and for testing the fixes. Pushed d498981:

  • Errored rows now count as wrong in both pools: a miss on invented-claim items as well as a false fail on gold-pass items, in arm_metrics, compare and by_category (wrong = o.errored or passed != expected). README:54 now says what that means.
  • Outcome.repeats records the requested --repeats. compare rejects a run that stopped between repeats, rows without it, and two files with different --repeats. Taking the max still lets a resumed run extend.
  • Revision pinning: the README now says it covers local-file only, and that --model should name the served model or Switch checkpoint with --backend-factory.
  • compare refuses an adapter file whose rows have no adapter provenance, so a run without the hooks extra can't be used for the adapter arm.
  • The report breaks hard failures down by error_type per arm.
  • Token means skip errored rows; the docstring usage uses uv run.

Your four tests are in, plus tests for the by_category change, the provenance check, the error_type breakdown and the token means: 315 passed, 1 skipped (cpex), ruff and mypy clean.

…'s open points

compare now gates a third error in strict mode: partial-only items that
passed (20 items, same 5-point margin, so one extra allowed). Lenient
mode is unchanged, since those items are gold-pass there.

README: the decision table names the mode and binding that decide, says
repeats 1 and 2 are reported only, states the hard-failure gate in
items, and says what a pass or a fail near the line means.

Signed-off-by: Sri Charan Sripadi <charansripadi@gmail.com>
@planetf1

Copy link
Copy Markdown
Contributor

Added some technical comments but in response to @jakelorocco 's comments about where..

I understand the concern. I myself had some some impromptu harness building to validate real end-end working of adapters which have been useful in finding genuine issues. So I do think having something in the mellea space could be useful, even though it's not 'core'.

In the pr we have

  • a labelled data set
  • an engine
  • a mellea runner

I think the tension is with the engine/framework that would equally sit in many projects - so the question becomes whether we could instead re-use someone that's already out there?

One possibility - take a framework like Inspect (MIT) the two arms become solvers, and repeats, resume, logs and re-scoring come for free. The decision rule is then a short script over samples_df() with statsmodels' McNemar and scipy's paired bootstrap.

That would remove the concern about the generic framework code? but still get us a similar end result?

@planetf1 planetf1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some technical comments but in response to @jakelorocco 's comments about where..

I understand the concern. I myself had some impromptu harness building to validate real end-end working of adapters which have been useful in finding genuine issues. So I do think having something in the mellea space could be useful, even though it's not 'core'.

In the pr we have

  • a labelled data set
  • an engine
  • a mellea runner

I think the tension is with the engine/framework that would equally sit in many projects - so the question becomes whether we could instead re-use something that's already out there? That would leave a light mellea runner plus the data.

For example, with Inspect (MIT) the two arms become solvers, and repeats, resume, logs and re-scoring come for free. The decision rule is then a short script over samples_df() with statsmodels' McNemar and scipy's paired bootstrap. Untested: a short trial would show whether Inspect's model layer gets in the way.


Re-checked ab06c5a: 322 pass with --extra hooks, and the re-review points are all fixed. On resume: keeping recorded adapter errors as done is right, so leave it.

The rule changes on #1721 need some harness work, and there's one new problem.

Before any run

  1. Arm 1 turns environment failures into hard failures. You narrowed this for arm 2, and arm 1 needs the same once #1718 is in. #1815 catches every exception per stage and returns it as error, and the necessity stage re-wraps it as ValueError("LLM judgment failed: ...") (rag.py:350). So an OOM on an 80-sentence item gets written as an errored row and never retried. Checking the type of result.error misses it because of the re-wrap. Walking the chain works:

    _JUDGMENT_ERRORS = (ValueError, AdapterSchemaMismatchError)  # JSONDecodeError is a ValueError
    
    
    def _exception_chain(exc: BaseException | None) -> list[BaseException]:
        seen: set[int] = set()
        chain: list[BaseException] = []
        while exc is not None and id(exc) not in seen:
            seen.add(id(exc))
            chain.append(exc)
            exc = exc.__cause__ or exc.__context__
        return chain
    
    
    # in run_prompts_arm, before returning an errored row
    if not all(isinstance(e, _JUDGMENT_ERRORS) for e in _exception_chain(error)):
        raise error

    I tested the filter on its own: wrapped bad JSON passes, a wrapped RuntimeError or OSError is re-raised. Worth re-checking against #1815 once it's final.

  2. Score arm 1's errors as catches in the miss and partial pools. Arm 2's stay misses. Test: arm 1 errored on stations-long-3 to -8, arm 2 passing -4, -6, -8 and one short invented item. That's go: true today and should be go: false.

  3. Gate lenient misses with the three strict checks, either by giving compare the lenient files too or with a small step that combines the two reports.

  4. Count the hard-failure gate in items: at most 2 items with an errored repeat. hard_ok allows 7 of 384 validations today.

  5. Report the three-way verdict: each check's excess in items, then go, inconclusive or no_go.

Smaller

  • Allowances as item counts. partial_miss sits exactly on 0.05; it works, but only because diff is sum(d)/n.
  • Read the Switch hard-failure check from arm 2's metrics, so it doesn't need an arm-1 run on Switch.
  • A README line saying find_citations errors count against arm 2.
  • #1815's input checks also raise ValueError, so a harness bug would get through the filter. Adding the first line of the message to the breakdown would show it.

Runner API use

I checked the runner against main. The calls are right: the public wrappers with matching arguments, caught errors that match their Raises: sections, and an adapter hook that runs to completion before the wrapper returns, so provenance is always recorded inside the scope. A few tidy-ups:

  • Import plugin_scope, hook and PluginMode from mellea.plugins, which exports them, rather than from mellea.plugins.registry.
  • Use async with plugin_scope(...) instead of calling __aenter__ and __aexit__ by hand.
  • Type backend as AdapterMixin rather than Any.
  • The adapter hook payload has an outcome field. Recording it would let the hard-failure count be checked against what mellea reported.

…e a three-way verdict

- An errored arm 1 row is a catch on gold-fail items (production fails closed);
  arm 2 errors stay wrong everywhere.
- Allowances in items: strict false fails +3/74, misses +1/34, partial-only
  passes +1/20; lenient misses +1/34; lenient false fails reported only.
- Hard-failure gate counts items with an errored repeat (at most 2 per binding,
  over both modes) instead of a share of validations.
- compare takes one binding's strict and lenient files and returns
  go / inconclusive (one item over) / no-go; the margin/ci switch is gone,
  CI and McNemar are still reported.
- README names the base model and adapter revision that decide.

Signed-off-by: Sri Charan Sripadi <charansripadi@gmail.com>
@Charansripadi

Copy link
Copy Markdown
Author

Thanks @planetf1, all four make sense. Pushed 772cba4.

  1. Errored arm-1 rows now count as catches in the miss and partial pools, and arm-2 errors stay misses. The cap is max_new_tokens=500 in rag.py. Your case is now a test (test_arm1_errors_do_not_make_room_for_arm2_misses): arm 1 erroring on the long failing items plus four arm-2 misses gives no-go.
  2. Lenient misses are gated at +1 of 34, and lenient false fails are report-only. test_lenient_gate_catches_unfaithful_labelled_partial covers the adapter that labels unfaithful as partial.
  3. I kept the margin and the set, and the verdict is now go / inconclusive / no-go. compare takes one binding's strict and lenient files and prints the verdict. CI and McNemar are still reported but decide nothing.
  4. Named in the README: base ibm-granite/granite-4.0-micro, adapters from granitelib-rag-r1.0 at 2f0b2c79 (the catalog pin; both publish 4.0-micro there). Hard failures now count items with an errored repeat, at most 2 per binding.

Two readings to confirm:

  • I read "one item over" as one item over in total, so two checks each one over (say strict partial and lenient miss) is no-go. If you meant the worst single check, it's a one-line change.
  • The 2-item hard-failure ceiling counts distinct items across the strict and lenient runs on a binding, not per mode.

334 pass.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants