Repository navigation
test(rag): labelled set and harness for groundedness prompts vs hallucination_detection - #1762
Charansripadi wants to merge 5 commits into
Conversation
…cination_detection Adds the measurement set and harness for generative-computing#1721, ahead of the arms themselves (which wait on generative-computing#1718 and generative-computing#1719): - labelled_set.jsonl: 128 hand-written items with fictional facts, so neither arm can pass a claim from world knowledge. Covers grounded, cross-document synthesis, genuinely partial, invented, non-factual, mixed, and long (20-80 sentence) responses. Every response carries per-sentence gold labels in the adapter's own label set, split exactly as the adapter input splits it (NLTK Punkt), checked by a unit test. - harness.py: runs arm 1 (GroundednessRequirement as shipped) or arm 2 (find_citations for the output, verdict from flag_hallucinated_content with the rule in the issue) in either allow_partial_support mode, with repeats, per-validation latency and token counts, and resumable JSONL output. compare() reports false fails and misses separately, exact McNemar on disagreements, a paired bootstrap CI, hard failures, label use and stability, and applies a pre-registered decision rule. - README.md: set design, how to run, and a proposed decision rule to be agreed on the issue before any run. Refs generative-computing#1721 Signed-off-by: Sri Charan Sripadi <charansripadi@gmail.com>
jakelorocco
left a comment
There was a problem hiding this comment.
Thanks for working on this issue!
I see Nigel opened this issue and has been working with you on it; so I will defer to him for now. Whatever the outcome, I don't think the code for determining that outcome should live in the main mellea codebase.
I do see the positives for saving harnesses like this; we should figure out where to save things like this so that we can reuse / repurpose them for future experiments.
planetf1
left a comment
There was a problem hiding this comment.
Thanks for this. The set design is careful: invented facts so neither arm can lean on the model's own knowledge, gold labels that line up with the verdicts, and the Punkt split pinned by a test. The unit tests are thorough, and the pure logic (McNemar, the verdict rule, resume) checks out.
Nothing here blocks merge. The inline comments are all about making the measurement trustworthy before the runs on #1721, plus a few nits.
- compare: reject files that mix arms, bindings, modes or models, hold the wrong arm, have missing, unknown or duplicate rows, stop partway through a repeat, or mix adapter revisions. - errors: only ValueError and AdapterSchemaMismatchError from the adapter calls become hard failures, and the row records the exception type. Anything else stops the run and is retried on resume. Adapters are resolved outside the try, since a failed download can be a ValueError. - provenance: record the model and each adapter's name, revision and binding type on every row. The model is part of the resume key. - tokens: store totals only when every generation call reported usage, and record both call counts. - rule: state the decision rule as the margin it is. Rename the rule to `margin` and pin that McNemar cannot fire inside it on this set. - only catch ImportError around the plugin scope, and warn when it happens. - pin that the sentences join back to the response with single spaces. - README: commands that work under zsh. Signed-off-by: Sri Charan Sripadi <charansripadi@gmail.com>
|
@planetf1 thanks for the review. All of it is fixed in 05357a9, as a new commit on top so your earlier comments still line up.
One choice to check. You offered two fixes for resume, and I took the second one. Recorded adapter errors stay done on resume. If they were retried, a flaky failure could keep rerunning until it passed and the 2% gate would never see it. Now that environment failures never get written, what's left in the file is the adapter's own. After an adapter change you write a new file. Say if you'd rather have it the other way. Rule. It's now a plain margin rule ( Testing. The folder's tests pass on this branch, with and without the Where it lives. @jakelorocco asked whether this belongs in the main codebase, and said that's your call. I'm happy to move it wherever you prefer, or to keep only the set and the results here. Let me know and I'll do it in this PR. |
There was a problem hiding this comment.
Re-reviewed 05357a9 on top of e9c8c72. Every point from the earlier review is fixed, and I've resolved those threads. Two new problems change what compare decides and need fixing before any run. They're contained to the harness, so I'm fine with this merging if the fixes follow before the runs. Both are small, and I've tested them (code and tests below).
Verified: 306 passed, 1 skipped (the cpex scoped-hooks test, which passes with --extra hooks); ruff and mypy clean. The new compare checks, token totals, narrowed hard-failure handling and provenance all behave as described, apart from the two problems below.
Fix before any run
-
An errored validation on an invented-claim item is scored as a catch.
When an arm can't produce a verdict (bad JSON, missing sentence records and so on), the row is written with
errored: trueandpassed: None. The README says an errored row "counts as a fail", and on gold-pass items it does: the false-fail check ispassed is not True(harness.py:618,:834), soNonecounts as a false fail. The miss check goes the other way. A miss is an invented-claim item that passed, tested aspassed is True(harness.py:621,:836).Noneis notTrue, so an errored row on an invented-claim item is scored as if the arm had caught the invented claim.For a groundedness check this is backwards. A validation that errored did not detect anything, so it should not improve the miss rate. As it stands, an adapter that returns unusable output on hard invented-claim items looks better than one that answers them and gets them wrong.
It is enough to flip the verdict. The miss pool is 34 items, so the margin allows arm 2 one extra miss. The 2% hard-failure gate allows 7 errored validations of 384, and at temperature 0 an item that errors once will usually error on every repeat, so 2 items can error and still pass the gate. Two misses hidden that way is 5.9 points, which is more than the margin. I checked it with synthetic rows:
- Arm 2 misses 3 invented-claim items that arm 1 catches: miss rate 8.8%,
go: false, which is correct. - Same arm, but 2 of those 3 items error instead of passing: miss rate 2.9%, hard failures 6 of 384 (under the gate),
go: true.
This is not just theoretical. The long items (40, 60 and 80 sentences) are where truncated or malformed adapter output is most likely, and half of them are invented-claim items.
The fix is to treat an errored row as wrong in both pools: a false fail on a gold-pass item, and a miss on an invented-claim item. For a no-worse test that is the conservative reading, and it keeps both arms scored on the same items, so the comparison stays paired:
# arm_metrics miss += int(o.passed is not False) ... pmiss += int(o.passed is not False) # compare.errors(), miss pool return [prim[i.id].passed is not False for i in pool]
by_categoryhas the same gap atharness.py:612(wrong = passed != expected).wrong = o.errored or passed != expectedshould fix it, though I haven't tested that one. README:54 would also need to say what "counts as a fail" means on invented-claim items. These two tests fail on the current code and pass with the change:def test_errored_invented_claim_counts_as_a_miss(): invented = [i.id for i in ITEMS if i.has_invented_claim] m = arm_metrics(ITEMS, _outcomes("adapter", errored_ids=invented[:2])) assert m.misses == 2 def test_errors_cannot_hide_misses(): invented = [i.id for i in ITEMS if i.has_invented_claim] second = _outcomes( "adapter", wrong_ids=invented[2:3], errored_ids=invented[:2], repeats=3 ) report = _compare(_outcomes("prompts", repeats=3), second) assert report["comparison"]["miss"]["arm2_rate"] == pytest.approx(3 / 34) assert report["go"] is False
- Arm 2 misses 3 invented-claim items that arm 1 catches: miss rate 8.8%,
-
A run that stops between repeats is accepted as finished.
run_armruns every item at repeat 0, then every item at repeat 1, and so on (harness.py:443). The unfinished-run check in_check_inputs(harness.py:767) asks one question: do all items have the same set of repeats? That catches a run that stopped partway through a repeat, where some items have{0, 1}and others only{0}. It cannot catch a run that stopped at the end of a repeat, because then every item has the same set. Nothing in the file records how many repeats were asked for, so a--repeats 3run killed after repeat 0 looks exactly like a finished--repeats 1run.That matters for both numbers that use the extra repeats:
- The hard-failure gate sees a third of the evidence. The threshold is still 2%, but it is measured over 128 validations instead of 384. Flaky adapter output, the kind that errors on some repeats and not others, gets one chance to show up instead of three: an item that errors one time in three is caught about 70% of the time over three repeats, and 33% over one. The pre-registered gate assumes three repeats.
- Stability is silently empty. With one repeat, no item can change verdict, so
unstable_itemsreports 0. That reads as "perfectly stable" when nothing was measured.
Separately,
comparenever checks that the two files have the same number of repeats. Arm 1 with three repeats against arm 2 with one is accepted, so the two arms' hard-failure and stability figures sit side by side in the report without measuring the same thing. The primary verdicts are unaffected, since they use repeat 0 only.Recording the requested repeats on each row fixes both. Add
repeats: int | None = NonetoOutcome, set it fromrun_arm'srepeatsargument, and check it in_check_inputs:requested = {o.repeats for o in outcomes} if None in requested: raise ValueError(f"{expected_arm}: rows do not record --repeats; rerun") wanted = max(r for r in requested if r is not None) if set(next(iter(repeats.values()))) != set(range(wanted)): raise ValueError( f"{expected_arm}: run asked for {wanted} repeats but has " f"{len(next(iter(repeats.values())))}; finish the run first" )
Then check across the two files, after the per-file loop:
if max(o.repeats or 0 for o in first) != max(o.repeats or 0 for o in second): raise ValueError("compare runs made with the same --repeats")
Taking the largest requested value means that resuming a run with a higher
--repeats(to extend it) still works. The_outcomestest helper needs to passrepeats=repeatstoOutcome. These two tests cover the new checks:def test_compare_rejects_a_run_stopped_after_a_whole_repeat(): stopped = [replace(o, repeats=3) for o in _outcomes("adapter")] with pytest.raises(ValueError, match="asked for 3 repeats"): _compare(_outcomes("prompts", repeats=3), stopped) def test_compare_rejects_files_with_different_repeats(): with pytest.raises(ValueError, match="same --repeats"): _compare(_outcomes("prompts", repeats=3), _outcomes("adapter"))
With both fixes and all four new tests, the suite gives 311 passed, and ruff and mypy are clean on the harness.
Also before the runs
- The revision check cannot see Granite Switch. The embedded path hard-codes
revision=None(mellea/backends/adapters/_core.py:1028), so every row on that binding hasrevision: nulland the mixed-revision check passes without checking anything. The only provenance left is--model, and with--backend-factorythat is a label nobody checks against the factory. Please record the served model id or the Switch checkpoint in the row, or say in the README that revision pinning coverslocal-fileonly. - A run without the
hooksextra only warns (harness.py:315). The README documents this, but such a file has no adapter provenance and no token counts, andcomparestill accepts it. That should not be possible for the adapter arm of a pre-registered run. Either make the missing extra an error for--arm adapter, or havecomparerefuse an adapter file whose rows have noadapters. - Hard failures by type.
error_typeis stored, but the report only gives a total.ValueErroris broad: it covers bad JSON and schema errors, which is the intent, but also the input checks inmellea/stdlib/components/intrinsic/_util.py(for example the "context forwards no history" check at line 160), so a harness or input problem would count against the adapter. ACounteroferror_typeper arm would let a reader audit the 2% figure.
Smaller
- Errored rows still carry the tokens of the calls that did run (say
find_citationssucceeded andflag_hallucinated_contentraised), andarm_metricsaverages over them, which lowers arm 2's cost. Averaging over rows that didn't error fixes it. - The usage in the
harness.pymodule docstring still sayspython -m; the README usesuv run python -m.
Assessment of the decision rule is on #1721.
…eats - An errored validation now counts as a miss on invented-claim items, not only as a false fail on gold-pass items, in arm_metrics, compare and by_category, so errors cannot hide misses. - Rows record the requested --repeats; compare rejects a run that stopped between repeats, rows without --repeats, and two files with different --repeats. - compare refuses an adapter file without adapter provenance (no hooks extra). - Hard failures are reported by error_type; token means skip errored rows. - README: errored scoring, revision pinning covers local-file only, repeats. - Module docstring usage uses uv run. Signed-off-by: Sri Charan Sripadi <charansripadi@gmail.com>
|
Thanks for the careful re-review and for testing the fixes. Pushed d498981:
Your four tests are in, plus tests for the |
…'s open points compare now gates a third error in strict mode: partial-only items that passed (20 items, same 5-point margin, so one extra allowed). Lenient mode is unchanged, since those items are gold-pass there. README: the decision table names the mode and binding that decide, says repeats 1 and 2 are reported only, states the hard-failure gate in items, and says what a pass or a fail near the line means. Signed-off-by: Sri Charan Sripadi <charansripadi@gmail.com>
|
Added some technical comments but in response to @jakelorocco 's comments about where.. I understand the concern. I myself had some some impromptu harness building to validate real end-end working of adapters which have been useful in finding genuine issues. So I do think having something in the mellea space could be useful, even though it's not 'core'. In the pr we have
I think the tension is with the engine/framework that would equally sit in many projects - so the question becomes whether we could instead re-use someone that's already out there? One possibility - take a framework like Inspect (MIT) the two arms become solvers, and repeats, resume, logs and re-scoring come for free. The decision rule is then a short script over samples_df() with statsmodels' McNemar and scipy's paired bootstrap. That would remove the concern about the generic framework code? but still get us a similar end result? |
planetf1
left a comment
There was a problem hiding this comment.
Added some technical comments but in response to @jakelorocco 's comments about where..
I understand the concern. I myself had some impromptu harness building to validate real end-end working of adapters which have been useful in finding genuine issues. So I do think having something in the mellea space could be useful, even though it's not 'core'.
In the pr we have
- a labelled data set
- an engine
- a mellea runner
I think the tension is with the engine/framework that would equally sit in many projects - so the question becomes whether we could instead re-use something that's already out there? That would leave a light mellea runner plus the data.
For example, with Inspect (MIT) the two arms become solvers, and repeats, resume, logs and re-scoring come for free. The decision rule is then a short script over samples_df() with statsmodels' McNemar and scipy's paired bootstrap. Untested: a short trial would show whether Inspect's model layer gets in the way.
Re-checked ab06c5a: 322 pass with --extra hooks, and the re-review points are all fixed. On resume: keeping recorded adapter errors as done is right, so leave it.
The rule changes on #1721 need some harness work, and there's one new problem.
Before any run
-
Arm 1 turns environment failures into hard failures. You narrowed this for arm 2, and arm 1 needs the same once #1718 is in. #1815 catches every exception per stage and returns it as
error, and the necessity stage re-wraps it asValueError("LLM judgment failed: ...")(rag.py:350). So an OOM on an 80-sentence item gets written as an errored row and never retried. Checking the type ofresult.errormisses it because of the re-wrap. Walking the chain works:_JUDGMENT_ERRORS = (ValueError, AdapterSchemaMismatchError) # JSONDecodeError is a ValueError def _exception_chain(exc: BaseException | None) -> list[BaseException]: seen: set[int] = set() chain: list[BaseException] = [] while exc is not None and id(exc) not in seen: seen.add(id(exc)) chain.append(exc) exc = exc.__cause__ or exc.__context__ return chain # in run_prompts_arm, before returning an errored row if not all(isinstance(e, _JUDGMENT_ERRORS) for e in _exception_chain(error)): raise error
I tested the filter on its own: wrapped bad JSON passes, a wrapped
RuntimeErrororOSErroris re-raised. Worth re-checking against #1815 once it's final. -
Score arm 1's errors as catches in the miss and partial pools. Arm 2's stay misses. Test: arm 1 errored on
stations-long-3to-8, arm 2 passing-4,-6,-8and one short invented item. That'sgo: truetoday and should bego: false. -
Gate lenient misses with the three strict checks, either by giving
comparethe lenient files too or with a small step that combines the two reports. -
Count the hard-failure gate in items: at most 2 items with an errored repeat.
hard_okallows 7 of 384 validations today. -
Report the three-way verdict: each check's excess in items, then
go,inconclusiveorno_go.
Smaller
- Allowances as item counts.
partial_misssits exactly on 0.05; it works, but only becausediffissum(d)/n. - Read the Switch hard-failure check from arm 2's metrics, so it doesn't need an arm-1 run on Switch.
- A README line saying
find_citationserrors count against arm 2. - #1815's input checks also raise
ValueError, so a harness bug would get through the filter. Adding the first line of the message to the breakdown would show it.
Runner API use
I checked the runner against main. The calls are right: the public wrappers with matching arguments, caught errors that match their Raises: sections, and an adapter hook that runs to completion before the wrapper returns, so provenance is always recorded inside the scope. A few tidy-ups:
- Import
plugin_scope,hookandPluginModefrommellea.plugins, which exports them, rather than frommellea.plugins.registry. - Use
async with plugin_scope(...)instead of calling__aenter__and__aexit__by hand. - Type
backendasAdapterMixinrather thanAny. - The adapter hook payload has an
outcomefield. Recording it would let the hard-failure count be checked against what mellea reported.
…e a three-way verdict - An errored arm 1 row is a catch on gold-fail items (production fails closed); arm 2 errors stay wrong everywhere. - Allowances in items: strict false fails +3/74, misses +1/34, partial-only passes +1/20; lenient misses +1/34; lenient false fails reported only. - Hard-failure gate counts items with an errored repeat (at most 2 per binding, over both modes) instead of a share of validations. - compare takes one binding's strict and lenient files and returns go / inconclusive (one item over) / no-go; the margin/ci switch is gone, CI and McNemar are still reported. - README names the base model and adapter revision that decide. Signed-off-by: Sri Charan Sripadi <charansripadi@gmail.com>
|
Thanks @planetf1, all four make sense. Pushed 772cba4.
Two readings to confirm:
334 pass. |
Pull Request
Issue
Part of #1721. This adds the labelled set and the harness; the per-arm results follow once #1718 has merged and the decision rule is agreed on #1721.
Description
Adds
test/stdlib/requirements/groundedness_eval/, the measurement set and harness for deciding #1669.labelled_set.jsonl: 128 hand-written items. Every fact is invented, so neither arm can pass a claim from the model's own knowledge, and the set has no third-party licence terms. It covers grounded (30), cross-document synthesis (20), genuinely partial (20), invented (30), needs-no-citation (10), mixed (10) and long responses of 20 to 80 sentences (8). Each sentence has a gold label in the adapter's own set (faithful,partial,unfaithful,NA). The responses are split exactly as the adapter input splits them (NLTK Punkt), and a unit test checks that for every item.harness.py:runvalidates the set with arm 1 (GroundednessRequirementas shipped) or arm 2 (find_citationsfor the citation output, verdict fromflag_hallucinated_contentwith the rule in the issue), in eitherallow_partial_supportmode, with repeats. Output is resumable JSONL. Each row records latency, the model, and each adapter's name, revision and binding type. Token totals come fromgeneration.usagethrough a scoped plugin, and are stored only when every generation call in the validation reported usage. Only the adapter's own output errors count as hard failures; anything else stops the run.comparerefuses inputs that can't give a meaningful comparison (mixed arms, bindings, modes, models or adapter revisions, missing or duplicate rows, swapped files, an unfinished repeat). It then reports, per arm:It then applies the decision rule, a plain 5-point margin, and reports the exact McNemar p-value and a paired bootstrap CI for information.
Backends other than local HF (e.g. Granite Switch) plug in with
--backend-factory module:function.README.md: set design, how to run, and the proposed decision rule (also posted on design(rag): measure hallucination_detection against the groundedness judgment prompts before swapping #1721 for agreement before any run).Arm 1 is meant to run after #1718, so that an errored result counts as a hard failure rather than a "not grounded" verdict.
Testing
test_groundedness_eval.pyhas 307 unit tests, no model needed. They cover set integrity, the arm-2 verdict rule (including no-record and malformed records), McNemar and the CI, metrics, the decision rule, input checks, error handling, call recording, and run/resume. Therunpaths that call a real backend are not covered by unit tests; they need a GPU and the adapters.Attribution
Adding a new component, requirement, sampling strategy, or tool?
If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.
NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing with this pull request.