Repository navigation
Conversation
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com> Assisted-by: Claude Code
ajbozarth
left a comment
There was a problem hiding this comment.
Some feedback from Claude: one question about the public API, plus a couple of nits inline.
| assert isinstance(self._call.action, GenerativeStub), ( | ||
| "enforce constraint that a computed ModelOutputThunk has a non-None parsed_repr" | ||
| ) | ||
| assert self._parsed, "computed thunk skipped _set_parsed_repr()" |
There was a problem hiding this comment.
Nothing tests the new assert self._parsed. test_component_without_parse_fails_loudly fails earlier, on the NotImplementedError from the base Component._parse, so it never gets as far as this assert. #1805 asked for that test to cover "parse never ran", and that case is still missing. Something like:
def test_computed_thunk_without_parse_trips_invariant(monkeypatch):
"""A thunk that computes without running `_set_parsed_repr` fails loudly."""
from mellea.stdlib.components import Instruction
monkeypatch.setattr(ModelOutputThunk, "_set_parsed_repr", lambda self: None)
with pytest.raises(AssertionError, match="skipped _set_parsed_repr"):
mfuncs.act(Instruction("describe"), SimpleContext(), _FixedJSONBackend("x"))I tried this locally and the assert fires as expected.
There was a problem hiding this comment.
Added test_computed_thunk_without_parse_trips_invariant, which stubs _set_parsed_repr and expects the assert's message.
| self._parsed_repr = value | ||
| self._parsed = True | ||
|
|
||
| def is_parsed(self) -> bool: |
There was a problem hiding this comment.
The description says the invariant is now "a computed thunk has been parsed". That's only enforced at the end of astream. A thunk that's built with a value already set is computed from the start and never gets parsed: the dummy backend, budget_forcing_alg.py, genstub.py:834/985, and every generate_from_raw until it assigns parsed_repr. ComputedModelOutputThunk(ModelOutputThunk("y")).is_parsed() is False.
The current callers handle this fine. The risk comes later, when someone treats is_parsed() as "this thunk is finished". Could the docstring say that thunks constructed with a value start out unparsed?
| self.parsed_repr: S | None = parsed_repr | ||
| """Will be non-`None` once computed.""" | ||
| self._parsed_repr: S | None = parsed_repr | ||
| self._parsed: bool = parsed_repr is not None |
There was a problem hiding this comment.
nit: Passing parsed_repr=None to the constructor means "unparsed", so a thunk that parsed to None can't be created that way. __copy__ has to fix up _parsed afterwards for this reason. Nothing else re-wraps through the constructor today, but the next site that does will drop the state quietly. A short comment at __copy__, or a keyword-only parsed argument, would make this harder to miss.
There was a problem hiding this comment.
Added the comment at __copy__
There was a problem hiding this comment.
I think this is fine for now but may need to be re-evaluated. I don't think there's necessary a use case for a parsed to None model output thunk, but there may actually be (like an empty message).
| ) # This is already entailed by c.is_computed(); the line is included here to satisfy the type-checker. | ||
|
|
||
| if c.parsed_repr is not None: | ||
| if c.is_parsed() and c.parsed_repr is not None: |
There was a problem hiding this comment.
nit: A non-None parsed_repr already implies is_parsed(), so this check (and the one in stdlib/components/chat.py:464) reduces to the old parsed_repr is not None / is None. Rendering hasn't changed. The two new tests in test_chat.py also pass against the code before this PR, which means they pin the documented rule without guarding against a regression. That's fine if it's intended. Otherwise the is_parsed() calls could go.
There was a problem hiding this comment.
is_parsed() is gone as per #1822 (comment)
| # may still be unparsed at wrap time). The guard is purely value-based, | ||
| # so it needs no "is-sealed" flag: at wrap time the base thunk is already computed | ||
| # with a str value, and the wrap itself writes no guarded field. | ||
| _INVARIANT_GUARDS: dict[str, Callable[[Any], bool]] = { |
There was a problem hiding this comment.
nit, optional: Sampling only ever sets _parsed to True after wrapping. A "_parsed": lambda v: v is True guard would stop a computed thunk from being flipped back to unparsed. The copy paths write _parsed before the class is reassigned, so the guard wouldn't affect them.
There was a problem hiding this comment.
_parsed is now in _INVARIANT_GUARDS as true-only with a test covering it.
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com> Assisted-by: IBM Bob
| @@ -458,11 +461,11 @@ def _to_msg(c: Span) -> Message: | |||
| return c.parsed_repr | |||
| if isinstance(c.parsed_repr, str): | |||
| return Message(role="assistant", content=c.parsed_repr) | |||
| # Use value if parsed_repr is None | |||
| if c.parsed_repr is None: | |||
There was a problem hiding this comment.
I think both here and in the chat formatter we should explicitly check for the _parsed flag. And then we need to make a decision about what to do if the value is actually None.
I'm thinking that in those cases maybe we actually just output an empty message.
There was a problem hiding this comment.
Both to_chat_messages and as_generic_chat_history now check _parsed explicitly and parsed-to-None renders an empty assistant message.
| self.parsed_repr: S | None = parsed_repr | ||
| """Will be non-`None` once computed.""" | ||
| self._parsed_repr: S | None = parsed_repr | ||
| self._parsed: bool = parsed_repr is not None |
There was a problem hiding this comment.
I think this is fine for now but may need to be re-evaluated. I don't think there's necessary a use case for a parsed to None model output thunk, but there may actually be (like an empty message).
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com> Assisted-by: Claude Code
ajbozarth
left a comment
There was a problem hiding this comment.
This LGTM, Claude helped me confirm all my Nigel and jakes review has been addressed, I'd wait for final approval from them before merging though
Pull Request
Thanks for sending a pull request! Please make sure you read the contributing guidelines.
Issue
Fixes #1805
Description
ModelOutputThunk.parsed_reprusedNonefor both "not parsed yet" and "parsed toNone", so core enforced "a computed thunk has a non-Noneparsed_repr" and had to exemptGenerativeStub. This tracks the parse separately.Core (
mellea/core/base.py)parsed_repris now a property. Its setter also sets a private_parsedflag, so every assignment site records the parse unchanged, including assigningNone. Passingparsed_repr=Noneto the constructor still means "unparsed".assert self._parsed). TheGenerativeStubexemption and its deferred stdlib import are removed, so any component whose_parsereturnsNonenow computes normally.__copy__,__deepcopy__and_copy_fromcarry the flag.ComputedModelOutputThunkguards_parsedas true-only, so a computed thunk can't be flipped back to unparsed. The guard's error message now names the field.Chat consumers (
ChatFormatter.to_chat_messages,as_generic_chat_history)_parsedexplicitly. A never-parsed thunk renders as its raw model output, as before. A thunk that parsed toNonerenders as an empty assistant message. A thunk that parsed toNoneused to render as its raw output in both consumers.ValueErrorinas_generic_chat_history; its message now includesis_computed().parsed_reprdescribe these rules.Testing
Attribution
Adding a new component, requirement, sampling strategy, or tool?
If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.
NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing.