Skip to content

fix(core): distinguish unparsed from None in ModelOutputThunk - #1822

Open
rafvasq wants to merge 3 commits into
generative-computing:mainfrom
rafvasq:feat/parsed-repr-flag
Open

rafvasq wants to merge 3 commits into
generative-computing:mainfrom
rafvasq:feat/parsed-repr-flag

Conversation

@rafvasq

@rafvasq rafvasq commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Pull Request

Thanks for sending a pull request! Please make sure you read the contributing guidelines.

Issue

Fixes #1805

Description

ModelOutputThunk.parsed_repr used None for both "not parsed yet" and "parsed to None", so core enforced "a computed thunk has a non-None parsed_repr" and had to exempt GenerativeStub. This tracks the parse separately.

Core (mellea/core/base.py)

  • parsed_repr is now a property. Its setter also sets a private _parsed flag, so every assignment site records the parse unchanged, including assigning None. Passing parsed_repr=None to the constructor still means "unparsed".
  • The invariant is now "a computed thunk has been parsed" (assert self._parsed). The GenerativeStub exemption and its deferred stdlib import are removed, so any component whose _parse returns None now computes normally.
  • __copy__, __deepcopy__ and _copy_from carry the flag.
  • ComputedModelOutputThunk guards _parsed as true-only, so a computed thunk can't be flipped back to unparsed. The guard's error message now names the field.

Chat consumers (ChatFormatter.to_chat_messages, as_generic_chat_history)

  • Both check _parsed explicitly. A never-parsed thunk renders as its raw model output, as before. A thunk that parsed to None renders as an empty assistant message. A thunk that parsed to None used to render as its raw output in both consumers.
  • An unparsed thunk with no value still raises ValueError in as_generic_chat_history; its message now includes is_computed().
  • Docstrings for both consumers and for parsed_repr describe these rules.

Testing

  • Tests added to the respective file if code was changed
  • New code has 100% coverage if code was added
  • Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

Attribution

  • AI coding assistants used

Adding a new component, requirement, sampling strategy, or tool?

If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.

  • Component
  • Requirement
  • Sampling Strategy
  • Tool

NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing.

Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
Assisted-by: Claude Code
@rafvasq
rafvasq requested a review from jakelorocco November 12, 2026 22:10
@github-actions github-actions Bot added the bug Something isn't working label Nov 12, 2026
@rafvasq
rafvasq marked this pull request as ready for review November 12, 2026 22:29
@rafvasq
rafvasq requested a review from a team as a code owner November 12, 2026 22:29

@ajbozarth ajbozarth left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some feedback from Claude: one question about the public API, plus a couple of nits inline.

Comment thread mellea/core/base.py Resolved
Comment thread mellea/core/base.py Resolved
Comment thread mellea/formatters/chat_formatter.py Resolved
Comment thread mellea/core/base.py
assert isinstance(self._call.action, GenerativeStub), (
"enforce constraint that a computed ModelOutputThunk has a non-None parsed_repr"
)
assert self._parsed, "computed thunk skipped _set_parsed_repr()"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing tests the new assert self._parsed. test_component_without_parse_fails_loudly fails earlier, on the NotImplementedError from the base Component._parse, so it never gets as far as this assert. #1805 asked for that test to cover "parse never ran", and that case is still missing. Something like:

def test_computed_thunk_without_parse_trips_invariant(monkeypatch):
    """A thunk that computes without running `_set_parsed_repr` fails loudly."""
    from mellea.stdlib.components import Instruction

    monkeypatch.setattr(ModelOutputThunk, "_set_parsed_repr", lambda self: None)
    with pytest.raises(AssertionError, match="skipped _set_parsed_repr"):
        mfuncs.act(Instruction("describe"), SimpleContext(), _FixedJSONBackend("x"))

I tried this locally and the assert fires as expected.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added test_computed_thunk_without_parse_trips_invariant, which stubs _set_parsed_repr and expects the assert's message.

Comment thread mellea/core/base.py Outdated
self._parsed_repr = value
self._parsed = True

def is_parsed(self) -> bool:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description says the invariant is now "a computed thunk has been parsed". That's only enforced at the end of astream. A thunk that's built with a value already set is computed from the start and never gets parsed: the dummy backend, budget_forcing_alg.py, genstub.py:834/985, and every generate_from_raw until it assigns parsed_repr. ComputedModelOutputThunk(ModelOutputThunk("y")).is_parsed() is False.

The current callers handle this fine. The risk comes later, when someone treats is_parsed() as "this thunk is finished". Could the docstring say that thunks constructed with a value start out unparsed?

Comment thread mellea/core/base.py
self.parsed_repr: S | None = parsed_repr
"""Will be non-`None` once computed."""
self._parsed_repr: S | None = parsed_repr
self._parsed: bool = parsed_repr is not None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Passing parsed_repr=None to the constructor means "unparsed", so a thunk that parsed to None can't be created that way. __copy__ has to fix up _parsed afterwards for this reason. Nothing else re-wraps through the constructor today, but the next site that does will drop the state quietly. A short comment at __copy__, or a keyword-only parsed argument, would make this harder to miss.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the comment at __copy__

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine for now but may need to be re-evaluated. I don't think there's necessary a use case for a parsed to None model output thunk, but there may actually be (like an empty message).

Comment thread mellea/formatters/chat_formatter.py Outdated
) # This is already entailed by c.is_computed(); the line is included here to satisfy the type-checker.

if c.parsed_repr is not None:
if c.is_parsed() and c.parsed_repr is not None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: A non-None parsed_repr already implies is_parsed(), so this check (and the one in stdlib/components/chat.py:464) reduces to the old parsed_repr is not None / is None. Rendering hasn't changed. The two new tests in test_chat.py also pass against the code before this PR, which means they pin the documented rule without guarding against a regression. That's fine if it's intended. Otherwise the is_parsed() calls could go.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is_parsed() is gone as per #1822 (comment)

Comment thread mellea/core/base.py
# may still be unparsed at wrap time). The guard is purely value-based,
# so it needs no "is-sealed" flag: at wrap time the base thunk is already computed
# with a str value, and the wrap itself writes no guarded field.
_INVARIANT_GUARDS: dict[str, Callable[[Any], bool]] = {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, optional: Sampling only ever sets _parsed to True after wrapping. A "_parsed": lambda v: v is True guard would stop a computed thunk from being flipped back to unparsed. The copy paths write _parsed before the class is reassigned, so the guard wouldn't affect them.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_parsed is now in _INVARIANT_GUARDS as true-only with a test covering it.

Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
Assisted-by: IBM Bob
Comment thread mellea/stdlib/components/chat.py Outdated
@@ -458,11 +461,11 @@ def _to_msg(c: Span) -> Message:
return c.parsed_repr
if isinstance(c.parsed_repr, str):
return Message(role="assistant", content=c.parsed_repr)
# Use value if parsed_repr is None
if c.parsed_repr is None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think both here and in the chat formatter we should explicitly check for the _parsed flag. And then we need to make a decision about what to do if the value is actually None.

I'm thinking that in those cases maybe we actually just output an empty message.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both to_chat_messages and as_generic_chat_history now check _parsed explicitly and parsed-to-None renders an empty assistant message.

Comment thread mellea/core/base.py
self.parsed_repr: S | None = parsed_repr
"""Will be non-`None` once computed."""
self._parsed_repr: S | None = parsed_repr
self._parsed: bool = parsed_repr is not None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine for now but may need to be re-evaluated. I don't think there's necessary a use case for a parsed to None model output thunk, but there may actually be (like an empty message).

Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
Assisted-by: Claude Code
@rafvasq
rafvasq requested a review from jakelorocco November 17, 2026 15:11

@ajbozarth ajbozarth left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM, Claude helped me confirm all my Nigel and jakes review has been addressed, I'd wait for final approval from them before merging though

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ModelOutputThunk.parsed_repr uses None for both "not parsed yet" and "parsed to None"

4 participants