Repository navigation
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1982 +/- ##
==========================================
- Coverage 87.67% 87.67% -0.01%
==========================================
Files 358 358
Lines 29039 29052 +13
==========================================
+ Hits 25461 25472 +11
- Misses 3578 3580 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Tomkess
force-pushed
the
fix/eval-failed-create-not-scored
branch
2 times, most recently
from
November 21, 2026 22:01
3e47766 to
5dc01a9
Compare
When an answer carries no visualization part, the SSE client falls back to
the arguments of the last create_adhoc_visualization call, so a chart the
agent built but could not persist (data source unavailable) still gets
scored. That fallback also scored definitions the backend rejected as
unbuildable against the workspace's model, e.g. a pre-aggregated metric
broken down by an attribute its fact cannot reach. Such a run recorded
executed=false and still passed 3/3 when a fixture carried the same
unbuildable shape.
The fallback now looks at the last call only, as before, but scores no
chart when that call's result is a definition rejection. The markers are
gen-ai's own compatibility rejections ("is not comparable to the
dimensionality", "has no witness"). It never falls back to an earlier
call, which may be a lookup the agent did not mean to show, and a last
call that is itself a lookup (show_to_user: false, gen-ai's flag for a
chart built only to read a value) scores no chart either. Environment
errors keep the old behaviour; the existing fallback test's tool result
was malformed JSON, so it now carries a well-formed error and a second
test pins the environment case explicitly.
A turn whose only chart was rejected now counts as a turn without a
chart, so the agentic loop continues with a simulated reply instead of
ending as if a chart was built. A loop-level test pins that.
The README and the ExpectationOptions docstring say rejected definitions
are not scored.
jira: trivial
risk: high
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tomkess
force-pushed
the
fix/eval-failed-create-not-scored
branch
from
November 21, 2026 22:08
5dc01a9 to
11c74fb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When an assistant answer carries no visualization part, the SSE client falls back to the arguments of the last
create_adhoc_visualizationcall. That way a chart the agent built but could not persist (for example, data source unavailable) still gets scored.That fallback also scored definitions the backend rejected as unbuildable against the workspace's model, such as a pre-aggregated metric broken down by an attribute its fact cannot reach:
In Services-AI-Evaluations, a Mastercard run recorded
executed: falsewith that error and still passed 3/3, because the fixture carried the same unbuildable shape.Now:
show_to_user: false(gen-ai's flag for a chart built only to read a value) is a lookup the user never saw, so it scores no chart either._COMPATIBILITY_REJECTIONS("is not comparable to the dimensionality","has no witness", increate_adhoc_visualization_tool.py).ExpectationOptionsdocstring say rejected definitions are not scored.Test plan
New tests, each failing on the base
sse_client.py:callId.show_to_user: falsegives no chart.test_agentic_visualization.py): a turn whose only chart was rejected continues to a simulated reply, and the run is scored on the next turn's chart.Environment errors still score the arguments.
test_parse_sse_lines_falls_back_to_adhoc_viz_when_multipart_viz_is_nullcarried a malformed tool result (unescaped quotes), so it never exercised the error. It now carries a well-formed{"status":"error","message":"Data source does not exist"}. A second test,..._still_scores_a_definition_an_environment_error_stopped, pins the case explicitly. Both fail if the check treats everystatus: errorresult as a rejection. These two pass on the base, since they guard unchanged behaviour.The rejected-then-accepted-retry case isn't a new test: the base already scores the last call, so it can't tell the two apart.
Commands:
make -C packages/gooddata-eval format-fix lint-fix type-check: cleanTEST_ENVS=py314 make -C packages/gooddata-eval test: 3657 passed, 1 skippedRisk
High: this changes default scoring behaviour, so please weigh the trade-off below.
Loop behaviour changes. A turn whose only chart was rejected used to count as "chart produced" and ended the run. Now it's a turn without a chart, so the agentic loop continues with simulated-user replies, and those replies know the expected metrics. Consequences:
Alternative considered. Keep the fallback as it was and lean on the opt-in
requires_execution/execution_signals, which already fail a chart that was never executed. That would change nothing by default.Why the default changes anyway. Without
requires_execution, production evals report false passes on charts the backend refused to build, and fixtures carrying the same unbuildable shape reinforce them. A rejected definition is not an answer under any option. Reviewers: please say if you'd rather make this opt-in.Services-AI-Evaluations pins gooddata-eval by revision and needs a re-pin after release to pick this up.
🤖 Generated with Claude Code