Skip to content

fix(gooddata-eval): score no chart for a rejected adhoc definition - #1982

Open
Tomkess wants to merge 1 commit into
masterfrom
fix/eval-failed-create-not-scored
Open

Tomkess wants to merge 1 commit into
masterfrom
fix/eval-failed-create-not-scored

Conversation

@Tomkess

@Tomkess Tomkess commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

When an assistant answer carries no visualization part, the SSE client falls back to the arguments of the last create_adhoc_visualization call. That way a chart the agent built but could not persist (for example, data source unavailable) still gets scored.

That fallback also scored definitions the backend rejected as unbuildable against the workspace's model, such as a pre-aggregated metric broken down by an attribute its fact cannot reach:

{"status":"error","message":"Aggregation dimension='[attribute/...]' is not comparable to the dimensionality='[dataset/fact_cuta_daily_summable_metrics]' of the subtree"}

In Services-AI-Evaluations, a Mastercard run recorded executed: false with that error and still passed 3/3, because the fixture carried the same unbuildable shape.

Now:

  • The fallback still looks only at the last call. If that call's result is a definition rejection, the turn has no chart.
  • It never falls back to an earlier call, which may be a lookup the agent did not mean to show. A successful retry is naturally the last call.
  • A last call with show_to_user: false (gen-ai's flag for a chart built only to read a value) is a lookup the user never saw, so it scores no chart either.
  • The rejection markers are gen-ai's own _COMPATIBILITY_REJECTIONS ("is not comparable to the dimensionality", "has no witness", in create_adhoc_visualization_tool.py).
  • Environment errors (e.g. "Data source does not exist") keep the old behaviour.
  • The README and the ExpectationOptions docstring say rejected definitions are not scored.

Test plan

New tests, each failing on the base sse_client.py:

  • Each marker on a rejected definition with no retry gives no chart.
  • Two rejections give no chart.
  • An accepted lookup followed by a rejected chart gives no chart; the earlier lookup is never scored.
  • With out-of-order parallel results, the last call's own result decides, matched by callId.
  • A last call with show_to_user: false gives no chart.
  • Loop level (test_agentic_visualization.py): a turn whose only chart was rejected continues to a simulated reply, and the run is scored on the next turn's chart.

Environment errors still score the arguments. test_parse_sse_lines_falls_back_to_adhoc_viz_when_multipart_viz_is_null carried a malformed tool result (unescaped quotes), so it never exercised the error. It now carries a well-formed {"status":"error","message":"Data source does not exist"}. A second test, ..._still_scores_a_definition_an_environment_error_stopped, pins the case explicitly. Both fail if the check treats every status: error result as a rejection. These two pass on the base, since they guard unchanged behaviour.

The rejected-then-accepted-retry case isn't a new test: the base already scores the last call, so it can't tell the two apart.

Commands:

  • make -C packages/gooddata-eval format-fix lint-fix type-check: clean
  • TEST_ENVS=py314 make -C packages/gooddata-eval test: 3657 passed, 1 skipped

Risk

High: this changes default scoring behaviour, so please weigh the trade-off below.

Loop behaviour changes. A turn whose only chart was rejected used to count as "chart produced" and ended the run. Now it's a turn without a chart, so the agentic loop continues with simulated-user replies, and those replies know the expected metrics. Consequences:

  • more turns, latency and cost for those runs;
  • some runs that failed before may now pass after the simulated user's guidance;
  • these extra turns use the multi-turn clarification budget.

Alternative considered. Keep the fallback as it was and lean on the opt-in requires_execution / execution_signals, which already fail a chart that was never executed. That would change nothing by default.

Why the default changes anyway. Without requires_execution, production evals report false passes on charts the backend refused to build, and fixtures carrying the same unbuildable shape reinforce them. A rejected definition is not an answer under any option. Reviewers: please say if you'd rather make this opt-in.

Services-AI-Evaluations pins gooddata-eval by revision and needs a re-pin after release to pick this up.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The visualization fallback now checks each definition's associated tool result. It skips definitions rejected as unbuildable, selects the last remaining definition, and leaves createdVisualizations unset if all definitions are rejected. Tests cover these cases and unrelated tool errors.

Changes: Visualization fallback

Layer / File(s) Summary
Track and classify tool-call results
packages/gooddata-eval/src/gooddata_eval/core/chat/sse_client.py
Visualization arguments now retain their tool-call event indices. A helper identifies results with an error status and a message containing a configured unbuildable marker.
Filter fallback candidates
packages/gooddata-eval/src/gooddata_eval/core/chat/sse_client.py, packages/gooddata-eval/tests/test_sse_client.py
The fallback skips definitions rejected as unbuildable and selects the last remaining definition. If all definitions are rejected, it does not add a chart. Tests cover rejection, later successful definitions, and unrelated errors.

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix

Merge Risk: Minimal · up to 63d51

The eval client's fallback no longer scores a chart definition the backend rejected as unbuildable. Tests cover the rejected, retried, and unrelated-error cases. The only open point is that the backend's exact error format is not visible in this repository. If the format differs, behavior simply stays as before. No merge-blocking risk remains.

Pre-merge checks: Passed 4

Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title names the main change: a rejected adhoc definition is no longer scored as a chart in the gooddata-eval SSE client fallback. The change matches this. The title is specific and concise. It use…
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Title check

Explanation

The title names the main change: a rejected adhoc definition is no longer scored as a chart in the gooddata-eval SSE client fallback. The change matches this. The title is specific and concise. It uses the conventional fix(scope) prefix and is 66 characters long.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

A rabbit checks each chart in line,
And skips the ones that fail to shine.
The last good chart is set to show,
Unless all candidates say no.
Then carrots wait, and logs will know.

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 3bd4f696-e1ed-4a42-bd01-3f7766955753
Commits

Reviewing files that changed from the base of the PR and between 0e2b94a and 63d51f5.

Files selected for processing: 2

  • packages/gooddata-eval/src/gooddata_eval/core/chat/sse_client.py
  • packages/gooddata-eval/tests/test_sse_client.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@codecov

codecov Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.67%. Comparing base (0e2b94a) to head (11c74fb).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1982      +/-   ##
==========================================
- Coverage   87.67%   87.67%   -0.01%     
==========================================
  Files         358      358              
  Lines       29039    29052      +13     
==========================================
+ Hits        25461    25472      +11     
- Misses       3578     3580       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Tomkess Tomkess changed the title fix(gooddata-eval): score no chart when the backend rejected its definition fix(gooddata-eval): score no chart for a rejected adhoc definition Nov 21, 2026
@Tomkess
Tomkess force-pushed the fix/eval-failed-create-not-scored branch 2 times, most recently from 3e47766 to 5dc01a9 Compare November 21, 2026 22:01
When an answer carries no visualization part, the SSE client falls back to
the arguments of the last create_adhoc_visualization call, so a chart the
agent built but could not persist (data source unavailable) still gets
scored. That fallback also scored definitions the backend rejected as
unbuildable against the workspace's model, e.g. a pre-aggregated metric
broken down by an attribute its fact cannot reach. Such a run recorded
executed=false and still passed 3/3 when a fixture carried the same
unbuildable shape.

The fallback now looks at the last call only, as before, but scores no
chart when that call's result is a definition rejection. The markers are
gen-ai's own compatibility rejections ("is not comparable to the
dimensionality", "has no witness"). It never falls back to an earlier
call, which may be a lookup the agent did not mean to show, and a last
call that is itself a lookup (show_to_user: false, gen-ai's flag for a
chart built only to read a value) scores no chart either. Environment
errors keep the old behaviour; the existing fallback test's tool result
was malformed JSON, so it now carries a well-formed error and a second
test pins the environment case explicitly.

A turn whose only chart was rejected now counts as a turn without a
chart, so the agentic loop continues with a simulated reply instead of
ending as if a chart was built. A loop-level test pins that.

The README and the ExpectationOptions docstring say rejected definitions
are not scored.

jira: trivial
risk: high
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Tomkess
Tomkess force-pushed the fix/eval-failed-create-not-scored branch from 5dc01a9 to 11c74fb Compare November 21, 2026 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant