Skip to content

feat(gooddata-eval): add agentic_summary and agentic_describe_object - #1980

Draft
myduong243 wants to merge 5 commits into
masterfrom
myd/QA-30179
Draft

myduong243 wants to merge 5 commits into
masterfrom
myd/QA-30179

Conversation

@myduong243

@myduong243 myduong243 commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • New agentic kind agentic_summary: the chat summary of a widget or a dashboard, sent the way the UI's "Summarize" items send it.
    • An item stores the UI's userContext without result ids, plus one execution per widget (agentic_summary_input.executions: AFM + result spec, filters included). Result ids expire, so each run re-executes the stored executions and puts the fresh ids back. Nothing resolves filters at run time; the stored execution is what the widget rendered with.
    • Follow-up questions run in the same conversation, each with its own expected output.
    • Code checks gate the verdict: shape by scope (chart: 4 headings, KPI: no headings and few sentences, dashboard: 3 headings, dashboard without data: one paragraph, or the user's own format), no reference tags or raw ids, no figures when every widget in scope returned no data, no tool calls on a follow-up marked no_tool_calls. Judge criteria are graded one by one with summary-specific steps.
  • New agentic kind agentic_describe_object: "describe / explain " with no context. The answer must rest on a workspace lookup (search_objects / get_object_with_children) unless the item expects the agent to ask back.
  • Both kinds only read, so they are in PARALLEL_SAFE_TEST_KINDS; agentic_summary is multi-turn, agentic_describe_object single-turn.
  • Not changed: agentic_dashboard_summary and dashboard_summary keep their own builder and rubric.
  • Test Plan

    • Unit tests: test_agentic_chat_summary.py (stored context + fresh result ids, missing executions, UI default question, execute errors, conversation stops on a failed turn, Langfuse item mapping) and test_chat_summary_evaluator.py (every shape rule, ids, figures on no data, follow-up grading, scope facts, describe lookup).
    • Live check on a staging workspace: replaying 4 captured UI states (dashboard, widget, tab, two date filters) through the stored executions gives the same result data as the UI for every widget; one item with a follow-up runs end to end.
    • Type check and the tox matrix run in CI only.

    Risk Assessment

    low — two new kinds in the eval package; existing kinds unchanged. Revert: full.

    jira: QA-30179
    risk: low

    🤖 Generated with Claude Code

agentic_summary evaluates the chat summary of a widget or a dashboard
the way the UI asks for it. An item stores the UI's userContext without
result ids and one execution (AFM + result spec, filters included) per
widget; every run re-executes them, because result ids expire, and puts
the fresh ids back. Follow-up questions run in the same conversation.
Code checks: shape by scope (chart, KPI, dashboard, no data, or the
user's own format), no ids, no figures on no data, no tool calls on a
follow-up that should answer from the conversation. Judge criteria are
graded one by one with summary-specific steps.

agentic_describe_object asks to describe or explain an analytics
object with no context; the answer must rest on a workspace lookup
unless the item expects the agent to ask back.

Both kinds only read, so they run in parallel.

jira: QA-30179
risk: low
@coderabbitai

coderabbitai Bot commented Nov 20, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

- summary turn keys at the top level (actual_output, shape_ok,
  must_include_0, ...), like the other kinds' detail
- follow-up i keys prefixed followup_<i>_
- tools that read detail[k] is False now see follow-up fails too

jira: QA-30179
risk: low
…hecks

- add both kinds to the registry coverage lists (dispatch, user
  context relay, trace linker)
- agentic_summary items carry their summary input and context; a
  missing context is rejected before the run
- agentic_describe_object asks with no context, like the Analyze page

jira: QA-30179
risk: nonprod
@codecov

codecov Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 65.99598% with 169 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.31%. Comparing base (0e2b94a) to head (636fdcc).

Files with missing lines Patch % Lines
...val/src/gooddata_eval/core/agentic/chat_summary.py 61.01% 92 Missing ⚠️
.../src/gooddata_eval/core/agentic/describe_object.py 29.29% 70 Missing ⚠️
.../src/gooddata_eval/core/evaluators/chat_summary.py 95.19% 5 Missing ⚠️
...c/gooddata_eval/core/evaluators/describe_object.py 92.59% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1980      +/-   ##
==========================================
- Coverage   87.67%   87.31%   -0.37%     
==========================================
  Files         358      362       +4     
  Lines       29039    29535     +496     
==========================================
+ Hits        25461    25788     +327     
- Misses       3578     3747     +169     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- a KPI (headline) has one result dimension holding only measureGroup
- a donut, waterfall or bullet of measures only has no attributes
  either, but a second dimension; it gets the chart shape check
- no result spec: no attributes still means KPI

jira: QA-30179
risk: low
- the summary prompt forbids reference tags; a follow-up is an
  ordinary chat answer whose tags the UI renders as links
- follow-ups keep the empty-answer, tool-call and judge checks

jira: QA-30179
risk: low
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant