Repository navigation
feat(gooddata-eval): add agentic_summary and agentic_describe_object - #1980
Draft
myduong243 wants to merge 5 commits into
Draft
myduong243 wants to merge 5 commits into
myduong243 wants to merge 5 commits into
Conversation
agentic_summary evaluates the chat summary of a widget or a dashboard the way the UI asks for it. An item stores the UI's userContext without result ids and one execution (AFM + result spec, filters included) per widget; every run re-executes them, because result ids expire, and puts the fresh ids back. Follow-up questions run in the same conversation. Code checks: shape by scope (chart, KPI, dashboard, no data, or the user's own format), no ids, no figures on no data, no tool calls on a follow-up that should answer from the conversation. Judge criteria are graded one by one with summary-specific steps. agentic_describe_object asks to describe or explain an analytics object with no context; the answer must rest on a workspace lookup unless the item expects the agent to ask back. Both kinds only read, so they run in parallel. jira: QA-30179 risk: low
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
|
- summary turn keys at the top level (actual_output, shape_ok, must_include_0, ...), like the other kinds' detail - follow-up i keys prefixed followup_<i>_ - tools that read detail[k] is False now see follow-up fails too jira: QA-30179 risk: low
…hecks - add both kinds to the registry coverage lists (dispatch, user context relay, trace linker) - agentic_summary items carry their summary input and context; a missing context is rejected before the run - agentic_describe_object asks with no context, like the Analyze page jira: QA-30179 risk: nonprod
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1980 +/- ##
==========================================
- Coverage 87.67% 87.31% -0.37%
==========================================
Files 358 362 +4
Lines 29039 29535 +496
==========================================
+ Hits 25461 25788 +327
- Misses 3578 3747 +169 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
- a KPI (headline) has one result dimension holding only measureGroup - a donut, waterfall or bullet of measures only has no attributes either, but a second dimension; it gets the chart shape check - no result spec: no attributes still means KPI jira: QA-30179 risk: low
- the summary prompt forbids reference tags; a follow-up is an ordinary chat answer whose tags the UI renders as links - follow-ups keep the empty-answer, tool-call and judge checks jira: QA-30179 risk: low
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agentic_summary: the chat summary of a widget or a dashboard, sent the way the UI's "Summarize" items send it.userContextwithout result ids, plus one execution per widget (agentic_summary_input.executions: AFM + result spec, filters included). Result ids expire, so each run re-executes the stored executions and puts the fresh ids back. Nothing resolves filters at run time; the stored execution is what the widget rendered with.no_tool_calls. Judge criteria are graded one by one with summary-specific steps.agentic_describe_object: "describe / explain " with no context. The answer must rest on a workspace lookup (search_objects/get_object_with_children) unless the item expects the agent to ask back.PARALLEL_SAFE_TEST_KINDS;agentic_summaryis multi-turn,agentic_describe_objectsingle-turn.agentic_dashboard_summaryanddashboard_summarykeep their own builder and rubric.Test Plan
test_agentic_chat_summary.py(stored context + fresh result ids, missing executions, UI default question, execute errors, conversation stops on a failed turn, Langfuse item mapping) andtest_chat_summary_evaluator.py(every shape rule, ids, figures on no data, follow-up grading, scope facts, describe lookup).Risk Assessment
low — two new kinds in the eval package; existing kinds unchanged. Revert: full.
jira: QA-30179
risk: low
🤖 Generated with Claude Code