Skip to content

[bot] Merge master/96f2bf73 into rel/dev - #1977

Merged
yenkins-admin merged 2 commits into
rel/devfrom
snapshot-master-96f2bf73-to-rel/dev
Nov 20, 2026
Merged

yenkins-admin merged 2 commits into
rel/devfrom
snapshot-master-96f2bf73-to-rel/dev

Conversation

@yenkins-admin

Copy link
Copy Markdown
Contributor

🚀 Automated PR to perform merge from master into rel/dev with changes up to 96f2bf7 (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/44710060512).

huyenthanh09 and others added 2 commits November 20, 2026 14:33
An expected_output dict that names an option key opts a metric_skill
item into further checks. A list of candidates, or a dict holding one,
runs and scores as before and makes no REST call.

- user_replies are sent in order before the simulated user;
  run_all_replies keeps sending them after a write; expected_metrics
  waits for several metrics, judges each id by its latest write and
  tries every assignment within a step budget, {metric/@n} standing
  for an earlier one
- checks: expect_no_create (+ expected_mentions; needs a completed
  answer with text, so a silent agent does not pass), expect_no_overwrite, expect_clarification,
  confirmation_turn, rubric (LLM judge over the whole conversation in
  order), expected_fields (the metric the agent wrote) and
  expected_update (requested fields change, every other writable one
  stays), expect_no_deletion (ids to guard), expect_executes (a 4xx
  rejection fails it; a rate limit or a 5xx is retried, never blamed
  on the metric), expect_suggestions; score_first_proposal is a
  diagnostic (the first formula shown, else the first write)
- an options run whose chat fails on a turn it still needed does not
  pass: it is a FAIL with that exit
- each run snapshots the workspace metrics through the SDK entity API
  and puts back every pre-existing metric it wrote over, with all
  writable attributes and transient retries, through the entity
  endpoint, never create_metric (GDAI-2838)
- a check the harness cannot read, a judge without a verdict or an
  undecided metric assignment leaves that check unscored, and the
  other checks are still evaluated; a failed snapshot or restore
  does the same through metric_scored. An unscored run never passes,
  is labelled ERROR unless a graded check or its chat already failed
  it (then FAIL), and its quality counts metric_scored alone; a graded
  run is preferred as the best run; a failed snapshot or an earlier
  failed restore skips the run
- the judge is built before the first request; a malformed or
  contradicting option, a misspelt option key beside an original-shape
  MAQL, or an item whose checks only forbid (no overwrite, no early
  write, no deletion) raises ValueError before any request
- an options item puts its checks at the top of the detail, so the
  quality score counts them

jira: QA-29907
risk: low
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ions

feat(gooddata-eval): opt-in metric-skill checks beyond the MAQL
@yenkins-admin
yenkins-admin merged commit 1ac40dc into rel/dev Nov 20, 2026
1 check passed
@coderabbitai

coderabbitai Bot commented Nov 20, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: cbb40ed3-5d32-44d6-ae38-c1c24a74b0c5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@yenkins-admin
yenkins-admin deleted the snapshot-master-96f2bf73-to-rel/dev branch November 20, 2026 09:22
@codecov

codecov Bot commented Nov 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.67%. Comparing base (26d6ff0) to head (96f2bf7).
⚠️ Report is 663 commits behind head on rel/dev.

Additional details and impacted files
@@             Coverage Diff             @@
##           rel/dev    #1977      +/-   ##
===========================================
+ Coverage    87.37%   87.67%   +0.30%     
===========================================
  Files          356      358       +2     
  Lines        28380    29039     +659     
===========================================
+ Hits         24798    25461     +663     
+ Misses        3582     3578       -4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants