[Sydney Lister (slister1001)]

Summary

Regenerates azure-ai-projects from the latest Foundry TypeSpec (feature/foundry-release, spec commit 96acdcd) to pick up the rubric evaluator contract changes. This PR contains only the emitted code, the patch fixes needed for it, and the CHANGELOG feature entries. The sample updates are split into a stacked PR so they can be reviewed on their own: #49931.

Rubric changes picked up

• RubricScoring and RubricScoringMode (LIKERT, BOOLEAN), plus RubricBasedEvaluatorDefinition.scoring and pass_threshold.
• Dimension.allowed (safety) and read-only Dimension.generated_weight. Dimension.weight is now optional.
• EvaluatorGenerationInputs.category, scoring, pass_threshold, and dimensions. model is now optional (required for quality, ignored for safety).

Commits

1. Part 1: Emit SDK from TypeSpec. Generated code only (tsp-client update).
2. Part 2: Update patches, docs, and tests. Handwritten _patch fixes for the regenerated agent insights operations, docs/public-methods.md, and a test helper update.
3. Part 3: Update changelog.

Overlap

The regen also pulls in non-rubric TypeSpec changes that overlap with #49876 (Howie Leung (@howieleung)) and #49905, #49916, #49918, #49924 (aprilk-ms). Whichever merges second will need a trivial rebase of generated files.

Validation

• Unit and playback tests pass for the package.
• black applied to edited files only.

Howie Leung (@howieleung) could you review and sign off by 9am PT 11/11? This is needed for the Ignite docs deadline.

[azure-sdk-automation]

Add sdk-api-review-ready when this working PR is ready for API review of azure-ai-projects. The review will compare target version 2.11.0 from this working PR against baseline version 2.10.0 (tag azure-ai-projects_2.10.0). If you need a different comparison, use the Azure SDK CLI (azsdk apireview create).

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Sydney Lister (slister1001)]

Companion .NET PRs: Azure/azure-sdk-for-net#64776 (regen) and Azure/azure-sdk-for-net#64775 (samples).

[github-actions]

[Pilot] PR Pipeline Failure Analysis

What failed

All failures on PR #49932 build 7060574 trace to one root cause: the regenerated azure-ai-projects operations surface does not match the names used by the handwritten patch/tests/samples that were updated in commit 2 ("Update patches, docs, and tests").

Test collection failures (all 6 platforms × sdist/whl = 12 failed test-file collections, 2 distinct tests each):

• sdk.ai.azure-ai-projects.tests.agent_insights.test_agent_insight_pollers
• sdk.ai.azure-ai-projects.tests.agent_insights.test_agent_insights_samples

Both fail identically on macos311, ubuntu2404_311, ubuntu2404_311_coverage, Ubuntu2404_313, Ubuntu2404_314, and windows2022_312:

ImportError: cannot import name 'BetaAgentInsightMonitorsOperations' from 'azure.ai.projects.operations'
ImportError: cannot import name 'BetaAgentInsightMonitorsOperations' from 'azure.ai.projects.aio.operations._patch_agent_insights_async'

BetaAgentInsightMonitorsOperations is referenced by the test modules but is not exported (or no longer named that) in the regenerated/patched operations modules — a single naming-drift bug causing the same two test files to fail collection everywhere, not 12 separate issues.

Static type-check failures (pyright + mypy, same build, azure-ai-projects package, 59 pyright errors / multiple mypy errors):

• azure/ai/projects/_unions.py:26 — "_unions" is not defined (reportUndefinedVariable) / mypy name-defined
• samples/agent_insights/*.py — AgentInsightMonitorCreate, AgentInsightMonitorUpdate, AgentInsightRunCreate, BetaAgentInsightMonitorsOperations are unknown import symbols; BetaOperations has no attribute agent_insight_monitors
• samples/skills/*.py, samples/agents/tools/*.py — BetaOperations has no attribute skills
• samples/evaluations/sample_rubric_evaluator_generation_iterate.py:126-127 — max()/min() called with a key returning int | None, which mypy/pyright reject because None isn't SupportsRichComparison

Relevant pipeline output

tests/agent_insights/test_agent_insight_pollers.py:23: in <module>
    from azure.ai.projects.aio.operations._patch_agent_insights_async import (
E   ImportError: cannot import name 'BetaAgentInsightMonitorsOperations' from 'azure.ai.projects.aio.operations._patch_agent_insights_async'

tests/agent_insights/test_agent_insights_samples.py:22: in <module>
    from azure.ai.projects.operations import BetaAgentInsightMonitorsOperations
E   ImportError: cannot import name 'BetaAgentInsightMonitorsOperations' from 'azure.ai.projects.operations'

[pyright] azure/ai/projects/_unions.py:26:37 - error: "_unions" is not defined (reportUndefinedVariable)
[pyright] samples/agent_insights/agent_insights_util.py:26:5 - error: "AgentInsightMonitorUpdate" is unknown import symbol (reportAttributeAccessIssue)
[pyright] samples/skills/sample_skills_crud.py:59:29 - error: Cannot access attribute "skills" for class "BetaOperations"
[mypy] samples/evaluations/sample_rubric_evaluator_generation_iterate.py:126: error: Argument "key" to "max" has incompatible type "Callable[[Dimension], int | None]"; expected "Callable[[Dimension], SupportsDunderLT[Any] | SupportsDunderGT[Any]]"  [arg-type]

Recommended next steps

• Fix the naming/export mismatch for the agent-insights operations patch: ensure azure/ai/projects/operations/__init__.py and azure/ai/projects/aio/operations/_patch_agent_insights_async.py export the class name actually used by tests/agent_insights/test_agent_insight_pollers.py and tests/agent_insights/test_agent_insights_samples.py (either rename the generated/patched class to BetaAgentInsightMonitorsOperations or update the test imports to the current name, whichever matches the intended public API per the TypeSpec regen).
• Fix azure/ai/projects/_unions.py:26 — _unions self-reference is undefined; likely a generator/patch artifact that needs correcting.
• Update the samples/agent_insights/*.py and samples/skills/*.py / samples/agents/tools/*.py samples to use the models/attributes actually present on the regenerated client (agent_insight_monitors, skills, AgentInsightMonitorCreate/Update, AgentInsightRunCreate) — these were likely renamed by the overlapping TypeSpec changes called out in the PR description (overlap with Update Agent Insights GA samples and tests #49876, [azure-ai-projects] Align Voice Agents GA names and preview opt-in scope #49905, [azure-ai-projects] External Agents GA: drop preview opt-in and update samples #49916, [azure-ai-projects] Add trace and dataset evaluation rule samples #49918, [azure-ai-projects] Update endpoint evaluator samples to schema 0.0.2 #49924).
• Fix samples/evaluations/sample_rubric_evaluator_generation_iterate.py:126-127 by giving the max/min key function a non-None sort key (e.g. filter None first or default to a sentinel) so pyright/mypy accept it.
• Re-run pytest sdk/ai/azure-ai-projects/tests/agent_insights/ and azpysdk pyright/mypy against sdk/ai/azure-ai-projects locally to verify after the fix.
• See https://aka.ms/ci-fix

Automated fix: Fix found, view and apply fix

Generated by Pipeline Analysis Next Steps · copilot · sonnet50 · 82.2 AIC · ⌖ 0.589 AIC · ⊞ 9.1K · ◷