Skip to content

Add privacy-preserving agent, action, and workflow usage reporting - #1943

Merged
Paul Lizer (paullizer) merged 1 commit into
microsoft:Developmentfrom
paullizer:paullizer-agent-usage-telemetry
Nov 17, 2026
Merged

Paul Lizer (paullizer) merged 1 commit into
microsoft:Developmentfrom
paullizer:paullizer-agent-usage-telemetry

Conversation

@paullizer

Copy link
Copy Markdown
Collaborator

Summary

  • Give administrators adoption and reliability reporting without requiring access to conversation content. Both Classic and V2 Control Center Activity Logs now show agent, action-operation, and workflow usage, filters, rankings, and metadata-only CSV exports.
  • Record actual invocation attempts, including delegated agents, retries, failures, cancellation, and measured duration. Preserve canonical configuration identity separately from the execution workspace and deduplicate repeated completion delivery.
  • Report configured action host, operation, and authentication category without full URLs, credentials, prompts, arguments, or results. Unknown historical metadata stays unknown; historical action use is not reconstructed from conversations.
  • Calculate complete filtered totals separately from the existing 5,000-record activity sample. Explicit range/query budgets reject incomplete calculations rather than returning misleading partial totals. Counts are operational telemetry, not token/cost allocation or an exactly-once billing ledger.

Linked issue

N/A. The user chose not to create an issue for this work.

Release Notes & Latest Features

  • New Feature
  • Bug Fix
  • UI Enhancement
  • Breaking Change
  • Internal only

Is this visible to end users?

  • Yes
  • No

Available to authorized Control Center Activity Logs readers, not ordinary chat users.

Is this admin-facing (Admin Settings, governance, deployment, config)?

  • Yes
  • No

Should this become a Latest Feature card?

  • Yes
  • No
  • Already added

Screenshot needed for the card?

  • Yes
  • No
  • Attached

Version bump

  • application/single_app/config.py VERSION third segment bumped, or not needed because this is docs-only
  • deployers/version.txt bumped, or not needed because deployers/ was not changed

Application version: 0.261.446 -> 0.261.447. No deployer changes.

Testing / validation

Run from the repository root; the offline functional suite used $env:PYTHONPATH="$PWD\application\single_app;$PWD\functional_tests".

  • python -m pytest -p test_support.cosmos_network_guard functional_tests\test_runtime_usage_reporting_owner.py functional_tests\test_agent_workflow_runtime_usage.py functional_tests\test_action_runtime_usage.py functional_tests\test_control_center_usage.py -q --disable-warnings --tb=short --maxfail=1 - 68 passed, with network blocked and real initialized owners/installed SDK point responses.
  • python -m pytest ui_tests\test_control_center_runtime_usage.py ui_tests\test_v2_control_center_activity_logs.py -q --disable-warnings - 18 passed, including Classic/V2 desktop/mobile usage interactions.
  • npm --prefix application\v2_ui run build - passed; existing large-bundle warning remains.
  • node --check application\single_app\static\js\control-center-usage.js and node --check application\single_app\static\js\control-center.js - passed.
  • python functional_tests\route_tests\test_route_blueprint_policy_inventory.py - 15/15 passed.
  • python functional_tests\route_tests\test_route_unauthenticated_policy_contract.py - 10/10 passed.
  • python functional_tests\route_tests\test_route_policy_test_coverage.py - 5/5 passed.
  • python functional_tests\test_docs_app_surface_coverage.py - 7/7 passed; python functional_tests\test_docs_site_quality.py - 6/6 passed.
  • git diff --check - passed. Repository XSS/access-control inspectors also passed against the actual changed/new lines.

Known validation limitations:

  • python scripts\check_cosmos_response_boundaries.py --run-contracts - blocked by 18 pre-existing fingerprint gaps in unchanged route_backend_orchestration.py. Confirmed that file matches the starting HEAD while HEAD's committed inventory already differs. Changed usage owners have zero review gaps; unrelated owner reviews were preserved rather than silently approved. A broader independent contract run was stopped after a failure; no full-suite pass is claimed.
  • python functional_tests\test_logging_tag_standardization.py - fails on existing nonstandard tags. The change introduces no new logging tags.
  • Existing OpenAPI outcome probes still mock the obsolete requests.get seam rather than guarded_request; the agent's broader outcome selection reported 9 passed, 2 failed. Focused new action telemetry coverage passes.

Documentation

  • Release notes updated, or not needed
  • Feature documentation updated, or not needed
  • Fix documentation updated, or not needed

Added runtime usage feature documentation and updated the administrator/V2 guides, including privacy boundaries, metric interpretation, legacy coverage, retention/storage gaps, and query limits. Separate fix documentation is not applicable. Release-note changes were offered but deferred while the user was unavailable; they are not included here.

Security checklist

  • New Flask routes include @swagger_route(security=get_auth_security())
  • Settings sent to non-admin frontends use sanitize_settings_for_user()
  • Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS
  • No secrets, keys, connection strings, or local-only artifacts are included

Both API aliases retain Activity Logs authorization; dashboard-reader access alone is insufficient. No settings payload was added. Usage events, report output, and CSV use explicit metadata allowlists; UI values use text-safe rendering and CSV cells receive formula-injection protection.

Track actual agent, action and workflow attempts and expose metadata-only reporting in Classic and V2 Activity Logs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread application/single_app/action_usage.py Resolved
Comment thread application/single_app/action_usage.py Resolved
Comment thread application/single_app/action_usage.py Resolved
Comment thread application/single_app/runtime_usage.py Resolved
@paullizer
Paul Lizer (paullizer) merged commit 0015082 into microsoft:Development Nov 17, 2026
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants