Skip to content

load-testing.md

Latest commit

 

History

History
52 lines (36 loc) · 4.47 KB

load-testing.md

File metadata and controls

52 lines (36 loc) · 4.47 KB
Raw

Load testing

The ServiceControl Testing Tool is a stateless .NET service that generates error load and real-world failure scenarios against a test ServiceControl instance, to validate error-ingestion performance. It has a web UI for manual control, built-in failure scenarios (third-party outage, timeout spike, poison message, and more), and OpenTelemetry telemetry.

The testing tool README is the full reference: every scenario, API endpoint, configuration setting, container usage, and scaling notes.

Quick start: full stack with Aspire

The Aspire AppHost starts the whole system with a single command: the ServiceControl platform (Learning transport), the testing tool, and a complete observability stack (OTel Collector, Jaeger, Prometheus, and Grafana with a prebuilt dashboard).

aspire run tools/testing-tool/TestingTool.AppHost/TestingTool.AppHost.csproj

Aspire assigns dynamic ports. Open the web UI of the testing tool through the link in the Aspire dashboard.

Useful flags, passed after --:

Flag Description
--tag <tag> Test a specific ServiceControl image tag, for example a PR prerelease like pr-1234
--persistence <type> Persistence for the error instance: RavenDb (default), SqlServer, or PostgreSql
--error-ingestion-scale-unit <n> Start n additional error-ingestion-only ServiceControl instances
--azure-blob-body-storage "<connection string>" Store message bodies in Azure Blob Storage instead of the container file system. Applies only to SqlServer and PostgreSql. The error instance and its scale-out units share one container, and each audit instance gets its own
--s3-body-storage <bucket> Store message bodies in S3 instead of the container file system. Applies only to SqlServer and PostgreSql, and is mutually exclusive with --azure-blob-body-storage. The error instance and its scale-out units share the error-bodies/ prefix, and each audit instance gets its own
--s3-region <region> Region for --s3-body-storage
--s3-service-url <url> S3-compatible endpoint, such as MinIO or LocalStack. It must be reachable from inside the containers, so localhost does not work
--s3-access-key-id <key>, --s3-secret-access-key <secret> Static credentials, set both or neither. Without them, the containers use the default AWS credential chain; host credentials are not forwarded

Run against an existing ServiceControl

dotnet build tools/testing-tool/TestingTool.slnx --configuration Release
dotnet run --project tools/testing-tool/TestingTool --configuration Release

Open http://localhost:5290. The tool defaults to a ServiceControl instance at http://localhost:33333; to use another instance, set the TestingTool__ServiceControlApiUrl environment variable.

Generating load

Everything is controllable from the web UI, or via the HTTP API:

  • Scenarios: the six built-in scenarios (third-party-outage, timeout-spike, poison-message, deserialization-failure, background-noise, large-messages), each producing errors that group naturally. large-messages starts at 3 messages per second unless rate is set, sends bodies of about 256 KiB (the wire envelope is slightly larger), fails 20% of them to the error queue, and audits the rest. The bypass writer fails every message by design, so it sends the large bodies only as failures. Start with POST /api/scenarios/{name}/start (optional rate and durationSeconds).
  • Bypass writer: high-throughput load that writes failed-message envelopes directly to the ServiceControl error queue, skipping the message handler (POST /api/bypass/start).
  • Jobs: retry, archive, search, retention sweep, and custom-check-failure jobs exercise ServiceControl's recoverability, FTS search, and retention pipelines against the load. Jobs do not start automatically. Start them from the UI or /api/jobs.
  • Release-test presets: named presets mapped from the testing scenarios, for example POST /api/release-tests/ingestion-load/start.

Observing the run

  • The Aspire dashboard shows logs and traces for every service.
  • Grafana (auto-provisioned; log in with admin/admin) has a prebuilt "Testing Tool" dashboard with errors/sec by scenario, search latency p95, replay/archive rates, and errors raised versus ingested.
  • Jaeger provides trace analysis and Prometheus provides metrics. The tool also exposes a Prometheus scraping endpoint at /metrics.