diff --git a/mkdocs/blog/posts/amd-pd-disaggregation.md b/mkdocs/blog/posts/amd-pd-disaggregation.md index 5e7cf7aa2..ede29c479 100644 --- a/mkdocs/blog/posts/amd-pd-disaggregation.md +++ b/mkdocs/blog/posts/amd-pd-disaggregation.md @@ -198,7 +198,6 @@ $ curl http://localhost:3000/proxy/services/main/amd-sglang-pd-service/v1/chat/c Requests are routed to SMG, which selects the prefill and decode workers for each request. The prefill worker processes the prompt, the decode worker continues generation, and Mooncake transfers the KV cache between them over RoCE. `dstack` registers and deregisters workers with SMG as replicas are added or removed, runs the `/health` probe on each replica, and scales each replica group independently. !!! info "Limitations" - - Currently, only one router replica per service is supported. - The example uses the SGLang inference backend for prefill and decode workers. vLLM backend support is coming soon. - Autoscaling supports the RPS metric. TTFT and ITL-based autoscaling support is coming soon. diff --git a/mkdocs/blog/posts/nvidia-dynamo.md b/mkdocs/blog/posts/nvidia-dynamo.md index 0a7ac8adb..1c0f52510 100644 --- a/mkdocs/blog/posts/nvidia-dynamo.md +++ b/mkdocs/blog/posts/nvidia-dynamo.md @@ -18,8 +18,8 @@ With the latest update, `dstack` added native support for NVIDIA Dynamo with Pre -!!! note "Deprecation notice" - The `DSTACK_ROUTER_INTERNAL_IP` environment variable used in this blog post is no longer supported. For the modern approach, see the [NVIDIA Dynamo example](../../docs/examples/inference/dynamo.md). +!!! note "Update" + Since this post was published, `dstack` has removed the `DSTACK_ROUTER_INTERNAL_IP` environment variable used here and lifted most of the limitations listed below: the router replica group can have multiple replicas, and services with a Dynamo router support `retry` and in-place updates. For the current approach, see the [NVIDIA Dynamo example](../../docs/examples/inference/dynamo.md). ## About NVIDIA Dynamo diff --git a/mkdocs/blog/posts/smg.md b/mkdocs/blog/posts/smg.md index 94a3b173f..d0e73dc32 100644 --- a/mkdocs/blog/posts/smg.md +++ b/mkdocs/blog/posts/smg.md @@ -40,7 +40,6 @@ env: - MODEL_ID=zai-org/GLM-4.5-Air-FP8 replicas: - count: 1 - # For now replica group with router must have count: 1 commands: - pip install smg - | diff --git a/mkdocs/docs/concepts/services.md b/mkdocs/docs/concepts/services.md index 44b1d0757..a9923e36b 100644 --- a/mkdocs/docs/concepts/services.md +++ b/mkdocs/docs/concepts/services.md @@ -344,7 +344,6 @@ Below is an example for running `zai-org/GLM-4.5-Air-FP8` on `H200`: groups: - replicas: 1 - # For now the router group must have replicas: 1 commands: - pip install smg - | @@ -788,8 +787,7 @@ scales the service to 4 replicas. Setting the minimum number of replicas to `0` allows the service to scale down to zero when there are no requests. Each [replica group](#replica-groups) can set its own `replicas` range and `scaling` rules, so -worker groups scale independently. For now, a group with [`router`](#router) must have -`replicas: 1` and cannot be scaled. +the router and worker groups scale independently. > The `scaling` property requires creating a [gateway](gateways.md). diff --git a/mkdocs/docs/examples/inference/sglang.md b/mkdocs/docs/examples/inference/sglang.md index e37d3bef6..94c337025 100644 --- a/mkdocs/docs/examples/inference/sglang.md +++ b/mkdocs/docs/examples/inference/sglang.md @@ -151,7 +151,6 @@ To run SGLang with [PD disaggregation](https://docs.sglang.io/advanced_features/ groups: - replicas: 1 - # For now replica group with router must have replicas: 1 commands: - pip install smg - | @@ -218,7 +217,6 @@ To run SGLang with [PD disaggregation](https://docs.sglang.io/advanced_features/ groups: - replicas: 1 - # For now replica group with router must have replicas: 1 python: "3.12" commands: - pip install smg diff --git a/src/dstack/_internal/core/models/configurations.py b/src/dstack/_internal/core/models/configurations.py index 93f32be41..3df7dc393 100644 --- a/src/dstack/_internal/core/models/configurations.py +++ b/src/dstack/_internal/core/models/configurations.py @@ -1541,10 +1541,6 @@ def validate_at_most_one_router_replica_group(self) -> Self: router_groups = [g for g in groups if g.router is not None] if len(router_groups) > 1: raise ValueError("At most one replica group may specify `router`.") - if router_groups: - router_group = router_groups[0] - if router_group.replicas.min != 1 or router_group.replicas.max != 1: - raise ValueError("For now replica group with `router` must have `replicas: 1`.") return self