From b0dba1314feb8f3583cb7a447321f0c243b3525a Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Mon, 2 Nov 2026 22:53:05 -0800 Subject: [PATCH 1/9] Add partition weaving post (routing series, post 5) Draft for review. Also points the pw and partition-weaving redirects at the new post. Signed-off-by: Sam Barker --- _data/redirects/blog.yaml | 4 +- ...weaving-stretching-the-seams-of-a-topic.md | 169 ++++++++++++++++++ 2 files changed, 171 insertions(+), 2 deletions(-) create mode 100644 _posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md diff --git a/_data/redirects/blog.yaml b/_data/redirects/blog.yaml index ff5fb0bc..fec8d849 100644 --- a/_data/redirects/blog.yaml +++ b/_data/redirects/blog.yaml @@ -29,7 +29,7 @@ mappings: absoluteTarget: https://kroxylicious.io/blog/kroxylicious-proxy/2026/11/02/topic-weaving-one-client-many-clusters.html - name: pw subgroup: routing - absoluteTarget: https://kroxylicious.io/blog/kroxylicious-proxy/2026/10/12/these-are-not-the-brokers-you-are-connecting-to.html + absoluteTarget: https://kroxylicious.io/blog/kroxylicious-proxy/2026/11/03/partition-weaving-stretching-the-seams-of-a-topic.html - name: partition-weaving subgroup: routing - absoluteTarget: https://kroxylicious.io/blog/kroxylicious-proxy/2026/10/12/these-are-not-the-brokers-you-are-connecting-to.html + absoluteTarget: https://kroxylicious.io/blog/kroxylicious-proxy/2026/11/03/partition-weaving-stretching-the-seams-of-a-topic.html diff --git a/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md b/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md new file mode 100644 index 00000000..3b04b238 --- /dev/null +++ b/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md @@ -0,0 +1,169 @@ +--- +layout: post +title: "Partition Weaving: Stretching the seams of a topic" +date: 2026-11-03 15:00:00 -0800 +author: "Sam Barker" +author_url: "https://github.com/sambarker" +# noinspection YAMLSchemaValidation +categories: blog kroxylicious-proxy +tags: [ "routing" ] +--- + +This is the fifth post in a series about routing in Kroxylicious, and it covers the last of the four patterns [Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) named: [connection switching]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}), [stream branching]({% post_url 2026-10-29-stream-branching-mind-the-gap %}), [topic weaving]({% post_url 2026-11-02-topic-weaving-one-client-many-clusters %}) and partition weaving. Like the rest of the series, this is design-level thinking rather than a description of something shipped, and we haven't tried any of it. + +Partition weaving is a cautionary tale. My first reaction was "that's a really neat idea". The more I looked at it, the harder it got. It is clearly part of the design space the Routing API opens up, which is why it's in the series, and it may well be possible. But it is a lot of effort to go to, and I haven't found a use case that justifies it to me. If you have one, I'd like to hear about it. + +## A train in two portions + +[Post 4]({% post_url 2026-11-02-topic-weaving-one-client-many-clusters %}) said the railway had run out of track. It had, for topics. For partitions there turns out to be a siding. + +[Post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) gave us a ticket: the 3:15 to Mos Eisley, carriage 13, seat 342323. In Kafka terms that's `topic-A.partition-13` at offset 342323. Take a train whose timetable says it has 32 carriages. Passengers are assigned a carriage by hashing their name and taking the remainder after dividing by 32, so everyone called Smith ends up in the same one, every time. Carriages 0 to 15 are one portion of the train, which started out in us-east. Carriages 16 to 31 are another, which started out in eu-west. The two are coupled together at a junction the passengers never see. Each portion keeps its own reservations, so seat 342323 only means something within its own carriage. The ticket says carriage 20. The guard knows that is carriage 4 of the eu-west portion. + +That's the scenario from Post 1: a risk engine in us-east, hardcoded to read a topic called `orders`, and an EU team producing to `orders-eu` on a local cluster. The router presents both as one `orders` with 32 partitions. + +{% comment %} +```mermaid +graph TB + subgraph Client view + L["orders: partitions 0 to 31"] + end + R[Router] + subgraph us-east + A["orders: partitions 0 to 15"] + end + subgraph eu-west + B["orders-eu: partitions 0 to 15"] + end + L --> R + R -- "p" --> A + R -- "p - 16" --> B +``` +{% endcomment %} +``` +Client sees orders: [ 0 1 2 ... 15 | 16 17 18 ... 31 ] 32 partitions + │ │ + │ p │ p − 16 + ▼ ▼ +Router maps us-east / orders eu-west / orders-eu + partitions 0 – 15 partitions 0 – 15 +``` + +The analogy breaks in one place, and it's worth stating because the difficulty lives there. The carriages of a real train are in a line and you can walk between them. Partitions have no order and nothing moves between them. The number is only a label, and the label is the thing the client's hashing depends on. Kafka's default partitioner takes a keyed record, computes a `murmur2` hash of the key bytes and takes the remainder after dividing by the partition count. The count comes from the `Metadata` response. Unkeyed records are spread over whichever partitions are available. A label that stops meaning the same carriage is a label that silently reshuffles every passenger. + +## What would you use it for? + +The best case I have is a migration. The runtime gives you nothing for it today. None of the others are much better, and I've set each against the thing that would usually beat it. + +**Moving a very large topic a piece at a time.** Replicate one range of partitions to the new cluster with something that preserves offsets, repoint that range, watch for a while, and repeat. This is connection switching with a finer lever, and it brings the problems from [Post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) with it, per range instead of all at once: committed offsets have to mean the same thing on the new cluster, and idempotent producers' sequence state stays behind. Compare it with connection switching for the whole topic, which is stateless and already exists. A range at a time buys you a canary, and it only seems worth building for a topic you are properly nervous about moving. + +**A topic bigger than a cluster.** Clients see one topic with N partitions, keys hash as normal, and per-key ordering holds. But [topic weaving]({% post_url 2026-11-02-topic-weaving-one-client-many-clusters %}) already spreads topics across clusters. Partition weaving only helps when a single topic outgrows what one cluster will hold or serve, and I'd want to look at that topic's design before building anything to accommodate it. + +**The name-bound consumer.** This is the risk engine above. Most consumers can subscribe to more than one topic, and where this one can't, replicating `orders-eu` into the consumer's cluster does the same job. The weave saves you the replica. In exchange the consumer reads the EU partitions across the link on every fetch. + +**Data residency by key.** This is the one people will suggest, and it fits worst. The default partitioner hashes keys without knowing where anything lives, so residency would need a custom partitioner that sends EU keys to EU partitions. Producers that already know the layout could write to separate topics, and topic weaving would stitch those together without any of this. + +That's the list. If you have a better one, tell me. + +## The layout is declared, not discovered + +Everything else depends on where the partition layout comes from. For topic weaving the map could follow the clusters, because topics appearing and disappearing is something clients cope with. A partition count is different. It's the divisor in the client's hash. + +Suppose the router discovered the layout by asking the clusters. If eu-west became unreachable, the router would find only 16 partitions and report a topic with 16 partitions. Every keyed record produced in that window would hash into 0 to 15, to the wrong carriage, with no error anywhere. So I'd declare it. A woven topic is an ordered list of constituents, each a route, a physical topic and a partition count, held in the router's configuration or a catalogue. Three consequences follow: + +- **The numbering has to be contiguous.** The default partitioner picks a number from 0 to N-1 using the count. A hole is a partition the producer will try to use and can't find. +- **An unreachable constituent still counts.** Its partitions stay in the `Metadata` response, reported with an error and no leader, the way a broker reports a partition whose leader is offline. N doesn't shrink because a cluster is down. +- **The declared count wins.** Kafka lets someone add partitions to the physical topic behind a constituent. The extra partitions stay invisible until the layout says otherwise. + +Every proxy instance must give the same identity the same answer, as in Post 4. Because the layout is configuration, every instance has the same list, so there's no state to share. + +Changing it is where the care goes. The rule is that a logical partition number must keep meaning the same log. Appending a constituent to the end satisfies that: existing numbers don't move and the new ones are simply new. It does grow N, which changes where keyed records land, exactly as adding partitions to any Kafka topic does. Data already written stays where it is. If the new constituent is on a cluster the router has no route to yet, `numRoutes` changes too, and every virtual node ID moves with it, a coordinated reconnect for all clients of that VKC. A rolling update is tolerable for an append. An instance that hasn't picked up the change returns errors for the new partitions, and clients retry. Reordering, removing, resizing or relocating a constituent is a migration, and no amount of careful rollout makes it anything else. The piecemeal migration above is one of those, so it needs a deliberate cutover and not a config push. + +## Two ways to number them + +The first is **ranges**, which is what the picture shows. Constituent *i* starts at the sum of the partition counts before it. A logical partition `p` falls in the range `[start_i, start_i + count_i)` and maps to physical partition `p - start_i`. The inverse adds `start_i` back. In the example, logical 20 is `eu-west` / `orders-eu` partition 4. Constituents can have different counts, and appending to the end leaves everything before it alone. + +The second is **interleaving**, and it's the tempting one, because it's the arithmetic from Post 2. With K constituents of equal size, logical `p` goes to constituent `p mod K` as physical partition `p div K`. It looks like the node ID bijection and it's just as stateless. + +Both are stateless. Interleaving isn't saving anything the declared list hasn't already saved, because the router needs that list to know the routes anyway. Where they differ is growth: + +| | Ranges | Interleaved | +|---|---|---| +| Logical 5 with K = 2 | constituent 0, partition 5 | constituent 1, partition 2 | +| Logical 5 after a third constituent joins (K = 3) | constituent 0, partition 5 | constituent 2, partition 1 | + +(Constituents are numbered from 0, and the ranges column assumes the first holds at least 6 partitions.) + +Interleaving moves every logical partition when K changes. The data hasn't moved. Committed offsets sit in `__consumer_offsets` keyed by the physical topic and partition, and earlier records were placed by an earlier hash. After the change the same logical number reads from a different log. In Post 2 the same formula was safe because node IDs are re-learned from the next `Metadata` response. Partition identity is stored, so it can't be re-learned. Ranges are only growable at the end, but at least they know what that means. + +Reshuffling isn't a quirk of weaving. A normal Kafka topic can't shrink either, and adding partitions already moves keys. A key with a hash of 50 lands in carriage 18 of 32 and carriage 2 of 48. The number is only illustrative. The principle is that a partition count is a promise to clients, and weaving doesn't let you break it any more cheaply. + +## Where the numbers appear + +Partition numbers turn up in nearly every RPC that names a partition. I'm sticking to produce and consume, which is the data path, and leaving the admin APIs out. These are the requests where the router has to translate: + +| Request | Where the partition number is | What the router does | +|---|---|---| +| `Metadata` response | Each partition's index, and its leader as a node ID | Emits logical 0 to N-1. Sets each leader to a virtual node on the owning route, with an error for an unreachable constituent | +| `Produce` | The partition index under each topic | Maps logical to (route, physical topic, physical partition), splits by route, rewrites the response the other way | +| `Fetch` | The partition under each topic, and the partitions to drop in the forgotten-topics list of an incremental fetch | The same translation. The session state is kept on both sides, as in the [topic routing proof of concept]({% post_url 2026-05-21-topic-routing %}) | +| `ListOffsets` | The partition index | The same translation. Consumers use it to find the start or end of a partition | +| `OffsetCommit`, `OffsetFetch` | The partition index under each topic | Splits by owning route, with the boss and standalone commits from Post 4 | + +Offsets, leader epochs and the record batches themselves belong to the physical partition and pass through untouched. Rewriting a partition number doesn't touch the batch, so unlike rewriting a producer ID it costs no CRC recalculation. + +A complete list would be longer. These are the ones that, if wrong, a client notices. + +## Topic IDs + +There's a second identifier that needs the same care, and it applies to topic weaving as much as to this pattern. [Post 4]({% post_url 2026-11-02-topic-weaving-one-client-many-clusters %}) renamed topics without mentioning it, so the right place for it is wherever you weave. + +Kafka gives each topic a UUID at creation, and a topic deleted and recreated under the same name gets a new one. That's the point. A late fetch for the old topic can't read the new log. Fetch has taken the ID instead of the name since Kafka 3.1. Produce followed in 4.1, and `OffsetCommit` and `OffsetFetch` can use it from version 10. `ListOffsets` still uses names. The router writes the `Metadata` response, so it chooses the UUID the client sees. When the client sends that UUID back in a `Produce` or `Fetch`, the router has to find the physical topic, and the physical topic's own UUID is the one it must send upstream. + +I'd treat the two sides differently: + +- **Client side: derived.** The router computes the ID for a logical topic from its name and something that says which incarnation it is. A generation number alongside the layout is one way to do that. Every instance computes the same value, with nothing stored. Going the other way, from a UUID back to a topic, you can't invert a hash. The router builds a table at startup by computing the ID for every topic it declares. The result must avoid the all-zero UUID, which the protocol uses to mean "no ID". Kafka's own generator also skips a few reserved values and any ID whose string form contains a dash, and it would be sensible to follow suit. +- **Broker side: learned.** The physical ID comes from the backing cluster's `Metadata` response, discovered in-band as in Post 2. If it goes stale, the broker answers `UNKNOWN_TOPIC_ID` and the router refreshes. The broker's own protection against stale topics stays intact on this leg. + +That's the opposite of a producer ID. The client-facing ID can be rebuilt from the declared layout by any instance, and the physical ID can be re-learned from the cluster. Nothing that can't be recomputed has to survive a restart. + +What you lose is the client-facing guarantee. The client's protection against a delete-and-recreate is only as good as the generation. Nobody really wants to delete and recreate a woven topic at will, so I wouldn't build anything to evolve client IDs automatically. Whoever does it is doing a migration, and bumping the generation is how they say so. It needs managing, for topic weaving as well as for this. + +The alternative is to avoid IDs altogether. The proposal notes that a router can cap the versions it advertises, for example `PRODUCE` at 12, so clients use names. It calls that "an implementation simplification rather than a fundamental necessity". The cost is exactly the guarantee above, and it's a choice about what the client leg promises. + +## Producing + +A producer writing through the weave picks a partition from 0 to 31 as it would for any topic. The router sends it wherever that number lives. For keyed records the hash decides, so a producer in eu-west has no reason to land on the eu-west partitions. For unkeyed records the client spreads load across available partitions, which includes the ones across the ocean. Writing through the weave gives you no locality unless something outside the router arranges it. + +Arranging it means a custom partitioner that knows which numbers are local, and that puts the layout the router hides back into the client. If you can change a producer's `partitioner.class`, you could change its topic name as well. So the producers that gain from partition weaving are, in practice, the ones that don't use it. In Post 1's scenario the EU team writes straight to `orders-eu`, and the weave serves readers only. + +## Consuming + +Consumer groups work as they did in [Post 4]({% post_url 2026-11-02-topic-weaving-one-client-many-clusters %}), with one change of granularity. Membership goes to the boss, and the group leader works out the assignment from the merged `Metadata`, so every member sees `orders` with 32 partitions and may be handed some from each cluster. Offsets split inside a topic now, not between topics. Commits for the partitions on the boss go with the member's ID and generation. Commits for the rest go as standalone commits. All of Post 4's caveats come along, including that fencing is partial and the new group protocol isn't covered. A woven topic that gains a constituent changes its partition count, and subscribed consumers rebalance as they would for any topic that gained partitions. + +Two smaller points. An `OffsetFetch` for every topic returns offsets under physical names, so the router has to translate or hide them, particularly if a group also reads `orders-eu` directly. And an unreachable constituent is reported the way a broker reports a partition whose leader is offline, which is the shape clients already expect. I'd start there and not invent something new. We haven't tested it with real clients. + +A `read_committed` consumer doesn't need any producer ID translation. The batch headers, the commit and abort markers and the aborted-transaction list for a partition all come from one cluster and agree with each other, so they pass through as in Post 4. + +## Identity and transactions + +Producer IDs add nothing new. Sequence numbers are counted per producer and partition, and a partition lives on one cluster, so the producer ID map from Post 4 covers it unchanged. I called the place that map lives a boss cluster. The requirement is a durable store that every instance can see, and a boss cluster is one way to get one. One producer ends up holding different identities on different partitions of the same topic, which is harmless for each partition and not something any client I know of depends on. + +Transactions are where the escape hatch closes. In topic weaving you could keep related topics on one cluster. Here a single topic's partitions span clusters, and a transactional producer chooses a partition by key without knowing which cluster owns it. It has no way to build a transaction that stays on one side. Post 4's two attempts failed with two clusters involved, and here the client can't even tell that it involves two. The router can at least fail loudly by rejecting transactional writes to a woven topic. Kafka Streams with exactly-once semantics is again the usual way to be running transactions without knowing it. + +## Egress, ingress, or does it matter? + +The answer is much as it was for topic weaving. Co-locating the proxy with one of the clusters puts the proxy in that cluster's failure domain. The client side keeps backing cluster failures isolated, and if you adopt a durable store like the boss, putting the proxy near it shortens the paths that use it. + +One difference is that a single topic's reads now cross whatever links join the clusters. Where the proxy sits decides which consumers pay for the long hop. No placement makes every consumer local, because a consumer in a group gets partitions from both sides. + +## Know when to pick a different chute + +If your problem is moving a workload, use [connection switching]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}). If your topics have distinct homes, [topic weaving]({% post_url 2026-11-02-topic-weaving-one-client-many-clusters %}) hides the clusters without hiding the partitions. If your problem is locality or disaster recovery, replicate. Weaving gives you unified access, not local copies. If you need atomic writes across clusters, no proxy can help, and with partition weaving you can't even tell where the seam is. + +The cost of getting this wrong is silent. The number a client hashes against must mean the same log forever, the layout can't be discovered, and the one thing that would justify it, moving a big topic carefully, is a job for a migration you plan. I'd only reach for partition weaving if you already know which of those you're buying. + +--- + +That's the four patterns. Each one asks the router to understand a little more of the protocol than the last, and partition weaving is where I think the cost outruns the benefit, for now. I'd be glad to be shown otherwise. + +If this is the sort of problem you're wrestling with, or you have a use case that makes partition weaving look worthwhile, come and talk to us on [GitHub](https://github.com/kroxylicious/kroxylicious), [Slack](https://kroxylicious.slack.com), or [Bluesky](https://bsky.app/profile/kroxylicious.io). From 7ce15046ff6a25c84307463ebee80e057145c9c0 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Tue, 3 Nov 2026 09:23:44 -0800 Subject: [PATCH 2/9] Rework partition weaving post: Tatooine train, no-split data plane, group.id prefixing Open the train analogy as a fetch from Tatooine (virtual cluster) with Mos Eisley and Mos Espa as the backing clusters, dropping the Post 2 ticket reference. Note that Produce, Fetch and ListOffsets never need splitting because each virtual node maps to one route. Add the group.id collision hazard for standalone commits and have the router prefix group IDs. Signed-off-by: Sam Barker --- ...artition-weaving-stretching-the-seams-of-a-topic.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md b/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md index 3b04b238..8eb84bf0 100644 --- a/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md +++ b/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md @@ -17,9 +17,9 @@ Partition weaving is a cautionary tale. My first reaction was "that's a really n [Post 4]({% post_url 2026-11-02-topic-weaving-one-client-many-clusters %}) said the railway had run out of track. It had, for topics. For partitions there turns out to be a siding. -[Post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) gave us a ticket: the 3:15 to Mos Eisley, carriage 13, seat 342323. In Kafka terms that's `topic-A.partition-13` at offset 342323. Take a train whose timetable says it has 32 carriages. Passengers are assigned a carriage by hashing their name and taking the remainder after dividing by 32, so everyone called Smith ends up in the same one, every time. Carriages 0 to 15 are one portion of the train, which started out in us-east. Carriages 16 to 31 are another, which started out in eu-west. The two are coupled together at a junction the passengers never see. Each portion keeps its own reservations, so seat 342323 only means something within its own carriage. The ticket says carriage 20. The guard knows that is carriage 4 of the eu-west portion. +Picture a train coming back from Tatooine with 32 carriages. A carriage is a partition and a seat within it is an offset, so a ticket for carriage 13, seat 342323 means partition 13 at offset 342323. Passengers were loaded into a carriage on the outbound journey, by hashing their name and taking the remainder after dividing by 32, so everyone called Smith ends up in the same one, every time. Carriages 0 to 15 are one portion of the train, which started out in Mos Eisley. Carriages 16 to 31 are another, which started out in Mos Espa. Two towns on one planet, with the portions coupled together at a junction that neither the passengers nor the consumer at the end of the line ever see. The consumer believes it is meeting one 32-carriage train from Tatooine. Each portion keeps its own reservations, so seat 342323 only means something within its own carriage. The consumer's ticket says carriage 20. The guard knows that is carriage 4 of the Mos Espa portion. -That's the scenario from Post 1: a risk engine in us-east, hardcoded to read a topic called `orders`, and an EU team producing to `orders-eu` on a local cluster. The router presents both as one `orders` with 32 partitions. +That's the scenario from Post 1: a risk engine in us-east, hardcoded to read a topic called `orders`, and an EU team producing to `orders-eu` on a local cluster. The router presents both as one `orders` with 32 partitions. In the story, Tatooine is the virtual cluster, Mos Eisley is us-east and Mos Espa is eu-west. {% comment %} ```mermaid @@ -104,11 +104,13 @@ Partition numbers turn up in nearly every RPC that names a partition. I'm sticki | Request | Where the partition number is | What the router does | |---|---|---| | `Metadata` response | Each partition's index, and its leader as a node ID | Emits logical 0 to N-1. Sets each leader to a virtual node on the owning route, with an error for an unreachable constituent | -| `Produce` | The partition index under each topic | Maps logical to (route, physical topic, physical partition), splits by route, rewrites the response the other way | +| `Produce` | The partition index under each topic | Maps logical to (route, physical topic, physical partition) and rewrites the response the other way | | `Fetch` | The partition under each topic, and the partitions to drop in the forgotten-topics list of an incremental fetch | The same translation. The session state is kept on both sides, as in the [topic routing proof of concept]({% post_url 2026-05-21-topic-routing %}) | | `ListOffsets` | The partition index | The same translation. Consumers use it to find the start or end of a partition | | `OffsetCommit`, `OffsetFetch` | The partition index under each topic | Splits by owning route, with the boss and standalone commits from Post 4 | +`Produce`, `Fetch` and `ListOffsets` never need to be split. A client sends each of them to the leader of the partitions involved. With the node ID mapping from Post 2 every virtual node belongs to exactly one route, so a client that follows the `Metadata` it was given sends a frame whose partitions all live on one cluster. The router rewrites the coordinates and forwards the whole frame. The requests that do get torn apart are `Metadata`, which merges several answers into one, and the offset requests, because a group has a single coordinator whichever clusters its partitions are on. + Offsets, leader epochs and the record batches themselves belong to the physical partition and pass through untouched. Rewriting a partition number doesn't touch the batch, so unlike rewriting a producer ID it costs no CRC recalculation. A complete list would be longer. These are the ones that, if wrong, a client notices. @@ -140,6 +142,8 @@ Arranging it means a custom partitioner that knows which numbers are local, and Consumer groups work as they did in [Post 4]({% post_url 2026-11-02-topic-weaving-one-client-many-clusters %}), with one change of granularity. Membership goes to the boss, and the group leader works out the assignment from the merged `Metadata`, so every member sees `orders` with 32 partitions and may be handed some from each cluster. Offsets split inside a topic now, not between topics. Commits for the partitions on the boss go with the member's ID and generation. Commits for the rest go as standalone commits. All of Post 4's caveats come along, including that fencing is partial and the new group protocol isn't covered. A woven topic that gains a constituent changes its partition count, and subscribed consumers rebalance as they would for any topic that gained partitions. +Standalone commits bring a hazard of their own. They land in the other clusters' `__consumer_offsets`, and the `group.id` namespace there is shared with every other application on the cluster. If one of them uses the same group ID, the commit is either rejected because that group has live members, or it quietly overwrites that application's offsets. So the router prefixes the group ID on the way to those clusters, and strips it again on the way back. Yes, yes, we could coordinate group names across the clusters instead. Then we're back on the same conference bridge at 2am, arguing about which team owns which name. + Two smaller points. An `OffsetFetch` for every topic returns offsets under physical names, so the router has to translate or hide them, particularly if a group also reads `orders-eu` directly. And an unreachable constituent is reported the way a broker reports a partition whose leader is offline, which is the shape clients already expect. I'd start there and not invent something new. We haven't tested it with real clients. A `read_committed` consumer doesn't need any producer ID translation. The batch headers, the commit and abort markers and the aborted-transaction list for a partition all come from one cluster and agree with each other, so they pass through as in Post 4. From b119ddabc573dc04f8aed01773b372fb120acd96 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Tue, 3 Nov 2026 09:33:19 -0800 Subject: [PATCH 3/9] Fix stream branching redirect to use the published 2026/10/29 URL Signed-off-by: Sam Barker --- _data/redirects/blog.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_data/redirects/blog.yaml b/_data/redirects/blog.yaml index fec8d849..55c237d6 100644 --- a/_data/redirects/blog.yaml +++ b/_data/redirects/blog.yaml @@ -17,10 +17,10 @@ mappings: absoluteTarget: https://kroxylicious.io/blog/kroxylicious-proxy/2026/10/19/connection-switching-identity-without-commitment.html - name: sb subgroup: routing - absoluteTarget: https://kroxylicious.io/blog/kroxylicious-proxy/2026/10/26/stream-branching-mind-the-gap.html + absoluteTarget: https://kroxylicious.io/blog/kroxylicious-proxy/2026/10/29/stream-branching-mind-the-gap.html - name: stream-branching subgroup: routing - absoluteTarget: https://kroxylicious.io/blog/kroxylicious-proxy/2026/10/26/stream-branching-mind-the-gap.html + absoluteTarget: https://kroxylicious.io/blog/kroxylicious-proxy/2026/10/29/stream-branching-mind-the-gap.html - name: tw subgroup: routing absoluteTarget: https://kroxylicious.io/blog/kroxylicious-proxy/2026/11/02/topic-weaving-one-client-many-clusters.html From 0217075fda3235ade6f92858dfd327dc58ce3fd8 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Tue, 3 Nov 2026 09:42:45 -0800 Subject: [PATCH 4/9] Use prose alt text for stream branching diagrams Signed-off-by: Sam Barker --- ...026-10-29-stream-branching-mind-the-gap.md | 32 +++---------------- 1 file changed, 4 insertions(+), 28 deletions(-) diff --git a/_posts/2026-10-29-stream-branching-mind-the-gap.md b/_posts/2026-10-29-stream-branching-mind-the-gap.md index 9687b667..967f19e6 100644 --- a/_posts/2026-10-29-stream-branching-mind-the-gap.md +++ b/_posts/2026-10-29-stream-branching-mind-the-gap.md @@ -46,13 +46,7 @@ graph LR style DR fill:#ffd,stroke:#333,stroke-width:2px style Router fill:#ddf,stroke:#333,stroke-width:2px {% endcomment %} -![ - ┌──(mainline)──► [ Primary cluster ] ──ack──┐ -[ Producer ] ──Produce──► [ Router ] │ - ▲ └──(branch)────► [ DR cluster ] (identical copy) - │ │ - └──────────────── ProduceResponse (primary's verdict only) ◄───────────────┘ -](/assets/blog/images/routing-series/naive-dr.svg) +![A producer sends a Produce request to the router. The router forwards it on the mainline to the primary cluster and, on a branch, to a DR cluster that receives an identical copy. The ProduceResponse returned to the producer carries only the primary cluster's verdict.](/assets/blog/images/routing-series/naive-dr.svg) Yes, I know this looks like a DR solution, it's really not. By definition the branch is a best-effort copy; the DR cluster can have holes and duplicates that the primary doesn't — [Mind the gap](#mind-the-gap) goes into the gory details. You can't trust it for failover — the only way to verify nothing's missing is to compare both copies, and if both copies are online you don't need to fail over. If you need reliable replication to a DR cluster, use MirrorMaker 2 or [KIP-1279](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1279%3A+Cluster+Mirroring) cluster mirroring — they were designed and built for that job. @@ -92,13 +86,7 @@ graph LR style Redactor fill:#fff4dd,stroke:#333 ``` {% endcomment %} -![ - ┌──(mainline)──► [ Primary cluster ] ──ack──┐ -[ Producer ] ──Produce──► [ Router ] │ - ▲ └──(branch)──► [sample 10%]──► [redact]──► [ Staging cluster ] - │ │ - └──────────────── ProduceResponse (primary's verdict only) ◄───────────────┘ -](/assets/blog/images/routing-series/staging-branchline.svg) +![A producer sends a Produce request to the router. The router forwards it on the mainline to the primary cluster. On a branch, it samples 10% of records, redacts them, and writes them to a staging cluster. The ProduceResponse returned to the producer carries only the primary cluster's verdict.](/assets/blog/images/routing-series/staging-branchline.svg) So far we have talked about taking the original record and forwarding it somewhere. That's powerful, but why stop there? Filters can observe, transform, or act on requests — and what's the use in observing a record without making that observation visible somewhere? We are all Kafka fans here, so of course we send our telemetry via Kafka too. A branch can derive some interesting information from the produce request and send it merrily on its way to the telemetry cluster. @@ -129,13 +117,7 @@ graph LR style Derive fill:#fff4dd,stroke:#333 ``` {% endcomment %} -![ - ┌──(mainline)──► [ Primary cluster ] ──ack────────────────┐ -[ Producer ] ──Produce──► [ Router ] │ - ▲ └──(branch)──► [derive telemetry]──► [ telemetry cluster ] │ - │ │ - └──────────────── ProduceResponse (primary's verdict only) ◄─────────────────────────────┘ -](/assets/blog/images/routing-series/telemetry-branchline.svg) +![A producer sends a Produce request to the router. The router forwards it on the mainline to the primary cluster. On a branch, it derives telemetry from the records and writes that to a telemetry cluster. The ProduceResponse returned to the producer carries only the primary cluster's verdict.](/assets/blog/images/routing-series/telemetry-branchline.svg) The branch writes belong to the router, not the client. The client didn't send them, never sees their responses, and has no idea they exist. The router opens its own connections to branch targets, dispatches its own copy of the request to each, and decides what to do when they fail. @@ -175,13 +157,7 @@ graph LR style Broker1 fill:#eee,stroke:#999,stroke-width:1px,stroke-dasharray: 5 5 ``` {% endcomment %} -![ -[ Consumer (AZ-a) ] ──Fetch──► [ Kroxylicious (AZ-a) ] ────► [ Broker 3 (AZ-a, Replica) ] - │ - (Skips AZ tax) - ▼ - [ Broker 1 (AZ-c, Leader) ] ← never sees the request -](/assets/blog/images/routing-series/replica-routing.svg) +![A consumer in availability zone A sends a Fetch request to a Kroxylicious proxy in the same zone. The proxy sends it to broker 3, a replica in zone A, instead of broker 1, the leader in zone C, which never sees the request. This avoids the cross-zone network charge.](/assets/blog/images/routing-series/replica-routing.svg) The router can do what it thinks is best here: it can honour a client rack, ignore it, or invent one if the client didn't set it. A router doesn't need to trust what the client says about its own location; it can route correctly regardless. The request payload is untouched, the original broker never receives the fetch, and the cross-AZ charge never reaches the invoice. What makes that possible: the proxy can hold two backend sockets — one to the leader, one to the local replica — and choose between them per request. To the client, that's still one broker on one cluster. From da3e293209ff7f519d6d8dcabad9bb4ba6853804 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Tue, 3 Nov 2026 09:42:46 -0800 Subject: [PATCH 5/9] Add partition ranges diagram to partition weaving post Signed-off-by: Sam Barker --- ...ion-weaving-stretching-the-seams-of-a-topic.md | 15 ++++----------- .../images/routing-series/partition-ranges.svg | 2 ++ 2 files changed, 6 insertions(+), 11 deletions(-) create mode 100644 assets/blog/images/routing-series/partition-ranges.svg diff --git a/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md b/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md index 8eb84bf0..7ab9a30c 100644 --- a/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md +++ b/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md @@ -25,28 +25,21 @@ That's the scenario from Post 1: a risk engine in us-east, hardcoded to read a t ```mermaid graph TB subgraph Client view - L["orders: partitions 0 to 31"] + L["orders: partitions 0 .. 31"] end R[Router] subgraph us-east - A["orders: partitions 0 to 15"] + A["orders: partitions 0 .. 15"] end subgraph eu-west - B["orders-eu: partitions 0 to 15"] + B["orders-eu: partitions 0 .. 15"] end L --> R R -- "p" --> A R -- "p - 16" --> B ``` {% endcomment %} -``` -Client sees orders: [ 0 1 2 ... 15 | 16 17 18 ... 31 ] 32 partitions - │ │ - │ p │ p − 16 - ▼ ▼ -Router maps us-east / orders eu-west / orders-eu - partitions 0 – 15 partitions 0 – 15 -``` +![The client sees one topic, orders, with 32 partitions numbered 0 to 31. The router sends partitions 0 to 15 unchanged to the orders topic on us-east, and partitions 16 to 31 to partitions 0 to 15 of the orders-eu topic on eu-west, subtracting 16 from the partition number.](/assets/blog/images/routing-series/partition-ranges.svg) The analogy breaks in one place, and it's worth stating because the difficulty lives there. The carriages of a real train are in a line and you can walk between them. Partitions have no order and nothing moves between them. The number is only a label, and the label is the thing the client's hashing depends on. Kafka's default partitioner takes a keyed record, computes a `murmur2` hash of the key bytes and takes the remainder after dividing by the partition count. The count comes from the `Metadata` response. Unkeyed records are spread over whichever partitions are available. A label that stops meaning the same carriage is a label that silently reshuffles every passenger. diff --git a/assets/blog/images/routing-series/partition-ranges.svg b/assets/blog/images/routing-series/partition-ranges.svg new file mode 100644 index 00000000..8fefe3b2 --- /dev/null +++ b/assets/blog/images/routing-series/partition-ranges.svg @@ -0,0 +1,2 @@ +pp - 16eu-westus-eastClient vieworders: partitions 0.. 31Routerorders: partitions 0.. 15orders-eu:partitions 0 .. 15 \ No newline at end of file From e76107e6e4b23d487dba0b54c9888dd12385f339 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Tue, 3 Nov 2026 09:53:49 -0800 Subject: [PATCH 6/9] Add partition weaving diagrams Signed-off-by: Sam Barker --- ...weaving-stretching-the-seams-of-a-topic.md | 52 +++++++++++++++++++ .../routing-series/consumer-groups-v1.svg | 2 + .../partition-range-management.svg | 2 + .../blog/images/routing-series/topicIds.svg | 2 + 4 files changed, 58 insertions(+) create mode 100644 assets/blog/images/routing-series/consumer-groups-v1.svg create mode 100644 assets/blog/images/routing-series/partition-range-management.svg create mode 100644 assets/blog/images/routing-series/topicIds.svg diff --git a/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md b/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md index 7ab9a30c..014d1195 100644 --- a/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md +++ b/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md @@ -86,6 +86,22 @@ Both are stateless. Interleaving isn't saving anything the declared list hasn't (Constituents are numbered from 0, and the ranges column assumes the first holds at least 6 partitions.) +{% comment %} +```mermaid +graph TB + subgraph Ranges + R1["logical 5, K = 2"] --> R2["constituent 0, partition 5"] + R3["logical 5, K = 3"] --> R4["constituent 0, partition 5"] + end + subgraph Interleaved + I1["logical 5, K = 2"] --> I2["constituent 1, partition 2"] + I3["logical 5, K = 3"] --> I4["constituent 2, partition 1"] + end + style I4 fill:#fdd,stroke:#a33,stroke-width:2px +``` +{% endcomment %} +![Logical partition 5 under the two numbering schemes, before and after a third constituent joins. With ranges, it stays on partition 5 of constituent 0 both times. With interleaving, it is partition 2 of constituent 1 with two constituents, and partition 1 of constituent 2 with three, so the same logical number now reads from a different log.](/assets/blog/images/routing-series/partition-range-management.svg) + Interleaving moves every logical partition when K changes. The data hasn't moved. Committed offsets sit in `__consumer_offsets` keyed by the physical topic and partition, and earlier records were placed by an earlier hash. After the change the same logical number reads from a different log. In Post 2 the same formula was safe because node IDs are re-learned from the next `Metadata` response. Partition identity is stored, so it can't be re-learned. Ranges are only growable at the end, but at least they know what that means. Reshuffling isn't a quirk of weaving. A normal Kafka topic can't shrink either, and adding partitions already moves keys. A key with a hash of 50 lands in carriage 18 of 32 and carriage 2 of 48. The number is only illustrative. The principle is that a partition count is a promise to clients, and weaving doesn't let you break it any more cheaply. @@ -119,6 +135,22 @@ I'd treat the two sides differently: - **Client side: derived.** The router computes the ID for a logical topic from its name and something that says which incarnation it is. A generation number alongside the layout is one way to do that. Every instance computes the same value, with nothing stored. Going the other way, from a UUID back to a topic, you can't invert a hash. The router builds a table at startup by computing the ID for every topic it declares. The result must avoid the all-zero UUID, which the protocol uses to mean "no ID". Kafka's own generator also skips a few reserved values and any ID whose string form contains a dash, and it would be sensible to follow suit. - **Broker side: learned.** The physical ID comes from the backing cluster's `Metadata` response, discovered in-band as in Post 2. If it goes stale, the broker answers `UNKNOWN_TOPIC_ID` and the router refreshes. The broker's own protection against stale topics stays intact on this leg. +{% comment %} +```mermaid +graph LR + C[Client] + R[Router] + L["Declared layout and generation"] + B["Backing cluster"] + L -- "derives the client topic ID" --> R + C -- "Produce or Fetch with the client topic ID" --> R + R -- "Produce or Fetch with the physical topic ID" --> B + B -- "Metadata: the physical topic ID" --> R + B -. "UNKNOWN_TOPIC_ID if stale, so the router refreshes" .-> R +``` +{% endcomment %} +![The client topic ID is derived by the router from the declared layout and its generation, so no instance needs to store it. The client sends Produce and Fetch requests with that ID to the router. The router swaps in the physical topic ID, which it learned from the backing cluster's Metadata response, and sends the request on. If the physical ID is stale, the cluster answers UNKNOWN_TOPIC_ID and the router refreshes its Metadata.](/assets/blog/images/routing-series/topicIds.svg) + That's the opposite of a producer ID. The client-facing ID can be rebuilt from the declared layout by any instance, and the physical ID can be re-learned from the cluster. Nothing that can't be recomputed has to survive a restart. What you lose is the client-facing guarantee. The client's protection against a delete-and-recreate is only as good as the generation. Nobody really wants to delete and recreate a woven topic at will, so I wouldn't build anything to evolve client IDs automatically. Whoever does it is doing a migration, and bumping the generation is how they say so. It needs managing, for topic weaving as well as for this. @@ -135,6 +167,26 @@ Arranging it means a custom partitioner that knows which numbers are local, and Consumer groups work as they did in [Post 4]({% post_url 2026-11-02-topic-weaving-one-client-many-clusters %}), with one change of granularity. Membership goes to the boss, and the group leader works out the assignment from the merged `Metadata`, so every member sees `orders` with 32 partitions and may be handed some from each cluster. Offsets split inside a topic now, not between topics. Commits for the partitions on the boss go with the member's ID and generation. Commits for the rest go as standalone commits. All of Post 4's caveats come along, including that fencing is partial and the new group protocol isn't covered. A woven topic that gains a constituent changes its partition count, and subscribed consumers rebalance as they would for any topic that gained partitions. +{% comment %} +```mermaid +graph TB + M[Group member] + R[Router] + subgraph Boss cluster + BC["Group coordinator: membership, generation, offsets for the partitions it holds"] + end + subgraph Other cluster + OC["__consumer_offsets: offsets for the partitions it holds"] + end + M -- "JoinGroup, SyncGroup, Heartbeat" --> R + M -- "OffsetCommit for orders, partitions 0 .. 31" --> R + R -- "membership" --> BC + R -- "commit with member ID and generation" --> BC + R -- "standalone commit: generation -1, no member ID, group ID prefixed, partitions renumbered" --> OC +``` +{% endcomment %} +![A group member sends its group membership requests and its offset commits for the woven topic to the router. The router sends membership requests to the boss cluster, which is the group coordinator. It sends commits for the partitions on the boss cluster there too, with the member ID and generation. Commits for the partitions on the other cluster go there as standalone commits with generation minus one and no member ID, with the group ID prefixed and the partitions renumbered.](/assets/blog/images/routing-series/consumer-groups-v1.svg) + Standalone commits bring a hazard of their own. They land in the other clusters' `__consumer_offsets`, and the `group.id` namespace there is shared with every other application on the cluster. If one of them uses the same group ID, the commit is either rejected because that group has live members, or it quietly overwrites that application's offsets. So the router prefixes the group ID on the way to those clusters, and strips it again on the way back. Yes, yes, we could coordinate group names across the clusters instead. Then we're back on the same conference bridge at 2am, arguing about which team owns which name. Two smaller points. An `OffsetFetch` for every topic returns offsets under physical names, so the router has to translate or hide them, particularly if a group also reads `orders-eu` directly. And an unreachable constituent is reported the way a broker reports a partition whose leader is offline, which is the shape clients already expect. I'd start there and not invent something new. We haven't tested it with real clients. diff --git a/assets/blog/images/routing-series/consumer-groups-v1.svg b/assets/blog/images/routing-series/consumer-groups-v1.svg new file mode 100644 index 00000000..aa84ebed --- /dev/null +++ b/assets/blog/images/routing-series/consumer-groups-v1.svg @@ -0,0 +1,2 @@ +JoinGroup, SyncGroup,HeartbeatOffsetCommit fororders, partitions 0 ..31membershipcommit with memberID and generationstandalone commit:generation -1, nomember ID, group IDprefixed, partitionsrenumberedOther clusterBoss clusterGroup memberRouterGroup coordinator:membership,generation, offsetsfor the partitions itholds__consumer_offsets:offsets for thepartitions it holds \ No newline at end of file diff --git a/assets/blog/images/routing-series/partition-range-management.svg b/assets/blog/images/routing-series/partition-range-management.svg new file mode 100644 index 00000000..8cd6c7e0 --- /dev/null +++ b/assets/blog/images/routing-series/partition-range-management.svg @@ -0,0 +1,2 @@ +InterleavedRangeslogical 5, K = 2constituent 0,partition 5logical 5, K = 3constituent 0,partition 5logical 5, K = 2constituent 1,partition 2logical 5, K = 3constituent 2,partition 1 \ No newline at end of file diff --git a/assets/blog/images/routing-series/topicIds.svg b/assets/blog/images/routing-series/topicIds.svg new file mode 100644 index 00000000..a6b829cd --- /dev/null +++ b/assets/blog/images/routing-series/topicIds.svg @@ -0,0 +1,2 @@ +derives the clienttopic IDProduce or Fetch withthe client topic IDProduce or Fetch withthe physical topic IDMetadata: thephysical topic IDUNKNOWN_TOPIC_IDif stale, so the routerrefreshesClientRouterDeclared layout andgenerationBacking cluster \ No newline at end of file From 49d8ec2776dfab0a45edcd3e9e114442fc581d5d Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Tue, 3 Nov 2026 19:48:22 -0800 Subject: [PATCH 7/9] Make re-numbering partitions the central difficulty in the partition weaving post Signed-off-by: Sam Barker --- ...-03-partition-weaving-stretching-the-seams-of-a-topic.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md b/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md index 014d1195..344e987c 100644 --- a/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md +++ b/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md @@ -41,7 +41,7 @@ graph TB {% endcomment %} ![The client sees one topic, orders, with 32 partitions numbered 0 to 31. The router sends partitions 0 to 15 unchanged to the orders topic on us-east, and partitions 16 to 31 to partitions 0 to 15 of the orders-eu topic on eu-west, subtracting 16 from the partition number.](/assets/blog/images/routing-series/partition-ranges.svg) -The analogy breaks in one place, and it's worth stating because the difficulty lives there. The carriages of a real train are in a line and you can walk between them. Partitions have no order and nothing moves between them. The number is only a label, and the label is the thing the client's hashing depends on. Kafka's default partitioner takes a keyed record, computes a `murmur2` hash of the key bytes and takes the remainder after dividing by the partition count. The count comes from the `Metadata` response. Unkeyed records are spread over whichever partitions are available. A label that stops meaning the same carriage is a label that silently reshuffles every passenger. +The analogy breaks in one place, and it's where the difficulty lives: once we've numbered the partitions, we can't re-number them. Nothing moves between carriages. Records stay in the partition they were written to, and the numbers we expose in `Metadata` are the only way a client can address them. Change the count, or what a number points at, and the data can't follow. Clients keep using the old addresses and read or write the wrong place. ## What would you use it for? @@ -59,7 +59,7 @@ That's the list. If you have a better one, tell me. ## The layout is declared, not discovered -Everything else depends on where the partition layout comes from. For topic weaving the map could follow the clusters, because topics appearing and disappearing is something clients cope with. A partition count is different. It's the divisor in the client's hash. +Everything else depends on where the partition layout comes from. For topic weaving the map could follow the clusters, because topics appearing and disappearing is something clients cope with. A partition count is different. A partition number is the only handle a client has on a record, and changing the handle without moving the data breaks all sorts of things. Suppose the router discovered the layout by asking the clusters. If eu-west became unreachable, the router would find only 16 partitions and report a topic with 16 partitions. Every keyed record produced in that window would hash into 0 to 15, to the wrong carriage, with no error anywhere. So I'd declare it. A woven topic is an ordered list of constituents, each a route, a physical topic and a partition count, held in the router's configuration or a catalogue. Three consequences follow: @@ -104,7 +104,7 @@ graph TB Interleaving moves every logical partition when K changes. The data hasn't moved. Committed offsets sit in `__consumer_offsets` keyed by the physical topic and partition, and earlier records were placed by an earlier hash. After the change the same logical number reads from a different log. In Post 2 the same formula was safe because node IDs are re-learned from the next `Metadata` response. Partition identity is stored, so it can't be re-learned. Ranges are only growable at the end, but at least they know what that means. -Reshuffling isn't a quirk of weaving. A normal Kafka topic can't shrink either, and adding partitions already moves keys. A key with a hash of 50 lands in carriage 18 of 32 and carriage 2 of 48. The number is only illustrative. The principle is that a partition count is a promise to clients, and weaving doesn't let you break it any more cheaply. +Re-numbering isn't a quirk of weaving. Changing the partition count of a normal Kafka topic is already a risky business: a key whose hash is 50 lands in carriage 18 of 32 and carriage 2 of 48. Weaving doesn't make that any worse, but it does make an inadvertent change much more likely. Changing the configuration is a much more serious activity than it appears. ## Where the numbers appear From b492b857235492f531a8066d25a018901e5e5d62 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Tue, 3 Nov 2026 20:02:45 -0800 Subject: [PATCH 8/9] Address review feedback on partition weaving post Drop the train and Star Wars analogy in favour of a direct description of the two clusters, define virtual and physical partitions at first use, use 'virtual' in place of 'logical' to match project terminology, and expand the use cases introduction. Assisted-by: IBM Bob Signed-off-by: Sam Barker --- ...weaving-stretching-the-seams-of-a-topic.md | 44 +++++++++---------- .../partition-range-management.svg | 2 +- 2 files changed, 22 insertions(+), 24 deletions(-) diff --git a/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md b/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md index 344e987c..f525f124 100644 --- a/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md +++ b/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md @@ -13,13 +13,11 @@ This is the fifth post in a series about routing in Kroxylicious, and it covers Partition weaving is a cautionary tale. My first reaction was "that's a really neat idea". The more I looked at it, the harder it got. It is clearly part of the design space the Routing API opens up, which is why it's in the series, and it may well be possible. But it is a lot of effort to go to, and I haven't found a use case that justifies it to me. If you have one, I'd like to hear about it. -## A train in two portions +## One topic, two clusters -[Post 4]({% post_url 2026-11-02-topic-weaving-one-client-many-clusters %}) said the railway had run out of track. It had, for topics. For partitions there turns out to be a siding. +Take the scenario from Post 1: a risk engine in us-east, hardcoded to read a topic called `orders`, and an EU team producing to `orders-eu` on a local cluster. The router presents both as one `orders` topic with 32 partitions. Partitions 0 to 15 are the `orders` partitions on us-east. Partitions 16 to 31 are partitions 0 to 15 of `orders-eu` on eu-west. A consumer that asks for partition 20 gets partition 4 of `orders-eu` and never knows. I'll call the numbers the client sees *virtual*, in the same sense as a virtual cluster, and the partitions and topics that actually exist on a backing cluster *physical*. Partition 20 is virtual, and partition 4 of `orders-eu` is the physical partition behind it. -Picture a train coming back from Tatooine with 32 carriages. A carriage is a partition and a seat within it is an offset, so a ticket for carriage 13, seat 342323 means partition 13 at offset 342323. Passengers were loaded into a carriage on the outbound journey, by hashing their name and taking the remainder after dividing by 32, so everyone called Smith ends up in the same one, every time. Carriages 0 to 15 are one portion of the train, which started out in Mos Eisley. Carriages 16 to 31 are another, which started out in Mos Espa. Two towns on one planet, with the portions coupled together at a junction that neither the passengers nor the consumer at the end of the line ever see. The consumer believes it is meeting one 32-carriage train from Tatooine. Each portion keeps its own reservations, so seat 342323 only means something within its own carriage. The consumer's ticket says carriage 20. The guard knows that is carriage 4 of the Mos Espa portion. - -That's the scenario from Post 1: a risk engine in us-east, hardcoded to read a topic called `orders`, and an EU team producing to `orders-eu` on a local cluster. The router presents both as one `orders` with 32 partitions. In the story, Tatooine is the virtual cluster, Mos Eisley is us-east and Mos Espa is eu-west. +Producers pick a partition by hashing the record key and taking the remainder after dividing by 32, so every record with a given key lands in the same partition, every time. Each cluster keeps its own offsets, so offset 54 only means something within its own partition. {% comment %} ```mermaid @@ -41,11 +39,11 @@ graph TB {% endcomment %} ![The client sees one topic, orders, with 32 partitions numbered 0 to 31. The router sends partitions 0 to 15 unchanged to the orders topic on us-east, and partitions 16 to 31 to partitions 0 to 15 of the orders-eu topic on eu-west, subtracting 16 from the partition number.](/assets/blog/images/routing-series/partition-ranges.svg) -The analogy breaks in one place, and it's where the difficulty lives: once we've numbered the partitions, we can't re-number them. Nothing moves between carriages. Records stay in the partition they were written to, and the numbers we expose in `Metadata` are the only way a client can address them. Change the count, or what a number points at, and the data can't follow. Clients keep using the old addresses and read or write the wrong place. +The difficulty lives in one place: once we've numbered the partitions, we can't re-number them. Records stay in the partition they were written to, and the numbers we expose in `Metadata` are the only way a client can address them. Change the count, or what a number points at, and the data can't follow. Clients keep using the old addresses and read or write the wrong place. ## What would you use it for? -The best case I have is a migration. The runtime gives you nothing for it today. None of the others are much better, and I've set each against the thing that would usually beat it. +The best case I have is a migration, for moving a very large topic between clusters. The runtime gives you nothing for it today. None of the other cases are much better, and I've set each against the solution that would usually beat it. **Moving a very large topic a piece at a time.** Replicate one range of partitions to the new cluster with something that preserves offsets, repoint that range, watch for a while, and repeat. This is connection switching with a finer lever, and it brings the problems from [Post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) with it, per range instead of all at once: committed offsets have to mean the same thing on the new cluster, and idempotent producers' sequence state stays behind. Compare it with connection switching for the whole topic, which is stateless and already exists. A range at a time buys you a canary, and it only seems worth building for a topic you are properly nervous about moving. @@ -61,7 +59,7 @@ That's the list. If you have a better one, tell me. Everything else depends on where the partition layout comes from. For topic weaving the map could follow the clusters, because topics appearing and disappearing is something clients cope with. A partition count is different. A partition number is the only handle a client has on a record, and changing the handle without moving the data breaks all sorts of things. -Suppose the router discovered the layout by asking the clusters. If eu-west became unreachable, the router would find only 16 partitions and report a topic with 16 partitions. Every keyed record produced in that window would hash into 0 to 15, to the wrong carriage, with no error anywhere. So I'd declare it. A woven topic is an ordered list of constituents, each a route, a physical topic and a partition count, held in the router's configuration or a catalogue. Three consequences follow: +Suppose the router discovered the layout by asking the clusters. If eu-west became unreachable, the router would find only 16 partitions and report a topic with 16 partitions. Every keyed record produced in that window would hash into 0 to 15, to the wrong partition, with no error anywhere. So I'd declare it. A woven topic is an ordered list of constituents, each a route, a physical topic and a partition count, held in the router's configuration or a catalogue. Three consequences follow: - **The numbering has to be contiguous.** The default partitioner picks a number from 0 to N-1 using the count. A hole is a partition the producer will try to use and can't find. - **An unreachable constituent still counts.** Its partitions stay in the `Metadata` response, reported with an error and no leader, the way a broker reports a partition whose leader is offline. N doesn't shrink because a cluster is down. @@ -69,20 +67,20 @@ Suppose the router discovered the layout by asking the clusters. If eu-west beca Every proxy instance must give the same identity the same answer, as in Post 4. Because the layout is configuration, every instance has the same list, so there's no state to share. -Changing it is where the care goes. The rule is that a logical partition number must keep meaning the same log. Appending a constituent to the end satisfies that: existing numbers don't move and the new ones are simply new. It does grow N, which changes where keyed records land, exactly as adding partitions to any Kafka topic does. Data already written stays where it is. If the new constituent is on a cluster the router has no route to yet, `numRoutes` changes too, and every virtual node ID moves with it, a coordinated reconnect for all clients of that VKC. A rolling update is tolerable for an append. An instance that hasn't picked up the change returns errors for the new partitions, and clients retry. Reordering, removing, resizing or relocating a constituent is a migration, and no amount of careful rollout makes it anything else. The piecemeal migration above is one of those, so it needs a deliberate cutover and not a config push. +Changing it is where the care goes. The rule is that a virtual partition number must keep meaning the same log. Appending a constituent to the end satisfies that: existing numbers don't move and the new ones are simply new. It does grow N, which changes where keyed records land, exactly as adding partitions to any Kafka topic does. Data already written stays where it is. If the new constituent is on a cluster the router has no route to yet, `numRoutes` changes too, and every virtual node ID moves with it, a coordinated reconnect for all clients of that VKC. A rolling update is tolerable for an append. An instance that hasn't picked up the change returns errors for the new partitions, and clients retry. Reordering, removing, resizing or relocating a constituent is a migration, and no amount of careful rollout makes it anything else. The piecemeal migration above is one of those, so it needs a deliberate cutover and not a config push. ## Two ways to number them -The first is **ranges**, which is what the picture shows. Constituent *i* starts at the sum of the partition counts before it. A logical partition `p` falls in the range `[start_i, start_i + count_i)` and maps to physical partition `p - start_i`. The inverse adds `start_i` back. In the example, logical 20 is `eu-west` / `orders-eu` partition 4. Constituents can have different counts, and appending to the end leaves everything before it alone. +The first is **ranges**, which is what the picture shows. Constituent *i* starts at the sum of the partition counts before it. A virtual partition `p` falls in the range `[start_i, start_i + count_i)` and maps to physical partition `p - start_i`. The inverse adds `start_i` back. In the example, virtual 20 is `eu-west` / `orders-eu` partition 4. Constituents can have different counts, and appending to the end leaves everything before it alone. -The second is **interleaving**, and it's the tempting one, because it's the arithmetic from Post 2. With K constituents of equal size, logical `p` goes to constituent `p mod K` as physical partition `p div K`. It looks like the node ID bijection and it's just as stateless. +The second is **interleaving**, and it's the tempting one, because it's the arithmetic from Post 2. With K constituents of equal size, virtual `p` goes to constituent `p mod K` as physical partition `p div K`. It looks like the node ID bijection and it's just as stateless. Both are stateless. Interleaving isn't saving anything the declared list hasn't already saved, because the router needs that list to know the routes anyway. Where they differ is growth: | | Ranges | Interleaved | |---|---|---| -| Logical 5 with K = 2 | constituent 0, partition 5 | constituent 1, partition 2 | -| Logical 5 after a third constituent joins (K = 3) | constituent 0, partition 5 | constituent 2, partition 1 | +| Virtual 5 with K = 2 | constituent 0, partition 5 | constituent 1, partition 2 | +| Virtual 5 after a third constituent joins (K = 3) | constituent 0, partition 5 | constituent 2, partition 1 | (Constituents are numbered from 0, and the ranges column assumes the first holds at least 6 partitions.) @@ -90,21 +88,21 @@ Both are stateless. Interleaving isn't saving anything the declared list hasn't ```mermaid graph TB subgraph Ranges - R1["logical 5, K = 2"] --> R2["constituent 0, partition 5"] - R3["logical 5, K = 3"] --> R4["constituent 0, partition 5"] + R1["virtual 5, K = 2"] --> R2["constituent 0, partition 5"] + R3["virtual 5, K = 3"] --> R4["constituent 0, partition 5"] end subgraph Interleaved - I1["logical 5, K = 2"] --> I2["constituent 1, partition 2"] - I3["logical 5, K = 3"] --> I4["constituent 2, partition 1"] + I1["virtual 5, K = 2"] --> I2["constituent 1, partition 2"] + I3["virtual 5, K = 3"] --> I4["constituent 2, partition 1"] end style I4 fill:#fdd,stroke:#a33,stroke-width:2px ``` {% endcomment %} -![Logical partition 5 under the two numbering schemes, before and after a third constituent joins. With ranges, it stays on partition 5 of constituent 0 both times. With interleaving, it is partition 2 of constituent 1 with two constituents, and partition 1 of constituent 2 with three, so the same logical number now reads from a different log.](/assets/blog/images/routing-series/partition-range-management.svg) +![Virtual partition 5 under the two numbering schemes, before and after a third constituent joins. With ranges, it stays on partition 5 of constituent 0 both times. With interleaving, it is partition 2 of constituent 1 with two constituents, and partition 1 of constituent 2 with three, so the same virtual number now reads from a different log.](/assets/blog/images/routing-series/partition-range-management.svg) -Interleaving moves every logical partition when K changes. The data hasn't moved. Committed offsets sit in `__consumer_offsets` keyed by the physical topic and partition, and earlier records were placed by an earlier hash. After the change the same logical number reads from a different log. In Post 2 the same formula was safe because node IDs are re-learned from the next `Metadata` response. Partition identity is stored, so it can't be re-learned. Ranges are only growable at the end, but at least they know what that means. +Interleaving moves every virtual partition when K changes. The data hasn't moved. Committed offsets sit in `__consumer_offsets` keyed by the physical topic and partition, and earlier records were placed by an earlier hash. After the change the same virtual number reads from a different log. In Post 2 the same formula was safe because node IDs are re-learned from the next `Metadata` response. Partition identity is stored, so it can't be re-learned. Ranges are only growable at the end, but at least they know what that means. -Re-numbering isn't a quirk of weaving. Changing the partition count of a normal Kafka topic is already a risky business: a key whose hash is 50 lands in carriage 18 of 32 and carriage 2 of 48. Weaving doesn't make that any worse, but it does make an inadvertent change much more likely. Changing the configuration is a much more serious activity than it appears. +Re-numbering isn't a quirk of weaving. Changing the partition count of a normal Kafka topic is already a risky business: a key whose hash is 50 lands in partition 18 of 32 and partition 2 of 48. Weaving doesn't make that any worse, but it does make an inadvertent change much more likely. Changing the configuration is a much more serious activity than it appears. ## Where the numbers appear @@ -112,8 +110,8 @@ Partition numbers turn up in nearly every RPC that names a partition. I'm sticki | Request | Where the partition number is | What the router does | |---|---|---| -| `Metadata` response | Each partition's index, and its leader as a node ID | Emits logical 0 to N-1. Sets each leader to a virtual node on the owning route, with an error for an unreachable constituent | -| `Produce` | The partition index under each topic | Maps logical to (route, physical topic, physical partition) and rewrites the response the other way | +| `Metadata` response | Each partition's index, and its leader as a node ID | Emits virtual 0 to N-1. Sets each leader to a virtual node on the owning route, with an error for an unreachable constituent | +| `Produce` | The partition index under each topic | Maps virtual to (route, physical topic, physical partition) and rewrites the response the other way | | `Fetch` | The partition under each topic, and the partitions to drop in the forgotten-topics list of an incremental fetch | The same translation. The session state is kept on both sides, as in the [topic routing proof of concept]({% post_url 2026-05-21-topic-routing %}) | | `ListOffsets` | The partition index | The same translation. Consumers use it to find the start or end of a partition | | `OffsetCommit`, `OffsetFetch` | The partition index under each topic | Splits by owning route, with the boss and standalone commits from Post 4 | @@ -132,7 +130,7 @@ Kafka gives each topic a UUID at creation, and a topic deleted and recreated und I'd treat the two sides differently: -- **Client side: derived.** The router computes the ID for a logical topic from its name and something that says which incarnation it is. A generation number alongside the layout is one way to do that. Every instance computes the same value, with nothing stored. Going the other way, from a UUID back to a topic, you can't invert a hash. The router builds a table at startup by computing the ID for every topic it declares. The result must avoid the all-zero UUID, which the protocol uses to mean "no ID". Kafka's own generator also skips a few reserved values and any ID whose string form contains a dash, and it would be sensible to follow suit. +- **Client side: derived.** The router computes the ID for a virtual topic from its name and something that says which incarnation it is. A generation number alongside the layout is one way to do that. Every instance computes the same value, with nothing stored. Going the other way, from a UUID back to a topic, you can't invert a hash. The router builds a table at startup by computing the ID for every topic it declares. The result must avoid the all-zero UUID, which the protocol uses to mean "no ID". Kafka's own generator also skips a few reserved values and any ID whose string form contains a dash, and it would be sensible to follow suit. - **Broker side: learned.** The physical ID comes from the backing cluster's `Metadata` response, discovered in-band as in Post 2. If it goes stale, the broker answers `UNKNOWN_TOPIC_ID` and the router refreshes. The broker's own protection against stale topics stays intact on this leg. {% comment %} diff --git a/assets/blog/images/routing-series/partition-range-management.svg b/assets/blog/images/routing-series/partition-range-management.svg index 8cd6c7e0..ee31b51a 100644 --- a/assets/blog/images/routing-series/partition-range-management.svg +++ b/assets/blog/images/routing-series/partition-range-management.svg @@ -1,2 +1,2 @@ InterleavedRangeslogical 5, K = 2constituent 0,partition 5logical 5, K = 3constituent 0,partition 5logical 5, K = 2constituent 1,partition 2logical 5, K = 3constituent 2,partition 1 \ No newline at end of file + @font-face { font-family: Excalifont; src: url(data:font/woff2;base64,d09GMgABAAAAABJ8AA4AAAAAH+AAABIoAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGhYbhVAcgUYGYACBJBEICq5MoiwLPgABNgIkA3gEIAWDGAcgG40YIwPBxgEEIXw72T8TbLr0CzxAGh7stGHTWLLYfPLfaAsa/zdE27wvnkiJVAHBSpzYRJqAMV2DVYhOFxHGqsJluariedr5+sUYWJ4EQd1NbEA40c3XZn3kCm9vVUXSyXZGOiFjkun5019ECHw5rZ/2cIdrcS1BIi1AvMdIujMiCtRK/ocDXL9fq6feLjslcolQCIkQ8fffyZ75O7GGahMNmaGU62gz8SahEBKNEkmJWqiUEjCRzdCw+XXWiYdinjfJASAAUOmgIwxCIgLRRQFAFkkD7aaMXCBa/T3VQLQ9JVVAdJ2NtUAEBQB432hySjy1gBgfQFABJxjhRCdWEsGlxXD0oA/aRsWEzP/zbqx5gYw3HlHPJrQ3ztwfERlwi9huHFxH7aQPTmB91C5YB4EPh2dU38hpkNWQUVDRMbB44eDi9Xz2MCEC3CYQVBEYEqOMgHiaHkNCQ0FGB0EImG6M6UhHADAEAOCuHACSg8wNoshDQF4o0E0gCMkKlYpaR3kIh25AEJhCJcKIYkB+dBhLOZ/RMkzibfR4imsB512mDaADAMq/ClFKXmIHKDWQ11ExSbmiVJx6orrkJENAIh1esZmwjfPIsCBKBaedxAGKQgj9gdxKNLlQS/2UmZACe7rOBpog2i/IC1m1YZizQAepptr0D8NqE9QkBRaGsr3puKSv3DgClA5pOapFJSWF7i9UjFjJrNI5OJUoU6GaW9OPzR3JLpLV8ny5ds1lnc7ocNDC2MjQQCwSBrkOqH9MYqaDkizBDg/fLSY+5aBYLkY1GNYOmFzQyeQykKcTJOdHFOTW+vGL8upiOLFJ9CBOvkii1At9ghmxmu7p5mhfMs+XKyezTbnWqGB3A5X3gp7KA464H6hjxYGUutBNK5aVYHmA4L8kBtu+hNDmcleIsjA5xZUrkqU8VZyfXmd6mhpPhXk5f3/7iYxPk+OELaV6Vrr4+i8YYVbwC3bV/n+1EKaPVsIb5fMqFxoPtuU4QcWpUU+SdL2oe42wFc5H6kh84VyFJr85vlzdunMOh0k4KLqgGGFMmLphEF5Z2v7g5tNvHirOAUqH3rSzUXEvvBo0JinI4CyrRY+LtIETGcdzlOf6yrW/4Dx+MK/f28eDDF+jGmUwU5yX5V5+5onBYsaQv6VxIcrBEyOW8Zf17UPBKn+4A6NZtng8LbqTwaMwBVFz9h297k07xXU9bdXnaH6OyWwb+ZdrDqRwMWtGD8eHDl3N3x/F9fB0eDqRcX20/7y537amDpBdttUCP8RVp5b31a1qOweiTtFaCcGnYf4ASQZ2AdErpCgRuLP6cNWZXZ8emWaVVE9NzEZLvbH2meaI1e30n9HaoghIeK9eZw/mMktP7/tv4ZnP5bhQtgNF0RQnUxaS5YJBKzm63+yrgSiLfUzGiUxkjOJldlPr15S+ysdqHyq8zzmFSjQtkjAkBGMZr2CwQmRMpHZaIetuyaJQNJ2IRhRSF4Q6BqeJnDTkenjvL4Rh68g33me7gOxZB4P6fL8tqlumeVnjq/2NIqm4lbY6FUjhTXP8i/hhrEaL0vDsWXLt2i3w6vCP0Xv09nVIF5Ba3fy8wn9jdOXjUX7wheGX/UDlzetN5+HOjja+VH9izGWs0w71pLRFM9ipuNdQsrRpCMM3+bivqpfKwdPaEN6vObBJlSzEzU4l6roDsgvCJAXSEEZZDULoSAeGJavjAtIK84OM+cy31Z64ucbVm4rTohEFMII0g45TCAkB4Rk2/W0g5Btpajja0w/uVu0eCtOV8Nnk8ODI8E84wXOUf7xt9u5+cFM0+4vhAc8rugCHDHURm4Pk1WGr7UHvqyjTTDqUrLNJCx9kyM/LBSFas6ykVs0q3wp6aiAKgUYX0NurW7qbFqXSCW+aQQ8QGddvGLdW0i/vGe3riqMKk8Ry/mlqbopPhKn24f1s70b6aSjjr+uutJF9MftG63/RL0nXTq+kZ1IjNyOF0shxlP7NckCz4N1Bo/iPJ0Ycj5a62uu/0+1hzVH7l4VoLPdoR59fyZe3xNpVxfnIApaMEfs83p00CCCE4qxgpDjiZVmsKtERiDppMdXbFw5PVsgfJH6C4m45uFqjnGK7kL5H2+DX8OAIsTxHfjlo0pLV8Tag1XYlghNqYcS2u/5mYUtUN8W+QHEOWBS2qLSeppKXdTolCshfQrxZlO9jXcRGMgapTtKiW3+/EAjVXmIFewydkidZbYmfP6/y8Vgdr/JoSkuZ1LI6kgdI2Howv7K9bDJbvWpeVZzso+x4qpMkfGA6QlSZ77MRymUMQkAGYLdI3I5EpzD6jRCpv8DcsHNb9ER56+03fcWpZa3Xejv9PMnfl61USltecfCHc0wYowef7ureO4tm9MfltZtfPHxnKFlgIOMYyUmjMQFkAtZL1CMu6db7uLFkdI1uOdB4BCmVrJbJA/X+e3Q2ix4qjsYDFOcoN9jIR921H4TpRE6Tnlynge+RJAwBceEiO5BJln4vPzf/y/wgu9Qt5/uDZR+FACdIvqtmPXVt5wu1MG4OSzSDtf7DfrM5qwxeHRF8Ggi0jXWERhoHqOY3ENQ4LZXWwWTw7EuS/AXHMWI+67LNQqrH8YjdQP6mLwq8UGD7+VjlM6Wipw4m0voRGlHFOSTMak/liqNwGsFhxQJtK/tzp0FamKHRnaP5sv9xt5yTZTzZyfVS1ip53hBSld/k5/k4UIPq5UO8RpXXjbXS39aBUgBClojxv2J/ieyCRzJ+U7APjftaf0exInY9/lSqn4QxU7l67e1rPfU9olsd4A70pD5Sg2+44nym0B1IZxEcFt2zJJ4jmSTJHxYdKo72zf70j3v5ydy0XyrOVHn8Hu3AqQ4madRK0xFL9vCpfKZLkGKkkUI/TMu8j941Uh3efm2UhHnV9NHn6p9jX4Hpzzc5EQlMEvZIGJmBRUlQCjtA7RtIO9qQzEUCOW4mu2EJ1COufJQGMAFB8i3TetevsmfSOAqZ3z20eWDcLB2FpWnMNUJKIuZQ2v6bZNdl9pCRG+fFuLbl9vRNvwhWBS0WySMiTs7YOm2YegLeC8+9/fsIELDYryED1OED5wdQCCFlkw3hISMbiH0X7fw5PCU0Z0tO0a/idowcG9mG38XtaDYFOWbtj4LJxBnvunLXdzHtgKe1LB0Hpk38o5387IN48Yr5p9p6+EE8SAnFSB1ajWqvMsLpSezMl8Y/2P7y2LeSwsHOpD3B72990Kqba7HWGd192M8Y0bUtaDvGPD6lzcQpomXAHjsIgxEj6sqv4D10IUpCav4414gC+WM7xyqMPNlbrbuwxIo7jY4JCYvLqJ5qcZritqTWgriEaRtSwh9+Tu7zsv3hJ46VuoTajVyxpzxRbRS4Oiie18bmR00VxJB8vVEE3srYK4oEhh3IsmvxxYucrP9/N0Prllym1+fkSIj5fv32fzCm/ejXzBeeuJyXxND+RAXBcBGFmomicK/N11nUmUuyyBZdedX30UJHDud1oHyX3YoFMNESGnYzIPT/kHAz00hCV+Ps9C2yO64ROVzgehpRkOfr8Aa2CsLnBSK28sSTRqiZpuAkxmgS7T5i5N2kLVGbmH0g90Au82Ev3bXOurlXbwfA6BX0AoHPK/FClDxkiioPFArjBrWWCXPDVrF2Gxn1lqd0ooHcSGdty1QRVSHOoRlcwMx3b14Qujg36mLLA4H1nuNqmXLLBwIbEUUjTD0TgQzQCOlrN1FDQOGTtmoaC2o7HLsGD5zgvOXhYJFIJMZ1+c4AwWC+v+3w9VV3+51nbddmJ2kCxhFu6BZ2OhZ6w+J18DLybCwDSjU0jbSNS+XndrdsGP9ONfxdhHG1MD0ff3HYVsnxFzSMfLvhsVxxBtUX5Oek/2a2yNaELuQ2XW5lgkEQ8WrsnWflebaGDNMvtuzYv5myTjVXevEfbbFyTcTg4DuhKq0X+2NCAZVUiy6HsfRYqZTN/YysdZL7EjfC5r2QQc5hNORp36NBzpjtpemYzc+GDffrFpZnjr0pYazb+vPVuGjKLR02nFAwJsVmTjbM9oIgaekwjBlkLAGHCY+1Av/JlTey87jBULBxd2ixvNXCyMZZFXPUZ3t0n8LP4su7CybTs9HfAAtxrTIdUahW87QB+vpcXU/WdWhQiQXph6vS0Jj2gq1utEIEetA6RPZPRczbvwim2qkjeL0DXmhOyEovpSpPdCCdX+M7p79s/Rio2gvrEpd9jv6dCW2pwjwqXNCvJ1jhFXXq1oafBHsyTsoeOSCL+jAtMAXBkf44IZJXnOXF7AUn7frgEM9UAZuSWqXwzVRonBFdJWJNms/rWJxT805YW8swW4+cUC1zb5fxv3cNTZ7OXQybSbklMSAg/6iitDLFniH4VNRRb0/1R+2FCI8bU33vFH2lK2Mke/iJ23qPUs55NMbGs2ejuiCDX7dXzVjDVy3ZG855EjOlnss5E122GHCa2Ka4bjVzNcIsJ99rr6HtuxwBt0aQMnt5oXZ4SpeoxLEJFnJsQfn3rDgzwCiRFKx5h9H291OVSW5ITLSuKd3Duq2Q6ji9fYlN85GVYxRArJ6X7VtBb5YWz70T+gJMPfTop7eUvGKEk82yOaZrxkUP8OsPxbO6ssYKzi4DNG8xbsMLcJH3S/9jF7g1dZfDiPIICI4IqgTD2fwT96DBAhpOppwUwZTzof15El94KIaYybQw2m58uhVCjNKNS2kDToCtMFnoVVrmpe3hv8HClxOIMm1crIMdnlW/2LDP10qZ7o1lIiSm/oP0SGUIPQ7Sb7nql1/nd0sjW1rZUrXFx5znZUcOQaJX0nJwBw2uOGD9jVZUBlds0AbOSk0bl3oixrQ+tWaw5GuJO1KaHpjPs6pNFO2iMkieGx7qe2wPK7fVQAwIoUDkmg9bqsA8j9t9K38/K8BMXkkEpSLI0YNkGPrhFNtIw9v7+kRa/lKHvKtNkd9yqOajJHiMDK9MKG6b3WXGgSQx12ox5W5QfqOkMxYOpg8AU/3a6QYOtllO5cfIadDSRfzlk7qP6M3BSKLJFg51lXnweDJbOQF4eLRJ8YnLH15KpiNWhAcbgNbIvqpUR7/2XrZZQ5Gcbo9tCluQqcsTaWNNrk4xoeZ5D0UOBsuQj1FbDU0WrmM0Zv+wre/v61dGf3+/Ne9jQZyu1yOcVS8bMgRw1Je6Kzt3KCkD/TmWfP3hGyTvza7bD/Tx7AJqh7psVzlvWXgA4i5VRt3arL20fC0Y7vxuYm7m9zQx1OTsjaJ/JarIz7at2lHG5opBZrQvZ0PHpbt2TZ+BcZm2eukQcItKwBO0418MLRE44JSB/X2ly7aMiY1Z1SqAjJQ8GiH8G1seNdyMSMNhhTIhwnHBFXiE5L0vrVSSFqg+tx5H0D5L6bKIEH81mkwzf/rjVQju4awju0+pewptG5v53sOBHJIBhe0riEb0wqqxY3ct+Pz4j1zQddF80bXuV57lEGZtlqIdGQz8gVrbz5Br7Dkp7q52BatYpo/UEnCg3+74oWxQ1V/mKscEyZv1c0fI7opHwFETDzLZsZoCF+s+CV9Lq21cEn+kKwUpojWR35N5k24tZc1Z4823fOZzvmMtzzv7LFriOlL4sHRYG69PQMtJA8QhhOrY4gKCaxIL3N+2rSgeoXRNH41fYHiNYo1RADfrvaSQG2TWTl2cyOcv/4M0In1sQGhaKNHEJdsYXK516LrbXShtXy9OT9WjMUgzmwwXXmhrzVFTnzQ9ZRaq0/0Ht5a45y0O8YuKuDHryPRh6gkAdmh9gfMF2iG9GUlfiRTkBQAAPOwfbAAAgEdLXg/8P+rfQPQVGg8AIAI4Bv6nyI15Mby/f7r/Z2a5PogLGKEWACAI+MOzQBK0FvDgfevt9QFCuBJIYX/gC8uAD1wCwuEHIA3WAH/ox6Pt9ARBsBCI4RAQAr5ggodABdNBPFwAIq3p4qB5IAr2XSwwQaAoTo1pgFZWkDgoD6oUKh/XEVBZGVAcaGBegOsGATUSAKBa50kD4TiggdFt0iCCjdGgFFwaTAoFEGj1ABj1U8SpWoVSdWo1CuP4D4zet8LJI/+CeDSwiJpKRQkXqUgiM1/Rn1u5U/aQHgPRGgIDevTwcYJSNyDtmlp9gxxmWU7PsbpSzyzoNXHrz0CFka08Jwg0gCBNokWuQXUAFztUGheKjSucmoUT0apWg6XxbQ0XUKLpJdBETYqFQ6Cg//9gAA==); }InterleavedRangesvirtual 5, K = 2constituent 0,partition 5virtual 5, K = 3constituent 0,partition 5virtual 5, K = 2constituent 1,partition 2virtual 5, K = 3constituent 2,partition 1 \ No newline at end of file From ca16c13dffd9ccdc67312d3dba82633c52e8ba35 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Tue, 3 Nov 2026 21:24:07 -0800 Subject: [PATCH 9/9] Tone down partition weaving in routing series and add sorting office imagery Signed-off-by: Sam Barker --- ...se-are-not-the-brokers-you-are-connecting-to.md | 2 ++ ...tion-weaving-stretching-the-seams-of-a-topic.md | 14 ++++++++++---- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/_posts/2026-10-12-these-are-not-the-brokers-you-are-connecting-to.md b/_posts/2026-10-12-these-are-not-the-brokers-you-are-connecting-to.md index 9052510a..363788f6 100644 --- a/_posts/2026-10-12-these-are-not-the-brokers-you-are-connecting-to.md +++ b/_posts/2026-10-12-these-are-not-the-brokers-you-are-connecting-to.md @@ -114,6 +114,8 @@ Instead, the proxy intercepts `OffsetCommit` and `OffsetFetch` requests, demuxes What the proxy has to do here is far more demanding than in any of the previous patterns: partition numbers must be rewritten consistently across every Kafka RPC that mentions them — and they get everywhere, as you might expect. That's the essential complexity of the pattern, not incidental overhead. It's a lot of surfaces to get right. The operational contract also shifts: both physical topics have to honour the mapping, and your router is now the source of truth for what `orders` means. Transactions don't cross cluster boundaries here either — same caveat as topic weaving, but worth repeating because with partition weaving it's easier to forget the seam is there. +Partition weaving is workable, with a lot of effort. The dedicated post goes into what's involved, and you can judge whether it's worth it. + --- These four patterns give us a working vocabulary for the rest of the series. The posts that follow take each one in turn — what it can do, what it can't, and why. Some of the limits are ours and will close over time. Some are the Kafka protocol's: transactions have no external coordinator, and no amount of clever routing changes that. And some would require the proxy to grow a consensus layer of its own — which is a whole different animal, and not one we're planning to domesticate any time soon. diff --git a/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md b/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md index f525f124..201795bd 100644 --- a/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md +++ b/_posts/2026-11-03-partition-weaving-stretching-the-seams-of-a-topic.md @@ -13,6 +13,8 @@ This is the fifth post in a series about routing in Kroxylicious, and it covers Partition weaving is a cautionary tale. My first reaction was "that's a really neat idea". The more I looked at it, the harder it got. It is clearly part of the design space the Routing API opens up, which is why it's in the series, and it may well be possible. But it is a lot of effort to go to, and I haven't found a use case that justifies it to me. If you have one, I'd like to hear about it. +Back at the sorting office from Post 4, the sorter picked a chute from the label on the parcel. Here every parcel carries the same label, `orders`, and the sorter has to read the house number as well. That number is the one thing it can never reassign once it's printed. + ## One topic, two clusters Take the scenario from Post 1: a risk engine in us-east, hardcoded to read a topic called `orders`, and an EU team producing to `orders-eu` on a local cluster. The router presents both as one `orders` topic with 32 partitions. Partitions 0 to 15 are the `orders` partitions on us-east. Partitions 16 to 31 are partitions 0 to 15 of `orders-eu` on eu-west. A consumer that asks for partition 20 gets partition 4 of `orders-eu` and never knows. I'll call the numbers the client sees *virtual*, in the same sense as a virtual cluster, and the partitions and topics that actually exist on a backing cluster *physical*. Partition 20 is virtual, and partition 4 of `orders-eu` is the physical partition behind it. @@ -53,7 +55,7 @@ The best case I have is a migration, for moving a very large topic between clust **Data residency by key.** This is the one people will suggest, and it fits worst. The default partitioner hashes keys without knowing where anything lives, so residency would need a custom partitioner that sends EU keys to EU partitions. Producers that already know the layout could write to separate topics, and topic weaving would stitch those together without any of this. -That's the list. If you have a better one, tell me. +That's everything I've found, and I'd be glad to be wrong about it. If you have another idea, I'd like to hear about it. ## The layout is declared, not discovered @@ -73,9 +75,13 @@ Changing it is where the care goes. The rule is that a virtual partition number The first is **ranges**, which is what the picture shows. Constituent *i* starts at the sum of the partition counts before it. A virtual partition `p` falls in the range `[start_i, start_i + count_i)` and maps to physical partition `p - start_i`. The inverse adds `start_i` back. In the example, virtual 20 is `eu-west` / `orders-eu` partition 4. Constituents can have different counts, and appending to the end leaves everything before it alone. -The second is **interleaving**, and it's the tempting one, because it's the arithmetic from Post 2. With K constituents of equal size, virtual `p` goes to constituent `p mod K` as physical partition `p div K`. It looks like the node ID bijection and it's just as stateless. +The second is **interleaving**, and it's the tempting one, because it's the same arithmetic we use to map broker IDs to virtual node IDs. With K constituents of equal size, virtual `p` goes to constituent `p mod K` as physical partition `p div K`. + +Both are stateless. Interleaving isn't saving anything the declared list hasn't already saved, because the router needs that list to know the routes anyway. + +Think of numbering the houses on a street. Ranges build new houses at the far end, so number 5 is still the same house. Interleaving renumbers every house whenever one is built, but the letters already on the doormat stay where they were, and the post, addressed by number, now arrives at the wrong door. -Both are stateless. Interleaving isn't saving anything the declared list hasn't already saved, because the router needs that list to know the routes anyway. Where they differ is growth: +The numbers say the same thing: | | Ranges | Interleaved | |---|---|---| @@ -159,7 +165,7 @@ The alternative is to avoid IDs altogether. The proposal notes that a router can A producer writing through the weave picks a partition from 0 to 31 as it would for any topic. The router sends it wherever that number lives. For keyed records the hash decides, so a producer in eu-west has no reason to land on the eu-west partitions. For unkeyed records the client spreads load across available partitions, which includes the ones across the ocean. Writing through the weave gives you no locality unless something outside the router arranges it. -Arranging it means a custom partitioner that knows which numbers are local, and that puts the layout the router hides back into the client. If you can change a producer's `partitioner.class`, you could change its topic name as well. So the producers that gain from partition weaving are, in practice, the ones that don't use it. In Post 1's scenario the EU team writes straight to `orders-eu`, and the weave serves readers only. +Arranging it means a custom partitioner that knows which numbers are local, and that puts the layout the router hides back into the client. The weave exists for the consumer you can't touch, the vendor's risk engine, while the producers are yours to change. If you can change a producer's `partitioner.class`, you could change its topic name as well. So the producers that gain from partition weaving are, in practice, the ones that don't use it. In Post 1's scenario the EU team writes straight to `orders-eu`, and the weave serves readers only. ## Consuming