From c240bb6f5953e643d919fd1ef5454e61196bc2d5 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Wed, 28 Oct 2026 17:25:04 +1300 Subject: [PATCH 01/21] post 4: stub with section outline for topic weaving post Signed-off-by: Sam Barker --- ...-topic-weaving-one-client-many-clusters.md | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 _posts/2026-11-03-topic-weaving-one-client-many-clusters.md diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md new file mode 100644 index 00000000..2fea688e --- /dev/null +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -0,0 +1,41 @@ +--- +layout: post +title: "Topic Weaving: One client, many clusters" +date: 2026-11-03 15:00:00 +1200 +author: "Sam Barker" +author_url: "https://github.com/sambarker" +# noinspection YAMLSchemaValidation +categories: blog kroxylicious-proxy +tags: [ "routing" ] +--- + +[Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API and named the patterns. [Post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) covered **connection switching** — the router picks a cluster at connect time. [Post 3]({% post_url 2026-10-26-stream-branching-mind-the-gap %}) covered **stream branching** — the router fans traffic across multiple clusters, but the client only knows about one of them. + +This is the fourth post: **topic weaving**. The client knows about all of the clusters. It isn't running a simple point-to-point stream anymore — it's producing to and consuming from topics that are scattered across several physical clusters, and it knows some of those clusters exist. What the proxy supplies is the illusion that those clusters are one coherent thing: a single set of topic names, a single set of offsets, a single authentication handshake. + +## TODO + +- [ ] Introduction: what the sorting office does that the railway station didn't +- [ ] Refresher: how the existing patterns are different (what the client knows, what it doesn't) +- [ ] The topic map: how the router decides which topic lives where +- [ ] The client's view: Metadata, ListTopics, and the virtual namespace +- [ ] Producing: why node ID translation isn't enough, and why PID translation is hard + - Pull in stash from `stash-replica-routing.md` — "why PIDs aren't node IDs" +- [ ] Consuming: the simpler side — passing upstream PIDs through, `read_committed` caveats +- [ ] Transactions: the coordinator lives on one cluster, and that's all + - Reference Post 3's transactions section; explain how weaving makes it worse +- [ ] KIP-939 and why it isn't magic + - Link/summarise the Gemini doc before quoting +- [ ] Proxy placement: run it in the client's network/trust zone, not co-located with any single cluster, so a cluster failure doesn't take the proxy with it +- [ ] When to use weaving vs replication vs connection switching +- [ ] Closing: what's next in the series + +--- + +*Stash and working notes follow below — remove before publishing.* + +--- + + + + From e9695536c6ed4f0a9b65fc53b33fceb886d773e9 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Wed, 28 Oct 2026 17:37:42 +1300 Subject: [PATCH 02/21] post 4: draft intro for topic weaving Signed-off-by: Sam Barker --- .../2026-11-03-topic-weaving-one-client-many-clusters.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index 2fea688e..6cfe41d3 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -9,13 +9,15 @@ categories: blog kroxylicious-proxy tags: [ "routing" ] --- -[Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API and named the patterns. [Post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) covered **connection switching** — the router picks a cluster at connect time. [Post 3]({% post_url 2026-10-26-stream-branching-mind-the-gap %}) covered **stream branching** — the router fans traffic across multiple clusters, but the client only knows about one of them. +[Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API. [Post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) covered **connection switching** — one client, one cluster, the proxy just picks which one and can change it without the client noticing. [Post 3]({% post_url 2026-10-26-stream-branching-mind-the-gap %}) covered **stream branching** — one client, several clusters, but only one of them is the primary; the rest are shadows the client never directly addresses. -This is the fourth post: **topic weaving**. The client knows about all of the clusters. It isn't running a simple point-to-point stream anymore — it's producing to and consuming from topics that are scattered across several physical clusters, and it knows some of those clusters exist. What the proxy supplies is the illusion that those clusters are one coherent thing: a single set of topic names, a single set of offsets, a single authentication handshake. +This post is about the third pattern: **topic weaving**. The client is still talking to a single Virtual Kafka Cluster, but what lives behind it is a virtual topic namespace stitched together from topics on several real clusters. Topic A might live on cluster 1, topic B on cluster 2. The router holds the map, fans produce and fetch traffic out to the right cluster, and merges the results back into responses that look like they came from one coherent namespace. + +The railway has served us well, but topic weaving needs a different frame. Topic weaving is the sorting office: parcels arrive addressed to a virtual destination and leave down the right chute for their actual one. Unlike stream branching, every cluster on the map accepts produce and fetch requests — there's no shadow, no best-effort copy. That's what gives it a different set of sharp edges. ## TODO -- [ ] Introduction: what the sorting office does that the railway station didn't +- [x] Introduction: what the sorting office does that the railway station didn't - [ ] Refresher: how the existing patterns are different (what the client knows, what it doesn't) - [ ] The topic map: how the router decides which topic lives where - [ ] The client's view: Metadata, ListTopics, and the virtual namespace From 7fc8de18d9188fcf7eccdea7b16d11c85686caee Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Thu, 29 Oct 2026 09:27:10 +1300 Subject: [PATCH 03/21] post 4: refine intro Signed-off-by: Sam Barker --- .../2026-11-03-topic-weaving-one-client-many-clusters.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index 6cfe41d3..e9e2a04e 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -9,11 +9,13 @@ categories: blog kroxylicious-proxy tags: [ "routing" ] --- -[Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API. [Post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) covered **connection switching** — one client, one cluster, the proxy just picks which one and can change it without the client noticing. [Post 3]({% post_url 2026-10-26-stream-branching-mind-the-gap %}) covered **stream branching** — one client, several clusters, but only one of them is the primary; the rest are shadows the client never directly addresses. +[Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API. [Post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) covered **connection switching** — one client, one cluster, the proxy just picks which one. [Post 3]({% post_url 2026-10-26-stream-branching-mind-the-gap %}) covered **stream branching** — one client, one primary cluster, with the router silently fanning writes to shadow clusters the client never knows about. -This post is about the third pattern: **topic weaving**. The client is still talking to a single Virtual Kafka Cluster, but what lives behind it is a virtual topic namespace stitched together from topics on several real clusters. Topic A might live on cluster 1, topic B on cluster 2. The router holds the map, fans produce and fetch traffic out to the right cluster, and merges the results back into responses that look like they came from one coherent namespace. +This post is about the third pattern: **topic weaving**. Connection switching and stream branching both treat the cluster as the unit of routing — you're deciding *where* traffic lands, not *which topics* the client is allowed to see. Topic weaving is different. The client still connects to a single Virtual Kafka Cluster, but the namespace behind it is stitched together from topics distributed across several real clusters. Topic A lives on cluster 1, topic B lives on cluster 2; the router holds the map and the client never has to know. -The railway has served us well, but topic weaving needs a different frame. Topic weaving is the sorting office: parcels arrive addressed to a virtual destination and leave down the right chute for their actual one. Unlike stream branching, every cluster on the map accepts produce and fetch requests — there's no shadow, no best-effort copy. That's what gives it a different set of sharp edges. +Kafka clusters accumulate. Teams get their own cluster for isolation, another spins up for a different region, a third appears because somebody hit a partition limit and it was quicker to provision hardware than fight for headroom. The data teams feel it first: every pipeline that crosses a cluster boundary needs its own bootstrap config, its own credentials, its own mental model of which topic lives where. A data engineer building a join across two domains has to know which physical cluster each topic lives on — not because it's relevant to their problem, but because Kafka makes it unavoidable. Topic weaving moves that knowledge into a routing layer where the platform team can encode it once — and control which applications can see which topics while they're at it. + +The railway metaphor has served us well but it runs out of track here. Think of a sorting office instead. Parcels arrive addressed to a virtual destination — `orders`, `payments`, `inventory` — and the sorter routes each one down the right chute to its actual cluster. Every backing cluster is a peer — every write counts. That's also why there are sharp edges. A transaction requires a coordinator, and a coordinator can only see its own cluster. If you need to write atomically to `orders` and `inventory` and they live on different clusters, the sorting office can't help you. ## TODO From 5f8d7896d2dbf896fa2932af68a0db2343e05268 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Thu, 29 Oct 2026 13:31:48 +1300 Subject: [PATCH 04/21] post 4: ramp into topic weaving and add client-knowledge section Signed-off-by: Sam Barker --- ...-topic-weaving-one-client-many-clusters.md | 26 ++++++++++++++++--- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index e9e2a04e..d7c828a7 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -9,18 +9,36 @@ categories: blog kroxylicious-proxy tags: [ "routing" ] --- -[Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API. [Post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) covered **connection switching** — one client, one cluster, the proxy just picks which one. [Post 3]({% post_url 2026-10-26-stream-branching-mind-the-gap %}) covered **stream branching** — one client, one primary cluster, with the router silently fanning writes to shadow clusters the client never knows about. +This is the fourth post in a series about routing in Kroxylicious. [Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API and named four patterns for using it: [connection switching]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}), [stream branching]({% post_url 2026-10-26-stream-branching-mind-the-gap %}), topic weaving and partition weaving. This post covers the third, **topic weaving**. -This post is about the third pattern: **topic weaving**. Connection switching and stream branching both treat the cluster as the unit of routing — you're deciding *where* traffic lands, not *which topics* the client is allowed to see. Topic weaving is different. The client still connects to a single Virtual Kafka Cluster, but the namespace behind it is stitched together from topics distributed across several real clusters. Topic A lives on cluster 1, topic B lives on cluster 2; the router holds the map and the client never has to know. +Kafka clusters accumulate. Teams get their own cluster for isolation, another spins up for a different region, a third appears because somebody hit a partition limit and it was quicker to provision hardware than fight for headroom. The data teams feel it first: every pipeline that crosses a cluster boundary needs its own bootstrap config, its own credentials, its own mental model of which topic lives where. A data engineer building a join across two domains has to know which physical cluster each topic lives on — not because it's relevant to their problem, but because Kafka makes it unavoidable. -Kafka clusters accumulate. Teams get their own cluster for isolation, another spins up for a different region, a third appears because somebody hit a partition limit and it was quicker to provision hardware than fight for headroom. The data teams feel it first: every pipeline that crosses a cluster boundary needs its own bootstrap config, its own credentials, its own mental model of which topic lives where. A data engineer building a join across two domains has to know which physical cluster each topic lives on — not because it's relevant to their problem, but because Kafka makes it unavoidable. Topic weaving moves that knowledge into a routing layer where the platform team can encode it once — and control which applications can see which topics while they're at it. +## A cluster is whatever the metadata says it is + +A Kafka client starts with a list of bootstrap addresses and picks one to talk to. It sends a `Metadata` request there and gets back a list of brokers, the topics it asked about, and for every partition of those topics, the broker that leads it. From then on it connects directly to those brokers and sends each request to the right one. Everything the client believes about the cluster comes from that one response. + +The proxy sits in the middle and speaks Kafka, so it gets to write that response. A Virtual Kafka Cluster (VKC) is the stable address clients connect to, and until recently all the proxy did with the response was swap the broker addresses for its own. The brokers and topics still came from one real cluster. The Routing API changes who answers. It lets a router, a component you plug into the proxy, decide where each request goes and build the response from whatever it likes. The proxy supplies the plumbing; the router supplies the policy. Nothing says the answers have to come from the same cluster, and the client takes the response at face value. + +That's topic weaving. The client connects to a single VKC, but the namespace behind it is stitched together from topics on several real clusters. Topic A lives on cluster 1, topic B lives on cluster 2, and the router holds the map. Connection switching and stream branching both treat the cluster as the unit of routing: you're deciding *where* traffic lands. Weaving decides *which topics* exist, where they live, and which applications can see them. The platform team encodes that once, in the routing layer, instead of every pipeline carrying its own copy. The railway metaphor has served us well but it runs out of track here. Think of a sorting office instead. Parcels arrive addressed to a virtual destination — `orders`, `payments`, `inventory` — and the sorter routes each one down the right chute to its actual cluster. Every backing cluster is a peer — every write counts. That's also why there are sharp edges. A transaction requires a coordinator, and a coordinator can only see its own cluster. If you need to write atomically to `orders` and `inventory` and they live on different clusters, the sorting office can't help you. +## What the client knows + +A VKC already keeps the real cluster from the client: clients dial the proxy, never the brokers. What the patterns change is what stands behind that address, and how much of it the client can see. + +**Connection switching** makes the choice of cluster dynamic. The client still sees one cluster. The proxy can change which real one is behind it. + +**Stream branching** adds connections the client never sees. They may go to other clusters, or to another topic or broker on the same one. The client has one stream, and nothing is allowed to break that. + +**Topic weaving** hides the *seams*. The `MetadataResponse` lists brokers from every backing cluster. The client connects to all of them (via the proxy!), sends them requests, and takes their answers at face value. It just believes they belong to one cluster. + +The client really is talking to several clusters. Some of what it carries with it, like a producer ID or an open transaction, was issued by one cluster and means nothing to the others. How much that matters depends on what you're doing with it. + ## TODO - [x] Introduction: what the sorting office does that the railway station didn't -- [ ] Refresher: how the existing patterns are different (what the client knows, what it doesn't) +- [x] Refresher: how the existing patterns are different (what the client knows, what it doesn't) - [ ] The topic map: how the router decides which topic lives where - [ ] The client's view: Metadata, ListTopics, and the virtual namespace - [ ] Producing: why node ID translation isn't enough, and why PID translation is hard From f69c8dfce4b234e675ccd437bd207e7ff5fc7388 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Thu, 29 Oct 2026 17:45:41 +1300 Subject: [PATCH 05/21] post 4: add transactions and KIP-939 sections Signed-off-by: Sam Barker --- ...-topic-weaving-one-client-many-clusters.md | 41 +++++++++++++++++-- 1 file changed, 37 insertions(+), 4 deletions(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index d7c828a7..d5806661 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -1,7 +1,7 @@ --- layout: post title: "Topic Weaving: One client, many clusters" -date: 2026-11-03 15:00:00 +1200 +date: 2026-11-03 15:00:00 +1300 author: "Sam Barker" author_url: "https://github.com/sambarker" # noinspection YAMLSchemaValidation @@ -15,7 +15,7 @@ Kafka clusters accumulate. Teams get their own cluster for isolation, another sp ## A cluster is whatever the metadata says it is -A Kafka client starts with a list of bootstrap addresses and picks one to talk to. It sends a `Metadata` request there and gets back a list of brokers, the topics it asked about, and for every partition of those topics, the broker that leads it. From then on it connects directly to those brokers and sends each request to the right one. Everything the client believes about the cluster comes from that one response. +A Kafka client starts with a list of bootstrap addresses and picks one to talk to. It sends a `Metadata` request there and gets back a list of brokers, the topics it asked about, and for every partition of those topics, the broker that leads it. From then on it connects directly to those brokers and sends each request to the right one. Everything the client believes about the cluster comes from `Metadata` responses. The proxy sits in the middle and speaks Kafka, so it gets to write that response. A Virtual Kafka Cluster (VKC) is the stable address clients connect to, and until recently all the proxy did with the response was swap the broker addresses for its own. The brokers and topics still came from one real cluster. The Routing API changes who answers. It lets a router, a component you plug into the proxy, decide where each request goes and build the response from whatever it likes. The proxy supplies the plumbing; the router supplies the policy. Nothing says the answers have to come from the same cluster, and the client takes the response at face value. @@ -35,6 +35,39 @@ A VKC already keeps the real cluster from the client: clients dial the proxy, ne The client really is talking to several clusters. Some of what it carries with it, like a producer ID or an open transaction, was issued by one cluster and means nothing to the others. How much that matters depends on what you're doing with it. +## Transactions stop at the cluster boundary + +A transactional producer starts by sending `FindCoordinator` for its `transactional.id`, then `InitProducerId` to whichever broker the answer names. From then on that coordinator tracks the transaction, and when the producer commits it writes the commit or abort markers to the partitions involved. The coordinator, its transaction log and those markers all belong to one cluster. + +A coordinator can only see its own cluster, so the obvious answer is to give the client one everywhere. The router fans `InitProducerId` out to every backing cluster and starts a transaction on each. As the client registers partitions and produces, the router sends each call to the cluster that owns the topic, using the producer ID and epoch that cluster issued. The client holds one producer ID and the router holds the mapping to the rest. That mapping has to survive a router restart and be visible to every proxy instance, so it needs a state store. That's real work, but it's work you can do. + +It falls over at commit. The client sends one `EndTxn` and the router has to turn it into one per cluster. On each cluster, commit is a point of no return: once the coordinator has written its decision, it will commit, whatever else happens. Until then it is free to refuse, and each cluster decides for itself. Cluster 1 can pass the point of no return while cluster 2 says no. None of the ways cluster 2 can say no are transient. The transaction may have timed out, since each cluster runs its own clock from the first partition added there. A zombie may have bumped the producer epoch. The ACLs may differ. Then `orders` has its records and `inventory` doesn't, and whatever the client is told is wrong for one of them. + +A state store doesn't rescue this. It can remember that the router meant to commit cluster 2, but it can't make cluster 2 agree. Reordering the commits doesn't help either, because whichever goes first can't be undone. + +So stop negotiating and pick a boss. One cluster is the primary, and its coordinator runs every transaction. The client's `InitProducerId`, `AddPartitionsToTxn` and `EndTxn` all go there, while the records still go to whichever cluster owns the topic. Commit is now a single decision on a single coordinator, so the atomicity problem is gone. + +It doesn't survive the first write to another cluster. The client registers each partition with the coordinator before it writes to it, and the primary's coordinator doesn't recognise `inventory`'s partitions, because they aren't in its cluster. It refuses them, and the transaction never gets as far as commit. + +Both attempts fail for the same reason. Somewhere there has to be a moment when every cluster has promised to commit before any of them does, and Kafka's transaction protocol has no such step. + +## KIP-939 might add the missing step + +[KIP-939](https://cwiki.apache.org/confluence/display/KAFKA/KIP-939%3A+Support+Participation+in+2PC) is the obvious objection. It's accepted, it's about two-phase commit, and it has not landed in Kafka yet. It lets Kafka take part in somebody else's two-phase commit: a producer can prepare a transaction, and a prepared transaction isn't aborted by the timeout while it waits for an external coordinator to decide. That is the promise our two attempts were missing. + +In principle the router could be that coordinator. It would start a transaction on each backing cluster, wait for all of them to prepare, record the decision, and then tell each one to commit. The client would still believe it was talking to a single coordinator. The hard part is the middle step. Someone has to durably record the decision between "everyone prepared" and "everyone committed", and survive a crash in the gap. A router that does that needs a durable log of its own, recovery after a restart, and a way to stop two proxy instances disagreeing about the outcome. That is the consensus layer from [post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}), and the Kafka community has just spent years moving from ZooKeeper to KRaft so that nobody has to run a second consensus system alongside Kafka. Asking a proxy to bring its own would be a step backwards. + +That said, this is where a boss cluster might earn its keep. Pick one cluster to hold the router's state, and keep the decision log in a compacted topic on it. Kafka already gives you durable, replicated storage there, with nothing new to deploy. The recovery logic is still yours to write. We haven't investigated this design in any depth, so treat it as a direction rather than a recipe. + +That gets harder still when the client is already a coordinator. A stream processor like Flink can run its own two-phase commit across Kafka and other systems. Put a weaving router in front of Kafka and Flink sees one participant, while the router has to act as a coordinator for the clusters behind it: two-phase commit nested inside two-phase commit. The KIP describes a flat arrangement of one coordinator and its participants, and says nothing about a participant in the middle. + +## What to do today + +The router can at least fail loudly. It has to answer `FindCoordinator` before it knows what the transaction will touch, so it picks using something it does know, like the `transactional.id` or the authenticated principal (the [topic routing proof of concept]({% post_url 2026-05-21-topic-routing %}) used the principal). It then rejects a write that strays onto a different cluster. An error in development beats a half-committed transaction in production. + +Does it matter? We don't know what your business does with Kafka. Transactions are fine within a cluster, so this is not a blanket ban on using them. The question is whether any single transaction needs to touch topics that live on different clusters. If your router keeps `orders` and `inventory` together, a transaction across them should work as it always did. But transactions may be more pervasive than a first glance suggests. Kafka Streams uses them to provide exactly-once semantics, so an application that never mentions a transaction can still be running one. If that sounds like your estate, check where the topics it reads and writes live before you weave. + + ## TODO - [x] Introduction: what the sorting office does that the railway station didn't @@ -44,9 +77,9 @@ The client really is talking to several clusters. Some of what it carries with i - [ ] Producing: why node ID translation isn't enough, and why PID translation is hard - Pull in stash from `stash-replica-routing.md` — "why PIDs aren't node IDs" - [ ] Consuming: the simpler side — passing upstream PIDs through, `read_committed` caveats -- [ ] Transactions: the coordinator lives on one cluster, and that's all +- [x] Transactions: the coordinator lives on one cluster, and that's all - Reference Post 3's transactions section; explain how weaving makes it worse -- [ ] KIP-939 and why it isn't magic +- [x] KIP-939 and why it isn't magic - Link/summarise the Gemini doc before quoting - [ ] Proxy placement: run it in the client's network/trust zone, not co-located with any single cluster, so a cluster failure doesn't take the proxy with it - [ ] When to use weaving vs replication vs connection switching From d7230370c575c9eaf3addda4c8d071bfe6bf4f53 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Sat, 31 Oct 2026 16:30:50 +1300 Subject: [PATCH 06/21] post 4: add 'Whose map is it anyway?' and router jobs sections Signed-off-by: Sam Barker --- ...-topic-weaving-one-client-many-clusters.md | 37 ++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index d5806661..b375d50f 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -35,6 +35,41 @@ A VKC already keeps the real cluster from the client: clients dial the proxy, ne The client really is talking to several clusters. Some of what it carries with it, like a producer ID or an open transaction, was issued by one cluster and means nothing to the others. How much that matters depends on what you're doing with it. +## Whose map is it anyway? + +All of this hangs on one fact: every router owns a map, the `Metadata` response the client ends up seeing. The proxy supplies the plumbing to send a request down a route and get the response back, and that's all it does. It doesn't merge anything. Merging, splitting a request across clusters and stitching the replies back together are the router's job, and so is deciding what the map says. + +That's true of every router, not just a weaving one. A connection-switching router owns a map too, because it decides which cluster's answer the client sees. Routers can also be stacked in the DAG, so a router further along may change the map again. Each router is responsible for the changes it makes. + +The map is also written for whoever is asking. Nothing says two clients get the same `Metadata` response. The router can give each identity its own view, as long as every proxy instance gives that identity the same answer. If two instances disagree, topics appear and disappear depending on who answered. It also means the router has to know who is asking, so this depends on authentication being in place. + +### Where the map comes from + +The simplest source is a table you write down: `orders` is visible from cluster A, `payments` from cluster B. That's a static, client-facing map. The config says what *should* be visible from each backing cluster, but it can't invent a topic. If `payments` doesn't exist on B, it doesn't appear, because the router can only show what the cluster actually has. The trouble is that it's static. Clusters drift. Somebody creates a topic the map has never heard of and it stays invisible until a human edits the config. Somebody deletes one and it quietly vanishes from the client's view. + +A rule lets the map follow the clusters instead. The simplest is a prefixing router. Say each of the clusters has an `orders` topic. The router gives every cluster a prefix and applies it to whatever that cluster reports, so the client sees `london_orders`, `tokyo_orders` and `newyork_orders`. Nobody has to tell it about a new topic, because it never had a list. The rule has to be deterministic, though: `orders` from the "London" cluster must be `london_orders` today, tomorrow, after every restart, and on every proxy instance. + +The rule has to run in both directions. The prefix goes on in `Metadata` responses. It comes off again in every request that names a topic, and the router uses it to pick the cluster. The brokers never hear of `london_orders`. The catch is that the topic name now says which cluster it lives on, which is a little of what weaving was meant to hide. Sometimes that's exactly what you want. + +A catalogue service is the same idea with the rule replaced by a lookup. It says which physical topic on which cluster is known to an identity as what, so the names don't have to follow a pattern at all, and it can change at runtime. Either way the router translates in both directions. + +### Who sees what + +Whatever the source, the router still decides what ends up in the response. When two clusters both have an `orders`, it can include only cluster A's, include both with a prefix, or exclude both. That is policy, and it belongs to the router. + +Because the map is per identity, the same machinery covers RBAC, tenant-specific rules and filtering. A topic can be visible to one identity and absent for another. What no map can do is move a topic. It only says where one is. + +## What a weaving router has to do + +Whichever way the map is built, the router has three jobs: + +- **Metadata.** Ask every backing cluster, keep the topics the map shows this identity, and return the lot as one response. +- **Produce and fetch.** Group the topics in a request by the cluster they belong to, send the pieces in parallel, and merge the replies into one. +- **Enforce the map.** Apply the same rules to every request that names a topic, not just `Metadata`. Hiding a topic from a response isn't enough if a `Produce` can still name it. + +This section describes what a router for this pattern has to do, not a list of features you can switch on today. + + ## Transactions stop at the cluster boundary A transactional producer starts by sending `FindCoordinator` for its `transactional.id`, then `InitProducerId` to whichever broker the answer names. From then on that coordinator tracks the transaction, and when the producer commits it writes the commit or abort markers to the partitions involved. The coordinator, its transaction log and those markers all belong to one cluster. @@ -72,7 +107,7 @@ Does it matter? We don't know what your business does with Kafka. Transactions a - [x] Introduction: what the sorting office does that the railway station didn't - [x] Refresher: how the existing patterns are different (what the client knows, what it doesn't) -- [ ] The topic map: how the router decides which topic lives where +- [x] The topic map: how the router decides which topic lives where - [ ] The client's view: Metadata, ListTopics, and the virtual namespace - [ ] Producing: why node ID translation isn't enough, and why PID translation is hard - Pull in stash from `stash-replica-routing.md` — "why PIDs aren't node IDs" From 40e98aab2aa38d293dcfeee294b57fba096f530c Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Sun, 1 Nov 2026 15:07:47 +1300 Subject: [PATCH 07/21] post 4: add broker ID collision section Signed-off-by: Sam Barker --- ...-03-topic-weaving-one-client-many-clusters.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index b375d50f..7f1f22bd 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -59,6 +59,18 @@ Whatever the source, the router still decides what ends up in the response. When Because the map is per identity, the same machinery covers RBAC, tenant-specific rules and filtering. A topic can be visible to one identity and absent for another. What no map can do is move a topic. It only says where one is. +## Broker IDs collide too + +Topic names aren't the only thing two clusters can have in common. Nearly every Kafka cluster numbers its brokers from 0 or 1, so cluster A has a broker 2 and so does cluster B. It's the Platform 2 problem from [post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) again, and it's no less fatal here: put both in one `MetadataResponse` and the client has two different brokers answering to the same number. Broker 2 is Broker 2, remember. + +The good news is that nobody writing a router has to deal with it. The runtime does, at every route boundary, with the same bijection as before: `downstreamId = (upstreamId * numRoutes) + routeIndex`. Broker 2 on route 0 becomes 4, broker 2 on route 1 becomes 5, and requests are mapped back again on their way upstream. A router never sees the formula, only the IDs on its side of the boundary. + +That last part is what allows us to compose routers into a DAG, and thus manage the complexity. What a router calls node 4 might be a real broker, or it might be a virtual node presented by another router further along. It neither knows nor cares. The IDs the client uses are the downstream side of the first router it meets; everything upstream of that is somebody else's business, with its own mapping at its own boundary. They compose without anyone having to coordinate. + +For weaving, that makes merging a non-event. Each route's `Metadata` arrives already mapped into the router's ID space, so the broker lists can be concatenated without a thought for collisions. It's a bigger deal than it was for connection switching, where the client only ever saw one cluster's brokers at a time. Here every cluster's brokers are live at once. The constraint from post 2 comes along too: change the number of routes and every ID moves. + +Node IDs are the easy case. They're a pure function of where a broker lives, so the runtime can translate them with arithmetic and no memory. Not everything the client carries around is that obliging. + ## What a weaving router has to do Whichever way the map is built, the router has three jobs: @@ -108,7 +120,9 @@ Does it matter? We don't know what your business does with Kafka. Transactions a - [x] Introduction: what the sorting office does that the railway station didn't - [x] Refresher: how the existing patterns are different (what the client knows, what it doesn't) - [x] The topic map: how the router decides which topic lives where -- [ ] The client's view: Metadata, ListTopics, and the virtual namespace +- [x] Broker ID collision: recap the post 2 formula, why weaving needs it permanently +- [x] The client's view: folded into the map and "What the client knows" sections (ListTopics is just unfiltered Metadata) +- [ ] Admin operations that name topics (CreateTopics, DeleteTopics, DescribeConfigs): which cluster owns a new topic? Add to the router's jobs or "Who sees what" - [ ] Producing: why node ID translation isn't enough, and why PID translation is hard - Pull in stash from `stash-replica-routing.md` — "why PIDs aren't node IDs" - [ ] Consuming: the simpler side — passing upstream PIDs through, `read_committed` caveats From 9450280658a8f0d7e24a644b19efd48b52aa4f41 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Sun, 1 Nov 2026 15:42:08 +1300 Subject: [PATCH 08/21] post 4: add 'Writing to the map' section on admin operations Signed-off-by: Sam Barker --- ...-topic-weaving-one-client-many-clusters.md | 20 +++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index 7f1f22bd..94185d97 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -49,7 +49,7 @@ The simplest source is a table you write down: `orders` is visible from cluster A rule lets the map follow the clusters instead. The simplest is a prefixing router. Say each of the clusters has an `orders` topic. The router gives every cluster a prefix and applies it to whatever that cluster reports, so the client sees `london_orders`, `tokyo_orders` and `newyork_orders`. Nobody has to tell it about a new topic, because it never had a list. The rule has to be deterministic, though: `orders` from the "London" cluster must be `london_orders` today, tomorrow, after every restart, and on every proxy instance. -The rule has to run in both directions. The prefix goes on in `Metadata` responses. It comes off again in every request that names a topic, and the router uses it to pick the cluster. The brokers never hear of `london_orders`. The catch is that the topic name now says which cluster it lives on, which is a little of what weaving was meant to hide. Sometimes that's exactly what you want. +The rule has to run in both directions. The prefix goes on in `Metadata` responses. It comes off again in every request that names a topic, and the router uses it to pick the cluster. The brokers never hear of `london_orders`. It could be argued that the topic name now leaks the physical reality, since it says which cluster the topic lives on. That may be intentional and is perfectly reasonable when it is. From the client's point of view it's a naming convention, not a routing decision. A catalogue service is the same idea with the rule replaced by a lookup. It says which physical topic on which cluster is known to an identity as what, so the names don't have to follow a pattern at all, and it can change at runtime. Either way the router translates in both directions. @@ -59,6 +59,22 @@ Whatever the source, the router still decides what ends up in the response. When Because the map is per identity, the same machinery covers RBAC, tenant-specific rules and filtering. A topic can be visible to one identity and absent for another. What no map can do is move a topic. It only says where one is. +### Writing to the map + +Everything so far has been about reading the map, and reading is aggregation. Each cluster reports what it has and the router combines the answers into one view. The clusters are the source of truth, and the router only decides which of their answers to show. Writes run the other way, and there is no source of truth to consult. A client calls `CreateTopics` for `X`: one request, which the router has to turn into work for some of the clusters. One, several or none. Neither the request nor the clusters say which. That's disaggregation, and the answer can only come from policy. There's no automatically correct one, and each option has a price: + +- **A default cluster.** Simple and predictable. But one cluster risks becoming a hotchpotch of disjointed topics, and nothing about where the data ought to live gets a say. +- **A rule on the name.** `london_orders` goes to London. No state, and the map follows for free. But the name is the placement, so moving a topic means renaming it, and a name that doesn't fit the convention needs somewhere to go. +- **A catalogue.** Placement is decided centrally and names can be anything. But a create is now two steps, update the catalogue and create the topic, and either can fail and leave the other behind. +- **Every cluster.** The name exists everywhere, which is convenient for reads. It doesn't share any data, since each cluster still holds its own. But the create is now a dual write, the problem from [stream branching]({% post_url 2026-10-26-stream-branching-mind-the-gap %}). If it works on A and fails on B, the topic half exists, and the router has to decide what to tell the client. +- **Refuse.** The platform team owns the topology and nothing surprising appears. But anything that creates topics for itself, Kafka Streams being the usual suspect, stops working. + +All of these are legitimate. Which is right depends on who owns the topics and how much you trust the clients, so it's the router's call and the proxy has no opinion. + +The one rule is that a write has to agree with the map. If the create lands on cluster B but the map says `X` lives on A, or doesn't mention `X` at all, the client is told it worked and then can't find its topic. A prefix rule gets this for free, because the prefix already says where `X` goes. A table or a catalogue has to be updated as part of the create, and that's the router's job too. + +`DeleteTopics`, `CreatePartitions` and `AlterConfigs` are the same question with a different verb: how many clusters, and which. Once the topic exists the map answers it. + ## Broker IDs collide too Topic names aren't the only thing two clusters can have in common. Nearly every Kafka cluster numbers its brokers from 0 or 1, so cluster A has a broker 2 and so does cluster B. It's the Platform 2 problem from [post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) again, and it's no less fatal here: put both in one `MetadataResponse` and the client has two different brokers answering to the same number. Broker 2 is Broker 2, remember. @@ -122,7 +138,7 @@ Does it matter? We don't know what your business does with Kafka. Transactions a - [x] The topic map: how the router decides which topic lives where - [x] Broker ID collision: recap the post 2 formula, why weaving needs it permanently - [x] The client's view: folded into the map and "What the client knows" sections (ListTopics is just unfiltered Metadata) -- [ ] Admin operations that name topics (CreateTopics, DeleteTopics, DescribeConfigs): which cluster owns a new topic? Add to the router's jobs or "Who sees what" +- [x] Admin operations that name topics: "Writing to the map" under "Who sees what" - [ ] Producing: why node ID translation isn't enough, and why PID translation is hard - Pull in stash from `stash-replica-routing.md` — "why PIDs aren't node IDs" - [ ] Consuming: the simpler side — passing upstream PIDs through, `read_committed` caveats From 93021ee9aece3dfdde9c4c88d977e9d0142b96da Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Sun, 1 Nov 2026 19:13:40 +1300 Subject: [PATCH 09/21] post 4: add producer ID section and tie the boss cluster through transactions and KIP-939 Signed-off-by: Sam Barker --- ...-topic-weaving-one-client-many-clusters.md | 40 ++++++++++++++++--- 1 file changed, 34 insertions(+), 6 deletions(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index 94185d97..241a4bfc 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -87,6 +87,34 @@ For weaving, that makes merging a non-event. Each route's `Metadata` arrives alr Node IDs are the easy case. They're a pure function of where a broker lives, so the runtime can translate them with arithmetic and no memory. Not everything the client carries around is that obliging. +## Producer IDs are not node IDs + +A producer ID looks like the same problem. It's an integer, a cluster hands it out, and the client carries it around and hands it back. So why not map it the same way? Because a node ID is spatial and a producer ID is temporal. A node ID is a coordinate: the proxy can lie about it as much as it likes, because no broker ever sees the lie, and if the proxy forgets its mapping it rebuilds the same one from the next `Metadata` response. A producer ID is a claim to be a particular live producer. The broker tracks it, uses it to discard retried batches, and fences anyone who presents it with a stale epoch. Lose the mapping and you can't recompute it, because the identity was negotiated rather than derived. + +That matters because every idempotent producer has one, and idempotence is the default. Each cluster issues its own `(producer ID, epoch)` pair, and the pair from cluster A means nothing to cluster B. The router has to give the client one identity and keep a map to the rest. + +It also costs more than node IDs did. The producer ID is inside the checksummed part of a record batch, so rewriting it means decoding the batch, changing the header and recomputing the CRC. Weaving gives up zero-copy on the produce path. + +### Where the map lives + +The map has to survive a proxy restart and be visible to every proxy instance, because a client holds one connection per partition leader and those can land on different instances. So it needs a store, and building a consensus layer into the proxy is the wrong answer. + +Kafka already has durable, replicated storage, so use it. Pick one cluster to be the **boss**. On it, keep a compacted topic keyed by the client's producer ID, whose value maps each backing cluster to the `(producer ID, epoch)` it issued. When the client calls `InitProducerId`, the router initialises a producer on every backing cluster, writes the map to the boss, waits for the ack, and only then replies. Any instance that sees an unfamiliar producer ID reads the topic to find the map, and caches it from then on. + +The map is small and changes only at init or when an epoch is bumped, so none of this is on the produce path. The sequence numbers are the part you'd expect to be hard, and they turn out not to be. They're counted per producer and partition, a partition lives on exactly one cluster, and weaving sends each client batch upstream as exactly one batch for the same partition. The router passes the sequence through untouched. Only the producer ID and epoch are rewritten. As long as every cluster's producer is initialised together, the counts start in step and stay there, so nothing needs storing per batch. + +That doesn't make it free: + +- **The boss is a dependency.** If it's down, no new producer can start. Existing producers carry on from each instance's cache, but a restarted instance has to read the topic before it can serve them. +- **Initialising everywhere is eager.** One backing cluster being unavailable blocks new producers on all the others, including ones that would never have written to it. +- **The map needs cleaning up.** Every short-lived producer leaves a key that compaction keeps until something removes it. + +We haven't built this, so treat it as a direction rather than a recipe. The boss will come up again. + +### Consuming is the easy side + +A consumer never hands a producer ID back to a broker. A `read_committed` consumer sees the one cluster's ID in the batch headers, the commit and abort markers and the aborted-transaction list of a fetch response, and all three come from the same cluster for the same partition. They already agree with each other, so the router passes them through. The one rule is for anything that rebuilds fetch batches: preserve the producer ID, epoch and control-batch flags, and keep the aborted list consistent. + ## What a weaving router has to do Whichever way the map is built, the router has three jobs: @@ -102,15 +130,15 @@ This section describes what a router for this pattern has to do, not a list of f A transactional producer starts by sending `FindCoordinator` for its `transactional.id`, then `InitProducerId` to whichever broker the answer names. From then on that coordinator tracks the transaction, and when the producer commits it writes the commit or abort markers to the partitions involved. The coordinator, its transaction log and those markers all belong to one cluster. -A coordinator can only see its own cluster, so the obvious answer is to give the client one everywhere. The router fans `InitProducerId` out to every backing cluster and starts a transaction on each. As the client registers partitions and produces, the router sends each call to the cluster that owns the topic, using the producer ID and epoch that cluster issued. The client holds one producer ID and the router holds the mapping to the rest. That mapping has to survive a router restart and be visible to every proxy instance, so it needs a state store. That's real work, but it's work you can do. +A coordinator can only see its own cluster, so the obvious answer is to give the client one everywhere. The router fans `InitProducerId` out to every backing cluster and starts a transaction on each. As the client registers partitions and produces, the router sends each call to the cluster that owns the topic, using the producer ID and epoch that cluster issued. The client holds one producer ID and the router holds the mapping to the rest. That mapping has to survive a router restart and be visible to every proxy instance, so it needs the kind of store described [above](#where-the-map-lives). That's real work, but it's work you can do. It falls over at commit. The client sends one `EndTxn` and the router has to turn it into one per cluster. On each cluster, commit is a point of no return: once the coordinator has written its decision, it will commit, whatever else happens. Until then it is free to refuse, and each cluster decides for itself. Cluster 1 can pass the point of no return while cluster 2 says no. None of the ways cluster 2 can say no are transient. The transaction may have timed out, since each cluster runs its own clock from the first partition added there. A zombie may have bumped the producer epoch. The ACLs may differ. Then `orders` has its records and `inventory` doesn't, and whatever the client is told is wrong for one of them. A state store doesn't rescue this. It can remember that the router meant to commit cluster 2, but it can't make cluster 2 agree. Reordering the commits doesn't help either, because whichever goes first can't be undone. -So stop negotiating and pick a boss. One cluster is the primary, and its coordinator runs every transaction. The client's `InitProducerId`, `AddPartitionsToTxn` and `EndTxn` all go there, while the records still go to whichever cluster owns the topic. Commit is now a single decision on a single coordinator, so the atomicity problem is gone. +So stop negotiating and pick a boss, as we did for producer IDs. This time the boss's coordinator runs every transaction. The client's `InitProducerId`, `AddPartitionsToTxn` and `EndTxn` all go there, while the records still go to whichever cluster owns the topic. Commit is now a single decision on a single coordinator, so the atomicity problem is gone. -It doesn't survive the first write to another cluster. The client registers each partition with the coordinator before it writes to it, and the primary's coordinator doesn't recognise `inventory`'s partitions, because they aren't in its cluster. It refuses them, and the transaction never gets as far as commit. +It doesn't survive the first write to another cluster. The client registers each partition with the coordinator before it writes to it, and the boss's coordinator doesn't recognise `inventory`'s partitions, because they aren't in its cluster. It refuses them, and the transaction never gets as far as commit. Both attempts fail for the same reason. Somewhere there has to be a moment when every cluster has promised to commit before any of them does, and Kafka's transaction protocol has no such step. @@ -120,7 +148,7 @@ Both attempts fail for the same reason. Somewhere there has to be a moment when In principle the router could be that coordinator. It would start a transaction on each backing cluster, wait for all of them to prepare, record the decision, and then tell each one to commit. The client would still believe it was talking to a single coordinator. The hard part is the middle step. Someone has to durably record the decision between "everyone prepared" and "everyone committed", and survive a crash in the gap. A router that does that needs a durable log of its own, recovery after a restart, and a way to stop two proxy instances disagreeing about the outcome. That is the consensus layer from [post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}), and the Kafka community has just spent years moving from ZooKeeper to KRaft so that nobody has to run a second consensus system alongside Kafka. Asking a proxy to bring its own would be a step backwards. -That said, this is where a boss cluster might earn its keep. Pick one cluster to hold the router's state, and keep the decision log in a compacted topic on it. Kafka already gives you durable, replicated storage there, with nothing new to deploy. The recovery logic is still yours to write. We haven't investigated this design in any depth, so treat it as a direction rather than a recipe. +That said, this is where the boss earns a second job. The decision log could live in a compacted topic on the same cluster that holds the producer ID map, with nothing new to deploy. The recovery logic is still yours to write. We haven't investigated this design in any depth, so treat it as a direction rather than a recipe. That gets harder still when the client is already a coordinator. A stream processor like Flink can run its own two-phase commit across Kafka and other systems. Put a weaving router in front of Kafka and Flink sees one participant, while the router has to act as a coordinator for the clusters behind it: two-phase commit nested inside two-phase commit. The KIP describes a flat arrangement of one coordinator and its participants, and says nothing about a participant in the middle. @@ -139,9 +167,9 @@ Does it matter? We don't know what your business does with Kafka. Transactions a - [x] Broker ID collision: recap the post 2 formula, why weaving needs it permanently - [x] The client's view: folded into the map and "What the client knows" sections (ListTopics is just unfiltered Metadata) - [x] Admin operations that name topics: "Writing to the map" under "Who sees what" -- [ ] Producing: why node ID translation isn't enough, and why PID translation is hard +- [x] Producing: why node ID translation isn't enough, and why PID translation is hard - Pull in stash from `stash-replica-routing.md` — "why PIDs aren't node IDs" -- [ ] Consuming: the simpler side — passing upstream PIDs through, `read_committed` caveats +- [x] Consuming: the simpler side — passing upstream PIDs through, `read_committed` caveats - [x] Transactions: the coordinator lives on one cluster, and that's all - Reference Post 3's transactions section; explain how weaving makes it worse - [x] KIP-939 and why it isn't magic From 996cc9bd57066f485bb6eb58c1304c87f3ae73bf Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Sun, 1 Nov 2026 19:24:00 +1300 Subject: [PATCH 10/21] post 4: define boss cluster concept and store-vs-participant test Signed-off-by: Sam Barker --- ...-topic-weaving-one-client-many-clusters.md | 26 ++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index 241a4bfc..dd51822d 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -115,6 +115,30 @@ We haven't built this, so treat it as a direction rather than a recipe. The boss A consumer never hands a producer ID back to a broker. A `read_committed` consumer sees the one cluster's ID in the batch headers, the commit and abort markers and the aborted-transaction list of a fetch response, and all three come from the same cluster for the same partition. They already agree with each other, so the router passes them through. The one rule is for anything that rebuilds fetch batches: preserve the producer ID, epoch and control-batch flags, and keep the aborted list consistent. +## Picking a boss + +The producer ID map is the first time we have reached for a **boss** cluster, but the underlying problem keeps recurring. + +Reads aggregate naturally. Every cluster reports what it has, and the router combines the answers into a single view. Writes and state do not aggregate, because there is no existing reality across the clusters to consult. When a client creates a topic, negotiates an identity, or coordinates a decision, the router cannot ask the backing clusters for an answer that does not exist yet. It has to make a choice, and it has to remember what it chose. That memory must survive proxy restarts and be shared across a fleet of proxy instances without inventing a new consensus layer. The Kafka-native answer is to pick one cluster to host that shared state: a **boss** cluster. + +None of the boss cluster roles are tested. They make sense in principle, but they split on what the cluster is asked to do: + +| Problem | Job | In principle? | +| --- | --- | --- | +| Topic creation (`CreateTopics`) | Default cluster for new topics (or catalogue store) | Works, with the hotchpotch caveat | +| Producer ID mapping | Registry in a compacted topic | Works | +| Transactions | The boss's coordinator runs every transaction | Fails (participant) | +| Distributed commit (KIP-939) | Decision log for two-phase commit | Feasible, untested | + +The test for whether a boss works is whether it acts as a **store** or as a **participant**. + +When the boss is a store, it holds proxy bookkeeping in internal topics. The proxy writes its state there and reads it back; the brokers on the boss are just durable storage for bytes, and the proxy enforces the semantics. When the boss is a participant, client traffic lands on it directly. Any write to the boss cluster makes it a participant. That works for `CreateTopics` where the boss hosts the newly created topic, but it fails when the boss is expected to participate on behalf of other clusters. A transaction coordinator on the boss can track and commit writes to its own partitions, but it rejects writes to partitions on other clusters because they do not exist in its metadata. + +Where that boss cluster lives is a separate choice: + +- **One of the data clusters.** The cheapest option: no extra infrastructure to deploy or monitor. But you couple the control plane of all your clusters to one of them. If that data cluster goes down or undergoes maintenance, administrative operations and new producers stall across the entire mesh. +- **A small dedicated cluster.** Isolates the router's state and blast radius from tenant data traffic. If a data cluster fails, only its own topics are affected. The cost is the operational overhead of running one more Kafka cluster. + ## What a weaving router has to do Whichever way the map is built, the router has three jobs: @@ -136,7 +160,7 @@ It falls over at commit. The client sends one `EndTxn` and the router has to tur A state store doesn't rescue this. It can remember that the router meant to commit cluster 2, but it can't make cluster 2 agree. Reordering the commits doesn't help either, because whichever goes first can't be undone. -So stop negotiating and pick a boss, as we did for producer IDs. This time the boss's coordinator runs every transaction. The client's `InitProducerId`, `AddPartitionsToTxn` and `EndTxn` all go there, while the records still go to whichever cluster owns the topic. Commit is now a single decision on a single coordinator, so the atomicity problem is gone. +So stop negotiating and pick a boss again, but this time as a participant rather than a store. The boss's coordinator runs every transaction. The client's `InitProducerId`, `AddPartitionsToTxn` and `EndTxn` all go there, while the records still go to whichever cluster owns the topic. Commit is now a single decision on a single coordinator, so the atomicity problem is gone. It doesn't survive the first write to another cluster. The client registers each partition with the coordinator before it writes to it, and the boss's coordinator doesn't recognise `inventory`'s partitions, because they aren't in its cluster. It refuses them, and the transaction never gets as far as commit. From 921aa3d1fe72ac2e77736c6c477c14b797a81140 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Sun, 1 Nov 2026 16:58:34 -0800 Subject: [PATCH 11/21] post 4: add proxy placement section Signed-off-by: Sam Barker --- ...-03-topic-weaving-one-client-many-clusters.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index dd51822d..29693d0a 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -182,6 +182,20 @@ The router can at least fail loudly. It has to answer `FindCoordinator` before i Does it matter? We don't know what your business does with Kafka. Transactions are fine within a cluster, so this is not a blanket ban on using them. The question is whether any single transaction needs to touch topics that live on different clusters. If your router keeps `orders` and `inventory` together, a transaction across them should work as it always did. But transactions may be more pervasive than a first glance suggests. Kafka Streams uses them to provide exactly-once semantics, so an application that never mentions a transaction can still be running one. If that sounds like your estate, check where the topics it reads and writes live before you weave. +## Egress, ingress, or does it matter? + +The question from the previous posts returns: should the proxy sit on the client side of the link (egress) or in front of the brokers (ingress)? + +The trade-off is very similar to the DR story from [connection switching]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}). Co-locating the proxy with an arbitrary data cluster puts them in the same failure domain. If that cluster has an outage, the proxy instances go down with it—cutting off access to healthy topics on all the other clusters too. + +Client-side egress avoids that coupling. When an individual backing cluster goes down, the proxy stays alive. It returns partition-level errors for the affected cluster while continuing to serve produce and fetch traffic to every other cluster normally. For most topologies, keeping backing cluster failures isolated makes client side the safer default. + +That doesn't make client side categorically better. If you adopt the boss pattern, co-locating the proxy fleet with the boss cluster has a compelling advantage: **latency**. + +Every new producer initialisation, transaction prepare, or catalogue lookup involves the boss. Putting the proxy next to the boss turns those round-trips into local network hops rather than cross-link calls. You accept that the proxy shares a failure domain with the boss—but if the boss is down, those control operations were going to stall anyway. + +If you don't use a boss, or if isolating data-cluster outages is paramount, client side is the natural home. If you do use a boss and want fast admin and init paths, co-locating with it is well worth considering. + ## TODO @@ -198,7 +212,7 @@ Does it matter? We don't know what your business does with Kafka. Transactions a - Reference Post 3's transactions section; explain how weaving makes it worse - [x] KIP-939 and why it isn't magic - Link/summarise the Gemini doc before quoting -- [ ] Proxy placement: run it in the client's network/trust zone, not co-located with any single cluster, so a cluster failure doesn't take the proxy with it +- [x] Proxy placement: run it in the client's network/trust zone, not co-located with any single cluster, so a cluster failure doesn't take the proxy with it - [ ] When to use weaving vs replication vs connection switching - [ ] Closing: what's next in the series From d29f2fddb54ad9f2b0fef804dd60053f20819ee1 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Sun, 1 Nov 2026 17:06:18 -0800 Subject: [PATCH 12/21] Add 'Know when to get off the train' section to topic weaving post Signed-off-by: Sam Barker --- ...-03-topic-weaving-one-client-many-clusters.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index 29693d0a..df3016f1 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -196,6 +196,20 @@ Every new producer initialisation, transaction prepare, or catalogue lookup invo If you don't use a boss, or if isolating data-cluster outages is paramount, client side is the natural home. If you do use a boss and want fast admin and init paths, co-locating with it is well worth considering. +## Know when to get off the train + +Every pattern in this series trades protocol simplicity for operational flexibility. Topic weaving sits near the top of that trade, which makes knowing when *not* to use it just as important as knowing how it works. + +If your problem is moving a whole workload, use [connection switching]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}). A blue/green cutover, a cluster migration, or insulating a legacy application from infrastructure churn only needs the router to throw points at connection time. It is completely stateless, involves zero metadata fabrication or PID rewriting, and adds no moving parts. If all the mail is going to the same city, you don't need a sorting office. + +If your problem is data locality or disaster recovery, use **replication**. MirrorMaker 2, KIP-1279 Cluster Mirroring, or your vendor's replication engine copy the committed log asynchronously after the fact. Topic weaving lets a client in London reach a topic in Tokyo without configuring a second bootstrap address, but the network packets still cross the ocean on every fetch. If you need local read latency or an independent replica that survives a regional outage, replicate the bytes. Weaving provides unified access; replication provides local copies. + +If your problem is tapping an auxiliary stream, use [stream branching]({% post_url 2026-10-26-stream-branching-mind-the-gap %}). Forking a produce stream to feed a staging cluster, an audit topic, or a PII-scrubbed shadow works best when the branch is best-effort and can tolerate the odd gap or duplicate. + +**Topic weaving** is the right choice when topics have distinct, legitimate homes across multiple clusters—by team, by tenant, or by compliance boundary—and you want clients to interact with them as a single coherent estate. It eliminates multi-client boilerplate in application code and hides physical cluster sprawl behind a clean virtual catalogue. + +The boundary is clear: topic weaving works as long as each topic lives entirely on one backing cluster. If your producers need atomic cross-topic transactions across cluster boundaries, no proxy can fix that; those topics must share a physical cluster. And if even a single topic cannot live on one cluster—if you need its partitions split across different physical environments—you have reached the edge of topic weaving. + ## TODO @@ -213,7 +227,7 @@ If you don't use a boss, or if isolating data-cluster outages is paramount, clie - [x] KIP-939 and why it isn't magic - Link/summarise the Gemini doc before quoting - [x] Proxy placement: run it in the client's network/trust zone, not co-located with any single cluster, so a cluster failure doesn't take the proxy with it -- [ ] When to use weaving vs replication vs connection switching +- [x] When to use weaving vs replication vs connection switching - [ ] Closing: what's next in the series --- From 139afc6112c92006a5b1e5c92c2c902a69893987 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Sun, 1 Nov 2026 17:11:05 -0800 Subject: [PATCH 13/21] Add closing section and partition weaving teaser Signed-off-by: Sam Barker --- ...6-11-03-topic-weaving-one-client-many-clusters.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index df3016f1..eeb0b3b9 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -210,6 +210,16 @@ If your problem is tapping an auxiliary stream, use [stream branching]({% post_u The boundary is clear: topic weaving works as long as each topic lives entirely on one backing cluster. If your producers need atomic cross-topic transactions across cluster boundaries, no proxy can fix that; those topics must share a physical cluster. And if even a single topic cannot live on one cluster—if you need its partitions split across different physical environments—you have reached the edge of topic weaving. +--- + +Topic weaving gives you a single logical cluster stitched from whole topics scattered across many physical backends. The client gets one bootstrap address, one cluster view, and zero awareness of the seams behind it. The proxy takes care of the routing table, broker ID collisions, and (with a boss) producer identities. + +Which leaves one final question. If we can weave whole topics together to build a virtual catalogue, why stop at whole topics? Kafka’s real unit of storage, throughput, and parallelism isn’t the topic—it’s the partition. + +In the final post of the series, we go all the way down into the protocol engine room: **partition weaving**. We’ll look at what happens when a single logical topic is assembled from physical partitions living on different clusters, under different names, and how the proxy survives rewriting partition numbers across every RPC in the Kafka protocol. + +If you’re weaving topics already, or exploring how to manage multi-cluster topologies, find us on [GitHub](https://github.com/kroxylicious/kroxylicious), [Slack](https://kroxylicious.slack.com), or [Bluesky](https://bsky.app/profile/kroxylicious.io). + ## TODO @@ -228,7 +238,7 @@ The boundary is clear: topic weaving works as long as each topic lives entirely - Link/summarise the Gemini doc before quoting - [x] Proxy placement: run it in the client's network/trust zone, not co-located with any single cluster, so a cluster failure doesn't take the proxy with it - [x] When to use weaving vs replication vs connection switching -- [ ] Closing: what's next in the series +- [x] Closing: what's next in the series --- From 1c3ee2d9b5a2239ef1df4e101b136fc5f96b4315 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Sun, 1 Nov 2026 17:18:12 -0800 Subject: [PATCH 14/21] Remove TODO list and stash notes; refine community invite Signed-off-by: Sam Barker --- ...-topic-weaving-one-client-many-clusters.md | 31 +------------------ 1 file changed, 1 insertion(+), 30 deletions(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index eeb0b3b9..ca0f5a31 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -218,34 +218,5 @@ Which leaves one final question. If we can weave whole topics together to build In the final post of the series, we go all the way down into the protocol engine room: **partition weaving**. We’ll look at what happens when a single logical topic is assembled from physical partitions living on different clusters, under different names, and how the proxy survives rewriting partition numbers across every RPC in the Kafka protocol. -If you’re weaving topics already, or exploring how to manage multi-cluster topologies, find us on [GitHub](https://github.com/kroxylicious/kroxylicious), [Slack](https://kroxylicious.slack.com), or [Bluesky](https://bsky.app/profile/kroxylicious.io). - - -## TODO - -- [x] Introduction: what the sorting office does that the railway station didn't -- [x] Refresher: how the existing patterns are different (what the client knows, what it doesn't) -- [x] The topic map: how the router decides which topic lives where -- [x] Broker ID collision: recap the post 2 formula, why weaving needs it permanently -- [x] The client's view: folded into the map and "What the client knows" sections (ListTopics is just unfiltered Metadata) -- [x] Admin operations that name topics: "Writing to the map" under "Who sees what" -- [x] Producing: why node ID translation isn't enough, and why PID translation is hard - - Pull in stash from `stash-replica-routing.md` — "why PIDs aren't node IDs" -- [x] Consuming: the simpler side — passing upstream PIDs through, `read_committed` caveats -- [x] Transactions: the coordinator lives on one cluster, and that's all - - Reference Post 3's transactions section; explain how weaving makes it worse -- [x] KIP-939 and why it isn't magic - - Link/summarise the Gemini doc before quoting -- [x] Proxy placement: run it in the client's network/trust zone, not co-located with any single cluster, so a cluster failure doesn't take the proxy with it -- [x] When to use weaving vs replication vs connection switching -- [x] Closing: what's next in the series +If this is the sort of problem you're wrestling with, or if topic weaving looks like something you could use, come and talk to us on [GitHub](https://github.com/kroxylicious/kroxylicious), [Slack](https://kroxylicious.slack.com), or [Bluesky](https://bsky.app/profile/kroxylicious.io). ---- - -*Stash and working notes follow below — remove before publishing.* - ---- - - - - From 44a7557830d4672082b287433142e86e663064de Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Sun, 1 Nov 2026 21:18:27 -0800 Subject: [PATCH 15/21] Post 4: consistency and voice pass Signed-off-by: Sam Barker --- ...-topic-weaving-one-client-many-clusters.md | 51 +++++++++---------- 1 file changed, 24 insertions(+), 27 deletions(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index ca0f5a31..e6b0f1ec 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -9,7 +9,7 @@ categories: blog kroxylicious-proxy tags: [ "routing" ] --- -This is the fourth post in a series about routing in Kroxylicious. [Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API and named four patterns for using it: [connection switching]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}), [stream branching]({% post_url 2026-10-26-stream-branching-mind-the-gap %}), topic weaving and partition weaving. This post covers the third, **topic weaving**. +This is the fourth post in a series about routing in Kroxylicious. [Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API and named four patterns for using it: [connection switching]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}), [stream branching]({% post_url 2026-10-26-stream-branching-mind-the-gap %}), topic weaving and partition weaving. This post covers the third, topic weaving. Kafka clusters accumulate. Teams get their own cluster for isolation, another spins up for a different region, a third appears because somebody hit a partition limit and it was quicker to provision hardware than fight for headroom. The data teams feel it first: every pipeline that crosses a cluster boundary needs its own bootstrap config, its own credentials, its own mental model of which topic lives where. A data engineer building a join across two domains has to know which physical cluster each topic lives on — not because it's relevant to their problem, but because Kafka makes it unavoidable. @@ -23,6 +23,8 @@ That's topic weaving. The client connects to a single VKC, but the namespace beh The railway metaphor has served us well but it runs out of track here. Think of a sorting office instead. Parcels arrive addressed to a virtual destination — `orders`, `payments`, `inventory` — and the sorter routes each one down the right chute to its actual cluster. Every backing cluster is a peer — every write counts. That's also why there are sharp edges. A transaction requires a coordinator, and a coordinator can only see its own cluster. If you need to write atomically to `orders` and `inventory` and they live on different clusters, the sorting office can't help you. +The sorting office also has to solve a naming problem. One parcel is addressed `orders`. Fine — except this is an office that handles international mail, and two different countries have sent you a parcel addressed `orders`. The sorter can't just route one of them, because both exist. It has to decide: rename one, suppress one, or return both with distinct labels so the recipient knows which country each came from. That decision is policy, and it belongs to whoever runs the office. The parcel doesn't get a vote. + ## What the client knows A VKC already keeps the real cluster from the client: clients dial the proxy, never the brokers. What the patterns change is what stands behind that address, and how much of it the client can see. @@ -37,11 +39,13 @@ The client really is talking to several clusters. Some of what it carries with i ## Whose map is it anyway? -All of this hangs on one fact: every router owns a map, the `Metadata` response the client ends up seeing. The proxy supplies the plumbing to send a request down a route and get the response back, and that's all it does. It doesn't merge anything. Merging, splitting a request across clusters and stitching the replies back together are the router's job, and so is deciding what the map says. +All of this hangs on one fact: every router owns a map — the `Metadata` response the client ends up seeing. The proxy supplies the plumbing to send a request down a route and get the answer back, and that's all it does. It doesn't merge anything. Splitting a request across clusters and stitching the replies together are the router's job, and so is deciding what goes in the map. + +### The map is per identity -That's true of every router, not just a weaving one. A connection-switching router owns a map too, because it decides which cluster's answer the client sees. Routers can also be stacked in the DAG, so a router further along may change the map again. Each router is responsible for the changes it makes. +A realisation from drafting a weaving router: nothing says two clients have to get the same `Metadata` response. The router can give each identity its own view of the namespace — different topic sets, different names for the same physical topic, or a completely different slice of the backing clusters. This is the same machinery that covers RBAC and multi-tenancy: a topic can be visible to one identity and absent for another. What no map can do is move a topic. It only says where one is. -The map is also written for whoever is asking. Nothing says two clients get the same `Metadata` response. The router can give each identity its own view, as long as every proxy instance gives that identity the same answer. If two instances disagree, topics appear and disappear depending on who answered. It also means the router has to know who is asking, so this depends on authentication being in place. +One constraint follows directly: **every proxy instance must give the same identity the same answer**. If two instances return different maps for the same identity, topics appear and disappear depending on which instance answered. A static config satisfies this for free. The router has to know who is asking, so this also depends on authentication being in place. ### Where the map comes from @@ -55,9 +59,7 @@ A catalogue service is the same idea with the rule replaced by a lookup. It says ### Who sees what -Whatever the source, the router still decides what ends up in the response. When two clusters both have an `orders`, it can include only cluster A's, include both with a prefix, or exclude both. That is policy, and it belongs to the router. - -Because the map is per identity, the same machinery covers RBAC, tenant-specific rules and filtering. A topic can be visible to one identity and absent for another. What no map can do is move a topic. It only says where one is. +This is where the sorting office comes into its own. When two clusters both have an `orders`, the sorter has to decide: route only cluster A's parcels as `orders`, route cluster B's as `orders` too (now there's a collision), give each a distinct label — `london_orders` and `tokyo_orders` — or reject the ambiguity entirely and return neither. Those are all legitimate choices; which is right depends on policy. The router makes the call. The proxy has no opinion. ### Writing to the map @@ -77,13 +79,13 @@ The one rule is that a write has to agree with the map. If the create lands on c ## Broker IDs collide too -Topic names aren't the only thing two clusters can have in common. Nearly every Kafka cluster numbers its brokers from 0 or 1, so cluster A has a broker 2 and so does cluster B. It's the Platform 2 problem from [post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) again, and it's no less fatal here: put both in one `MetadataResponse` and the client has two different brokers answering to the same number. Broker 2 is Broker 2, remember. +Topic names aren't the only thing two clusters can have in common. Many Kafka clusters number their brokers from 0 or 1, so cluster A has a broker 2 and so does cluster B. It's the Platform 2 problem from [Post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) again: put both in one `MetadataResponse` and the client has two different brokers answering to the same number. Broker 2 is Broker 2, remember. The good news is that nobody writing a router has to deal with it. The runtime does, at every route boundary, with the same bijection as before: `downstreamId = (upstreamId * numRoutes) + routeIndex`. Broker 2 on route 0 becomes 4, broker 2 on route 1 becomes 5, and requests are mapped back again on their way upstream. A router never sees the formula, only the IDs on its side of the boundary. That last part is what allows us to compose routers into a DAG, and thus manage the complexity. What a router calls node 4 might be a real broker, or it might be a virtual node presented by another router further along. It neither knows nor cares. The IDs the client uses are the downstream side of the first router it meets; everything upstream of that is somebody else's business, with its own mapping at its own boundary. They compose without anyone having to coordinate. -For weaving, that makes merging a non-event. Each route's `Metadata` arrives already mapped into the router's ID space, so the broker lists can be concatenated without a thought for collisions. It's a bigger deal than it was for connection switching, where the client only ever saw one cluster's brokers at a time. Here every cluster's brokers are live at once. The constraint from post 2 comes along too: change the number of routes and every ID moves. +For weaving, that makes merging a non-event. Each route's `Metadata` arrives already mapped into the router's ID space, so the broker lists can be concatenated without a thought for collisions. It's a bigger deal than it was for connection switching, where the client only ever saw one cluster's brokers at a time. Here every cluster's brokers are live at once. The constraint from [Post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) comes along too: change the number of routes and every ID moves. Node IDs are the easy case. They're a pure function of where a broker lives, so the runtime can translate them with arithmetic and no memory. Not everything the client carries around is that obliging. @@ -109,7 +111,7 @@ That doesn't make it free: - **Initialising everywhere is eager.** One backing cluster being unavailable blocks new producers on all the others, including ones that would never have written to it. - **The map needs cleaning up.** Every short-lived producer leaves a key that compaction keeps until something removes it. -We haven't built this, so treat it as a direction rather than a recipe. The boss will come up again. +We haven't built this, so treat it as a direction rather than a recipe. ### Consuming is the easy side @@ -147,9 +149,6 @@ Whichever way the map is built, the router has three jobs: - **Produce and fetch.** Group the topics in a request by the cluster they belong to, send the pieces in parallel, and merge the replies into one. - **Enforce the map.** Apply the same rules to every request that names a topic, not just `Metadata`. Hiding a topic from a response isn't enough if a `Produce` can still name it. -This section describes what a router for this pattern has to do, not a list of features you can switch on today. - - ## Transactions stop at the cluster boundary A transactional producer starts by sending `FindCoordinator` for its `transactional.id`, then `InitProducerId` to whichever broker the answer names. From then on that coordinator tracks the transaction, and when the producer commits it writes the commit or abort markers to the partitions involved. The coordinator, its transaction log and those markers all belong to one cluster. @@ -166,27 +165,27 @@ It doesn't survive the first write to another cluster. The client registers each Both attempts fail for the same reason. Somewhere there has to be a moment when every cluster has promised to commit before any of them does, and Kafka's transaction protocol has no such step. -## KIP-939 might add the missing step +## What to do today -[KIP-939](https://cwiki.apache.org/confluence/display/KAFKA/KIP-939%3A+Support+Participation+in+2PC) is the obvious objection. It's accepted, it's about two-phase commit, and it has not landed in Kafka yet. It lets Kafka take part in somebody else's two-phase commit: a producer can prepare a transaction, and a prepared transaction isn't aborted by the timeout while it waits for an external coordinator to decide. That is the promise our two attempts were missing. +The router can at least fail loudly. It has to answer `FindCoordinator` before it knows what the transaction will touch, so it picks using something it does know, like the `transactional.id` or the authenticated principal (the [topic routing proof of concept]({% post_url 2026-05-21-topic-routing %}) used the principal). It then rejects a write that strays onto a different cluster. An error in development beats a half-committed transaction in production. -In principle the router could be that coordinator. It would start a transaction on each backing cluster, wait for all of them to prepare, record the decision, and then tell each one to commit. The client would still believe it was talking to a single coordinator. The hard part is the middle step. Someone has to durably record the decision between "everyone prepared" and "everyone committed", and survive a crash in the gap. A router that does that needs a durable log of its own, recovery after a restart, and a way to stop two proxy instances disagreeing about the outcome. That is the consensus layer from [post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}), and the Kafka community has just spent years moving from ZooKeeper to KRaft so that nobody has to run a second consensus system alongside Kafka. Asking a proxy to bring its own would be a step backwards. +Does it matter for your deployment? Transactions are fine within a cluster, so this is not a blanket ban on using them. The question is whether any single transaction needs to touch topics that live on different clusters. If your router keeps `orders` and `inventory` together, a transaction across them should work as it always did. But transactions may be more pervasive than a first glance suggests. Kafka Streams uses them to provide exactly-once semantics, so an application that never mentions a transaction can still be running one. If that sounds like your estate, check where the topics it reads and writes live before you weave. -That said, this is where the boss earns a second job. The decision log could live in a compacted topic on the same cluster that holds the producer ID map, with nothing new to deploy. The recovery logic is still yours to write. We haven't investigated this design in any depth, so treat it as a direction rather than a recipe. +## KIP-939 might add the missing step -That gets harder still when the client is already a coordinator. A stream processor like Flink can run its own two-phase commit across Kafka and other systems. Put a weaving router in front of Kafka and Flink sees one participant, while the router has to act as a coordinator for the clusters behind it: two-phase commit nested inside two-phase commit. The KIP describes a flat arrangement of one coordinator and its participants, and says nothing about a participant in the middle. +[KIP-939](https://cwiki.apache.org/confluence/display/KAFKA/KIP-939%3A+Support+Participation+in+2PC) is the obvious objection to leaving it there. It's accepted, it's about two-phase commit, but has not landed in Kafka yet. It lets Kafka take part in somebody else's two-phase commit: a producer can prepare a transaction, and a prepared transaction isn't aborted by the timeout while it waits for an external coordinator to decide. That is the promise our two attempts were missing. -## What to do today +In principle the router could be that coordinator. It would start a transaction on each backing cluster, wait for all of them to prepare, record the decision, and then tell each one to commit. The client would still believe it was talking to a single coordinator. The hard part is the middle step. Someone has to durably record the decision between "everyone prepared" and "everyone committed", and survive a crash in the gap. A router that does that needs a durable log of its own, recovery after a restart, and a way to stop two proxy instances disagreeing about the outcome. That is the consensus layer from [Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}), and the Kafka community has just spent years moving from ZooKeeper to KRaft so that nobody has to run a second consensus system alongside Kafka. Whether asking a proxy to bring its own is acceptable depends on your constraints — it is not something we are planning to do. -The router can at least fail loudly. It has to answer `FindCoordinator` before it knows what the transaction will touch, so it picks using something it does know, like the `transactional.id` or the authenticated principal (the [topic routing proof of concept]({% post_url 2026-05-21-topic-routing %}) used the principal). It then rejects a write that strays onto a different cluster. An error in development beats a half-committed transaction in production. +That said, this is where the boss earns a second job. The decision log could live in a compacted topic on the same cluster that holds the producer ID map, with nothing new to deploy. The recovery logic is still yours to write. None of this has been built or tested — treat it as a sketch of a direction, not a recipe. -Does it matter? We don't know what your business does with Kafka. Transactions are fine within a cluster, so this is not a blanket ban on using them. The question is whether any single transaction needs to touch topics that live on different clusters. If your router keeps `orders` and `inventory` together, a transaction across them should work as it always did. But transactions may be more pervasive than a first glance suggests. Kafka Streams uses them to provide exactly-once semantics, so an application that never mentions a transaction can still be running one. If that sounds like your estate, check where the topics it reads and writes live before you weave. +That gets harder still when the client is already a coordinator. A stream processor like Flink can run its own two-phase commit across Kafka and other systems. Put a weaving router in front of Kafka and Flink sees one participant, while the router has to act as a coordinator for the clusters behind it: two-phase commit nested inside two-phase commit. The KIP describes a flat arrangement of one coordinator and its participants, and says nothing about a participant in the middle. ## Egress, ingress, or does it matter? -The question from the previous posts returns: should the proxy sit on the client side of the link (egress) or in front of the brokers (ingress)? +The question from [Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) returns: should the proxy sit on the client side of the link (egress) or in front of the brokers (ingress)? [Post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) covered this for connection switching; the answer for topic weaving is similar in shape but lands in a different place. -The trade-off is very similar to the DR story from [connection switching]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}). Co-locating the proxy with an arbitrary data cluster puts them in the same failure domain. If that cluster has an outage, the proxy instances go down with it—cutting off access to healthy topics on all the other clusters too. +Co-locating the proxy with an arbitrary data cluster puts them in the same failure domain. If that cluster has an outage, the proxy instances go down with it—cutting off access to healthy topics on all the other clusters too. Client-side egress avoids that coupling. When an individual backing cluster goes down, the proxy stays alive. It returns partition-level errors for the affected cluster while continuing to serve produce and fetch traffic to every other cluster normally. For most topologies, keeping backing cluster failures isolated makes client side the safer default. @@ -196,7 +195,7 @@ Every new producer initialisation, transaction prepare, or catalogue lookup invo If you don't use a boss, or if isolating data-cluster outages is paramount, client side is the natural home. If you do use a boss and want fast admin and init paths, co-locating with it is well worth considering. -## Know when to get off the train +## Know when to pick a different chute Every pattern in this series trades protocol simplicity for operational flexibility. Topic weaving sits near the top of that trade, which makes knowing when *not* to use it just as important as knowing how it works. @@ -214,9 +213,7 @@ The boundary is clear: topic weaving works as long as each topic lives entirely Topic weaving gives you a single logical cluster stitched from whole topics scattered across many physical backends. The client gets one bootstrap address, one cluster view, and zero awareness of the seams behind it. The proxy takes care of the routing table, broker ID collisions, and (with a boss) producer identities. -Which leaves one final question. If we can weave whole topics together to build a virtual catalogue, why stop at whole topics? Kafka’s real unit of storage, throughput, and parallelism isn’t the topic—it’s the partition. - -In the final post of the series, we go all the way down into the protocol engine room: **partition weaving**. We’ll look at what happens when a single logical topic is assembled from physical partitions living on different clusters, under different names, and how the proxy survives rewriting partition numbers across every RPC in the Kafka protocol. +Which leaves one final question. If we can weave whole topics together to build a virtual catalogue, why stop at whole topics? Kafka's real unit of storage, throughput, and parallelism isn't the topic—it's the partition. Next: **partition weaving**. If this is the sort of problem you're wrestling with, or if topic weaving looks like something you could use, come and talk to us on [GitHub](https://github.com/kroxylicious/kroxylicious), [Slack](https://kroxylicious.slack.com), or [Bluesky](https://bsky.app/profile/kroxylicious.io). From c3710969de941a3a1a8850fafa9d7e91d80eb326 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Sun, 1 Nov 2026 21:33:15 -0800 Subject: [PATCH 16/21] update stash Signed-off-by: Sam Barker --- _drafts/stash-replica-routing.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/_drafts/stash-replica-routing.md b/_drafts/stash-replica-routing.md index 64e890bf..970ca3ad 100644 --- a/_drafts/stash-replica-routing.md +++ b/_drafts/stash-replica-routing.md @@ -85,3 +85,6 @@ The PID argument bites on the produce side only. The consume side works without TODO: link the Gemini doc from the weaving post: Point to make: KIP-939 (two-phase commit support) still does not fix this. If we went anywhere near it we would be in full distributed consensus pain. Not read or verified yet. Summarise from the doc before quoting, and check its claims against the KIP. + +## Topic ID is also temporal +1. Topic IDs / UUIDs (KIP-516)At first glance, a Topic ID looks spatial (it’s just a coordinate for a topic). But it is actually a temporal lifecycle marker generated by the KRaft controller.The Semantics: If you delete a topic and recreate it with the same name, it gets a new UUID to prevent zombie delayed fetches from corrupting the new log.Why it breaks stateless math: You can mathematically fake a Node ID using $V = id + S \times t$ because Node IDs are predictable integers. You cannot mathematically derive a random backend UUID from a frontend virtual UUID.The Proxy Seam: To support API versions v13+ (where Produce and Fetch requests use UUIDs instead of strings), the proxy must maintain a stateful [Virtual Topic UUID -> Physical Topic UUID] mapping table in memory. If a proxy replica restarts and loses that map, it cannot route the frame. This is exactly why a proxy might intentionally cap PRODUCE API versions to v12—forcing clients to use string names—just to avoid handling cluster-specific Topic IDs. From b2a32828d5e5f465036285c3bfa4bd0a24f1f678 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Sun, 1 Nov 2026 21:34:07 -0800 Subject: [PATCH 17/21] remove cruft Signed-off-by: Sam Barker --- proposals/289-review-comments.json | 51 ------------------------------ 1 file changed, 51 deletions(-) delete mode 100644 proposals/289-review-comments.json diff --git a/proposals/289-review-comments.json b/proposals/289-review-comments.json deleted file mode 100644 index 551912b0..00000000 --- a/proposals/289-review-comments.json +++ /dev/null @@ -1,51 +0,0 @@ -[ - { - "type": "review_thread_reply", - "thread": "T5", - "replyToId": 4210755206, - "body": "Good point \u2014 you're right that we don't want to focus on record-level routing here. The design space does cover it (a router reconstituting a branch request can go as fine-grained as it likes), but that's not where current implementations sit. I've tweaked the language in the axis paragraph to avoid the ambiguity, and the stream branching section now covers the routing granularity point in more detail.", - "status": "posted" - }, - { - "type": "review_thread_reply", - "thread": "T6", - "replyToId": 4211918741, - "body": "Horses for courses \u2014 Kafka Streams is absolutely the right answer for some of this. Where the proxy approach has the edge: no duplicate storage cost (you're deriving data in flight rather than materialising a new topic), and no framework overhead for what are often stateless transforms. Kafka Streams and Connect are powerful but they're not lightweight \u2014 if all you need is \"emit a record count to a metrics topic as this batch passes through\", standing up a Streams application is a lot of machinery for a stateless operation. I've added a sentence to the section to make this clearer.", - "status": "posted" - }, - { - "type": "review_thread_reply", - "thread": "T9", - "replyToId": 4212088090, - "body": "Oops that wasn't quite the example I was trying to make. I've tweaked the text to make it clearer the woven topic is only on the consumer side.", - "status": "posted" - }, - { - "type": "review_thread_reply", - "thread": "T4", - "replyToId": 4210723236, - "body": "Glad the others landed. *Stream branching* was the best I could come up with \u2014 open to better suggestions. *Branching streams* is another candidate; the difference is whether you're naming the act (branching a stream) or the result (streams that branch). I went with the former but I'm not attached to it.", - "status": "posted" - }, - { - "type": "review_thread_reply", - "thread": "T1", - "replyToId": 4210638469, - "body": "Fixed in df2f68b \u2014 updated to broker rather than cluster.", - "status": "posted" - }, - { - "type": "review_thread_reply", - "thread": "T8", - "replyToId": 4223067354, - "body": "Fair point on consumer lag \u2014 you're right that cluster-side observation is the proper place for complete lag metrics. On counts/telemetry, the proxy isn't doing the final aggregation \u2014 it's just emitting inline record telemetry for downstream systems to aggregate. I've updated the examples in this section to focus on clean, stateless inline operations (a redaction filter for PII, a sampling filter to thin traffic for smaller staging environments, and derivation filters for audit logs/telemetry).", - "status": "posted" - }, - { - "type": "review_thread_reply", - "thread": "T9", - "replyToId": 4223173007, - "body": "Yep, exactly \u2014 if you control the consumer and don't mind it managing multiple topic subscriptions/assignments, topic weaving gets you part of the way there. Partition weaving is what lets you keep that complexity entirely in the infrastructure so downstream consumers don't even need to know the second topic exists.\n\nI've reworked this section to frame it around extending weaving down to Kafka's fundamental unit of parallelism (partitions), swapped the example to a third-party risk engine consuming an `orders` topic, and explained how consumer group coordination remains pinned to a single cluster without the coordinator needing to know about the seam.", - "status": "posted" - } -] From 19e54764e87a4a1a013521ec23edb856ab2afeb9 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Mon, 2 Nov 2026 07:42:18 -0800 Subject: [PATCH 18/21] Fix stream-branching post slug (date changed to 2026-10-29 before merge) Signed-off-by: Sam Barker --- _posts/2026-11-03-topic-weaving-one-client-many-clusters.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index e6b0f1ec..c2b952d6 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -9,7 +9,7 @@ categories: blog kroxylicious-proxy tags: [ "routing" ] --- -This is the fourth post in a series about routing in Kroxylicious. [Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API and named four patterns for using it: [connection switching]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}), [stream branching]({% post_url 2026-10-26-stream-branching-mind-the-gap %}), topic weaving and partition weaving. This post covers the third, topic weaving. +This is the fourth post in a series about routing in Kroxylicious. [Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API and named four patterns for using it: [connection switching]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}), [stream branching]({% post_url 2026-10-29-stream-branching-mind-the-gap %}), topic weaving and partition weaving. This post covers the third, topic weaving. Kafka clusters accumulate. Teams get their own cluster for isolation, another spins up for a different region, a third appears because somebody hit a partition limit and it was quicker to provision hardware than fight for headroom. The data teams feel it first: every pipeline that crosses a cluster boundary needs its own bootstrap config, its own credentials, its own mental model of which topic lives where. A data engineer building a join across two domains has to know which physical cluster each topic lives on — not because it's relevant to their problem, but because Kafka makes it unavoidable. @@ -68,7 +68,7 @@ Everything so far has been about reading the map, and reading is aggregation. Ea - **A default cluster.** Simple and predictable. But one cluster risks becoming a hotchpotch of disjointed topics, and nothing about where the data ought to live gets a say. - **A rule on the name.** `london_orders` goes to London. No state, and the map follows for free. But the name is the placement, so moving a topic means renaming it, and a name that doesn't fit the convention needs somewhere to go. - **A catalogue.** Placement is decided centrally and names can be anything. But a create is now two steps, update the catalogue and create the topic, and either can fail and leave the other behind. -- **Every cluster.** The name exists everywhere, which is convenient for reads. It doesn't share any data, since each cluster still holds its own. But the create is now a dual write, the problem from [stream branching]({% post_url 2026-10-26-stream-branching-mind-the-gap %}). If it works on A and fails on B, the topic half exists, and the router has to decide what to tell the client. +- **Every cluster.** The name exists everywhere, which is convenient for reads. It doesn't share any data, since each cluster still holds its own. But the create is now a dual write, the problem from [stream branching]({% post_url 2026-10-29-stream-branching-mind-the-gap %}). If it works on A and fails on B, the topic half exists, and the router has to decide what to tell the client. - **Refuse.** The platform team owns the topology and nothing surprising appears. But anything that creates topics for itself, Kafka Streams being the usual suspect, stops working. All of these are legitimate. Which is right depends on who owns the topics and how much you trust the clients, so it's the router's call and the proxy has no opinion. @@ -203,7 +203,7 @@ If your problem is moving a whole workload, use [connection switching]({% post_u If your problem is data locality or disaster recovery, use **replication**. MirrorMaker 2, KIP-1279 Cluster Mirroring, or your vendor's replication engine copy the committed log asynchronously after the fact. Topic weaving lets a client in London reach a topic in Tokyo without configuring a second bootstrap address, but the network packets still cross the ocean on every fetch. If you need local read latency or an independent replica that survives a regional outage, replicate the bytes. Weaving provides unified access; replication provides local copies. -If your problem is tapping an auxiliary stream, use [stream branching]({% post_url 2026-10-26-stream-branching-mind-the-gap %}). Forking a produce stream to feed a staging cluster, an audit topic, or a PII-scrubbed shadow works best when the branch is best-effort and can tolerate the odd gap or duplicate. +If your problem is tapping an auxiliary stream, use [stream branching]({% post_url 2026-10-29-stream-branching-mind-the-gap %}). Forking a produce stream to feed a staging cluster, an audit topic, or a PII-scrubbed shadow works best when the branch is best-effort and can tolerate the odd gap or duplicate. **Topic weaving** is the right choice when topics have distinct, legitimate homes across multiple clusters—by team, by tenant, or by compliance boundary—and you want clients to interact with them as a single coherent estate. It eliminates multi-client boilerplate in application code and hides physical cluster sprawl behind a clean virtual catalogue. From 6495ae7ae2f123a47d45f8ef6c36cbbab0e1d524 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Mon, 2 Nov 2026 10:50:04 -0800 Subject: [PATCH 19/21] Topic weaving: restructure sections and sharpen sorting office analogy Move 'What a weaving router has to do' ahead of the broker ID section and 'Picking a boss' after KIP-939. Use the sorting office counter to explain per-identity maps, rename 'Who sees what' to 'When names collide', clarify the node ID example and add a note on epoch bumps. Signed-off-by: Sam Barker --- ...-topic-weaving-one-client-many-clusters.md | 80 ++++++++++--------- 1 file changed, 41 insertions(+), 39 deletions(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index c2b952d6..eb384105 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -23,7 +23,7 @@ That's topic weaving. The client connects to a single VKC, but the namespace beh The railway metaphor has served us well but it runs out of track here. Think of a sorting office instead. Parcels arrive addressed to a virtual destination — `orders`, `payments`, `inventory` — and the sorter routes each one down the right chute to its actual cluster. Every backing cluster is a peer — every write counts. That's also why there are sharp edges. A transaction requires a coordinator, and a coordinator can only see its own cluster. If you need to write atomically to `orders` and `inventory` and they live on different clusters, the sorting office can't help you. -The sorting office also has to solve a naming problem. One parcel is addressed `orders`. Fine — except this is an office that handles international mail, and two different countries have sent you a parcel addressed `orders`. The sorter can't just route one of them, because both exist. It has to decide: rename one, suppress one, or return both with distinct labels so the recipient knows which country each came from. That decision is policy, and it belongs to whoever runs the office. The parcel doesn't get a vote. +The sorting office also has to solve a naming problem. One parcel is addressed `orders`. Fine — except this is an office that handles international mail, and two different countries have sent you a parcel addressed `orders`. The sorter can't just route one of them, because both exist. Choosing is policy, and it belongs to whoever runs the office. The parcel doesn't get a vote. ## What the client knows @@ -43,7 +43,7 @@ All of this hangs on one fact: every router owns a map — the `Metadata` respon ### The map is per identity -A realisation from drafting a weaving router: nothing says two clients have to get the same `Metadata` response. The router can give each identity its own view of the namespace — different topic sets, different names for the same physical topic, or a completely different slice of the backing clusters. This is the same machinery that covers RBAC and multi-tenancy: a topic can be visible to one identity and absent for another. What no map can do is move a topic. It only says where one is. +A realisation from drafting a weaving router: nothing says two clients have to get the same `Metadata` response. In sorting office terms, labels are applied at the counter, for whoever is standing there. The same physical `orders` can go out as `london_orders` to one customer, as plain `orders` to another, and not at all to a third who has no business seeing it. The router can give each identity its own view of the namespace — different topic sets, different names for the same physical topic, or a completely different slice of the backing clusters. This is the same machinery that covers RBAC and multi-tenancy: a topic can be visible to one identity and absent for another. What no map can do is move a topic. It only says where one is. One constraint follows directly: **every proxy instance must give the same identity the same answer**. If two instances return different maps for the same identity, topics appear and disappear depending on which instance answered. A static config satisfies this for free. The router has to know who is asking, so this also depends on authentication being in place. @@ -57,9 +57,11 @@ The rule has to run in both directions. The prefix goes on in `Metadata` respons A catalogue service is the same idea with the rule replaced by a lookup. It says which physical topic on which cluster is known to an identity as what, so the names don't have to follow a pattern at all, and it can change at runtime. Either way the router translates in both directions. -### Who sees what +### When names collide -This is where the sorting office comes into its own. When two clusters both have an `orders`, the sorter has to decide: route only cluster A's parcels as `orders`, route cluster B's as `orders` too (now there's a collision), give each a distinct label — `london_orders` and `tokyo_orders` — or reject the ambiguity entirely and return neither. Those are all legitimate choices; which is right depends on policy. The router makes the call. The proxy has no opinion. +This is where the sorting office comes into its own. When two clusters both have an `orders`, and each label has to mean one topic, the sorter has three choices. It can pick a winner and only ever hand out cluster A's `orders`, leaving B's off the counter. It can give each a distinct label — `london_orders` and `tokyo_orders`. Or it can refuse the ambiguity and hand out neither. Which is right depends on policy. The router makes the call. The proxy has no opinion. + +A fourth choice is to put `orders` on both and let the label cover two addresses. That's a different kind of office, because the sorter now has to work out which address each individual parcel belongs to. It's partition weaving, the final pattern in the series. ### Writing to the map @@ -77,11 +79,19 @@ The one rule is that a write has to agree with the map. If the create lands on c `DeleteTopics`, `CreatePartitions` and `AlterConfigs` are the same question with a different verb: how many clusters, and which. Once the topic exists the map answers it. +## What a weaving router has to do + +Whichever way the map is built, the router has three jobs: + +- **Metadata.** Ask every backing cluster, keep the topics the map shows this identity, and return the lot as one response. +- **Produce and fetch.** Group the topics in a request by the cluster they belong to, send the pieces in parallel, and merge the replies into one. +- **Enforce the map.** Apply the same rules to every request that names a topic, not just `Metadata`. Hiding a topic from a response isn't enough if a `Produce` can still name it. + ## Broker IDs collide too Topic names aren't the only thing two clusters can have in common. Many Kafka clusters number their brokers from 0 or 1, so cluster A has a broker 2 and so does cluster B. It's the Platform 2 problem from [Post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) again: put both in one `MetadataResponse` and the client has two different brokers answering to the same number. Broker 2 is Broker 2, remember. -The good news is that nobody writing a router has to deal with it. The runtime does, at every route boundary, with the same bijection as before: `downstreamId = (upstreamId * numRoutes) + routeIndex`. Broker 2 on route 0 becomes 4, broker 2 on route 1 becomes 5, and requests are mapped back again on their way upstream. A router never sees the formula, only the IDs on its side of the boundary. +The good news is that nobody writing a router has to deal with it. The runtime does, at every route boundary, with the same bijection as before: `downstreamId = (upstreamId * numRoutes) + routeIndex`. With two routes, broker 2 on route 0 becomes 4 and broker 2 on route 1 becomes 5, and requests are mapped back again on their way upstream. A router never sees the formula, only the IDs on its side of the boundary. That last part is what allows us to compose routers into a DAG, and thus manage the complexity. What a router calls node 4 might be a real broker, or it might be a virtual node presented by another router further along. It neither knows nor cares. The IDs the client uses are the downstream side of the first router it meets; everything upstream of that is somebody else's business, with its own mapping at its own boundary. They compose without anyone having to coordinate. @@ -103,7 +113,7 @@ The map has to survive a proxy restart and be visible to every proxy instance, b Kafka already has durable, replicated storage, so use it. Pick one cluster to be the **boss**. On it, keep a compacted topic keyed by the client's producer ID, whose value maps each backing cluster to the `(producer ID, epoch)` it issued. When the client calls `InitProducerId`, the router initialises a producer on every backing cluster, writes the map to the boss, waits for the ack, and only then replies. Any instance that sees an unfamiliar producer ID reads the topic to find the map, and caches it from then on. -The map is small and changes only at init or when an epoch is bumped, so none of this is on the produce path. The sequence numbers are the part you'd expect to be hard, and they turn out not to be. They're counted per producer and partition, a partition lives on exactly one cluster, and weaving sends each client batch upstream as exactly one batch for the same partition. The router passes the sequence through untouched. Only the producer ID and epoch are rewritten. As long as every cluster's producer is initialised together, the counts start in step and stay there, so nothing needs storing per batch. +The map is small and changes only at init or when an epoch is bumped, so none of this is on the produce path. The sequence numbers are the part you'd expect to be hard, and they turn out not to be. They're counted per producer and partition, a partition lives on exactly one cluster, and weaving sends each client batch upstream as exactly one batch for the same partition. The router passes the sequence through untouched. Only the producer ID and epoch are rewritten. As long as every cluster's producer is initialised together, the counts start in step and stay there, so nothing needs storing per batch. An epoch bump follows the same rule. The client re-sends `InitProducerId` with its current ID and epoch, and the router has to bump every cluster and update the map, because the client resets its sequence numbers for all partitions at once. That doesn't make it free: @@ -117,38 +127,6 @@ We haven't built this, so treat it as a direction rather than a recipe. A consumer never hands a producer ID back to a broker. A `read_committed` consumer sees the one cluster's ID in the batch headers, the commit and abort markers and the aborted-transaction list of a fetch response, and all three come from the same cluster for the same partition. They already agree with each other, so the router passes them through. The one rule is for anything that rebuilds fetch batches: preserve the producer ID, epoch and control-batch flags, and keep the aborted list consistent. -## Picking a boss - -The producer ID map is the first time we have reached for a **boss** cluster, but the underlying problem keeps recurring. - -Reads aggregate naturally. Every cluster reports what it has, and the router combines the answers into a single view. Writes and state do not aggregate, because there is no existing reality across the clusters to consult. When a client creates a topic, negotiates an identity, or coordinates a decision, the router cannot ask the backing clusters for an answer that does not exist yet. It has to make a choice, and it has to remember what it chose. That memory must survive proxy restarts and be shared across a fleet of proxy instances without inventing a new consensus layer. The Kafka-native answer is to pick one cluster to host that shared state: a **boss** cluster. - -None of the boss cluster roles are tested. They make sense in principle, but they split on what the cluster is asked to do: - -| Problem | Job | In principle? | -| --- | --- | --- | -| Topic creation (`CreateTopics`) | Default cluster for new topics (or catalogue store) | Works, with the hotchpotch caveat | -| Producer ID mapping | Registry in a compacted topic | Works | -| Transactions | The boss's coordinator runs every transaction | Fails (participant) | -| Distributed commit (KIP-939) | Decision log for two-phase commit | Feasible, untested | - -The test for whether a boss works is whether it acts as a **store** or as a **participant**. - -When the boss is a store, it holds proxy bookkeeping in internal topics. The proxy writes its state there and reads it back; the brokers on the boss are just durable storage for bytes, and the proxy enforces the semantics. When the boss is a participant, client traffic lands on it directly. Any write to the boss cluster makes it a participant. That works for `CreateTopics` where the boss hosts the newly created topic, but it fails when the boss is expected to participate on behalf of other clusters. A transaction coordinator on the boss can track and commit writes to its own partitions, but it rejects writes to partitions on other clusters because they do not exist in its metadata. - -Where that boss cluster lives is a separate choice: - -- **One of the data clusters.** The cheapest option: no extra infrastructure to deploy or monitor. But you couple the control plane of all your clusters to one of them. If that data cluster goes down or undergoes maintenance, administrative operations and new producers stall across the entire mesh. -- **A small dedicated cluster.** Isolates the router's state and blast radius from tenant data traffic. If a data cluster fails, only its own topics are affected. The cost is the operational overhead of running one more Kafka cluster. - -## What a weaving router has to do - -Whichever way the map is built, the router has three jobs: - -- **Metadata.** Ask every backing cluster, keep the topics the map shows this identity, and return the lot as one response. -- **Produce and fetch.** Group the topics in a request by the cluster they belong to, send the pieces in parallel, and merge the replies into one. -- **Enforce the map.** Apply the same rules to every request that names a topic, not just `Metadata`. Hiding a topic from a response isn't enough if a `Produce` can still name it. - ## Transactions stop at the cluster boundary A transactional producer starts by sending `FindCoordinator` for its `transactional.id`, then `InitProducerId` to whichever broker the answer names. From then on that coordinator tracks the transaction, and when the producer commits it writes the commit or abort markers to the partitions involved. The coordinator, its transaction log and those markers all belong to one cluster. @@ -159,7 +137,7 @@ It falls over at commit. The client sends one `EndTxn` and the router has to tur A state store doesn't rescue this. It can remember that the router meant to commit cluster 2, but it can't make cluster 2 agree. Reordering the commits doesn't help either, because whichever goes first can't be undone. -So stop negotiating and pick a boss again, but this time as a participant rather than a store. The boss's coordinator runs every transaction. The client's `InitProducerId`, `AddPartitionsToTxn` and `EndTxn` all go there, while the records still go to whichever cluster owns the topic. Commit is now a single decision on a single coordinator, so the atomicity problem is gone. +So stop negotiating and pick a boss again, but this time as a participant rather than a store: not a place to keep the map, but a coordinator that runs every transaction. The client's `InitProducerId`, `AddPartitionsToTxn` and `EndTxn` all go there, while the records still go to whichever cluster owns the topic. Commit is now a single decision on a single coordinator, so the atomicity problem is gone. It doesn't survive the first write to another cluster. The client registers each partition with the coordinator before it writes to it, and the boss's coordinator doesn't recognise `inventory`'s partitions, because they aren't in its cluster. It refuses them, and the transaction never gets as far as commit. @@ -181,6 +159,30 @@ That said, this is where the boss earns a second job. The decision log could liv That gets harder still when the client is already a coordinator. A stream processor like Flink can run its own two-phase commit across Kafka and other systems. Put a weaving router in front of Kafka and Flink sees one participant, while the router has to act as a coordinator for the clusters behind it: two-phase commit nested inside two-phase commit. The KIP describes a flat arrangement of one coordinator and its participants, and says nothing about a participant in the middle. +## Picking a boss + +The producer ID map is the first time we have reached for a **boss** cluster, but the underlying problem keeps recurring. + +Reads aggregate naturally. Every cluster reports what it has, and the router combines the answers into a single view. Writes and state do not aggregate, because there is no existing reality across the clusters to consult. When a client creates a topic, negotiates an identity, or coordinates a decision, the router cannot ask the backing clusters for an answer that does not exist yet. It has to make a choice, and it has to remember what it chose. That memory must survive proxy restarts and be shared across a fleet of proxy instances without inventing a new consensus layer. The Kafka-native answer is to pick one cluster to host that shared state: a **boss** cluster. + +None of the boss cluster roles are tested. They make sense in principle, but they split on what the cluster is asked to do: + +| Problem | Job | In principle? | +| --- | --- | --- | +| Topic creation (`CreateTopics`) | Default cluster for new topics (or catalogue store) | Works, with the hotchpotch caveat | +| Producer ID mapping | Registry in a compacted topic | Works | +| Transactions | The boss's coordinator runs every transaction | Fails (participant) | +| Distributed commit (KIP-939) | Decision log for two-phase commit | Feasible, untested | + +The test for whether a boss works is whether it acts as a **store** or as a **participant**. + +When the boss is a store, it holds proxy bookkeeping in internal topics. The proxy writes its state there and reads it back; the brokers on the boss are just durable storage for bytes, and the proxy enforces the semantics. When the boss is a participant, client traffic lands on it directly. Any write to the boss cluster makes it a participant. That works for `CreateTopics` where the boss hosts the newly created topic, but it fails when the boss is expected to participate on behalf of other clusters. A transaction coordinator on the boss can track and commit writes to its own partitions, but it rejects writes to partitions on other clusters because they do not exist in its metadata. + +Where that boss cluster lives is a separate choice: + +- **One of the data clusters.** The cheapest option: no extra infrastructure to deploy or monitor. But you couple the control plane of all your clusters to one of them. If that data cluster goes down or undergoes maintenance, administrative operations and new producers stall across the entire mesh. +- **A small dedicated cluster.** Isolates the router's state and blast radius from tenant data traffic. If a data cluster fails, only its own topics are affected. The cost is the operational overhead of running one more Kafka cluster. + ## Egress, ingress, or does it matter? The question from [Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) returns: should the proxy sit on the client side of the link (egress) or in front of the brokers (ingress)? [Post 2]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}) covered this for connection switching; the answer for topic weaving is similar in shape but lands in a different place. From 89a890e7f00e9a805d8c3e29f7d282cd4407eba6 Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Mon, 2 Nov 2026 11:50:22 -0800 Subject: [PATCH 20/21] Add consumer groups section to topic weaving post Cover the classic group protocol with a boss for membership and offsets split by topic, ahead of transactions as the contrasting case. Replace the store/participant test in 'Picking a boss' with local versus remote content, state the design-level status once in the intro, and trim the repeated 'not built' caveats. Signed-off-by: Sam Barker --- ...-topic-weaving-one-client-many-clusters.md | 49 ++++++++++++++++--- 1 file changed, 41 insertions(+), 8 deletions(-) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md index eb384105..a70beb42 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md @@ -9,7 +9,7 @@ categories: blog kroxylicious-proxy tags: [ "routing" ] --- -This is the fourth post in a series about routing in Kroxylicious. [Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API and named four patterns for using it: [connection switching]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}), [stream branching]({% post_url 2026-10-29-stream-branching-mind-the-gap %}), topic weaving and partition weaving. This post covers the third, topic weaving. +This is the fourth post in a series about routing in Kroxylicious. [Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}) introduced the Routing API and named four patterns for using it: [connection switching]({% post_url 2026-10-19-connection-switching-identity-without-commitment %}), [stream branching]({% post_url 2026-10-29-stream-branching-mind-the-gap %}), topic weaving and partition weaving. This post covers the third, topic weaving. Like the rest of the series, it is design-level thinking rather than a description of something shipped, and we haven't tried it in practice. Kafka clusters accumulate. Teams get their own cluster for isolation, another spins up for a different region, a third appears because somebody hit a partition limit and it was quicker to provision hardware than fight for headroom. The data teams feel it first: every pipeline that crosses a cluster boundary needs its own bootstrap config, its own credentials, its own mental model of which topic lives where. A data engineer building a join across two domains has to know which physical cluster each topic lives on — not because it's relevant to their problem, but because Kafka makes it unavoidable. @@ -121,25 +121,57 @@ That doesn't make it free: - **Initialising everywhere is eager.** One backing cluster being unavailable blocks new producers on all the others, including ones that would never have written to it. - **The map needs cleaning up.** Every short-lived producer leaves a key that compaction keeps until something removes it. -We haven't built this, so treat it as a direction rather than a recipe. +We haven't built this. ### Consuming is the easy side A consumer never hands a producer ID back to a broker. A `read_committed` consumer sees the one cluster's ID in the batch headers, the commit and abort markers and the aborted-transaction list of a fetch response, and all three come from the same cluster for the same partition. They already agree with each other, so the router passes them through. The one rule is for anything that rebuilds fetch batches: preserve the producer ID, epoch and control-batch flags, and keep the aborted list consistent. +## Consumer groups need a coordinator too + +Fetching is independent per partition, so the router can send each piece of a `Fetch` to the cluster that owns it. A consumer group is the part that isn't. As a reminder, this is how the classic group protocol runs: + +1. `FindCoordinator` for the `group.id` returns the broker that coordinates the group. +2. Each consumer sends `JoinGroup` to the coordinator, with its subscription as opaque bytes. The coordinator hands out member IDs and elects one member as the leader. +3. The leader divides the partitions among the members and sends the result back in `SyncGroup`. The coordinator passes each member its share. +4. From then on `Heartbeat` keeps the membership alive and `OffsetCommit` records progress in the coordinator's `__consumer_offsets`. Any change in membership starts a new generation and sends everyone back to step 2. + +All of that state lives on one broker, in one cluster. + +The obvious answer is to fan membership out to every cluster. Each cluster's coordinator would then run its own copy of the group, with its own member IDs, generation and leader, and decide for itself when a member has gone quiet. Nothing keeps those copies in step. If cluster B decides a member has gone quiet and A doesn't, B starts a rebalance and A doesn't, and the client's one membership is something two clusters disagree about. + +There's a way out, because nothing in a consumer group has to be atomic. An `OffsetCommit` reports success or failure per partition, so there is no point of no return to line up. Pick one cluster to be the boss for membership, and split the offsets by topic: + +- **`FindCoordinator`** for a group answers with the boss's coordinator. Every proxy instance has to give the same answer, so derive it from something the router knows, like the `group.id` or the principal. Nothing needs storing. +- **`JoinGroup`, `SyncGroup`, `Heartbeat` and `LeaveGroup`** go to the boss and nowhere else. The coordinator treats each member's subscription as opaque bytes, and the group leader works out the assignment client-side from the merged `Metadata`. It doesn't matter that the boss has never heard of some of the topics. +- **`OffsetCommit` and `OffsetFetch`** are split by topic, like `Produce`. The boss rejects topics it doesn't have, and the other clusters have never seen the member. So the router sends their share as standalone commits, with no member ID and generation -1, which a cluster accepts for a group with no active members. An `OffsetFetch` that asks for every topic goes to every cluster, and the replies are merged. + +This works because the boss is never asked about anything that isn't its own. Membership names no topics, and the offsets it receives are for topics it hosts. + +The single boss is cheaper than fanning out, but not free: + +- **The boss is a dependency.** If its cluster is down, no group can join or rebalance, even one that only reads topics elsewhere. +- **Fencing is partial.** The boss checks the generation, but a standalone commit to another cluster isn't checked, so a zombie consumer can still commit offsets there. +- **The group is in pieces.** Another cluster shows offsets and no members, and offsets expire on each cluster separately. Lag tooling has to ask through the proxy. +- **The new protocol is out.** With KIP-848's `ConsumerGroupHeartbeat`, the broker computes the assignment itself, so the boss would have to resolve topics it doesn't have. + +If you'd rather not depend on this, keep each group's topics on one cluster, or use `assign()` and store the offsets yourself. + ## Transactions stop at the cluster boundary A transactional producer starts by sending `FindCoordinator` for its `transactional.id`, then `InitProducerId` to whichever broker the answer names. From then on that coordinator tracks the transaction, and when the producer commits it writes the commit or abort markers to the partitions involved. The coordinator, its transaction log and those markers all belong to one cluster. +Consumer groups got away with a single coordinator because nothing in them had to be atomic. A transaction is the opposite: its whole purpose is that several writes land together or not at all. + A coordinator can only see its own cluster, so the obvious answer is to give the client one everywhere. The router fans `InitProducerId` out to every backing cluster and starts a transaction on each. As the client registers partitions and produces, the router sends each call to the cluster that owns the topic, using the producer ID and epoch that cluster issued. The client holds one producer ID and the router holds the mapping to the rest. That mapping has to survive a router restart and be visible to every proxy instance, so it needs the kind of store described [above](#where-the-map-lives). That's real work, but it's work you can do. It falls over at commit. The client sends one `EndTxn` and the router has to turn it into one per cluster. On each cluster, commit is a point of no return: once the coordinator has written its decision, it will commit, whatever else happens. Until then it is free to refuse, and each cluster decides for itself. Cluster 1 can pass the point of no return while cluster 2 says no. None of the ways cluster 2 can say no are transient. The transaction may have timed out, since each cluster runs its own clock from the first partition added there. A zombie may have bumped the producer epoch. The ACLs may differ. Then `orders` has its records and `inventory` doesn't, and whatever the client is told is wrong for one of them. A state store doesn't rescue this. It can remember that the router meant to commit cluster 2, but it can't make cluster 2 agree. Reordering the commits doesn't help either, because whichever goes first can't be undone. -So stop negotiating and pick a boss again, but this time as a participant rather than a store: not a place to keep the map, but a coordinator that runs every transaction. The client's `InitProducerId`, `AddPartitionsToTxn` and `EndTxn` all go there, while the records still go to whichever cluster owns the topic. Commit is now a single decision on a single coordinator, so the atomicity problem is gone. +So stop negotiating and do what worked for consumer groups: pick a boss, and let its coordinator run every transaction. The client's `InitProducerId`, `AddPartitionsToTxn` and `EndTxn` all go there, while the records still go to whichever cluster owns the topic. Commit is now a single decision on a single coordinator, so the atomicity problem is gone. -It doesn't survive the first write to another cluster. The client registers each partition with the coordinator before it writes to it, and the boss's coordinator doesn't recognise `inventory`'s partitions, because they aren't in its cluster. It refuses them, and the transaction never gets as far as commit. +It doesn't survive the first write to another cluster. The consumer group boss was never asked about another cluster's topics, but here the client registers each partition with the coordinator before it writes to it. The boss's coordinator doesn't recognise `inventory`'s partitions, because they aren't in its cluster. It refuses them, and the transaction never gets as far as commit. Both attempts fail for the same reason. Somewhere there has to be a moment when every cluster has promised to commit before any of them does, and Kafka's transaction protocol has no such step. @@ -155,7 +187,7 @@ Does it matter for your deployment? Transactions are fine within a cluster, so t In principle the router could be that coordinator. It would start a transaction on each backing cluster, wait for all of them to prepare, record the decision, and then tell each one to commit. The client would still believe it was talking to a single coordinator. The hard part is the middle step. Someone has to durably record the decision between "everyone prepared" and "everyone committed", and survive a crash in the gap. A router that does that needs a durable log of its own, recovery after a restart, and a way to stop two proxy instances disagreeing about the outcome. That is the consensus layer from [Post 1]({% post_url 2026-10-12-these-are-not-the-brokers-you-are-connecting-to %}), and the Kafka community has just spent years moving from ZooKeeper to KRaft so that nobody has to run a second consensus system alongside Kafka. Whether asking a proxy to bring its own is acceptable depends on your constraints — it is not something we are planning to do. -That said, this is where the boss earns a second job. The decision log could live in a compacted topic on the same cluster that holds the producer ID map, with nothing new to deploy. The recovery logic is still yours to write. None of this has been built or tested — treat it as a sketch of a direction, not a recipe. +That said, this is where the boss earns a second job. The decision log could live in a compacted topic on the same cluster that holds the producer ID map, with nothing new to deploy. The recovery logic is still yours to write. That gets harder still when the client is already a coordinator. A stream processor like Flink can run its own two-phase commit across Kafka and other systems. Put a weaving router in front of Kafka and Flink sees one participant, while the router has to act as a coordinator for the clusters behind it: two-phase commit nested inside two-phase commit. The KIP describes a flat arrangement of one coordinator and its participants, and says nothing about a participant in the middle. @@ -171,12 +203,13 @@ None of the boss cluster roles are tested. They make sense in principle, but the | --- | --- | --- | | Topic creation (`CreateTopics`) | Default cluster for new topics (or catalogue store) | Works, with the hotchpotch caveat | | Producer ID mapping | Registry in a compacted topic | Works | -| Transactions | The boss's coordinator runs every transaction | Fails (participant) | +| Consumer groups | The boss's coordinator runs membership; offsets are split by topic | Works in principle (classic protocol only) | +| Transactions | The boss's coordinator runs every transaction | Fails (remote content) | | Distributed commit (KIP-939) | Decision log for two-phase commit | Feasible, untested | -The test for whether a boss works is whether it acts as a **store** or as a **participant**. +The test for whether a boss works is whether the content it is asked to hold or decide is **local** to its cluster or **remote**. -When the boss is a store, it holds proxy bookkeeping in internal topics. The proxy writes its state there and reads it back; the brokers on the boss are just durable storage for bytes, and the proxy enforces the semantics. When the boss is a participant, client traffic lands on it directly. Any write to the boss cluster makes it a participant. That works for `CreateTopics` where the boss hosts the newly created topic, but it fails when the boss is expected to participate on behalf of other clusters. A transaction coordinator on the boss can track and commit writes to its own partitions, but it rejects writes to partitions on other clusters because they do not exist in its metadata. +Local content is anything that lives on the boss itself. Proxy bookkeeping in internal topics is the clearest case: the proxy writes its state there and reads it back, and the brokers on the boss are just durable storage for bytes. A newly created topic that the boss hosts is local too, and so is consumer group membership, because it names no topics. Remote content is anything the boss would have to answer for on behalf of another cluster. A transaction coordinator on the boss can track and commit writes to its own partitions, but it rejects writes to partitions on other clusters because they do not exist in its metadata. Where that boss cluster lives is a separate choice: From 7a53446ac9383345c4690b6342bad81cd86b3ede Mon Sep 17 00:00:00 2001 From: Sam Barker Date: Mon, 2 Nov 2026 11:57:01 -0800 Subject: [PATCH 21/21] Update publication date/time to 2026-11-02 15:00 PST Signed-off-by: Sam Barker --- ....md => 2026-11-02-topic-weaving-one-client-many-clusters.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename _posts/{2026-11-03-topic-weaving-one-client-many-clusters.md => 2026-11-02-topic-weaving-one-client-many-clusters.md} (99%) diff --git a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md b/_posts/2026-11-02-topic-weaving-one-client-many-clusters.md similarity index 99% rename from _posts/2026-11-03-topic-weaving-one-client-many-clusters.md rename to _posts/2026-11-02-topic-weaving-one-client-many-clusters.md index a70beb42..2fc52cb7 100644 --- a/_posts/2026-11-03-topic-weaving-one-client-many-clusters.md +++ b/_posts/2026-11-02-topic-weaving-one-client-many-clusters.md @@ -1,7 +1,7 @@ --- layout: post title: "Topic Weaving: One client, many clusters" -date: 2026-11-03 15:00:00 +1300 +date: 2026-11-02 15:00:00 -0800 author: "Sam Barker" author_url: "https://github.com/sambarker" # noinspection YAMLSchemaValidation