Repository navigation
Conversation
Draft for review. Also points the pw and partition-weaving redirects at the new post. Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
…roup.id prefixing Open the train analogy as a fetch from Tatooine (virtual cluster) with Mos Eisley and Mos Espa as the backing clusters, dropping the Post 2 ticket reference. Note that Produce, Fetch and ListOffsets never need splitting because each virtual node maps to one route. Add the group.id collision hazard for standalone commits and have the router prefix group IDs. Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
|
|
||
| [Post 4]({% post_url 2026-11-02-topic-weaving-one-client-many-clusters %}) said the railway had run out of track. It had, for topics. For partitions there turns out to be a siding. | ||
|
|
||
| Picture a train coming back from Tatooine with 32 carriages. A carriage is a partition and a seat within it is an offset, so a ticket for carriage 13, seat 342323 means partition 13 at offset 342323. Passengers were loaded into a carriage on the outbound journey, by hashing their name and taking the remainder after dividing by 32, so everyone called Smith ends up in the same one, every time. Carriages 0 to 15 are one portion of the train, which started out in Mos Eisley. Carriages 16 to 31 are another, which started out in Mos Espa. Two towns on one planet, with the portions coupled together at a junction that neither the passengers nor the consumer at the end of the line ever see. The consumer believes it is meeting one 32-carriage train from Tatooine. Each portion keeps its own reservations, so seat 342323 only means something within its own carriage. The consumer's ticket says carriage 20. The guard knows that is carriage 4 of the Mos Espa portion. |
There was a problem hiding this comment.
maybe dropping the seat number to something more realistic for a train carriage of the analogy would help?
There was a problem hiding this comment.
Good point. The seat number went along with the train: it's now a plain offset 54. Fixed in b38638b.
|
|
||
| [Post 4]({% post_url 2026-11-02-topic-weaving-one-client-many-clusters %}) said the railway had run out of track. It had, for topics. For partitions there turns out to be a siding. | ||
|
|
||
| Picture a train coming back from Tatooine with 32 carriages. A carriage is a partition and a seat within it is an offset, so a ticket for carriage 13, seat 342323 means partition 13 at offset 342323. Passengers were loaded into a carriage on the outbound journey, by hashing their name and taking the remainder after dividing by 32, so everyone called Smith ends up in the same one, every time. Carriages 0 to 15 are one portion of the train, which started out in Mos Eisley. Carriages 16 to 31 are another, which started out in Mos Espa. Two towns on one planet, with the portions coupled together at a junction that neither the passengers nor the consumer at the end of the line ever see. The consumer believes it is meeting one 32-carriage train from Tatooine. Each portion keeps its own reservations, so seat 342323 only means something within its own carriage. The consumer's ticket says carriage 20. The guard knows that is carriage 4 of the Mos Espa portion. |
There was a problem hiding this comment.
space train!? Star wars is a nice neutral location but I'm suddenly thinking planets and spaceships, even with the description of towns.
Maybe we could go for the Japanese bullet train and real locations? I guess that brings baggage where the real geography won't match the idea.
Go LOTR/Middle Earth instead? It's pastoral and land-based, ready for industrial rail I reckon.
There was a problem hiding this comment.
I've dropped the train entirely rather than re-skin it. Once the made-up setting was gone it wasn't adding anything the plain description didn't, and mixing real cluster names into a metaphor seemed worse than either. The section now just walks through the orders / orders-eu scenario. Fixed in b38638b.
|
|
||
| [Post 4]({% post_url 2026-11-02-topic-weaving-one-client-many-clusters %}) said the railway had run out of track. It had, for topics. For partitions there turns out to be a siding. | ||
|
|
||
| Picture a train coming back from Tatooine with 32 carriages. A carriage is a partition and a seat within it is an offset, so a ticket for carriage 13, seat 342323 means partition 13 at offset 342323. Passengers were loaded into a carriage on the outbound journey, by hashing their name and taking the remainder after dividing by 32, so everyone called Smith ends up in the same one, every time. Carriages 0 to 15 are one portion of the train, which started out in Mos Eisley. Carriages 16 to 31 are another, which started out in Mos Espa. Two towns on one planet, with the portions coupled together at a junction that neither the passengers nor the consumer at the end of the line ever see. The consumer believes it is meeting one 32-carriage train from Tatooine. Each portion keeps its own reservations, so seat 342323 only means something within its own carriage. The consumer's ticket says carriage 20. The guard knows that is carriage 4 of the Mos Espa portion. |
There was a problem hiding this comment.
something else isn't sitting quite right:
Picture a train coming back from Tatooine with 32 carriages
makes it sound like tatooine exists at the start, then later the reader has to figure out that Tatooine is fabricated. I wonder if we could make that clearer earlier?
There was a problem hiding this comment.
Resolved by the same change. There's no fictional location left, so nothing for the reader to untangle.
|
|
||
| ## What would you use it for? | ||
|
|
||
| The best case I have is a migration. The runtime gives you nothing for it today. None of the others are much better, and I've set each against the thing that would usually beat it. |
There was a problem hiding this comment.
| The best case I have is a migration. The runtime gives you nothing for it today. None of the others are much better, and I've set each against the thing that would usually beat it. | |
| The best case I have is a migration, for moving a very large topic between clusters. The runtime gives you nothing for it today. None of the other cases are much better, and I've set each against the thing that would usually beat it. |
I found this a bit too abbreviated and think it's good to expand the first sentence to link to the bullets better. Maybe the thing should also be the solution?
There was a problem hiding this comment.
Applied, including "the solution".
|
|
||
| Every proxy instance must give the same identity the same answer, as in Post 4. Because the layout is configuration, every instance has the same list, so there's no state to share. | ||
|
|
||
| Changing it is where the care goes. The rule is that a logical partition number must keep meaning the same log. Appending a constituent to the end satisfies that: existing numbers don't move and the new ones are simply new. It does grow N, which changes where keyed records land, exactly as adding partitions to any Kafka topic does. Data already written stays where it is. If the new constituent is on a cluster the router has no route to yet, `numRoutes` changes too, and every virtual node ID moves with it, a coordinated reconnect for all clients of that VKC. A rolling update is tolerable for an append. An instance that hasn't picked up the change returns errors for the new partitions, and clients retry. Reordering, removing, resizing or relocating a constituent is a migration, and no amount of careful rollout makes it anything else. The piecemeal migration above is one of those, so it needs a deliberate cutover and not a config push. |
There was a problem hiding this comment.
first appearance of the term logical partition number, we've not used logical/physical terms here yet, might be worth defining somewhere to keep this blog self-contained
There was a problem hiding this comment.
Added a definition of virtual and physical at first use, in the opening section. I also switched "logical" to "virtual" throughout, since that's the project's term (virtual cluster, virtual node ID).
…weaving post Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Drop the train and Star Wars analogy in favour of a direct description of the two clusters, define virtual and physical partitions at first use, use 'virtual' in place of 'logical' to match project terminology, and expand the use cases introduction. Assisted-by: IBM Bob Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
b38638b to
b492b85
Compare
…imagery Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
Adds the fifth post in the routing series, Partition Weaving: Stretching the seams of a topic, dated 2026-11-03.
partition-ranges,partition-range-management,topicIds,consumer-groups-v1), with prose alt text.pwandpartition-weavingentries to_data/redirects/blog.yaml.sb/stream-branchingredirect to the published 2026/10/29 URL.