Skip to content

Routing series/post4 topic weaving - #298

Merged
SamBarker merged 21 commits into
kroxylicious:mainfrom
SamBarker:routing-series/post4-topic-weaving
Nov 3, 2026
Merged

Routing series/post4 topic weaving#298
SamBarker merged 21 commits into
kroxylicious:mainfrom
SamBarker:routing-series/post4-topic-weaving

Conversation

@SamBarker

@SamBarker SamBarker commented Nov 1, 2026

Copy link
Copy Markdown
Member

Summary

Post 4 of the routing series: Topic Weaving: One client, many clusters. It is a work in progress, but the first half is written and I'd like feedback on it now rather than at the end.

What's ready for feedback

  • Introduction and refresher
  • Who sees what: the topic map, and how admin operations (create, delete, alter) have to agree with it
  • Broker ID collision, and why the runtime rather than the router does the translation
  • What a weaving router has to do
  • Transactions and KIP-939

Where I'd most like input

  • Technical accuracy of the framing: reads are aggregation (the clusters are the source of truth), admin writes are disaggregation (no source of truth, so the answer is policy).
  • Admin operation options (default cluster, rule on the name, catalogue, every cluster, refuse): are the trade-offs fair, and is anything missing?
  • Voice and length: does it hold together as a standalone read after posts 1 to 3?

Still to write

Producing (why PID translation is hard), Consuming, proxy placement, weaving vs replication vs connection switching, and the closing section. The working notes and HTML comment placeholders at the bottom of the file will be removed before publishing.

The filename is dated 2026-11-03, so the date needs revisiting if it slips.

@SamBarker
SamBarker force-pushed the routing-series/post4-topic-weaving branch from 3e0790a to 0ed04a9 Compare November 2, 2026 05:34
k-wall
k-wall reviewed Nov 2, 2026
View reviewed changes
Comment thread _drafts/stash-replica-routing.md
Point to make: KIP-939 (two-phase commit support) still does not fix this. If we went anywhere near it we would be in full distributed consensus pain. Not read or verified yet. Summarise from the doc before quoting, and check its claims against the KIP.

## Topic ID is also temporal
1. Topic IDs / UUIDs (KIP-516)At first glance, a Topic ID looks spatial (it’s just a coordinate for a topic). But it is actually a temporal lifecycle marker generated by the KRaft controller.The Semantics: If you delete a topic and recreate it with the same name, it gets a new UUID to prevent zombie delayed fetches from corrupting the new log.Why it breaks stateless math: You can mathematically fake a Node ID using $V = id + S \times t$ because Node IDs are predictable integers. You cannot mathematically derive a random backend UUID from a frontend virtual UUID.The Proxy Seam: To support API versions v13+ (where Produce and Fetch requests use UUIDs instead of strings), the proxy must maintain a stateful [Virtual Topic UUID -> Physical Topic UUID] mapping table in memory. If a proxy replica restarts and loses that map, it cannot route the frame. This is exactly why a proxy might intentionally cap PRODUCE API versions to v12—forcing clients to use string names—just to avoid handling cluster-specific Topic IDs.

@k-wall k-wall Nov 2, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: fetches don't mutate the transaction log in anyway.

k-wall
k-wall approved these changes Nov 2, 2026
View reviewed changes

@k-wall k-wall left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @SamBarker, another very thorough piece of work.

@SamBarker
SamBarker marked this pull request as ready for review November 2, 2026 17:22
@SamBarker
SamBarker requested a review from a team as a code owner November 2, 2026 17:22
@SamBarker
post 4: stub with section outline for topic weaving post
c240bb6 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
post 4: draft intro for topic weaving
e969553 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
post 4: refine intro
7fc8de1 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
post 4: ramp into topic weaving and add client-knowledge section
5f8d789 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
post 4: add transactions and KIP-939 sections
f69c8df 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
post 4: add 'Whose map is it anyway?' and router jobs sections
d723037 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
post 4: add broker ID collision section
40e98aa 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
post 4: add 'Writing to the map' section on admin operations
9450280 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
post 4: add producer ID section and tie the boss cluster through tran…
93021ee 
…sactions and KIP-939

Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
post 4: define boss cluster concept and store-vs-participant test
996cc9b 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
post 4: add proxy placement section
921aa3d 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
Add 'Know when to get off the train' section to topic weaving post
d29f2fd 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
Add closing section and partition weaving teaser
139afc6 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
Remove TODO list and stash notes; refine community invite
1c3ee2d 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
Post 4: consistency and voice pass
44a7557 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
update stash
c371096 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
remove cruft
b2a3282 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
Fix stream-branching post slug (date changed to 2026-10-29 before merge)
19e5476 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
Topic weaving: restructure sections and sharpen sorting office analogy
6495ae7 
Move 'What a weaving router has to do' ahead of the broker ID section and 'Picking a boss' after KIP-939. Use the sorting office counter to explain per-identity maps, rename 'Who sees what' to 'When names collide', clarify the node ID example and add a note on epoch bumps.

Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
Add consumer groups section to topic weaving post
89a890e 
Cover the classic group protocol with a boss for membership and offsets
split by topic, ahead of transactions as the contrasting case. Replace
the store/participant test in 'Picking a boss' with local versus remote
content, state the design-level status once in the intro, and trim the
repeated 'not built' caveats.

Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
Update publication date/time to 2026-11-02 15:00 PST
7a53446 
Signed-off-by: Sam Barker <sam@quadrocket.co.uk>
@SamBarker
SamBarker force-pushed the routing-series/post4-topic-weaving branch from 6da746f to 7a53446 Compare November 3, 2026 02:06
@SamBarker
SamBarker enabled auto-merge November 3, 2026 02:07
@SamBarker
SamBarker disabled auto-merge November 3, 2026 02:08
@SamBarker
SamBarker merged commit e4fe6eb into kroxylicious:main Nov 3, 2026
2 checks passed
@SamBarker
SamBarker deleted the routing-series/post4-topic-weaving branch November 3, 2026 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@k-wall k-wall k-wall approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@SamBarker @k-wall