Repository navigation
Fix concurrent metadata snapshot mutations - #329
Conversation
novatechflow
left a comment
There was a problem hiding this comment.
Review completed on fdb2743. No remaining blockers after the review/fix/regression cycle.
Validation: full repository go test -race, go vet ./..., published Go API compatibility, the 100% authentication coverage gate, and hosted CI/CodeQL pass. Deterministic regressions use real embedded etcd and force competing writes before commit. The original implementation failed the concurrent-create, expansion/delete versus create, startup-topic leakage, and failed-initial-read regressions. Additional cases cover same-name creates, deleted/recreated topics, read-to-watch replay, failed persistence, large expansion batches, and preservation of existing partition state. The issue-specific multi-broker kind reproduction was not run locally.
Release-note draft updated:
- Concurrent topic creates preserve every successful creation and the authoritative broker topology and partition counts. Each mutation reads etcd and compares the snapshot mod revision; a conflict rebuilds the requested change from a fresh read.
- Startup demo topics are not persisted. Failed reads/writes do not expose speculative topics, and an already-existing topic refreshes a stale broker cache.
- Partition expansion and topic retirement use the same snapshot fencing. A stale operation cannot modify a deleted and recreated topic. Expansion publishes missing partition state atomically in batches and preserves any existing state.
- Operator CAS retries merge the latest snapshot with the original desired state, so an auto-created topic retired during a conflict stays deleted. Unreadable snapshots produce an error instead of being overwritten. A deterministic regression reproduced the original resurrection bug.
- Snapshot watches retry failed initial reads and replay updates committed between their full read and watch registration.
No configuration migration is required.
PaxMachinaOne
left a comment
There was a problem hiding this comment.
Reviewed the fixes and regressions. Concurrent snapshot updates preserve topics, broker topology, and partition counts. Stale operations cannot modify recreated topics, and operator retries no longer resurrect deleted topics.
No remaining blockers found. CI and CodeQL pass, including race tests and 100% authentication coverage. Release-note draft updated.
Summary
Fixes #325. Concurrent topic creates now read the authoritative etcd snapshot and commit with a comparison against its mod revision. Conflicts retry from a fresh read, preserving other topics, brokers, and partition counts. Failed reads and writes do not expose an uncommitted topic. Bootstrap snapshots contain only the requested topic, never the startup demo topics.
Partition expansion and topic retirement use the same snapshot CAS path, so either operation cannot erase a concurrent create. They pin the observed topic generation to avoid modifying a deleted and recreated topic. Expansion writes its new partition states atomically with the snapshot in bounded batches and preserves existing partition state.
Operator publication also rebuilds each CAS retry from its original desired state and the latest snapshot, preventing resurrection of auto-created topics retired during a conflict. Unreadable snapshots are preserved and reported as errors.
Snapshot watches reseed after a failed startup read and start at the read revision plus one, closing the read-to-watch gap. The unreleased notes are updated.
Validation
go test -race -p 1 ./...,go vet ./..., and published Go API compatibility pass onfdb2743.fdb2743, including the full race suite, 100% authentication coverage, Kafka/librdkafka/schema-registry/consumer-group end-to-end tests, Kubernetes API tests, and nested modules.