Skip to content

Fix concurrent metadata snapshot mutations - #329

Merged
PaxMachinaOne merged 3 commits into
KafScale:mainfrom
novatechflow:fix/325-metadata-snapshot-cas
Nov 14, 2026
Merged

PaxMachinaOne merged 3 commits into
KafScale:mainfrom
novatechflow:fix/325-metadata-snapshot-cas

Conversation

@novatechflow

@novatechflow novatechflow commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Fixes #325. Concurrent topic creates now read the authoritative etcd snapshot and commit with a comparison against its mod revision. Conflicts retry from a fresh read, preserving other topics, brokers, and partition counts. Failed reads and writes do not expose an uncommitted topic. Bootstrap snapshots contain only the requested topic, never the startup demo topics.

Partition expansion and topic retirement use the same snapshot CAS path, so either operation cannot erase a concurrent create. They pin the observed topic generation to avoid modifying a deleted and recreated topic. Expansion writes its new partition states atomically with the snapshot in bounded batches and preserves existing partition state.

Operator publication also rebuilds each CAS retry from its original desired state and the latest snapshot, preventing resurrection of auto-created topics retired during a conflict. Unreadable snapshots are preserved and reported as errors.

Snapshot watches reseed after a failed startup read and start at the read revision plus one, closing the read-to-watch gap. The unreleased notes are updated.

Validation

  • Deterministic embedded-etcd regressions reproduce lost concurrent creates, stale snapshot writes by expansion/deletion, startup topic leakage, and missed startup recovery on the original implementation.
  • Fixed regressions pass under the race detector, including same-name creates, delete/recreate races, large expansion batches, existing partition-state preservation, persistence failures, and read-to-watch replay.
  • Metadata package race suite, repository-wide vet, and published Go API compatibility pass.
  • The operator retry regression reproduces resurrection with the original merge behavior and passes three race-detector runs with the fix; unreadable-snapshot preservation is covered.
  • Full go test -race -p 1 ./..., go vet ./..., and published Go API compatibility pass on fdb2743.
  • Hosted CI and CodeQL pass on fdb2743, including the full race suite, 100% authentication coverage, Kafka/librdkafka/schema-registry/consumer-group end-to-end tests, Kubernetes API tests, and nested modules.
  • The issue-specific multi-broker kind reproduction was not run locally.

@PaxMachinaOne
PaxMachinaOne marked this pull request as ready for review November 13, 2026 15:27

@novatechflow novatechflow left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed on fdb2743. No remaining blockers after the review/fix/regression cycle.

Validation: full repository go test -race, go vet ./..., published Go API compatibility, the 100% authentication coverage gate, and hosted CI/CodeQL pass. Deterministic regressions use real embedded etcd and force competing writes before commit. The original implementation failed the concurrent-create, expansion/delete versus create, startup-topic leakage, and failed-initial-read regressions. Additional cases cover same-name creates, deleted/recreated topics, read-to-watch replay, failed persistence, large expansion batches, and preservation of existing partition state. The issue-specific multi-broker kind reproduction was not run locally.

Release-note draft updated:

  • Concurrent topic creates preserve every successful creation and the authoritative broker topology and partition counts. Each mutation reads etcd and compares the snapshot mod revision; a conflict rebuilds the requested change from a fresh read.
  • Startup demo topics are not persisted. Failed reads/writes do not expose speculative topics, and an already-existing topic refreshes a stale broker cache.
  • Partition expansion and topic retirement use the same snapshot fencing. A stale operation cannot modify a deleted and recreated topic. Expansion publishes missing partition state atomically in batches and preserves any existing state.
  • Operator CAS retries merge the latest snapshot with the original desired state, so an auto-created topic retired during a conflict stays deleted. Unreadable snapshots produce an error instead of being overwritten. A deterministic regression reproduced the original resurrection bug.
  • Snapshot watches retry failed initial reads and replay updates committed between their full read and watch registration.

No configuration migration is required.

@PaxMachinaOne PaxMachinaOne left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the fixes and regressions. Concurrent snapshot updates preserve topics, broker topology, and partition counts. Stale operations cannot modify recreated topics, and operator retries no longer resurrect deleted topics.

No remaining blockers found. CI and CodeQL pass, including race tests and 100% authentication coverage. Release-note draft updated.

@PaxMachinaOne
PaxMachinaOne merged commit 7d3271d into KafScale:main Nov 14, 2026
14 checks passed
@novatechflow
novatechflow deleted the fix/325-metadata-snapshot-cas branch November 14, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

broker: CreateTopic overwrites the etcd metadata snapshot without compare-and-swap (stale broker view, lost topics)

2 participants