[novatechflow]

Summary

Fixes #321. Authoritative multipart sessions and checksums live in trusted etcd, so requests can move between LFS proxy replicas or survive a pod restart. S3 remains blob storage. Per-upload leased locks and revision/owner comparisons fence stale session writes and deletes, and lock-key/lease cleanup remains bounded after cancellation.

Part identities are persisted before S3 writes. All retries for a part must use identical bytes; conflicting bytes return 409 part_conflict, and failed attempts do not advance the checksum. Completion requires every part exactly once in order and checks the expected checksum before finalizing S3. A lost successful S3 completion response can be recovered by checking the existing unique object key and size. Durable publication/discard markers prevent stale cleanup from deleting possibly Kafka-referenced blobs. Kafka retry semantics remain at-least-once.

Expiry uses persisted deadlines. The orphan sweep checks a trusted upload index, preserves active sessions through TTL changes, and performs no abort when ownership cannot be read. Session records/indexes are updated atomically. Record-size headroom supports metadata for all 10,000 S3 parts within default etcd request limits.

Deployment and release notes

• Multipart requires KAFSCALE_PROXY_ETCD_ENDPOINTS, even with static brokers. Operator and Docker Compose deployments already configure it.
• Restart uploads from the earlier unsigned S3 session implementation or development etcd format without part identities (410 upload_restart_required). Unknown future session versions are preserved rather than modified.
• etcd credentials need the session, index, lock-prefix, and lease permissions documented in docs/operations.md.
• Orphan cleanup needs s3:ListBucketMultipartUploads and s3:AbortMultipartUpload; ambiguous-completion recovery uses s3:GetObject.
• Parts remain sequential. Ambiguous Kafka acknowledgements can lead to duplicate envelopes; possibly referenced blobs are preserved.
• docs/releases/unreleased.md, operations documentation, and multipart API responses are updated.

Validation

• Fault-injection regressions cover S3-session tampering, deferred metadata transactions, changed retry bytes, cancellation during lock cleanup, invalid completion lists, checksum mismatch cleanup, lost S3 completion responses, and missing/wrong-size recovery objects.
• A real TCP Kafka acknowledgement followed by failed metadata deletion and a later abort verifies the acknowledged blob remains intact.
• Further tests cover TTL differences, ownership-read outages, revoked leases, legacy/future formats, 10,000-part metadata, and cross-replica SHA-256/MD5/CRC32 uploads.
• Tests use real embedded etcd and shared fake S3.
• go vet ./... and the published Go API compatibility gate pass.
• Full go test -race ./..., go vet ./..., and the published API gate pass on implementation commit 78f6ca5. Follow-up f43dc3d adds a stale-discard regression for the 100% security coverage gate.
• Hosted CI and CodeQL pass on f43dc3d, including the full race suite, 100% authentication coverage, Kafka/librdkafka/schema-registry/consumer-group end-to-end tests, Kubernetes API tests, and nested modules. Dedicated MinIO/kind multipart fault-injection QA was not run locally.

[novatechflow]

Review completed on f43dc3d. The review/fix/regression cycle found no remaining blockers in the multipart flow.

Validation: full go test -race ./..., full go vet ./..., the published Go API gate, and hosted CI/CodeQL pass. The authentication gate covers 100% of its required statements (736/736 across 15 files). Regression coverage uses real embedded etcd, shared fake S3, and a TCP Kafka acknowledgement fixture, including delayed RPCs, lost responses, stale owners, cancellation, preservation of an acknowledged blob after failed metadata deletion, and stale discard rejection before any S3 cleanup. Dedicated live MinIO/kind multipart fault injection was not run locally.

Release-note draft updated, with matching operations/API docs:

• Multipart sessions and authoritative checksums are shared through trusted etcd, allowing requests across replicas and pod restarts. etcd is required even with static brokers; operator and Compose deployments already configure it.
• Revision/owner fences, bounded cancellation cleanup, immutable per-part hashes, and identical-byte retries prevent stale writes and conflicting retries. Conflicting bytes return 409 part_conflict.
• Completion requires every part exactly once in order and checks the expected checksum before S3 finalization. Lost successful S3 completion responses can be recovered; publication/discard markers prevent cleanup from deleting possibly Kafka-referenced blobs. Kafka retries remain at-least-once.
• Expiry/orphan cleanup honors persisted deadlines, protects active uploads through TTL changes, and fails closed on ownership-read failure. Required S3 permissions and etcd prefix permissions are documented.
• Restart uploads from earlier unsigned S3 sessions or development etcd records without durable part identities (410 upload_restart_required).