LFS proxy multipart uploads fail with more than one proxy replica (sessions are kept in pod memory) #321

Found during QA of #319 on kind (head 6f7340b), but it is not caused by that PR; the code is the same on main (bf2136c).

Problem. The LFS proxy keeps multipart upload sessions in memory in each pod (uploadSessions in cmd/proxy/lfs.go). The operator runs the LFS proxy as a Deployment with 2 replicas by default, behind the kafscale-lfs-proxy Service with no session affinity. So after POST /lfs/uploads, the PUT /lfs/uploads/{id}/parts/{n} and POST /lfs/uploads/{id}/complete calls can land on the other pod, which answers 404 upload_not_found. The started S3 multipart upload then stays in flight in the bucket.

Reproduction on kind. KafscaleCluster with spec.lfsProxy.enabled: true and http.enabled: true (2 proxy replicas), then from a pod in the cluster: init through the Service returns 200 with an upload_id, and the part upload through the same Service URL returned 404 upload_not_found in 3 of 4 attempts. aws s3api list-multipart-uploads on the data bucket then lists the started uploads.

Expected. A multipart upload through the Service works with any number of LFS proxy replicas. KafScale's stateless model keeps shared state in etcd or S3 rather than in pod memory, so the session state (or enough of it to resume: upload id, S3 key, topic, sizes, parts) should live there, or the session should be recoverable from S3 by any pod. Session affinity on the Service would only be a stopgap, since it breaks on pod restarts and scale events.

Also worth covering: in-flight S3 multipart uploads whose session is lost are never aborted.