Skip to content

feat(sync): preserve not-ready clusters in local kubeconfig - #87

Merged
onuryilmaz merged 5 commits into
mainfrom
feat/preserve-notready-on-sync
Nov 10, 2026
Merged

onuryilmaz merged 5 commits into
mainfrom
feat/preserve-notready-on-sync

Conversation

@onuryilmaz

Copy link
Copy Markdown
Contributor

Summary

  • Not-ready clusters are no longer deleted from the local kubeconfig on sync. A transient Ready=False condition (e.g. during a Greenhouse controller upgrade or a missing secret key) previously caused cloudctl to remove the local entry, making the cluster unreachable.
  • Existing local entries for not-ready clusters are preserved via stubs injected into the server-side config before the stale-cleanup pass — no content is overwritten.
  • A new --include-not-ready flag fully merges not-ready cluster spec data, for cases where the CRD spec is known-good but the controller is temporarily degraded.

Motivation

During the Greenhouse v0.18.0 → v0.19.0 rollout (PR cloudoperators/greenhouse#2245), all 208 ClusterKubeconfig resources in the sci namespace report Ready=False with reason KubeconfigMissing. Their spec.kubeconfig is fully populated. Running cloudctl sync in this window would delete all 208 entries from users' local kubeconfigs, cutting off cluster access at the worst possible time.

Behaviour change

Scenario Before After
Cluster Ready=False, exists in local kubeconfig Deleted on next sync Preserved
Cluster Ready=False, not yet in local kubeconfig Not added Not added (unchanged)
--include-not-ready flag N/A Spec data merged even if Ready=False
Cluster genuinely removed from Greenhouse Deleted Deleted (unchanged)

Test plan

  • TestInjectNotReadyStubs_PreservesLocalEntries — stub injection keeps existing local entries
  • TestInjectNotReadyStubs_SkipsUnknownClusters — no stubs for clusters not yet in local config
  • TestIncludeNotReady_MergesSpecData — --include-not-ready merges spec data from not-ready CRDs
  • All existing cmd tests pass

Previously, clusters whose Ready condition was False were silently
removed from the local kubeconfig on the next sync. This meant a
transient controller failure (e.g. missing secret key during a
Greenhouse upgrade) would cut off access to the cluster, making it
impossible to reach and investigate the issue.

This change keeps not-ready entries intact in the local kubeconfig:
- notReady clusters that already exist locally are preserved via stub
  entries injected into the server-side config before the stale-cleanup
  pass in mergeKubeconfig. The stubs prevent deletion without touching
  the local entry's content.
- A new --include-not-ready flag opts into fully merging not-ready
  cluster spec data, useful when the controller is temporarily degraded
  but the CRD spec is known-good (e.g. during a Greenhouse rollout).

Signed-off-by: onuryilmaz <onur.yilmaz@sap.com>
@onuryilmaz
onuryilmaz requested review from a team as code owners November 7, 2026 14:09
Copilot AI balanced review requested due to automatic review settings November 7, 2026 14:09

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The sync logic can abort on prefixed references, create dangling references, and delete managed entries when resource and context names differ.

6 open issues
What changed in this PR

This PR aims to preserve local Kubernetes access during transient Greenhouse readiness failures by keeping existing kubeconfig entries for not-ready clusters.

Changes:

  • Adds local-entry stubs intended to prevent stale cleanup from deleting not-ready clusters.
  • Introduces --include-not-ready to opt into merging their spec data.
  • Adds tests for stub creation, unknown-cluster handling, and not-ready spec conversion.
File Description
cmd/​sync.go Adds preservation logic, the opt-in flag, and updated command help.
cmd/​sync_test.go Adds tests for preservation stubs and not-ready spec merging.

Review effort: Balanced

Comment thread cmd/sync.go Resolved
Comment thread cmd/sync.go Resolved
Comment thread cmd/sync.go Resolved
Comment thread cmd/sync.go Resolved
Comment thread cmd/sync.go Resolved
Comment thread cmd/sync.go Resolved
Six issues identified in PR review:

- Match stubs by spec context names (not CKC resource name) so clusters
  whose context name differs from the resource name (e.g. e2e "demo"/
  "ctx1") are correctly protected from stale cleanup.
- Strip managed prefix from stub context references to prevent
  double-prefixed names (cloudctl:cloudctl:...) when mergeIdenticalUsers
  is disabled or when the merger re-applies the prefix.
- Read includeNotReady from Viper so CLOUDCTL_INCLUDE_NOT_READY env var
  and config-file values are honoured, consistent with all other flags.
- Allow --include-not-ready to bypass the len(ready)==0 early return;
  previously the flag had no effect when every cluster was not-ready,
  which is the primary motivation for the flag.
- Pass the actual merge/skip sets to result builders so clusters merged
  via --include-not-ready are reported as synced, not skipped.
- Skip unmanaged contexts in injectNotReadyStubs to avoid accidentally
  converting a manually-created entry into a managed one.

Signed-off-by: onuryilmaz <onur.yilmaz@sap.com>
Signed-off-by: onuryilmaz <onur.yilmaz@sap.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread cmd/sync.go Resolved
Comment thread cmd/sync.go Resolved
Comment thread cmd/sync_test.go Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Preserved not-ready contexts still enter authentication merging and deduplication, which can replace their credentials and change the identity used to access clusters without --include-not-ready.

3 open issues

Review effort: Balanced

- Bulgu 1: notReady entries with empty spec are kept on the preservation
  path even with --include-not-ready; previously they would be merged with
  no incoming data, causing their existing local entries to be deleted.
  Introduces hasSpecData() helper and filters the merged/skipped split.

- Bulgu 2: stub authinfo entries are no longer added to serverConfig to
  avoid mergeKubeconfig's deduplication pass assigning a new hash-derived
  managed name and redirecting the preserved context to a different identity.
  injectNotReadyStubs now returns a preSeed map (server-side name →
  local managed name) that is pre-populated into authInfoMap before merge.
  The no-merge mode cleanup path also honours the pre-seed map so it does
  not delete protected managed authinfos.

- Bulgu 3: TestIncludeNotReady_BypassesEarlyReturnWhenAllNotReady now
  calls the production shouldSkipMerge() helper (extracted from runSync)
  instead of evaluating a copied expression, so it would catch a revert.

Signed-off-by: onuryilmaz <onur.yilmaz@sap.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

With --merge-identical-users=false, context resolution ignores pre-seeded user mappings and can rewrite a reused unmanaged user reference to a nonexistent user, breaking cluster access.

1 open issue
3 resolved since last review

Review effort: Balanced

Comment thread cmd/sync.go Resolved
When --merge-identical-users=false, context merging and stale-context
validation were ignoring preSeedAuthInfoMap and falling back to
managedNameFunc, which would rewrite a stub context's authinfo reference
to cloudctl:<name> even though that entry does not exist in localConfig,
breaking cluster access.

Both the context-merge path (lines 775-776) and the stale-context
validation path (lines 872-873) now check preSeedAuthInfoMap first and
use the pre-seeded local managed name when present.

Signed-off-by: onuryilmaz <onur.yilmaz@sap.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because critical preservation and user-reference edge cases could delete managed entries or change a context’s identity.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@onuryilmaz
onuryilmaz merged commit ab26bd1 into main Nov 10, 2026
8 checks passed
@onuryilmaz
onuryilmaz deleted the feat/preserve-notready-on-sync branch November 10, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants