Unable to remove cilium EtcdCluster

[jgrasett]

/kind bug

1. What kops version are you running? The command kops version, will display
this information.
v1.30.3

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
1.30.9

3. What cloud provider are you using?
AWS

4. What commands did you run? What is the simplest way to reproduce this issue?
Testing migration from KubeNet to Cilium. This was successful.
Testing rollback in case of failure.
Reverted to KubeNet and removed all Cilium components.
Issued:
kops edit cluster <clustername> and removed cilium etcdCluster entries. On saving configuration get:

A copy of your changes has been stored to "/tmp/kops-edit-461843561yaml"
Error: spec.etcdClusters[cilium]: Forbidden: EtcdClusters cannot be removed

5. What happened after the commands executed?
See Error from 4.

6. What did you expect to happen?
Configuration to be updated and when update applied, cilium etcdCluster to be removed.

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2024-07-01T16:45:23Z"
  generation: 14
  name: my.cluster
spec:
  api:
    dns: {}
  authorization:
    rbac: {}
  certManager:
    enabled: true
  channel: stable
  cloudProvider: aws
  clusterAutoscaler:
    balanceSimilarNodeGroups: false
    cpuRequest: 100m
    enabled: true
    expander: least-waste
    image: registry.k8s.io/autoscaling/cluster-autoscaler:v1.30.3
    memoryRequest: 300Mi
    newPodScaleUpDelay: 0s
    scaleDownDelayAfterAdd: 10m0s
    scaleDownUtilizationThreshold: "0.5"
    skipNodesWithLocalStorage: true
    skipNodesWithSystemPods: true
  configBase: s3://my-cluster-io-state-store/my.cluster
  etcdClusters:
  - cpuRequest: 100m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: control-plane-eu-west-2a
      name: a
    manager:
      backupRetentionDays: 90
      env:
      - name: ETCD_AUTO_COMPACTION_MODE
        value: revision
      - name: ETCD_AUTO_COMPACTION_RETENTION
        value: "2500"
    memoryRequest: 100Mi
    name: cilium
  - cpuRequest: 200m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: control-plane-eu-west-2a
      name: a
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: control-plane-eu-west-2a
      name: a
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeControllerManager:
    allocateNodeCIDRs: true
  kubeProxy:
    enabled: true
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  kubernetesVersion: 1.30.9
  masterPublicName: api.my.cluster
  networkCIDR: 172.20.0.0/16
  networking:
    kubenet: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  podIdentityWebhook:
    enabled: true
  sshAccess:
  subnets:
  - cidr: a.b.c.d/19
    name: eu-west-2a
    type: Public
    zone: eu-west-2a
  topology:
    dns:
      type: Public

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-07-01T16:45:23Z"
  generation: 2
  labels:
    kops.k8s.io/cluster: my.cluster
  name: control-plane-eu-west-2a
spec:
  image: ami-07e0ad8f78f635e60
  machineType: c5.large
  maxSize: 1
  minSize: 1
  role: Master
  rootVolumeType: gp3
  subnets:
  - eu-west-2a

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-07-01T16:45:23Z"
  generation: 3
  labels:
    kops.k8s.io/cluster: my.cluster
  name: nodes-eu-west-2a
spec:
  image: ami-0e60dc20ad389771d
  machineType: t3.medium
  maxSize: 3
  minSize: 3
  role: Node
  rootVolumeType: gp3
  subnets:
  - eu-west-2a

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.
I notice this when using -v 10 in kops edit cluster and wonder if removing the volumes might change the behaviour?

I0325 16:20:54.904932 191783 status.go:59] Querying AWS for etcd volumes I0325 16:20:54.904981 191783 status.go:70] Listing EC2 Volumes I0325 16:20:55.098982 191783 status.go:42] Cluster status (from cloud): {"etcdClusters":[{"name":"main","etcdMembers":[{"name":"a","volumeID":"vol-0034e90d8576f cd62"}]},{"name":"cilium","etcdMembers":[{"name":"a","volumeID":"vol-0bd75fec7d3efc0a5"}]},{"name":"events","etcdMembers":[{"name":"a","volumeID":"vol-07924f7ab3b75c24f"}]}]} I0325 16:20:55.099078 191783 s3fs.go:385] Reading file "s3://yucca-cyberowl-io-state-store/yucca.cyberowl.io/config" A copy of your changes has been stored to "/tmp/kops-edit-4217680820yaml" Error: spec.etcdClusters[cilium]: Forbidden: EtcdClusters cannot be removed

9. Anything else do we need to know?
This is only for testing, but I would like a complete rollback plan to undo everything in case we have to revert this for some reason when we roll it out. This is a nuclear testing cluster I can delete and recreate easily.