Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-63320

[AWS][CAPI]loadbalancers are deleted in master machine when changing authoritativeAPI from MachineAPI to ClusterAPI

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • None
    • CLOUD Sprint 278, CLOUD Sprint 279
    • 2
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

          loadbalancers are deleted in master machine when changing authoritativeAPI from MachineAPI to ClusterAPI, then it will cause sync error when changing authoritativeAPI back to MachineAPI

      Version-Release number of selected component (if applicable):

          4.21.0-0.nightly-2025-10-19-181151

      How reproducible:

          always

      Steps to Reproduce:

          1.Install an AWS techpreview cluster, we can see there are loadbalancers in the AWSCluster
      
      liuhuali@Lius-MacBook-Pro huali-test % oc get awscluster huliu-aws1020b-rsrcn -n openshift-cluster-api -oyaml
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
      kind: AWSCluster
      metadata:
        annotations:
          cluster.x-k8s.io/managed-by: cluster-capi-operator-infracluster-controller
        creationTimestamp: "2025-10-20T03:41:22Z"
        generation: 1
        labels:
          cluster.x-k8s.io/cluster-name: huliu-aws1020b-rsrcn
        name: huliu-aws1020b-rsrcn
        namespace: openshift-cluster-api
        ownerReferences:
        - apiVersion: cluster.x-k8s.io/v1beta1
          blockOwnerDeletion: true
          controller: true
          kind: Cluster
          name: huliu-aws1020b-rsrcn
          uid: fbb38b3c-3cad-402b-9163-149f6092af17
        resourceVersion: "20146"
        uid: 4030074c-39d4-4e20-a2f9-615e60449ba4
      spec:
        bastion:
          enabled: false
        controlPlaneEndpoint:
          host: api-int.huliu-aws1020b.qe.devcluster.openshift.com
          port: 6443
        controlPlaneLoadBalancer:
          crossZoneLoadBalancing: false
          loadBalancerType: nlb
          name: huliu-aws1020b-rsrcn-int
          scheme: internet-facing
        identityRef:
          kind: AWSClusterControllerIdentity
          name: default
        network:
          vpc:
            availabilityZoneSelection: Ordered
            availabilityZoneUsageLimit: 3
            subnetSchema: PreferPrivate
        region: us-east-2
        secondaryControlPlaneLoadBalancer:
          crossZoneLoadBalancing: false
          loadBalancerType: nlb
          name: huliu-aws1020b-rsrcn-ext
          scheme: internet-facing
      status:
        ready: true
      
          2.Delete controlplanemachineset, then we can see master machines are sync to CAPI 
      
      liuhuali@Lius-MacBook-Pro huali-test % oc get machine huliu-aws1020b-rsrcn-master-0 -n openshift-machine-api -oyaml
      apiVersion: machine.openshift.io/v1beta1
      kind: Machine
      metadata:
        annotations:
          machine.openshift.io/instance-state: running
        creationTimestamp: "2025-10-20T03:38:28Z"
        finalizers:
        - sync.machine.openshift.io/finalizer
        - machine.machine.openshift.io
        generation: 3
        labels:
          machine.openshift.io/cluster-api-cluster: huliu-aws1020b-rsrcn
          machine.openshift.io/cluster-api-machine-role: master
          machine.openshift.io/cluster-api-machine-type: master
          machine.openshift.io/instance-type: m6i.xlarge
          machine.openshift.io/region: us-east-2
          machine.openshift.io/zone: us-east-2c
        name: huliu-aws1020b-rsrcn-master-0
        namespace: openshift-machine-api
        resourceVersion: "48288"
        uid: 10fc9a3d-e439-4710-9d9f-6e136969db73
      spec:
        authoritativeAPI: MachineAPI
        lifecycleHooks:
          preDrain:
          - name: EtcdQuorumOperator
            owner: clusteroperator/etcd
        metadata: {}
        providerID: aws:///us-east-2c/i-0fa92d671ac915000
        providerSpec:
          value:
            ami:
              id: ami-034f9c3fdfc80a314
            apiVersion: machine.openshift.io/v1beta1
            blockDevices:
            - ebs:
                encrypted: true
                iops: 0
                kmsKey:
                  arn: ""
                volumeSize: 120
                volumeType: gp3
            capacityReservationId: ""
            credentialsSecret:
              name: aws-cloud-credentials
            deviceIndex: 0
            iamInstanceProfile:
              id: huliu-aws1020b-rsrcn-master-profile
            instanceType: m6i.xlarge
            kind: AWSMachineProviderConfig
            loadBalancers:
            - name: huliu-aws1020b-rsrcn-int
              type: network
            - name: huliu-aws1020b-rsrcn-ext
              type: network
            metadata:
              creationTimestamp: null
            metadataServiceOptions: {}
            placement:
              availabilityZone: us-east-2b
              region: us-east-2
            securityGroups:
            - filters:
              - name: tag:Name
                values:
                - huliu-aws1020b-rsrcn-node
            - filters:
              - name: tag:Name
                values:
                - huliu-aws1020b-rsrcn-lb
            - filters:
              - name: tag:Name
                values:
                - huliu-aws1020b-rsrcn-controlplane
            subnet:
              filters:
              - name: tag:Name
                values:
                - huliu-aws1020b-rsrcn-subnet-private-us-east-2b
            tags:
            - name: kubernetes.io/cluster/huliu-aws1020b-rsrcn
              value: owned
            userDataSecret:
              name: master-user-data
      status:
        addresses:
        - address: 10.0.92.153
          type: InternalIP
        - address: ip-10-0-92-153.us-east-2.compute.internal
          type: InternalDNS
        - address: ip-10-0-92-153.us-east-2.compute.internal
          type: Hostname
        authoritativeAPI: MachineAPI
        conditions:
        - lastTransitionTime: "2025-10-20T03:42:58Z"
          message: 'Drain operation currently blocked by: [{Name:EtcdQuorumOperator Owner:clusteroperator/etcd}]'
          reason: HookPresent
          severity: Warning
          status: "False"
          type: Drainable
        - lastTransitionTime: "2025-10-20T03:42:50Z"
          status: "True"
          type: InstanceExists
        - lastTransitionTime: "2025-10-20T03:42:48Z"
          message: The AuthoritativeAPI status is set to 'MachineAPI'
          reason: AuthoritativeAPIMachineAPI
          severity: Info
          status: "False"
          type: Paused
        - lastTransitionTime: "2025-10-20T04:49:19Z"
          message: Successfully synchronized MAPI Machine to CAPI
          reason: ResourceSynchronized
          severity: ""
          status: "True"
          type: Synchronized
        - lastTransitionTime: "2025-10-20T03:42:48Z"
          status: "True"
          type: Terminable
        lastUpdated: "2025-10-20T03:42:58Z"
        nodeRef:
          kind: Node
          name: ip-10-0-92-153.us-east-2.compute.internal
          uid: e181b1a6-aeb1-4fd1-aa78-86b1e480dd33
        phase: Running
        providerStatus:
          conditions:
          - lastTransitionTime: "2025-10-20T03:42:50Z"
            message: Machine successfully created
            reason: MachineCreationSucceeded
            status: "True"
            type: MachineCreation
          instanceId: i-0fa92d671ac915000
          instanceState: running
        synchronizedGeneration: 3
      
          3.Change one master machine's authoritativeAPI from MachineAPI to ClusterAPI, sync successfully, but loadbalancers are deleted in the MAPI machine
      
      liuhuali@Lius-MacBook-Pro huali-test % oc edit machine huliu-aws1020b-rsrcn-master-0 -n openshift-machine-api      
      machine.machine.openshift.io/huliu-aws1020b-rsrcn-master-0 edited
      
      liuhuali@Lius-MacBook-Pro huali-test % oc get machine huliu-aws1020b-rsrcn-master-0 -n openshift-machine-api -oyaml
      apiVersion: machine.openshift.io/v1beta1
      kind: Machine
      metadata:
        annotations:
          machine.openshift.io/instance-state: running
        creationTimestamp: "2025-10-20T03:38:28Z"
        finalizers:
        - sync.machine.openshift.io/finalizer
        - machine.machine.openshift.io
        generation: 5
        labels:
          machine.openshift.io/cluster-api-cluster: huliu-aws1020b-rsrcn
          machine.openshift.io/cluster-api-machine-role: master
          machine.openshift.io/cluster-api-machine-type: master
          machine.openshift.io/instance-type: m6i.xlarge
          machine.openshift.io/region: us-east-2
          machine.openshift.io/zone: us-east-2c
        name: huliu-aws1020b-rsrcn-master-0
        namespace: openshift-machine-api
        resourceVersion: "132718"
        uid: 10fc9a3d-e439-4710-9d9f-6e136969db73
      spec:
        authoritativeAPI: ClusterAPI
        lifecycleHooks:
          preDrain:
          - name: EtcdQuorumOperator
            owner: clusteroperator/etcd
        metadata:
          annotations:
            machine.openshift.io/instance-state: running
          labels:
            machine.openshift.io/cluster-api-cluster: huliu-aws1020b-rsrcn
            machine.openshift.io/instance-type: m6i.xlarge
            machine.openshift.io/region: us-east-2
            machine.openshift.io/zone: us-east-2c
            node-role.kubernetes.io/master: ""
        providerID: aws:///us-east-2c/i-0fa92d671ac915000
        providerSpec:
          value:
            ami:
              id: ami-034f9c3fdfc80a314
            apiVersion: machine.openshift.io/v1beta1
            blockDevices:
            - ebs:
                encrypted: true
                kmsKey:
                  id: ""
                volumeSize: 120
                volumeType: gp3
            capacityReservationId: ""
            credentialsSecret:
              name: aws-cloud-credentials
            deviceIndex: 0
            iamInstanceProfile:
              id: huliu-aws1020b-rsrcn-master-profile
            instanceType: m6i.xlarge
            kind: AWSMachineProviderConfig
            metadata:
              creationTimestamp: null
            metadataServiceOptions:
              authentication: Optional
            placement:
              availabilityZone: us-east-2b
              region: us-east-2
            securityGroups:
            - filters:
              - name: tag:Name
                values:
                - huliu-aws1020b-rsrcn-node
            - filters:
              - name: tag:Name
                values:
                - huliu-aws1020b-rsrcn-lb
            - filters:
              - name: tag:Name
                values:
                - huliu-aws1020b-rsrcn-controlplane
            subnet:
              filters:
              - name: tag:Name
                values:
                - huliu-aws1020b-rsrcn-subnet-private-us-east-2b
            tags:
            - name: kubernetes.io/cluster/huliu-aws1020b-rsrcn
              value: owned
            userDataSecret:
              name: master-user-data
      status:
        addresses:
        - address: ip-10-0-92-153.us-east-2.compute.internal
          type: InternalDNS
        - address: 10.0.92.153
          type: InternalIP
        authoritativeAPI: ClusterAPI
        conditions:
        - lastTransitionTime: "2025-10-20T03:42:58Z"
          message: 'Drain operation currently blocked by: [{Name:EtcdQuorumOperator Owner:clusteroperator/etcd}]'
          reason: HookPresent
          severity: Warning
          status: "False"
          type: Drainable
        - lastTransitionTime: "2025-10-20T03:42:50Z"
          status: "True"
          type: InstanceExists
        - lastTransitionTime: "2025-10-20T08:05:52Z"
          message: The AuthoritativeAPI status is set to 'ClusterAPI'
          reason: AuthoritativeAPINotMachineAPI
          status: "True"
          type: Paused
        - lastTransitionTime: "2025-10-20T08:05:54Z"
          message: Successfully synchronized CAPI Machine to MAPI
          reason: ResourceSynchronized
          severity: ""
          status: "True"
          type: Synchronized
        - lastTransitionTime: "2025-10-20T03:42:48Z"
          status: "True"
          type: Terminable
        lastUpdated: "2025-10-20T03:42:58Z"
        nodeRef:
          kind: Node
          name: ip-10-0-92-153.us-east-2.compute.internal
          uid: e181b1a6-aeb1-4fd1-aa78-86b1e480dd33
        phase: Running
        providerStatus:
          conditions:
          - lastTransitionTime: "2025-10-20T03:42:50Z"
            message: Machine successfully created
            reason: MachineCreationSucceeded
            status: "True"
            type: MachineCreation
          instanceId: i-0fa92d671ac915000
          instanceState: running
        synchronizedGeneration: 1    
      
      4.Change the machine's authoritativeAPI back to MachineAPI, it reports sync error
      
      liuhuali@Lius-MacBook-Pro huali-test % oc edit machine huliu-aws1020b-rsrcn-master-0 -n openshift-machine-api      
      machine.machine.openshift.io/huliu-aws1020b-rsrcn-master-0 edited
      liuhuali@Lius-MacBook-Pro huali-test % oc get machine huliu-aws1020b-rsrcn-master-0 -n openshift-machine-api -oyaml
      apiVersion: machine.openshift.io/v1beta1
      kind: Machine
      metadata:
        annotations:
          machine.openshift.io/instance-state: running
        creationTimestamp: "2025-10-20T03:38:28Z"
        finalizers:
        - sync.machine.openshift.io/finalizer
        - machine.machine.openshift.io
        generation: 6
        labels:
          machine.openshift.io/cluster-api-cluster: huliu-aws1020b-rsrcn
          machine.openshift.io/cluster-api-machine-role: master
          machine.openshift.io/cluster-api-machine-type: master
          machine.openshift.io/instance-type: m6i.xlarge
          machine.openshift.io/region: us-east-2
          machine.openshift.io/zone: us-east-2c
        name: huliu-aws1020b-rsrcn-master-0
        namespace: openshift-machine-api
        resourceVersion: "133340"
        uid: 10fc9a3d-e439-4710-9d9f-6e136969db73
      spec:
        authoritativeAPI: MachineAPI
        lifecycleHooks:
          preDrain:
          - name: EtcdQuorumOperator
            owner: clusteroperator/etcd
        metadata:
          annotations:
            machine.openshift.io/instance-state: running
          labels:
            machine.openshift.io/cluster-api-cluster: huliu-aws1020b-rsrcn
            machine.openshift.io/instance-type: m6i.xlarge
            machine.openshift.io/region: us-east-2
            machine.openshift.io/zone: us-east-2c
            node-role.kubernetes.io/master: ""
        providerID: aws:///us-east-2c/i-0fa92d671ac915000
        providerSpec:
          value:
            ami:
              id: ami-034f9c3fdfc80a314
            apiVersion: machine.openshift.io/v1beta1
            blockDevices:
            - ebs:
                encrypted: true
                kmsKey:
                  id: ""
                volumeSize: 120
                volumeType: gp3
            capacityReservationId: ""
            credentialsSecret:
              name: aws-cloud-credentials
            deviceIndex: 0
            iamInstanceProfile:
              id: huliu-aws1020b-rsrcn-master-profile
            instanceType: m6i.xlarge
            kind: AWSMachineProviderConfig
            metadata:
              creationTimestamp: null
            metadataServiceOptions:
              authentication: Optional
            placement:
              availabilityZone: us-east-2b
              region: us-east-2
            securityGroups:
            - filters:
              - name: tag:Name
                values:
                - huliu-aws1020b-rsrcn-node
            - filters:
              - name: tag:Name
                values:
                - huliu-aws1020b-rsrcn-lb
            - filters:
              - name: tag:Name
                values:
                - huliu-aws1020b-rsrcn-controlplane
            subnet:
              filters:
              - name: tag:Name
                values:
                - huliu-aws1020b-rsrcn-subnet-private-us-east-2b
            tags:
            - name: kubernetes.io/cluster/huliu-aws1020b-rsrcn
              value: owned
            userDataSecret:
              name: master-user-data
      status:
        addresses:
        - address: 10.0.92.153
          type: InternalIP
        - address: ip-10-0-92-153.us-east-2.compute.internal
          type: InternalDNS
        - address: ip-10-0-92-153.us-east-2.compute.internal
          type: Hostname
        authoritativeAPI: MachineAPI
        conditions:
        - lastTransitionTime: "2025-10-20T03:42:58Z"
          message: 'Drain operation currently blocked by: [{Name:EtcdQuorumOperator Owner:clusteroperator/etcd}]'
          reason: HookPresent
          severity: Warning
          status: "False"
          type: Drainable
        - lastTransitionTime: "2025-10-20T03:42:50Z"
          status: "True"
          type: InstanceExists
        - lastTransitionTime: "2025-10-20T08:06:46Z"
          message: The AuthoritativeAPI status is set to 'MachineAPI'
          reason: AuthoritativeAPIMachineAPI
          severity: Info
          status: "False"
          type: Paused
        - lastTransitionTime: "2025-10-20T08:06:46Z"
          message: 'failed to convert Machine API machine to Cluster API machine: [spec.providerSpec.value.loadBalancers:
            Invalid value: []v1beta1.LoadBalancerReference(nil): must include load balancer
            named "huliu-aws1020b-rsrcn-int", spec.providerSpec.value.loadBalancers: Invalid
            value: []v1beta1.LoadBalancerReference(nil): must include load balancer named
            "huliu-aws1020b-rsrcn-ext"]'
          reason: FailedToConvertMAPIMachineToCAPI
          severity: Error
          status: "False"
          type: Synchronized
        - lastTransitionTime: "2025-10-20T03:42:48Z"
          status: "True"
          type: Terminable
        lastUpdated: "2025-10-20T08:06:46Z"
        nodeRef:
          kind: Node
          name: ip-10-0-92-153.us-east-2.compute.internal
          uid: e181b1a6-aeb1-4fd1-aa78-86b1e480dd33
        phase: Running
        providerStatus:
          conditions:
          - lastTransitionTime: "2025-10-20T03:42:50Z"
            message: Machine successfully created
            reason: MachineCreationSucceeded
            status: "True"
            type: MachineCreation
          instanceId: i-0fa92d671ac915000
          instanceState: running
        synchronizedGeneration: 0 

      Actual results:

          loadbalancers are deleted in master machine when changing authoritativeAPI from MachineAPI to ClusterAPI, then it will cause sync error when changing authoritativeAPI back to MachineAPI

      Expected results:

          loadbalancers should not be deleted in master machine when changing authoritativeAPI from MachineAPI to ClusterAPI, it should sync successfully when changing authoritativeAPI back to MachineAPI

      Additional info:

          new feature testing for https://issues.redhat.com/browse/OCPCLOUD-2709

              rmanak@redhat.com Radek Manak
              huliu@redhat.com Huali Liu
              None
              None
              Huali Liu Huali Liu
              None
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated: