Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-54333

[CAPI] Update labels and annotations will sync to existing machines and nodes

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

          The changes to machine set labels and annotations in MAPI are not synchronized to existing machines and nodes, they only affect new machines created after the update. But for CAPI they are synchronized to existing machines and nodes. There appear to be some gaps between MAPI and CAPI that may cause inconsistencies when migrating.
      
          awsmachine will not be consistent with others (machineset, machine, node) on removing labels or annotations, but for adding and updating labels or annotations, it will be consistent with others.

      Version-Release number of selected component (if applicable):

          4.19.0-0.nightly-2025-03-26-070608

      How reproducible:

      always    

      Steps to Reproduce:

          1.Create a awsmachinetemplate, then a capi machineset.
      
      liuhuali@Lius-MacBook-Pro huali-test % cat capimachineset926.yaml 
      apiVersion: cluster.x-k8s.io/v1beta1
      kind: MachineSet
      metadata:
        labels:
          cluster.x-k8s.io/cluster-name: huliu-aws327a-gt8j4
        name: capi-machineset
        namespace: openshift-cluster-api
      spec:
        clusterName: huliu-aws327a-gt8j4
        deletePolicy: Newest
        replicas: 1
        selector:
          matchLabels:
            cluster.x-k8s.io/cluster-name: huliu-aws327a-gt8j4
            machine.openshift.io/cluster-api-cluster: huliu-aws327a-gt8j4
        template:
          metadata:
            annotations:
              anno1: ""
              anno3: ""
              anno5: anno5
              anno7: anno7
            labels:
              lable1: ""
              lable3: ""
              label5: label5
              label7: label7
              cluster.x-k8s.io/cluster-name: huliu-aws327a-gt8j4
              machine.openshift.io/cluster-api-cluster: huliu-aws327a-gt8j4
          spec:
            bootstrap:
              dataSecretName: worker-user-data
            clusterName: huliu-aws327a-gt8j4
            infrastructureRef:
              apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
              kind: AWSMachineTemplate
              name: aws-machinetemplate 
      
      liuhuali@Lius-MacBook-Pro huali-test % oc create -f capimachineset926.yaml
      machineset.cluster.x-k8s.io/capi-machineset created
      
      2. Check there are the labels and annotations on machine, awsmachine and node.
      
      liuhuali@Lius-MacBook-Pro huali-test % oc get machine.c capi-machineset-8jkzm -oyaml
      apiVersion: cluster.x-k8s.io/v1beta1
      kind: Machine
      metadata:
        annotations:
          anno1: ""
          anno3: ""
          anno5: anno5
          anno7: anno7
        creationTimestamp: "2025-03-27T11:33:34Z"
        finalizers:
        - machine.cluster.x-k8s.io
        generation: 2
        labels:
          cluster.x-k8s.io/cluster-name: huliu-aws327a-gt8j4
          cluster.x-k8s.io/set-name: capi-machineset
          label5: label5
          label7: label7
          lable1: ""
          lable3: ""
          machine.openshift.io/cluster-api-cluster: huliu-aws327a-gt8j4
        name: capi-machineset-8jkzm
        namespace: openshift-cluster-api
      ...
      liuhuali@Lius-MacBook-Pro huali-test % oc get awsmachine capi-machineset-8jkzm  -oyaml
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
      kind: AWSMachine
      metadata:
        annotations:
          anno1: ""
          anno3: ""
          anno5: anno5
          anno7: anno7
          cluster.x-k8s.io/cloned-from-groupkind: AWSMachineTemplate.infrastructure.cluster.x-k8s.io
          cluster.x-k8s.io/cloned-from-name: aws-machinetemplate
          sigs.k8s.io/cluster-api-provider-aws-last-applied-security-groups: '{"sg-0017fbfe45a8d1c43":{},"sg-05fe540dfac0dec53":{}}'
          sigs.k8s.io/cluster-api-provider-last-applied-tags-on-volumes: '{"vol-01238cbd748334f68":{}}'
        creationTimestamp: "2025-03-27T11:33:34Z"
        finalizers:
        - awsmachine.infrastructure.cluster.x-k8s.io
        generation: 2
        labels:
          cluster.x-k8s.io/cluster-name: huliu-aws327a-gt8j4
          cluster.x-k8s.io/set-name: capi-machineset
          label5: label5
          label7: label7
          lable1: ""
          lable3: ""
          machine.openshift.io/cluster-api-cluster: huliu-aws327a-gt8j4
        name: capi-machineset-8jkzm
        namespace: openshift-cluster-api
      ...
      
      liuhuali@Lius-MacBook-Pro huali-test % oc get node ip-10-0-8-9.us-east-2.compute.internal -oyaml
      apiVersion: v1
      kind: Node
      metadata:
        annotations:
          anno1: ""
          anno3: ""
          anno5: anno5
          anno7: anno7
      ...
        labels:
      ...
          label5: label5
          label7: label7
          lable1: ""
          lable3: ""
      ... 
      
      3. Update the machineset, change the labels and annotations to below
        template:
          metadata:
            annotations:
              anno1: newanno1
              anno6: anno6
            labels:
              cluster.x-k8s.io/cluster-name: huliu-aws327a-gt8j4
              label6: label6
              lable3: newlabel3
              machine.openshift.io/cluster-api-cluster: huliu-aws327a-gt8j4 
      
      4. Check on the machine, awsmachine, node. Found the updates are synchronized to the existing machine and node, but the deleted labels and annotations still shows on awsmachine.
      
      liuhuali@Lius-MacBook-Pro huali-test % oc get machine.c capi-machineset-8jkzm  -oyaml           
      apiVersion: cluster.x-k8s.io/v1beta1
      kind: Machine
      metadata:
        annotations:
          anno1: newanno1
          anno6: anno6
        creationTimestamp: "2025-03-27T11:33:34Z"
        finalizers:
        - machine.cluster.x-k8s.io
        generation: 2
        labels:
          cluster.x-k8s.io/cluster-name: huliu-aws327a-gt8j4
          cluster.x-k8s.io/set-name: capi-machineset
          label6: label6
          lable3: newlabel3
          machine.openshift.io/cluster-api-cluster: huliu-aws327a-gt8j4
        name: capi-machineset-8jkzm
        namespace: openshift-cluster-api 
      ...
      
      liuhuali@Lius-MacBook-Pro huali-test % oc get awsmachine capi-machineset-8jkzm  -oyaml          
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
      kind: AWSMachine
      metadata:
        annotations:
          anno1: newanno1
          anno3: ""
          anno5: anno5
          anno6: anno6
          anno7: anno7
          cluster.x-k8s.io/cloned-from-groupkind: AWSMachineTemplate.infrastructure.cluster.x-k8s.io
          cluster.x-k8s.io/cloned-from-name: aws-machinetemplate
          sigs.k8s.io/cluster-api-provider-aws-last-applied-security-groups: '{"sg-0017fbfe45a8d1c43":{},"sg-05fe540dfac0dec53":{}}'
          sigs.k8s.io/cluster-api-provider-last-applied-tags-on-volumes: '{"vol-01238cbd748334f68":{}}'
        creationTimestamp: "2025-03-27T11:33:34Z"
        finalizers:
        - awsmachine.infrastructure.cluster.x-k8s.io
        generation: 2
        labels:
          cluster.x-k8s.io/cluster-name: huliu-aws327a-gt8j4
          cluster.x-k8s.io/set-name: capi-machineset
          label5: label5
          label6: label6
          label7: label7
          lable1: ""
          lable3: newlabel3
          machine.openshift.io/cluster-api-cluster: huliu-aws327a-gt8j4
        name: capi-machineset-8jkzm
        namespace: openshift-cluster-api
      ...
      
      liuhuali@Lius-MacBook-Pro huali-test % oc get node ip-10-0-8-9.us-east-2.compute.internal -oyaml
      apiVersion: v1
      kind: Node
      metadata:
        annotations:
          anno1: newanno1
          anno6: anno6
      ...
        labels:
      ...
          label6: label6
          lable3: newlabel3
      ...
          

      Actual results:

          Update labels and annotations in CAPI are synchronized to existing machines and nodes, but in MAPI they are not synchronized to existing machines and nodes, they only affect new machines created after the update. 
      
          awsmachine will not be consistent with others (machineset, machine, node) on removing labels or annotations, but for adding and updating labels or annotations, it will be consistent with others.

      Expected results:

          CAPI and MAPI should be feature parity and behave the same.
      
          Labels and annotations changes should be consistent across all resources (machineset, machine, awsmachine, node).

      Additional info:

          must-gather: https://drive.google.com/file/d/1ybB2jmNCDprwbO_YLCVXHPnZyUM-XXUF/view?usp=sharing
          This is for testing https://issues.redhat.com/browse/OCPCLOUD-2680 and https://issues.redhat.com/browse/OCPCLOUD-2860

              rh-ee-nbrubake Nolan Brubaker
              huliu@redhat.com Huali Liu
              None
              None
              Huali Liu Huali Liu
              None
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated: