The changes to machine set labels and annotations in MAPI are not synchronized to existing machines and nodes, they only affect new machines created after the update. But for CAPI they are synchronized to existing machines and nodes. There appear to be some gaps between MAPI and CAPI that may cause inconsistencies when migrating.

    awsmachine will not be consistent with others (machineset, machine, node) on removing labels or annotations, but for adding and updating labels or annotations, it will be consistent with others.

    4.19.0-0.nightly-2025-03-26-070608

always    

    1.Create a awsmachinetemplate, then a capi machineset.

liuhuali@Lius-MacBook-Pro huali-test % cat capimachineset926.yaml 
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineSet
metadata:
  labels:
    cluster.x-k8s.io/cluster-name: huliu-aws327a-gt8j4
  name: capi-machineset
  namespace: openshift-cluster-api
spec:
  clusterName: huliu-aws327a-gt8j4
  deletePolicy: Newest
  replicas: 1
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: huliu-aws327a-gt8j4
      machine.openshift.io/cluster-api-cluster: huliu-aws327a-gt8j4
  template:
    metadata:
      annotations:
        anno1: ""
        anno3: ""
        anno5: anno5
        anno7: anno7
      labels:
        lable1: ""
        lable3: ""
        label5: label5
        label7: label7
        cluster.x-k8s.io/cluster-name: huliu-aws327a-gt8j4
        machine.openshift.io/cluster-api-cluster: huliu-aws327a-gt8j4
    spec:
      bootstrap:
        dataSecretName: worker-user-data
      clusterName: huliu-aws327a-gt8j4
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        kind: AWSMachineTemplate
        name: aws-machinetemplate 

liuhuali@Lius-MacBook-Pro huali-test % oc create -f capimachineset926.yaml
machineset.cluster.x-k8s.io/capi-machineset created

2. Check there are the labels and annotations on machine, awsmachine and node.

liuhuali@Lius-MacBook-Pro huali-test % oc get machine.c capi-machineset-8jkzm -oyaml
apiVersion: cluster.x-k8s.io/v1beta1
kind: Machine
metadata:
  annotations:
    anno1: ""
    anno3: ""
    anno5: anno5
    anno7: anno7
  creationTimestamp: "2025-03-27T11:33:34Z"
  finalizers:
  - machine.cluster.x-k8s.io
  generation: 2
  labels:
    cluster.x-k8s.io/cluster-name: huliu-aws327a-gt8j4
    cluster.x-k8s.io/set-name: capi-machineset
    label5: label5
    label7: label7
    lable1: ""
    lable3: ""
    machine.openshift.io/cluster-api-cluster: huliu-aws327a-gt8j4
  name: capi-machineset-8jkzm
  namespace: openshift-cluster-api
...
liuhuali@Lius-MacBook-Pro huali-test % oc get awsmachine capi-machineset-8jkzm  -oyaml
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSMachine
metadata:
  annotations:
    anno1: ""
    anno3: ""
    anno5: anno5
    anno7: anno7
    cluster.x-k8s.io/cloned-from-groupkind: AWSMachineTemplate.infrastructure.cluster.x-k8s.io
    cluster.x-k8s.io/cloned-from-name: aws-machinetemplate
    sigs.k8s.io/cluster-api-provider-aws-last-applied-security-groups: '{"sg-0017fbfe45a8d1c43":{},"sg-05fe540dfac0dec53":{}}'
    sigs.k8s.io/cluster-api-provider-last-applied-tags-on-volumes: '{"vol-01238cbd748334f68":{}}'
  creationTimestamp: "2025-03-27T11:33:34Z"
  finalizers:
  - awsmachine.infrastructure.cluster.x-k8s.io
  generation: 2
  labels:
    cluster.x-k8s.io/cluster-name: huliu-aws327a-gt8j4
    cluster.x-k8s.io/set-name: capi-machineset
    label5: label5
    label7: label7
    lable1: ""
    lable3: ""
    machine.openshift.io/cluster-api-cluster: huliu-aws327a-gt8j4
  name: capi-machineset-8jkzm
  namespace: openshift-cluster-api
...

liuhuali@Lius-MacBook-Pro huali-test % oc get node ip-10-0-8-9.us-east-2.compute.internal -oyaml
apiVersion: v1
kind: Node
metadata:
  annotations:
    anno1: ""
    anno3: ""
    anno5: anno5
    anno7: anno7
...
  labels:
...
    label5: label5
    label7: label7
    lable1: ""
    lable3: ""
... 

3. Update the machineset, change the labels and annotations to below
  template:
    metadata:
      annotations:
        anno1: newanno1
        anno6: anno6
      labels:
        cluster.x-k8s.io/cluster-name: huliu-aws327a-gt8j4
        label6: label6
        lable3: newlabel3
        machine.openshift.io/cluster-api-cluster: huliu-aws327a-gt8j4 

4. Check on the machine, awsmachine, node. Found the updates are synchronized to the existing machine and node, but the deleted labels and annotations still shows on awsmachine.

liuhuali@Lius-MacBook-Pro huali-test % oc get machine.c capi-machineset-8jkzm  -oyaml           
apiVersion: cluster.x-k8s.io/v1beta1
kind: Machine
metadata:
  annotations:
    anno1: newanno1
    anno6: anno6
  creationTimestamp: "2025-03-27T11:33:34Z"
  finalizers:
  - machine.cluster.x-k8s.io
  generation: 2
  labels:
    cluster.x-k8s.io/cluster-name: huliu-aws327a-gt8j4
    cluster.x-k8s.io/set-name: capi-machineset
    label6: label6
    lable3: newlabel3
    machine.openshift.io/cluster-api-cluster: huliu-aws327a-gt8j4
  name: capi-machineset-8jkzm
  namespace: openshift-cluster-api 
...

liuhuali@Lius-MacBook-Pro huali-test % oc get awsmachine capi-machineset-8jkzm  -oyaml          
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSMachine
metadata:
  annotations:
    anno1: newanno1
    anno3: ""
    anno5: anno5
    anno6: anno6
    anno7: anno7
    cluster.x-k8s.io/cloned-from-groupkind: AWSMachineTemplate.infrastructure.cluster.x-k8s.io
    cluster.x-k8s.io/cloned-from-name: aws-machinetemplate
    sigs.k8s.io/cluster-api-provider-aws-last-applied-security-groups: '{"sg-0017fbfe45a8d1c43":{},"sg-05fe540dfac0dec53":{}}'
    sigs.k8s.io/cluster-api-provider-last-applied-tags-on-volumes: '{"vol-01238cbd748334f68":{}}'
  creationTimestamp: "2025-03-27T11:33:34Z"
  finalizers:
  - awsmachine.infrastructure.cluster.x-k8s.io
  generation: 2
  labels:
    cluster.x-k8s.io/cluster-name: huliu-aws327a-gt8j4
    cluster.x-k8s.io/set-name: capi-machineset
    label5: label5
    label6: label6
    label7: label7
    lable1: ""
    lable3: newlabel3
    machine.openshift.io/cluster-api-cluster: huliu-aws327a-gt8j4
  name: capi-machineset-8jkzm
  namespace: openshift-cluster-api
...

liuhuali@Lius-MacBook-Pro huali-test % oc get node ip-10-0-8-9.us-east-2.compute.internal -oyaml
apiVersion: v1
kind: Node
metadata:
  annotations:
    anno1: newanno1
    anno6: anno6
...
  labels:
...
    label6: label6
    lable3: newlabel3
...
    

    Update labels and annotations in CAPI are synchronized to existing machines and nodes, but in MAPI they are not synchronized to existing machines and nodes, they only affect new machines created after the update. 

    awsmachine will not be consistent with others (machineset, machine, node) on removing labels or annotations, but for adding and updating labels or annotations, it will be consistent with others.

    CAPI and MAPI should be feature parity and behave the same.

    Labels and annotations changes should be consistent across all resources (machineset, machine, awsmachine, node).

    must-gather: https://drive.google.com/file/d/1ybB2jmNCDprwbO_YLCVXHPnZyUM-XXUF/view?usp=sharing
    This is for testing https://issues.redhat.com/browse/OCPCLOUD-2680 and https://issues.redhat.com/browse/OCPCLOUD-2860