-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
4.19
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
Moderate
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
After enabling CustomNoUpgrade, creating a machine set using a custom secret, no machines were created for it. However, a standalone machine using a custom secret was created successfully. The machines using a custom secret cannot be deleted, they stuck in Deleting.
Also, before enabling CustomNoUpgrade, the machine set using a custom secret and its machines were created successfully.
Version-Release number of selected component (if applicable):
4.19.0-0.ci-2025-04-21-160508
How reproducible:
always
Steps to Reproduce:
1.Install an AWS cluster, then create a custom secret in openshift-machine-api, here I copied from aws-cloud-credentials
liuhuali@Lius-MacBook-Pro huali-test % oc get secret aws-cloud-credentials -oyaml>sec.yaml
liuhuali@Lius-MacBook-Pro huali-test % vim sec.yaml
liuhuali@Lius-MacBook-Pro huali-test % oc create -f sec.yaml
secret/aws1 created
liuhuali@Lius-MacBook-Pro huali-test % oc get secret
NAME TYPE DATA AGE
aws-cloud-credentials Opaque 3 109m
aws1 Opaque 3 4s
2.Create a machineset using the custom secret, the machine set and its machine created successfully
liuhuali@Lius-MacBook-Pro huali-test % oc get machineset
NAME DESIRED CURRENT READY AVAILABLE AGE
huliu-aws422a-7l2xd-customsecret 1 1 1 1 4m11s
huliu-aws422a-7l2xd-worker-us-east-2a 1 1 1 1 116m
huliu-aws422a-7l2xd-worker-us-east-2b 1 1 1 1 116m
huliu-aws422a-7l2xd-worker-us-east-2c 1 1 1 1 116m
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME PHASE TYPE REGION ZONE AGE
huliu-aws422a-7l2xd-customsecret-kjcq7 Running m6i.xlarge us-east-2 us-east-2a 4m59s
huliu-aws422a-7l2xd-master-0 Running m6i.xlarge us-east-2 us-east-2a 117m
huliu-aws422a-7l2xd-master-1 Running m6i.xlarge us-east-2 us-east-2b 117m
huliu-aws422a-7l2xd-master-2 Running m6i.xlarge us-east-2 us-east-2c 117m
huliu-aws422a-7l2xd-worker-us-east-2a-mdmwj Running m6i.xlarge us-east-2 us-east-2a 113m
huliu-aws422a-7l2xd-worker-us-east-2b-z2djd Running m6i.xlarge us-east-2 us-east-2b 113m
huliu-aws422a-7l2xd-worker-us-east-2c-6lmlt Running m6i.xlarge us-east-2 us-east-2c 113m
3.Enable CustomNoUpgrade, and wait for the cluster stable
4.Found the machine set using the custom secret and its machine cannot be converted to capi, this is as expected
status:
authoritativeAPI: MachineAPI
availableReplicas: 1
conditions:
- lastTransitionTime: "2025-04-22T04:09:32Z"
message: 'failed to convert MAPI machine set to CAPI machine set: spec.providerSpec.value.credentialsSecret:
Invalid value: "aws1": credential secret does not match the default of "aws-cloud-credentials",
manual conversion is necessary. Please see https://access.redhat.com/articles/7116313
for more details.'
reason: FailedToConvertMAPIMachineSetToCAPI
severity: Error
status: "False"
type: Synchronized
fullyLabeledReplicas: 1
observedGeneration: 1
readyReplicas: 1
replicas: 1
synchronizedGeneration: 0
5. Create another machineset using the custom secret, there is no machine created for it but it should.
liuhuali@Lius-MacBook-Pro huali-test % cat ms1.yaml
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
name: huliu-aws422a-7l2xd-customsecret1
namespace: openshift-machine-api
spec:
replicas: 1
selector:
matchLabels:
machine.openshift.io/cluster-api-cluster: huliu-aws422a-7l2xd
machine.openshift.io/cluster-api-machineset: huliu-aws422a-7l2xd-customsecret1
template:
metadata:
labels:
machine.openshift.io/cluster-api-cluster: huliu-aws422a-7l2xd
machine.openshift.io/cluster-api-machine-role: worker
machine.openshift.io/cluster-api-machine-type: worker
machine.openshift.io/cluster-api-machineset: huliu-aws422a-7l2xd-customsecret1
spec:
lifecycleHooks: {}
metadata: {}
providerSpec:
value:
ami:
id: ami-0f13d2cbfbca6203b
apiVersion: machine.openshift.io/v1beta1
blockDevices:
- ebs:
encrypted: true
iops: 0
kmsKey:
arn: ""
volumeSize: 120
volumeType: gp3
capacityReservationId: ""
credentialsSecret:
name: aws1
deviceIndex: 0
iamInstanceProfile:
id: huliu-aws422a-7l2xd-worker-profile
instanceType: m6i.xlarge
kind: AWSMachineProviderConfig
metadata:
creationTimestamp: null
metadataServiceOptions: {}
placement:
availabilityZone: us-east-2a
region: us-east-2
securityGroups:
- filters:
- name: tag:Name
values:
- huliu-aws422a-7l2xd-node
- filters:
- name: tag:Name
values:
- huliu-aws422a-7l2xd-lb
subnet:
filters:
- name: tag:Name
values:
- huliu-aws422a-7l2xd-subnet-private-us-east-2a
tags:
- name: kubernetes.io/cluster/huliu-aws422a-7l2xd
value: owned
userDataSecret:
name: worker-user-data
liuhuali@Lius-MacBook-Pro huali-test %
liuhuali@Lius-MacBook-Pro huali-test % oc create -f ms1.yaml
machineset.machine.openshift.io/huliu-aws422a-7l2xd-customsecret1 created
liuhuali@Lius-MacBook-Pro huali-test % oc get machineset
NAME DESIRED CURRENT READY AVAILABLE AGE
huliu-aws422a-7l2xd-customsecret 1 1 1 1 4h8m
huliu-aws422a-7l2xd-customsecret1 1 71m
huliu-aws422a-7l2xd-worker-us-east-2a 1 1 1 1 6h1m
huliu-aws422a-7l2xd-worker-us-east-2b 1 1 1 1 6h1m
huliu-aws422a-7l2xd-worker-us-east-2c 1 1 1 1 6h1m
status:
authoritativeAPI: MachineAPI
conditions:
- lastTransitionTime: "2025-04-22T06:32:39Z"
message: 'failed to convert MAPI machine set to CAPI machine set: spec.providerSpec.value.credentialsSecret:
Invalid value: "aws1": credential secret does not match the default of "aws-cloud-credentials",
manual conversion is necessary. Please see https://access.redhat.com/articles/7116313
for more details.'
reason: FailedToConvertMAPIMachineSetToCAPI
severity: Error
status: "False"
type: Synchronized
synchronizedGeneration: 0
6. Create a standalone machine using the custom secret, the machine create successfully, convert to capi failed, this is as expected.
liuhuali@Lius-MacBook-Pro huali-test % cat machine.yaml
apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
name: huliu-aws422a-7l2xd-csm1
namespace: openshift-machine-api
spec:
lifecycleHooks: {}
metadata: {}
providerSpec:
value:
ami:
id: ami-0f13d2cbfbca6203b
apiVersion: machine.openshift.io/v1beta1
blockDevices:
- ebs:
encrypted: true
iops: 0
kmsKey:
arn: ""
volumeSize: 120
volumeType: gp3
capacityReservationId: ""
credentialsSecret:
name: aws1
deviceIndex: 0
iamInstanceProfile:
id: huliu-aws422a-7l2xd-worker-profile
instanceType: m6i.xlarge
kind: AWSMachineProviderConfig
metadata:
creationTimestamp: null
metadataServiceOptions: {}
placement:
availabilityZone: us-east-2a
region: us-east-2
securityGroups:
- filters:
- name: tag:Name
values:
- huliu-aws422a-7l2xd-node
- filters:
- name: tag:Name
values:
- huliu-aws422a-7l2xd-lb
subnet:
filters:
- name: tag:Name
values:
- huliu-aws422a-7l2xd-subnet-private-us-east-2a
tags:
- name: kubernetes.io/cluster/huliu-aws422a-7l2xd
value: owned
userDataSecret:
name: worker-user-data
liuhuali@Lius-MacBook-Pro huali-test %
liuhuali@Lius-MacBook-Pro huali-test % oc create -f machine.yaml
machine.machine.openshift.io/huliu-aws422a-7l2xd-csm1 created
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME PHASE TYPE REGION ZONE AGE
huliu-aws422a-7l2xd-csm1 Running m6i.xlarge us-east-2 us-east-2a 70m
huliu-aws422a-7l2xd-customsecret-kjcq7 Running m6i.xlarge us-east-2 us-east-2a 4h14m
huliu-aws422a-7l2xd-master-0 Running m6i.xlarge us-east-2 us-east-2a 6h6m
huliu-aws422a-7l2xd-master-1 Running m6i.xlarge us-east-2 us-east-2b 6h6m
huliu-aws422a-7l2xd-master-2 Running m6i.xlarge us-east-2 us-east-2c 6h6m
huliu-aws422a-7l2xd-worker-us-east-2a-mdmwj Running m6i.xlarge us-east-2 us-east-2a 6h3m
huliu-aws422a-7l2xd-worker-us-east-2b-z2djd Running m6i.xlarge us-east-2 us-east-2b 6h3m
huliu-aws422a-7l2xd-worker-us-east-2c-6lmlt Running m6i.xlarge us-east-2 us-east-2c 6h3m
authoritativeAPI: MachineAPI
conditions:
- lastTransitionTime: "2025-04-22T06:39:47Z"
status: "True"
type: Drainable
- lastTransitionTime: "2025-04-22T06:40:20Z"
status: "True"
type: InstanceExists
- lastTransitionTime: "2025-04-22T06:39:47Z"
message: The AuthoritativeAPI is set to MachineAPI
reason: AuthoritativeAPIMachineAPI
severity: Info
status: "False"
type: Paused
- lastTransitionTime: "2025-04-22T06:39:47Z"
message: 'failed to convert Machine API machine to Cluster API machine: spec.providerSpec.value.credentialsSecret:
Invalid value: "aws1": credential secret does not match the default of "aws-cloud-credentials",
manual conversion is necessary. Please see https://access.redhat.com/articles/7116313
for more details.'
reason: FailedToConvertMAPIMachineToCAPI
severity: Error
status: "False"
type: Synchronized
- lastTransitionTime: "2025-04-22T06:39:47Z"
status: "True"
type: Terminable
lastUpdated: "2025-04-22T06:43:58Z"
nodeRef:
kind: Node
name: ip-10-0-8-55.us-east-2.compute.internal
uid: df6f0ba8-f098-4fc9-962a-f2e246146356
phase: Running
providerStatus:
conditions:
- lastTransitionTime: "2025-04-22T06:39:49Z"
message: Machine successfully created
reason: MachineCreationSucceeded
status: "True"
type: MachineCreation
instanceId: i-0b62a90100702ba83
instanceState: running
synchronizedGeneration: 0
7. Delete the machine set, the machine set is deleted, but the machine stuck in Deleting;
Delete the standalone machine, the machine stuck in Deleting.
I checked on AWS console, they are still there.
liuhuali@Lius-MacBook-Pro huali-test % oc delete machineset huliu-aws422a-7l2xd-customsecret
machineset.machine.openshift.io "huliu-aws422a-7l2xd-customsecret" deleted
liuhuali@Lius-MacBook-Pro huali-test % oc delete machine huliu-aws422a-7l2xd-csm1
machine.machine.openshift.io "huliu-aws422a-7l2xd-csm1" deleted
^C
liuhuali@Lius-MacBook-Pro huali-test % oc get machineset
NAME DESIRED CURRENT READY AVAILABLE AGE
huliu-aws422a-7l2xd-customsecret1 1 4h51m
huliu-aws422a-7l2xd-worker-us-east-2a 1 1 1 1 9h
huliu-aws422a-7l2xd-worker-us-east-2b 1 1 1 1 9h
huliu-aws422a-7l2xd-worker-us-east-2c 1 1 1 1 9h
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME PHASE TYPE REGION ZONE AGE
huliu-aws422a-7l2xd-csm1 Deleting m6i.xlarge us-east-2 us-east-2a 4h44m
huliu-aws422a-7l2xd-customsecret-kjcq7 Deleting m6i.xlarge us-east-2 us-east-2a 7h48m
huliu-aws422a-7l2xd-master-0 Running m6i.xlarge us-east-2 us-east-2a 9h
huliu-aws422a-7l2xd-master-1 Running m6i.xlarge us-east-2 us-east-2b 9h
huliu-aws422a-7l2xd-master-2 Running m6i.xlarge us-east-2 us-east-2c 9h
huliu-aws422a-7l2xd-worker-us-east-2a-mdmwj Running m6i.xlarge us-east-2 us-east-2a 9h
huliu-aws422a-7l2xd-worker-us-east-2b-z2djd Running m6i.xlarge us-east-2 us-east-2b 9h
huliu-aws422a-7l2xd-worker-us-east-2c-6lmlt Running m6i.xlarge us-east-2 us-east-2c 9h
liuhuali@Lius-MacBook-Pro huali-test %
liuhuali@Lius-MacBook-Pro huali-test % oc get node
NAME STATUS ROLES AGE VERSION
ip-10-0-27-160.us-east-2.compute.internal Ready,SchedulingDisabled worker 7h50m v1.32.3
ip-10-0-29-160.us-east-2.compute.internal Ready control-plane,master 9h v1.32.3
ip-10-0-40-61.us-east-2.compute.internal Ready control-plane,master 9h v1.32.3
ip-10-0-43-113.us-east-2.compute.internal Ready worker 9h v1.32.3
ip-10-0-6-147.us-east-2.compute.internal Ready worker 9h v1.32.3
ip-10-0-8-55.us-east-2.compute.internal Ready,SchedulingDisabled worker 4h45m v1.32.3
ip-10-0-91-109.us-east-2.compute.internal Ready worker 9h v1.32.3
ip-10-0-92-165.us-east-2.compute.internal Ready control-plane,master 9h v1.32.3
liuhuali@Lius-MacBook-Pro huali-test % oc logs machine-api-controllers-7856dcf465-ftbnp -c machine-controller
...
E0422 11:24:18.312151 1 controller.go:341] "msg"="Reconciler error" "error"="could not update machine status: Machine.machine.openshift.io \"huliu-aws422a-7l2xd-csm1\" is invalid: status.synchronizedGeneration: Invalid value: \"object\": synchronizedGeneration must not decrease unless authoritativeAPI is transitioning from Migrating to another value" "controller"="machine-drain-controller" "name"="huliu-aws422a-7l2xd-csm1" "namespace"="openshift-machine-api" "object"={"name":"huliu-aws422a-7l2xd-csm1","namespace":"openshift-machine-api"} "reconcileID"="54f2f45f-1db8-4415-b9fd-4fe1daf352d6"
I0422 11:30:51.376449 1 drain_controller.go:190] drain successful for machine "huliu-aws422a-7l2xd-customsecret-kjcq7"
I0422 11:30:51.376596 1 recorder.go:104] "msg"="Node \"ip-10-0-27-160.us-east-2.compute.internal\" drained" "logger"="events" "object"={"kind":"Machine","namespace":"openshift-machine-api","name":"huliu-aws422a-7l2xd-customsecret-kjcq7","uid":"4661921f-55b8-48b2-aeba-f2abf45de2f7","apiVersion":"machine.openshift.io/v1beta1","resourceVersion":"161881"} "reason"="Deleted" "type"="Normal"
I0422 11:30:51.376632 1 recorder.go:104] "msg"="Node drain succeeded" "logger"="events" "object"={"kind":"Machine","namespace":"openshift-machine-api","name":"huliu-aws422a-7l2xd-customsecret-kjcq7","uid":"4661921f-55b8-48b2-aeba-f2abf45de2f7","apiVersion":"machine.openshift.io/v1beta1","resourceVersion":"161881"} "reason"="DrainSucceeded" "type"="Normal"
E0422 11:30:51.385543 1 controller.go:341] "msg"="Reconciler error" "error"="could not update machine status: Machine.machine.openshift.io \"huliu-aws422a-7l2xd-customsecret-kjcq7\" is invalid: status.synchronizedGeneration: Invalid value: \"object\": synchronizedGeneration must not decrease unless authoritativeAPI is transitioning from Migrating to another value" "controller"="machine-drain-controller" "name"="huliu-aws422a-7l2xd-customsecret-kjcq7" "namespace"="openshift-machine-api" "object"={"name":"huliu-aws422a-7l2xd-customsecret-kjcq7","namespace":"openshift-machine-api"} "reconcileID"="653ec156-8bb5-4330-aae8-935badf00be1"
Actual results:
After enabling CustomNoUpgrade, creating a machine set using a custom secret, no machines were created for it. The machines using a custom secret cannot be deleted, they stuck in Deleting.
Expected results:
The machine set using a custom secret and its machine should create successfully; and the machines using a custom secret should delete successfully. Because the conversion should not block mapi features.
Additional info:
new feature testing for https://issues.redhat.com/browse/OCPCLOUD-2713