Details
-
Bug
-
Resolution: Done
-
Undefined
-
None
-
4.13
-
None
-
Moderate
-
False
-
Description
This is a clone of issue OCPBUGS-4297. The following is the description of the original issue:
—
Description of problem:
OnDelete update strategy create two replace machines when deleting a master machine
Version-Release number of selected component (if applicable):
4.13.0-0.nightly-2022-11-29-035943
How reproducible:
Not sure, I met twice on this template cluster https://gitlab.cee.redhat.com/aosqe/flexy-templates/-/blob/master/functionality-testing/aos-4_13/ipi-on-vsphere/versioned-installer-vmc7_techpreview
Steps to Reproduce:
1.Launch a 4.13 cluster on Vsphere with techpreview enabled, we use automated template: https://gitlab.cee.redhat.com/aosqe/flexy-templates/-/blob/master/functionality-testing/aos-4_13/ipi-on-vsphere/versioned-installer-vmc7_techpreview liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.13.0-0.nightly-2022-11-29-035943 True False 56m Cluster version is 4.13.0-0.nightly-2022-11-29-035943 2.Replace master machines one by one with index 3,4,5 liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-vs13d-rcr7z-master-3 Running 57m huliu-vs13d-rcr7z-master-4 Running 35m huliu-vs13d-rcr7z-master-5 Running 12m huliu-vs13d-rcr7z-worker-ngw2j Running 7h12m huliu-vs13d-rcr7z-worker-p2xd7 Running 7h12m liuhuali@Lius-MacBook-Pro huali-test % oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.13.0-0.nightly-2022-11-29-035943 True False False 29m baremetal 4.13.0-0.nightly-2022-11-29-035943 True False False 7h33m cloud-controller-manager 4.13.0-0.nightly-2022-11-29-035943 True False False 7h35m cloud-credential 4.13.0-0.nightly-2022-11-29-035943 True False False 7h37m cluster-api 4.13.0-0.nightly-2022-11-29-035943 True False False 7h33m cluster-autoscaler 4.13.0-0.nightly-2022-11-29-035943 True False False 7h32m config-operator 4.13.0-0.nightly-2022-11-29-035943 True False False 7h34m console 4.13.0-0.nightly-2022-11-29-035943 True False False 28m control-plane-machine-set 4.13.0-0.nightly-2022-11-29-035943 True False False 5h12m csi-snapshot-controller 4.13.0-0.nightly-2022-11-29-035943 True False False 7h33m dns 4.13.0-0.nightly-2022-11-29-035943 True False False 7h32m etcd 4.13.0-0.nightly-2022-11-29-035943 True False False 7h31m image-registry 4.13.0-0.nightly-2022-11-29-035943 True False False 74m ingress 4.13.0-0.nightly-2022-11-29-035943 True False False 7h21m insights 4.13.0-0.nightly-2022-11-29-035943 True False False 7h26m kube-apiserver 4.13.0-0.nightly-2022-11-29-035943 True False False 7h22m kube-controller-manager 4.13.0-0.nightly-2022-11-29-035943 True False False 7h31m kube-scheduler 4.13.0-0.nightly-2022-11-29-035943 True False False 7h30m kube-storage-version-migrator 4.13.0-0.nightly-2022-11-29-035943 True False False 74m machine-api 4.13.0-0.nightly-2022-11-29-035943 True False False 7h23m machine-approver 4.13.0-0.nightly-2022-11-29-035943 True False False 7h33m machine-config 4.13.0-0.nightly-2022-11-29-035943 True False False 27m marketplace 4.13.0-0.nightly-2022-11-29-035943 True False False 7h32m monitoring 4.13.0-0.nightly-2022-11-29-035943 True False False 7h19m network 4.13.0-0.nightly-2022-11-29-035943 True False False 7h34m node-tuning 4.13.0-0.nightly-2022-11-29-035943 True False False 7h32m openshift-apiserver 4.13.0-0.nightly-2022-11-29-035943 True False False 30m openshift-controller-manager 4.13.0-0.nightly-2022-11-29-035943 True False False 7h26m openshift-samples 4.13.0-0.nightly-2022-11-29-035943 True False False 7h25m operator-lifecycle-manager 4.13.0-0.nightly-2022-11-29-035943 True False False 7h33m operator-lifecycle-manager-catalog 4.13.0-0.nightly-2022-11-29-035943 True False False 7h33m operator-lifecycle-manager-packageserver 4.13.0-0.nightly-2022-11-29-035943 True False False 7h26m platform-operators-aggregated 4.13.0-0.nightly-2022-11-29-035943 True False False 20m service-ca 4.13.0-0.nightly-2022-11-29-035943 True False False 7h34m storage 4.13.0-0.nightly-2022-11-29-035943 True False False 5h16m liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-vs13d-rcr7z-master-3 Running 77m huliu-vs13d-rcr7z-master-4 Running 55m huliu-vs13d-rcr7z-master-5 Running 32m huliu-vs13d-rcr7z-worker-ngw2j Running 7h32m huliu-vs13d-rcr7z-worker-p2xd7 Running 7h32m 3.Create CPMS, yaml as below: apiVersion: machine.openshift.io/v1 kind: ControlPlaneMachineSet metadata: name: cluster namespace: openshift-machine-api spec: replicas: 3 state: Active strategy: type: OnDelete selector: matchLabels: machine.openshift.io/cluster-api-machine-role: master machine.openshift.io/cluster-api-machine-type: master template: machineType: machines_v1beta1_machine_openshift_io machines_v1beta1_machine_openshift_io: metadata: labels: machine.openshift.io/cluster-api-machine-role: master machine.openshift.io/cluster-api-machine-type: master machine.openshift.io/cluster-api-cluster: huliu-vs13d-rcr7z spec: providerSpec: value: apiVersion: machine.openshift.io/v1beta1 credentialsSecret: name: vsphere-cloud-credentials diskGiB: 120 kind: VSphereMachineProviderSpec memoryMiB: 16384 metadata: creationTimestamp: null network: devices: - networkName: qe-segment numCPUs: 4 numCoresPerSocket: 4 snapshot: "" template: huliu-vs13d-rcr7z-rhcos userDataSecret: name: master-user-data workspace: datacenter: SDDC-Datacenter datastore: WorkloadDatastore folder: /SDDC-Datacenter/vm/huliu-vs13d-rcr7z resourcePool: /SDDC-Datacenter/host/Cluster-1/Resources server: vcenter.sddc-44-236-21-251.vmwarevmc.com liuhuali@Lius-MacBook-Pro huali-test % oc create -f controlplanemachineset_vsphere.yaml controlplanemachineset.machine.openshift.io/cluster created liuhuali@Lius-MacBook-Pro huali-test % oc get controlplanemachineset NAME DESIRED CURRENT READY UPDATED UNAVAILABLE STATE AGE cluster 3 3 3 3 Active 9s 4.Edit CPMS, change numCPUs to 8 to trigger update liuhuali@Lius-MacBook-Pro huali-test % oc edit controlplanemachineset cluster controlplanemachineset.machine.openshift.io/cluster edited liuhuali@Lius-MacBook-Pro huali-test % oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.13.0-0.nightly-2022-11-29-035943 True False False 31m baremetal 4.13.0-0.nightly-2022-11-29-035943 True False False 7h34m cloud-controller-manager 4.13.0-0.nightly-2022-11-29-035943 True False False 7h37m cloud-credential 4.13.0-0.nightly-2022-11-29-035943 True False False 7h38m cluster-api 4.13.0-0.nightly-2022-11-29-035943 True False False 7h34m cluster-autoscaler 4.13.0-0.nightly-2022-11-29-035943 True False False 7h34m config-operator 4.13.0-0.nightly-2022-11-29-035943 True False False 7h35m console 4.13.0-0.nightly-2022-11-29-035943 True False False 29m control-plane-machine-set 4.13.0-0.nightly-2022-11-29-035943 True True False 5h14m Observed 3 replica(s) in need of update csi-snapshot-controller 4.13.0-0.nightly-2022-11-29-035943 True False False 7h35m dns 4.13.0-0.nightly-2022-11-29-035943 True False False 7h34m etcd 4.13.0-0.nightly-2022-11-29-035943 True False False 7h33m image-registry 4.13.0-0.nightly-2022-11-29-035943 True False False 75m ingress 4.13.0-0.nightly-2022-11-29-035943 True False False 7h23m insights 4.13.0-0.nightly-2022-11-29-035943 True False False 7h27m kube-apiserver 4.13.0-0.nightly-2022-11-29-035943 True False False 7h24m kube-controller-manager 4.13.0-0.nightly-2022-11-29-035943 True False False 7h32m kube-scheduler 4.13.0-0.nightly-2022-11-29-035943 True False False 7h32m kube-storage-version-migrator 4.13.0-0.nightly-2022-11-29-035943 True False False 75m machine-api 4.13.0-0.nightly-2022-11-29-035943 True False False 7h24m machine-approver 4.13.0-0.nightly-2022-11-29-035943 True False False 7h35m machine-config 4.13.0-0.nightly-2022-11-29-035943 True False False 28m marketplace 4.13.0-0.nightly-2022-11-29-035943 True False False 7h34m monitoring 4.13.0-0.nightly-2022-11-29-035943 True False False 7h21m network 4.13.0-0.nightly-2022-11-29-035943 True False False 7h35m node-tuning 4.13.0-0.nightly-2022-11-29-035943 True False False 7h34m openshift-apiserver 4.13.0-0.nightly-2022-11-29-035943 True False False 31m openshift-controller-manager 4.13.0-0.nightly-2022-11-29-035943 True False False 7h27m openshift-samples 4.13.0-0.nightly-2022-11-29-035943 True False False 7h26m operator-lifecycle-manager 4.13.0-0.nightly-2022-11-29-035943 True False False 7h35m operator-lifecycle-manager-catalog 4.13.0-0.nightly-2022-11-29-035943 True False False 7h35m operator-lifecycle-manager-packageserver 4.13.0-0.nightly-2022-11-29-035943 True False False 7h27m platform-operators-aggregated 4.13.0-0.nightly-2022-11-29-035943 True False False 21m service-ca 4.13.0-0.nightly-2022-11-29-035943 True False False 7h35m storage 4.13.0-0.nightly-2022-11-29-035943 True False False 5h18m liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-vs13d-rcr7z-master-3 Running 79m huliu-vs13d-rcr7z-master-4 Running 57m huliu-vs13d-rcr7z-master-5 Running 33m huliu-vs13d-rcr7z-worker-ngw2j Running 7h34m huliu-vs13d-rcr7z-worker-p2xd7 Running 7h34m 5.Delete master machine one by one, found it create two master machines when delete huliu-vs13d-rcr7z-master-4 liuhuali@Lius-MacBook-Pro huali-test % oc delete machine huliu-vs13d-rcr7z-master-5 machine.machine.openshift.io "huliu-vs13d-rcr7z-master-5" deleted ^C liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-vs13d-rcr7z-master-3 Running 79m huliu-vs13d-rcr7z-master-4 Running 57m huliu-vs13d-rcr7z-master-5 Deleting 33m huliu-vs13d-rcr7z-master-6b9x7-5 Provisioning 5s huliu-vs13d-rcr7z-worker-ngw2j Running 7h34m huliu-vs13d-rcr7z-worker-p2xd7 Running 7h34m liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-vs13d-rcr7z-master-3 Running 91m huliu-vs13d-rcr7z-master-4 Running 69m huliu-vs13d-rcr7z-master-6b9x7-5 Running 12m huliu-vs13d-rcr7z-worker-ngw2j Running 7h46m huliu-vs13d-rcr7z-worker-p2xd7 Running 7h46m liuhuali@Lius-MacBook-Pro huali-test % oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.13.0-0.nightly-2022-11-29-035943 True False False 53m baremetal 4.13.0-0.nightly-2022-11-29-035943 True False False 7h56m cloud-controller-manager 4.13.0-0.nightly-2022-11-29-035943 True False False 7h59m cloud-credential 4.13.0-0.nightly-2022-11-29-035943 True False False 8h cluster-api 4.13.0-0.nightly-2022-11-29-035943 True False False 7h56m cluster-autoscaler 4.13.0-0.nightly-2022-11-29-035943 True False False 7h56m config-operator 4.13.0-0.nightly-2022-11-29-035943 True False False 7h57m console 4.13.0-0.nightly-2022-11-29-035943 True False False 18m control-plane-machine-set 4.13.0-0.nightly-2022-11-29-035943 True True False 18m Observed 2 replica(s) in need of update csi-snapshot-controller 4.13.0-0.nightly-2022-11-29-035943 True False False 7h57m dns 4.13.0-0.nightly-2022-11-29-035943 True False False 7h56m etcd 4.13.0-0.nightly-2022-11-29-035943 True False False 7h55m image-registry 4.13.0-0.nightly-2022-11-29-035943 True False False 97m ingress 4.13.0-0.nightly-2022-11-29-035943 True False False 7h45m insights 4.13.0-0.nightly-2022-11-29-035943 True False False 7h49m kube-apiserver 4.13.0-0.nightly-2022-11-29-035943 True False False 7h46m kube-controller-manager 4.13.0-0.nightly-2022-11-29-035943 True False False 7h54m kube-scheduler 4.13.0-0.nightly-2022-11-29-035943 True False False 7h54m kube-storage-version-migrator 4.13.0-0.nightly-2022-11-29-035943 True False False 97m machine-api 4.13.0-0.nightly-2022-11-29-035943 True False False 7h46m machine-approver 4.13.0-0.nightly-2022-11-29-035943 True False False 7h56m machine-config 4.13.0-0.nightly-2022-11-29-035943 True False False 50m marketplace 4.13.0-0.nightly-2022-11-29-035943 True False False 7h56m monitoring 4.13.0-0.nightly-2022-11-29-035943 True False False 7h42m network 4.13.0-0.nightly-2022-11-29-035943 True False False 7h57m node-tuning 4.13.0-0.nightly-2022-11-29-035943 True False False 7h56m openshift-apiserver 4.13.0-0.nightly-2022-11-29-035943 True False False 53m openshift-controller-manager 4.13.0-0.nightly-2022-11-29-035943 True False False 7h49m openshift-samples 4.13.0-0.nightly-2022-11-29-035943 True False False 7h48m operator-lifecycle-manager 4.13.0-0.nightly-2022-11-29-035943 True False False 7h57m operator-lifecycle-manager-catalog 4.13.0-0.nightly-2022-11-29-035943 True False False 7h57m operator-lifecycle-manager-packageserver 4.13.0-0.nightly-2022-11-29-035943 True False False 7h49m platform-operators-aggregated 4.13.0-0.nightly-2022-11-29-035943 True False False 10m service-ca 4.13.0-0.nightly-2022-11-29-035943 True False False 7h57m storage 4.13.0-0.nightly-2022-11-29-035943 True False False 5h40m liuhuali@Lius-MacBook-Pro huali-test % oc delete machine huliu-vs13d-rcr7z-master-4 machine.machine.openshift.io "huliu-vs13d-rcr7z-master-4" deleted ^C liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-vs13d-rcr7z-master-3 Running 101m huliu-vs13d-rcr7z-master-4 Deleting 79m huliu-vs13d-rcr7z-master-6b9x7-5 Running 22m huliu-vs13d-rcr7z-master-8h9p9-4 Provisioning 6s huliu-vs13d-rcr7z-master-df78v-4 Provisioning 6s huliu-vs13d-rcr7z-worker-ngw2j Running 7h56m huliu-vs13d-rcr7z-worker-p2xd7 Running 7h56m liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-vs13d-rcr7z-master-3 Running 115m huliu-vs13d-rcr7z-master-6b9x7-5 Running 36m huliu-vs13d-rcr7z-master-8h9p9-4 Running 14m huliu-vs13d-rcr7z-master-df78v-4 Running 14m huliu-vs13d-rcr7z-worker-ngw2j Running 8h huliu-vs13d-rcr7z-worker-p2xd7 Running 8h
Actual results:
When deleting a mater machine, two replace machines created
Expected results:
When deleting a mater machine, only one replace machine created
Additional info:
Must-gather https://drive.google.com/file/d/1VVxGPW0WNDc3CxiJIg90dAQckEWhYy8i/view?usp=sharing
Attachments
Issue Links
- clones
-
-
- Closed
-
- is blocked by
-
-
- Closed
-
- links to
