[OCPBUGS-29249] CPMS leaves only 2 masters during update

Type: Bug
Resolution: Done-Errata
Priority: Critical
Fix Version/s: 4.16.0
Affects Version/s: 4.13, 4.12, 4.14, 4.15, 4.16
Component/s: Cloud Compute / Unknown
Labels:
None

Regression:
No
Sprint:
CLOUD Sprint 249
sprint_count:
1
Release Blocker:
Rejected
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Release Note Text:

Hide
* Previously, when a control plane machine was marked as unready and a change was initiated by the modifying the control plane machine set, the unready machine was removed prematurely.
This premature action caused multiple indexes to be replaced simultaneously.
With this release, the control plane machine set no longer deletes a machine when only a single machine exists within the index.
This change prevents premature roll-out of changes and prevents more than one index from being replaced at a time.
(link:https://issues.redhat.com/browse/OCPBUGS-29249[*~~OCPBUGS-29249~~*])

Show
* Previously, when a control plane machine was marked as unready and a change was initiated by the modifying the control plane machine set, the unready machine was removed prematurely. This premature action caused multiple indexes to be replaced simultaneously. With this release, the control plane machine set no longer deletes a machine when only a single machine exists within the index. This change prevents premature roll-out of changes and prevents more than one index from being replaced at a time. (link: https://issues.redhat.com/browse/OCPBUGS-29249 [* OCPBUGS-29249 *])
Release Note Type:
Bug Fix
Release Note Status:
Done
Target Version:

4.16.0
Target Backport Versions:

4.13.z, 4.12.z, 4.14.z, 4.15.z

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Observed during testing of candidate-4.15 image as of 2024-02-08.

This is an incomplete report as I haven't verified the reproducer yet or attempted to get a must-gather. I have observed this multiple times now, so I am confident it's a thing. I can't be confident that the procedure described here reliably reproduces it, or that all the described steps are required.

I have been using MCO to apply machine config to masters. This involves a rolling reboot of all masters.

During a rolling reboot I applied an update to CPMS. I observed the following sequence of events:

master-1 was NotReady as it was rebooting
I modified CPMS
CPMS immediately started provisioning a new master-0
CPMS immediately started deleting master-1
CPMS started provisioning a new master-1

At this point there were only 2 nodes in the cluster:

old master-0
old master-2

and machines provisioning:

new master-0
new master-1

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

98-var-lib-etcd.yaml
4 kB
2024/02/08 3:57 PM

blocks

OCPBUGS-29419 CPMS leaves only 2 masters during update

Closed

is cloned by

OCPBUGS-29419 CPMS leaves only 2 masters during update

Closed

links to

openshift/cluster-control-plane-machine-set-operator#278: OCPBUGS-29249: Never delete a Machine when there's a single Machine in an index

RHEA-2024:0041 OpenShift Container Platform 4.16.z bug fix update

Errata Tool added a comment - 2024/06/27 11:37 AM

Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

For information on the advisory (Critical: OpenShift Container Platform 4.16.0 bug fix and security update), and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2024:0041

Errata Tool added a comment - 2024/06/27 11:37 AM Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Critical: OpenShift Container Platform 4.16.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:0041

Huali Liu added a comment - 2024/02/20 7:51 AM

Verified on 4.16.0-0.nightly-2024-02-17-094036

1.Create the MCO attached by Matthew or delete the MCO to apply a MachineConfig update

liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.16.0-0.nightly-2024-02-17-094036   True        False         16m     Cluster version is 4.16.0-0.nightly-2024-02-17-094036
liuhuali@Lius-MacBook-Pro huali-test % oc get machine -o wide
NAME                                         PHASE     TYPE         REGION      ZONE         AGE    NODE                                        PROVIDERID                              STATE
huliu-aws20b-pswxj-master-b7ff5-2            Running   m6i.xlarge   us-east-2   us-east-2c   30m    ip-10-0-67-67.us-east-2.compute.internal    aws:///us-east-2c/i-031f500a19c7a1108   running
huliu-aws20b-pswxj-master-gq62c-1            Running   m6i.xlarge   us-east-2   us-east-2b   49m    ip-10-0-51-210.us-east-2.compute.internal   aws:///us-east-2b/i-0c6806cb676ffb3db   running
huliu-aws20b-pswxj-master-t57cx-0            Running   m6i.xlarge   us-east-2   us-east-2a   78m    ip-10-0-6-94.us-east-2.compute.internal     aws:///us-east-2a/i-0c3d7e19db15e2882   running
huliu-aws20b-pswxj-worker-us-east-2a-sxxn6   Running   m6i.xlarge   us-east-2   us-east-2a   4h5m   ip-10-0-9-68.us-east-2.compute.internal     aws:///us-east-2a/i-0529aed6f7d47143f   running
huliu-aws20b-pswxj-worker-us-east-2b-q58rv   Running   m6i.xlarge   us-east-2   us-east-2b   4h5m   ip-10-0-48-76.us-east-2.compute.internal    aws:///us-east-2b/i-080e733b1fbf10a66   running
huliu-aws20b-pswxj-worker-us-east-2c-t2v7r   Running   m6i.xlarge   us-east-2   us-east-2c   4h5m   ip-10-0-81-13.us-east-2.compute.internal    aws:///us-east-2c/i-07eb22c7a030e09ed   running
liuhuali@Lius-MacBook-Pro huali-test % oc create -f ~/Downloads/98-var-lib-etcd.yaml 
machineconfig.machineconfiguration.openshift.io/98-var-lib-etcd created

2.When the master-1 shows NotReady, update CPMS to trigger CPMS update. Observed that masters are updated sequentially from 0 to 2

liuhuali@Lius-MacBook-Pro huali-test % oc get node
NAME                                        STATUS                        ROLES                  AGE    VERSION
ip-10-0-48-76.us-east-2.compute.internal    Ready                         worker                 4h9m   v1.29.1+edc2c12
ip-10-0-51-210.us-east-2.compute.internal   NotReady,SchedulingDisabled   control-plane,master   54m    v1.29.1+edc2c12
ip-10-0-6-94.us-east-2.compute.internal     Ready                         control-plane,master   84m    v1.29.1+edc2c12
ip-10-0-67-67.us-east-2.compute.internal    Ready                         control-plane,master   35m    v1.29.1+edc2c12
ip-10-0-81-13.us-east-2.compute.internal    Ready                         worker                 4h7m   v1.29.1+edc2c12
ip-10-0-9-68.us-east-2.compute.internal     Ready                         worker                 4h9m   v1.29.1+edc2c12
liuhuali@Lius-MacBook-Pro huali-test % oc edit controlplanemachineset
controlplanemachineset.machine.openshift.io/cluster edited
liuhuali@Lius-MacBook-Pro huali-test % oc get node                   
NAME                                        STATUS                        ROLES                  AGE     VERSION
ip-10-0-48-76.us-east-2.compute.internal    Ready                         worker                 4h10m   v1.29.1+edc2c12
ip-10-0-51-210.us-east-2.compute.internal   NotReady,SchedulingDisabled   control-plane,master   55m     v1.29.1+edc2c12
ip-10-0-6-94.us-east-2.compute.internal     Ready                         control-plane,master   84m     v1.29.1+edc2c12
ip-10-0-67-67.us-east-2.compute.internal    Ready                         control-plane,master   36m     v1.29.1+edc2c12
ip-10-0-81-13.us-east-2.compute.internal    Ready                         worker                 4h8m    v1.29.1+edc2c12
ip-10-0-9-68.us-east-2.compute.internal     Ready                         worker                 4h10m   v1.29.1+edc2c12
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME                                         PHASE          TYPE         REGION      ZONE         AGE
huliu-aws20b-pswxj-master-b7ff5-2            Running        m6i.xlarge   us-east-2   us-east-2c   38m
huliu-aws20b-pswxj-master-gq62c-1            Running        m6i.xlarge   us-east-2   us-east-2b   57m
huliu-aws20b-pswxj-master-mf48d-0            Provisioning   m5.xlarge    us-east-2   us-east-2a   7s
huliu-aws20b-pswxj-master-t57cx-0            Running        m6i.xlarge   us-east-2   us-east-2a   87m
huliu-aws20b-pswxj-worker-us-east-2a-sxxn6   Running        m6i.xlarge   us-east-2   us-east-2a   4h13m
huliu-aws20b-pswxj-worker-us-east-2b-q58rv   Running        m6i.xlarge   us-east-2   us-east-2b   4h13m
huliu-aws20b-pswxj-worker-us-east-2c-t2v7r   Running        m6i.xlarge   us-east-2   us-east-2c   4h13m
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME                                         PHASE      TYPE         REGION      ZONE         AGE
huliu-aws20b-pswxj-master-b7ff5-2            Running    m6i.xlarge   us-east-2   us-east-2c   61m
huliu-aws20b-pswxj-master-gq62c-1            Running    m6i.xlarge   us-east-2   us-east-2b   80m
huliu-aws20b-pswxj-master-mf48d-0            Running    m5.xlarge    us-east-2   us-east-2a   22m
huliu-aws20b-pswxj-master-t57cx-0            Deleting   m6i.xlarge   us-east-2   us-east-2a   109m
huliu-aws20b-pswxj-worker-us-east-2a-sxxn6   Running    m6i.xlarge   us-east-2   us-east-2a   4h36m
huliu-aws20b-pswxj-worker-us-east-2b-q58rv   Running    m6i.xlarge   us-east-2   us-east-2b   4h36m
huliu-aws20b-pswxj-worker-us-east-2c-t2v7r   Running    m6i.xlarge   us-east-2   us-east-2c   4h36m
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME                                         PHASE      TYPE         REGION      ZONE         AGE
huliu-aws20b-pswxj-master-8gcm4-1            Running    m5.xlarge    us-east-2   us-east-2b   9m6s
huliu-aws20b-pswxj-master-b7ff5-2            Running    m6i.xlarge   us-east-2   us-east-2c   71m
huliu-aws20b-pswxj-master-gq62c-1            Deleting   m6i.xlarge   us-east-2   us-east-2b   90m
huliu-aws20b-pswxj-master-mf48d-0            Running    m5.xlarge    us-east-2   us-east-2a   32m
huliu-aws20b-pswxj-worker-us-east-2a-sxxn6   Running    m6i.xlarge   us-east-2   us-east-2a   4h46m
huliu-aws20b-pswxj-worker-us-east-2b-q58rv   Running    m6i.xlarge   us-east-2   us-east-2b   4h46m
huliu-aws20b-pswxj-worker-us-east-2c-t2v7r   Running    m6i.xlarge   us-east-2   us-east-2c   4h46m
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME                                         PHASE      TYPE         REGION      ZONE         AGE
huliu-aws20b-pswxj-master-8gcm4-1            Running    m5.xlarge    us-east-2   us-east-2b   21m
huliu-aws20b-pswxj-master-b7ff5-2            Running    m6i.xlarge   us-east-2   us-east-2c   84m
huliu-aws20b-pswxj-master-gq62c-1            Deleting   m6i.xlarge   us-east-2   us-east-2b   103m
huliu-aws20b-pswxj-master-mf48d-0            Running    m5.xlarge    us-east-2   us-east-2a   45m
huliu-aws20b-pswxj-worker-us-east-2a-sxxn6   Running    m6i.xlarge   us-east-2   us-east-2a   4h59m
huliu-aws20b-pswxj-worker-us-east-2b-q58rv   Running    m6i.xlarge   us-east-2   us-east-2b   4h59m
huliu-aws20b-pswxj-worker-us-east-2c-t2v7r   Running    m6i.xlarge   us-east-2   us-east-2c   4h59m
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME                                         PHASE      TYPE         REGION      ZONE         AGE
huliu-aws20b-pswxj-master-8gcm4-1            Running    m5.xlarge    us-east-2   us-east-2b   29m
huliu-aws20b-pswxj-master-b7ff5-2            Deleting   m6i.xlarge   us-east-2   us-east-2c   92m
huliu-aws20b-pswxj-master-kfb94-2            Running    m5.xlarge    us-east-2   us-east-2c   6m49s
huliu-aws20b-pswxj-master-mf48d-0            Running    m5.xlarge    us-east-2   us-east-2a   53m
huliu-aws20b-pswxj-worker-us-east-2a-sxxn6   Running    m6i.xlarge   us-east-2   us-east-2a   5h7m
huliu-aws20b-pswxj-worker-us-east-2b-q58rv   Running    m6i.xlarge   us-east-2   us-east-2b   5h7m
huliu-aws20b-pswxj-worker-us-east-2c-t2v7r   Running    m6i.xlarge   us-east-2   us-east-2c   5h7m

Huali Liu added a comment - 2024/02/20 7:51 AM Verified on 4.16.0-0.nightly-2024-02-17-094036 1.Create the MCO attached by Matthew or delete the MCO to apply a MachineConfig update liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.16.0-0.nightly-2024-02-17-094036 True False 16m Cluster version is 4.16.0-0.nightly-2024-02-17-094036 liuhuali@Lius-MacBook-Pro huali-test % oc get machine -o wide NAME PHASE TYPE REGION ZONE AGE NODE PROVIDERID STATE huliu-aws20b-pswxj-master-b7ff5-2 Running m6i.xlarge us-east-2 us-east-2c 30m ip-10-0-67-67.us-east-2.compute.internal aws: ///us-east-2c/i-031f500a19c7a1108 running huliu-aws20b-pswxj-master-gq62c-1 Running m6i.xlarge us-east-2 us-east-2b 49m ip-10-0-51-210.us-east-2.compute.internal aws: ///us-east-2b/i-0c6806cb676ffb3db running huliu-aws20b-pswxj-master-t57cx-0 Running m6i.xlarge us-east-2 us-east-2a 78m ip-10-0-6-94.us-east-2.compute.internal aws: ///us-east-2a/i-0c3d7e19db15e2882 running huliu-aws20b-pswxj-worker-us-east-2a-sxxn6 Running m6i.xlarge us-east-2 us-east-2a 4h5m ip-10-0-9-68.us-east-2.compute.internal aws: ///us-east-2a/i-0529aed6f7d47143f running huliu-aws20b-pswxj-worker-us-east-2b-q58rv Running m6i.xlarge us-east-2 us-east-2b 4h5m ip-10-0-48-76.us-east-2.compute.internal aws: ///us-east-2b/i-080e733b1fbf10a66 running huliu-aws20b-pswxj-worker-us-east-2c-t2v7r Running m6i.xlarge us-east-2 us-east-2c 4h5m ip-10-0-81-13.us-east-2.compute.internal aws: ///us-east-2c/i-07eb22c7a030e09ed running liuhuali@Lius-MacBook-Pro huali-test % oc create -f ~/Downloads/98- var -lib-etcd.yaml machineconfig.machineconfiguration.openshift.io/98- var -lib-etcd created 2.When the master-1 shows NotReady, update CPMS to trigger CPMS update. Observed that masters are updated sequentially from 0 to 2 liuhuali@Lius-MacBook-Pro huali-test % oc get node NAME STATUS ROLES AGE VERSION ip-10-0-48-76.us-east-2.compute.internal Ready worker 4h9m v1.29.1+edc2c12 ip-10-0-51-210.us-east-2.compute.internal NotReady,SchedulingDisabled control-plane,master 54m v1.29.1+edc2c12 ip-10-0-6-94.us-east-2.compute.internal Ready control-plane,master 84m v1.29.1+edc2c12 ip-10-0-67-67.us-east-2.compute.internal Ready control-plane,master 35m v1.29.1+edc2c12 ip-10-0-81-13.us-east-2.compute.internal Ready worker 4h7m v1.29.1+edc2c12 ip-10-0-9-68.us-east-2.compute.internal Ready worker 4h9m v1.29.1+edc2c12 liuhuali@Lius-MacBook-Pro huali-test % oc edit controlplanemachineset controlplanemachineset.machine.openshift.io/cluster edited liuhuali@Lius-MacBook-Pro huali-test % oc get node NAME STATUS ROLES AGE VERSION ip-10-0-48-76.us-east-2.compute.internal Ready worker 4h10m v1.29.1+edc2c12 ip-10-0-51-210.us-east-2.compute.internal NotReady,SchedulingDisabled control-plane,master 55m v1.29.1+edc2c12 ip-10-0-6-94.us-east-2.compute.internal Ready control-plane,master 84m v1.29.1+edc2c12 ip-10-0-67-67.us-east-2.compute.internal Ready control-plane,master 36m v1.29.1+edc2c12 ip-10-0-81-13.us-east-2.compute.internal Ready worker 4h8m v1.29.1+edc2c12 ip-10-0-9-68.us-east-2.compute.internal Ready worker 4h10m v1.29.1+edc2c12 liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-aws20b-pswxj-master-b7ff5-2 Running m6i.xlarge us-east-2 us-east-2c 38m huliu-aws20b-pswxj-master-gq62c-1 Running m6i.xlarge us-east-2 us-east-2b 57m huliu-aws20b-pswxj-master-mf48d-0 Provisioning m5.xlarge us-east-2 us-east-2a 7s huliu-aws20b-pswxj-master-t57cx-0 Running m6i.xlarge us-east-2 us-east-2a 87m huliu-aws20b-pswxj-worker-us-east-2a-sxxn6 Running m6i.xlarge us-east-2 us-east-2a 4h13m huliu-aws20b-pswxj-worker-us-east-2b-q58rv Running m6i.xlarge us-east-2 us-east-2b 4h13m huliu-aws20b-pswxj-worker-us-east-2c-t2v7r Running m6i.xlarge us-east-2 us-east-2c 4h13m liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-aws20b-pswxj-master-b7ff5-2 Running m6i.xlarge us-east-2 us-east-2c 61m huliu-aws20b-pswxj-master-gq62c-1 Running m6i.xlarge us-east-2 us-east-2b 80m huliu-aws20b-pswxj-master-mf48d-0 Running m5.xlarge us-east-2 us-east-2a 22m huliu-aws20b-pswxj-master-t57cx-0 Deleting m6i.xlarge us-east-2 us-east-2a 109m huliu-aws20b-pswxj-worker-us-east-2a-sxxn6 Running m6i.xlarge us-east-2 us-east-2a 4h36m huliu-aws20b-pswxj-worker-us-east-2b-q58rv Running m6i.xlarge us-east-2 us-east-2b 4h36m huliu-aws20b-pswxj-worker-us-east-2c-t2v7r Running m6i.xlarge us-east-2 us-east-2c 4h36m liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-aws20b-pswxj-master-8gcm4-1 Running m5.xlarge us-east-2 us-east-2b 9m6s huliu-aws20b-pswxj-master-b7ff5-2 Running m6i.xlarge us-east-2 us-east-2c 71m huliu-aws20b-pswxj-master-gq62c-1 Deleting m6i.xlarge us-east-2 us-east-2b 90m huliu-aws20b-pswxj-master-mf48d-0 Running m5.xlarge us-east-2 us-east-2a 32m huliu-aws20b-pswxj-worker-us-east-2a-sxxn6 Running m6i.xlarge us-east-2 us-east-2a 4h46m huliu-aws20b-pswxj-worker-us-east-2b-q58rv Running m6i.xlarge us-east-2 us-east-2b 4h46m huliu-aws20b-pswxj-worker-us-east-2c-t2v7r Running m6i.xlarge us-east-2 us-east-2c 4h46m liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-aws20b-pswxj-master-8gcm4-1 Running m5.xlarge us-east-2 us-east-2b 21m huliu-aws20b-pswxj-master-b7ff5-2 Running m6i.xlarge us-east-2 us-east-2c 84m huliu-aws20b-pswxj-master-gq62c-1 Deleting m6i.xlarge us-east-2 us-east-2b 103m huliu-aws20b-pswxj-master-mf48d-0 Running m5.xlarge us-east-2 us-east-2a 45m huliu-aws20b-pswxj-worker-us-east-2a-sxxn6 Running m6i.xlarge us-east-2 us-east-2a 4h59m huliu-aws20b-pswxj-worker-us-east-2b-q58rv Running m6i.xlarge us-east-2 us-east-2b 4h59m huliu-aws20b-pswxj-worker-us-east-2c-t2v7r Running m6i.xlarge us-east-2 us-east-2c 4h59m liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-aws20b-pswxj-master-8gcm4-1 Running m5.xlarge us-east-2 us-east-2b 29m huliu-aws20b-pswxj-master-b7ff5-2 Deleting m6i.xlarge us-east-2 us-east-2c 92m huliu-aws20b-pswxj-master-kfb94-2 Running m5.xlarge us-east-2 us-east-2c 6m49s huliu-aws20b-pswxj-master-mf48d-0 Running m5.xlarge us-east-2 us-east-2a 53m huliu-aws20b-pswxj-worker-us-east-2a-sxxn6 Running m6i.xlarge us-east-2 us-east-2a 5h7m huliu-aws20b-pswxj-worker-us-east-2b-q58rv Running m6i.xlarge us-east-2 us-east-2b 5h7m huliu-aws20b-pswxj-worker-us-east-2c-t2v7r Running m6i.xlarge us-east-2 us-east-2c 5h7m

Huali Liu added a comment - 2024/02/20 7:19 AM

Seems I reproduced this on 4.15.0-0.nightly-2024-02-19-162915, because I observed CPMS immediately started provisioning a new master-0 and CPMS immediately started deleting master-1.

liuhuali@Lius-MacBook-Pro huali-test % oc get node
NAME                                        STATUS                        ROLES                  AGE    VERSION
ip-10-0-22-26.us-east-2.compute.internal    Ready                         worker                 163m   v1.28.6+6216ea1
ip-10-0-31-203.us-east-2.compute.internal   Ready                         control-plane,master   114m   v1.28.6+6216ea1
ip-10-0-33-44.us-east-2.compute.internal    Ready                         worker                 163m   v1.28.6+6216ea1
ip-10-0-55-61.us-east-2.compute.internal    NotReady,SchedulingDisabled   control-plane,master   93m    v1.28.6+6216ea1
ip-10-0-76-37.us-east-2.compute.internal    Ready                         worker                 162m   v1.28.6+6216ea1
ip-10-0-86-103.us-east-2.compute.internal   Ready                         control-plane,master   73m    v1.28.6+6216ea1
liuhuali@Lius-MacBook-Pro huali-test % oc edit controlplanemachineset
controlplanemachineset.machine.openshift.io/cluster edited
liuhuali@Lius-MacBook-Pro huali-test % oc get node                   
NAME                                        STATUS                        ROLES                  AGE    VERSION
ip-10-0-22-26.us-east-2.compute.internal    Ready                         worker                 163m   v1.28.6+6216ea1
ip-10-0-31-203.us-east-2.compute.internal   Ready                         control-plane,master   115m   v1.28.6+6216ea1
ip-10-0-33-44.us-east-2.compute.internal    Ready                         worker                 163m   v1.28.6+6216ea1
ip-10-0-55-61.us-east-2.compute.internal    NotReady,SchedulingDisabled   control-plane,master   93m    v1.28.6+6216ea1
ip-10-0-76-37.us-east-2.compute.internal    Ready                         worker                 163m   v1.28.6+6216ea1
ip-10-0-86-103.us-east-2.compute.internal   Ready                         control-plane,master   73m    v1.28.6+6216ea1
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME                                         PHASE      TYPE         REGION      ZONE         AGE
huliu-aws20a-fvq57-master-cpwv4-2            Running    m5.xlarge    us-east-2   us-east-2c   76m
huliu-aws20a-fvq57-master-g4b45-0                                                             7s
huliu-aws20a-fvq57-master-kttvb-0            Running    m5.xlarge    us-east-2   us-east-2a   118m
huliu-aws20a-fvq57-master-s8t4c-1            Deleting   m5.xlarge    us-east-2   us-east-2b   96m
huliu-aws20a-fvq57-worker-us-east-2a-49kct   Running    m6i.xlarge   us-east-2   us-east-2a   165m
huliu-aws20a-fvq57-worker-us-east-2b-rjh5n   Running    m6i.xlarge   us-east-2   us-east-2b   165m
huliu-aws20a-fvq57-worker-us-east-2c-gfbwj   Running    m6i.xlarge   us-east-2   us-east-2c   165m
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME                                         PHASE         TYPE         REGION      ZONE         AGE
huliu-aws20a-fvq57-master-cpwv4-2            Running       m5.xlarge    us-east-2   us-east-2c   77m
huliu-aws20a-fvq57-master-g4b45-0            Provisioned   m6i.xlarge   us-east-2   us-east-2a   97s
huliu-aws20a-fvq57-master-kttvb-0            Running       m5.xlarge    us-east-2   us-east-2a   119m
huliu-aws20a-fvq57-master-s8t4c-1            Deleting      m5.xlarge    us-east-2   us-east-2b   97m
huliu-aws20a-fvq57-worker-us-east-2a-49kct   Running       m6i.xlarge   us-east-2   us-east-2a   167m
huliu-aws20a-fvq57-worker-us-east-2b-rjh5n   Running       m6i.xlarge   us-east-2   us-east-2b   167m
huliu-aws20a-fvq57-worker-us-east-2c-gfbwj   Running       m6i.xlarge   us-east-2   us-east-2c   167m
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME                                         PHASE          TYPE         REGION      ZONE         AGE
huliu-aws20a-fvq57-master-7dfhw-1            Provisioning   m6i.xlarge   us-east-2   us-east-2b   15s
huliu-aws20a-fvq57-master-cpwv4-2            Running        m5.xlarge    us-east-2   us-east-2c   81m
huliu-aws20a-fvq57-master-g4b45-0            Running        m6i.xlarge   us-east-2   us-east-2a   5m42s
huliu-aws20a-fvq57-master-kttvb-0            Deleting       m5.xlarge    us-east-2   us-east-2a   123m
huliu-aws20a-fvq57-worker-us-east-2a-49kct   Running        m6i.xlarge   us-east-2   us-east-2a   171m
huliu-aws20a-fvq57-worker-us-east-2b-rjh5n   Running        m6i.xlarge   us-east-2   us-east-2b   171m
huliu-aws20a-fvq57-worker-us-east-2c-gfbwj   Running        m6i.xlarge   us-east-2   us-east-2c   171m
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME                                         PHASE         TYPE         REGION      ZONE         AGE
huliu-aws20a-fvq57-master-7dfhw-1            Provisioned   m6i.xlarge   us-east-2   us-east-2b   5m54s
huliu-aws20a-fvq57-master-cpwv4-2            Deleting      m5.xlarge    us-east-2   us-east-2c   87m
huliu-aws20a-fvq57-master-g4b45-0            Running       m6i.xlarge   us-east-2   us-east-2a   11m
huliu-aws20a-fvq57-master-kttvb-0            Deleting      m5.xlarge    us-east-2   us-east-2a   129m
huliu-aws20a-fvq57-worker-us-east-2a-49kct   Running       m6i.xlarge   us-east-2   us-east-2a   177m
huliu-aws20a-fvq57-worker-us-east-2b-rjh5n   Running       m6i.xlarge   us-east-2   us-east-2b   177m
huliu-aws20a-fvq57-worker-us-east-2c-gfbwj   Running       m6i.xlarge   us-east-2   us-east-2c   177m

Huali Liu added a comment - 2024/02/20 7:19 AM Seems I reproduced this on 4.15.0-0.nightly-2024-02-19-162915, because I observed CPMS immediately started provisioning a new master-0 and CPMS immediately started deleting master-1. liuhuali@Lius-MacBook-Pro huali-test % oc get node NAME STATUS ROLES AGE VERSION ip-10-0-22-26.us-east-2.compute.internal Ready worker 163m v1.28.6+6216ea1 ip-10-0-31-203.us-east-2.compute.internal Ready control-plane,master 114m v1.28.6+6216ea1 ip-10-0-33-44.us-east-2.compute.internal Ready worker 163m v1.28.6+6216ea1 ip-10-0-55-61.us-east-2.compute.internal NotReady,SchedulingDisabled control-plane,master 93m v1.28.6+6216ea1 ip-10-0-76-37.us-east-2.compute.internal Ready worker 162m v1.28.6+6216ea1 ip-10-0-86-103.us-east-2.compute.internal Ready control-plane,master 73m v1.28.6+6216ea1 liuhuali@Lius-MacBook-Pro huali-test % oc edit controlplanemachineset controlplanemachineset.machine.openshift.io/cluster edited liuhuali@Lius-MacBook-Pro huali-test % oc get node NAME STATUS ROLES AGE VERSION ip-10-0-22-26.us-east-2.compute.internal Ready worker 163m v1.28.6+6216ea1 ip-10-0-31-203.us-east-2.compute.internal Ready control-plane,master 115m v1.28.6+6216ea1 ip-10-0-33-44.us-east-2.compute.internal Ready worker 163m v1.28.6+6216ea1 ip-10-0-55-61.us-east-2.compute.internal NotReady,SchedulingDisabled control-plane,master 93m v1.28.6+6216ea1 ip-10-0-76-37.us-east-2.compute.internal Ready worker 163m v1.28.6+6216ea1 ip-10-0-86-103.us-east-2.compute.internal Ready control-plane,master 73m v1.28.6+6216ea1 liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-aws20a-fvq57-master-cpwv4-2 Running m5.xlarge us-east-2 us-east-2c 76m huliu-aws20a-fvq57-master-g4b45-0 7s huliu-aws20a-fvq57-master-kttvb-0 Running m5.xlarge us-east-2 us-east-2a 118m huliu-aws20a-fvq57-master-s8t4c-1 Deleting m5.xlarge us-east-2 us-east-2b 96m huliu-aws20a-fvq57-worker-us-east-2a-49kct Running m6i.xlarge us-east-2 us-east-2a 165m huliu-aws20a-fvq57-worker-us-east-2b-rjh5n Running m6i.xlarge us-east-2 us-east-2b 165m huliu-aws20a-fvq57-worker-us-east-2c-gfbwj Running m6i.xlarge us-east-2 us-east-2c 165m liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-aws20a-fvq57-master-cpwv4-2 Running m5.xlarge us-east-2 us-east-2c 77m huliu-aws20a-fvq57-master-g4b45-0 Provisioned m6i.xlarge us-east-2 us-east-2a 97s huliu-aws20a-fvq57-master-kttvb-0 Running m5.xlarge us-east-2 us-east-2a 119m huliu-aws20a-fvq57-master-s8t4c-1 Deleting m5.xlarge us-east-2 us-east-2b 97m huliu-aws20a-fvq57-worker-us-east-2a-49kct Running m6i.xlarge us-east-2 us-east-2a 167m huliu-aws20a-fvq57-worker-us-east-2b-rjh5n Running m6i.xlarge us-east-2 us-east-2b 167m huliu-aws20a-fvq57-worker-us-east-2c-gfbwj Running m6i.xlarge us-east-2 us-east-2c 167m liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-aws20a-fvq57-master-7dfhw-1 Provisioning m6i.xlarge us-east-2 us-east-2b 15s huliu-aws20a-fvq57-master-cpwv4-2 Running m5.xlarge us-east-2 us-east-2c 81m huliu-aws20a-fvq57-master-g4b45-0 Running m6i.xlarge us-east-2 us-east-2a 5m42s huliu-aws20a-fvq57-master-kttvb-0 Deleting m5.xlarge us-east-2 us-east-2a 123m huliu-aws20a-fvq57-worker-us-east-2a-49kct Running m6i.xlarge us-east-2 us-east-2a 171m huliu-aws20a-fvq57-worker-us-east-2b-rjh5n Running m6i.xlarge us-east-2 us-east-2b 171m huliu-aws20a-fvq57-worker-us-east-2c-gfbwj Running m6i.xlarge us-east-2 us-east-2c 171m liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-aws20a-fvq57-master-7dfhw-1 Provisioned m6i.xlarge us-east-2 us-east-2b 5m54s huliu-aws20a-fvq57-master-cpwv4-2 Deleting m5.xlarge us-east-2 us-east-2c 87m huliu-aws20a-fvq57-master-g4b45-0 Running m6i.xlarge us-east-2 us-east-2a 11m huliu-aws20a-fvq57-master-kttvb-0 Deleting m5.xlarge us-east-2 us-east-2a 129m huliu-aws20a-fvq57-worker-us-east-2a-49kct Running m6i.xlarge us-east-2 us-east-2a 177m huliu-aws20a-fvq57-worker-us-east-2b-rjh5n Running m6i.xlarge us-east-2 us-east-2b 177m huliu-aws20a-fvq57-worker-us-east-2c-gfbwj Running m6i.xlarge us-east-2 us-east-2c 177m

OpenShift Jira Bot added a comment - 2024/02/13 2:32 PM

Hi joelspeed,

Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.

OpenShift Jira Bot added a comment - 2024/02/13 2:32 PM Hi joelspeed , Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.

Joel Speed added a comment - 2024/02/09 11:08 AM

CPMS currently has the ability, during a rollout, to delete a Machine in an index that it should not yet be deleting. This fix is important and should be backported through to 4.12. I'm not marking it as a release blocker, as, while the consequences of this could be pretty bad, CPMS rollouts are not an every day occurrence, even less frequent that they will coincide with MCO updates.

Joel Speed added a comment - 2024/02/09 11:08 AM CPMS currently has the ability, during a rollout, to delete a Machine in an index that it should not yet be deleting. This fix is important and should be backported through to 4.12. I'm not marking it as a release blocker, as, while the consequences of this could be pretty bad, CPMS rollouts are not an every day occurrence, even less frequent that they will coincide with MCO updates.

Matthew Booth added a comment - 2024/02/08 3:58 PM

I have attached the MCO I applying as an example. I don't think the specific contents are relevant.

Matthew Booth added a comment - 2024/02/08 3:58 PM I have attached the MCO I applying as an example. I don't think the specific contents are relevant.

Assignee:: Joel Speed

Reporter:: Matthew Booth

QA Contact:: Huali Liu

Doc Contact:: Jeana Routh

Votes:: 0 Vote for this issue

Watchers:: 7 Start watching this issue

Created:: 2024/02/08 3:31 PM

Updated:: 2024/06/27 11:37 AM

Resolved:: 2024/06/27 11:37 AM

Details

Description

Attachments

Attachments

Issue Links

Easy Agile Planning Poker

Activity

Collapse comment: Errata Tool added a comment - 2024/06/27 11:37 AM

Expand comment: Errata Tool added a comment - 2024/06/27 11:37 AM

Collapse comment: Huali Liu added a comment - 2024/02/20 7:51 AM

Expand comment: Huali Liu added a comment - 2024/02/20 7:51 AM

Collapse comment: Huali Liu added a comment - 2024/02/20 7:19 AM

Expand comment: Huali Liu added a comment - 2024/02/20 7:19 AM

Collapse comment: OpenShift Jira Bot added a comment - 2024/02/13 2:32 PM

Expand comment: OpenShift Jira Bot added a comment - 2024/02/13 2:32 PM

Collapse comment: Joel Speed added a comment - 2024/02/09 11:08 AM

Expand comment: Joel Speed added a comment - 2024/02/09 11:08 AM

Collapse comment: Matthew Booth added a comment - 2024/02/08 3:58 PM

Expand comment: Matthew Booth added a comment - 2024/02/08 3:58 PM

People

Dates