Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-31808

control-plane-machine-set operator pod stuck into crashloopbackoff state with the nil pointer dereference runtime error

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Major Major
    • 4.16.0
    • 4.15.z
    • Cloud Compute / Unknown
    • None
    • No
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the Control Plane Machine Set Operator crashed when attempting to update a {vmw-first} cluster because the infrastructure resource was not configured.
      With this release, the Operator is able to handle this scenario so that the cluster update is able to proceed.
      (link:https://issues.redhat.com/browse/OCPBUGS-31808[*OCPBUGS-31808*])
      Show
      * Previously, the Control Plane Machine Set Operator crashed when attempting to update a {vmw-first} cluster because the infrastructure resource was not configured. With this release, the Operator is able to handle this scenario so that the cluster update is able to proceed. (link: https://issues.redhat.com/browse/OCPBUGS-31808 [* OCPBUGS-31808 *])
    • Bug Fix
    • Done

      Description of problem:

      control-plane-machine-set operator pod stuck into crashloopbackoff state with panic: runtime error: invalid memory address or nil pointer dereference while extracting the failureDomain from the controlplanemachineset. Below is the error trace for reference.
~~~
2024-04-04T09:32:23.594257072Z I0404 09:32:23.594176       1 controller.go:146]  "msg"="Finished reconciling control plane machine set" "controller"="controlplanemachinesetgenerator" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="c282f3e3-9f9d-40df-a24e-417ba2ea4106"
2024-04-04T09:32:23.594257072Z I0404 09:32:23.594221       1 controller.go:125]  "msg"="Reconciling control plane machine set" "controller"="controlplanemachinesetgenerator" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="7f03c05f-2717-49e0-95f8-3e8b2ce2fc55"
2024-04-04T09:32:23.594274974Z I0404 09:32:23.594257       1 controller.go:146]  "msg"="Finished reconciling control plane machine set" "controller"="controlplanemachinesetgenerator" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="7f03c05f-2717-49e0-95f8-3e8b2ce2fc55"
2024-04-04T09:32:23.597509741Z I0404 09:32:23.597426       1 watch_filters.go:179] reconcile triggered by infrastructure change
2024-04-04T09:32:23.606311553Z I0404 09:32:23.606243       1 controller.go:220]  "msg"="Starting workers" "controller"="controlplanemachineset" "worker count"=1
2024-04-04T09:32:23.606360950Z I0404 09:32:23.606340       1 controller.go:169]  "msg"="Reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="5dac54f4-57ab-419b-b258-79136ca8b400"
2024-04-04T09:32:23.609322467Z I0404 09:32:23.609217       1 panic.go:884]  "msg"="Finished reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="5dac54f4-57ab-419b-b258-79136ca8b400"
2024-04-04T09:32:23.609322467Z I0404 09:32:23.609271       1 controller.go:115]  "msg"="Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference" "controller"="controlplanemachineset" "reconcileID"="5dac54f4-57ab-419b-b258-79136ca8b400"
2024-04-04T09:32:23.612540681Z panic: runtime error: invalid memory address or nil pointer dereference [recovered]
2024-04-04T09:32:23.612540681Z     panic: runtime error: invalid memory address or nil pointer dereference
2024-04-04T09:32:23.612540681Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1a5911c]
2024-04-04T09:32:23.612540681Z 
2024-04-04T09:32:23.612540681Z goroutine 255 [running]:
2024-04-04T09:32:23.612540681Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
2024-04-04T09:32:23.612571624Z     /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116 +0x1fa
2024-04-04T09:32:23.612571624Z panic({0x1c8ac60, 0x31c6ea0})
2024-04-04T09:32:23.612571624Z     /usr/lib/golang/src/runtime/panic.go:884 +0x213
2024-04-04T09:32:23.612571624Z github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig.VSphereProviderConfig.ExtractFailureDomain(...)
2024-04-04T09:32:23.612571624Z     /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig/vsphere.go:120
2024-04-04T09:32:23.612571624Z github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig.providerConfig.ExtractFailureDomain({{0x1f2a71a, 0x7}, {{{{...}, {...}}, {{...}, {...}, {...}, {...}, {...}, {...}, ...}, ...}}, ...})
2024-04-04T09:32:23.612588145Z     /go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig/providerconfig.go:212 +0x23c
~~~

      Version-Release number of selected component (if applicable):

      How reproducible:

      Steps to Reproduce:

          1.
    2.
    3.

      Actual results:

      control-plane-machine-set operator stuck into crashloopback off state while cluster upgrade.

      Expected results:

      control-plane-machine-set operator should be upgraded without any errors.

      Additional info:

      This is happening during the cluster upgrade of Vsphere IPI cluster from OCP version 4.14.z to 4.15.6 and may impact other z stream releases. 
from the official docs[1]  I see providing the failure domain for the Vsphere platform is tech preview feature.
[1] https://docs.openshift.com/container-platform/4.15/machine_management/control_plane_machine_management/cpmso-configuration.html#cpmso-yaml-failure-domain-vsphere_cpmso-configuration

            [OCPBUGS-31808] control-plane-machine-set operator pod stuck into crashloopbackoff state with the nil pointer dereference runtime error

            Errata Tool added a comment -

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory (Critical: OpenShift Container Platform 4.16.0 bug fix and security update), and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHSA-2024:0041

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Critical: OpenShift Container Platform 4.16.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:0041

            Huali Liu added a comment -

            Pre-merge verified the upgrade from 4.14 to 4.15 and continue verified the upgrade from 4.15 to 4.16 in OCPBUGS-32414

            Huali Liu added a comment - Pre-merge verified the upgrade from 4.14 to 4.15 and continue verified the upgrade from 4.15 to 4.16 in OCPBUGS-32414

            Huali Liu added a comment -

            Thank you rhn-support-zhsun for pointing there is a difference in the infrastructure, there is no .spec.platformSpec.vsphere part in the infrastructure resource pasted by Nirupma, but mine have. I can reproduce the issue after deleting .spec.platformSpec.vsphere part in the infrastructure.
            Steps:
            1. Install a vSphere 4.14.18 cluster, we use automated template: ipi-on-vsphere/versioned-installer

            liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.18   True        False         22m     Cluster version is 4.14.18

            2. Delete .spec.platformSpec.vsphere part in the infrastructure

            liuhuali@Lius-MacBook-Pro huali-test % oc edit infrastructure cluster                  
infrastructure.config.openshift.io/cluster edited
liuhuali@Lius-MacBook-Pro huali-test % oc get infrastructure cluster -oyaml
apiVersion: config.openshift.io/v1
kind: Infrastructure
metadata:
  creationTimestamp: "2024-04-24T03:42:01Z"
  generation: 2
  name: cluster
  resourceVersion: "38316"
  uid: 7b2608b3-89cc-4eab-abe4-0ec9c44c7043
spec:
  cloudConfig:
    key: config
    name: cloud-provider-config
  platformSpec:
    type: VSphere
status:
  apiServerInternalURI: https://api-int.huliu-vs424a.qe.devcluster.openshift.com:6443
  apiServerURL: https://api.huliu-vs424a.qe.devcluster.openshift.com:6443
  controlPlaneTopology: HighlyAvailable
  cpuPartitioning: None
  etcdDiscoveryDomain: ""
  infrastructureName: huliu-vs424a-ttzg8
  infrastructureTopology: HighlyAvailable
  platform: VSphere
  platformStatus:
    type: VSphere
    vsphere:
      apiServerInternalIP: 192.168.221.5
      apiServerInternalIPs:
      - 192.168.221.5
      ingressIP: 192.168.221.7
      ingressIPs:
      - 192.168.221.7

            3. Create the controlplanemachineset

            liuhuali@Lius-MacBook-Pro huali-test % oc create -f controlplanemachineset_vsphere.yaml 
controlplanemachineset.machine.openshift.io/cluster created
liuhuali@Lius-MacBook-Pro huali-test % oc get controlplanemachineset
NAME      DESIRED   CURRENT   READY   UPDATED   UNAVAILABLE   STATE    AGE
cluster   3         3         3       3                       Active   7s
liuhuali@Lius-MacBook-Pro huali-test % oc get controlplanemachineset -oyaml
apiVersion: v1
items:
- apiVersion: machine.openshift.io/v1
  kind: ControlPlaneMachineSet
  metadata:
    creationTimestamp: "2024-04-24T04:37:31Z"
    finalizers:
    - controlplanemachineset.machine.openshift.io
    generation: 2
    name: cluster
    namespace: openshift-machine-api
    resourceVersion: "40161"
    uid: 940392c1-b717-4e8c-810b-5ee2a3dac1c9
  spec:
    replicas: 3
    selector:
      matchLabels:
        machine.openshift.io/cluster-api-cluster: huliu-vs424a-ttzg8
        machine.openshift.io/cluster-api-machine-role: master
        machine.openshift.io/cluster-api-machine-type: master
    state: Active
    strategy:
      type: RollingUpdate
    template:
      machineType: machines_v1beta1_machine_openshift_io
      machines_v1beta1_machine_openshift_io:
        failureDomains:
          platform: ""
        metadata:
          labels:
            machine.openshift.io/cluster-api-cluster: huliu-vs424a-ttzg8
            machine.openshift.io/cluster-api-machine-role: master
            machine.openshift.io/cluster-api-machine-type: master
        spec:
          lifecycleHooks: {}
          metadata: {}
          providerSpec:
            value:
              apiVersion: machine.openshift.io/v1beta1
              credentialsSecret:
                name: vsphere-cloud-credentials
              diskGiB: 120
              kind: VSphereMachineProviderSpec
              memoryMiB: 16384
              metadata:
                creationTimestamp: null
              network:
                devices:
                - networkName: devqe-segment-221
              numCPUs: 4
              numCoresPerSocket: 4
              snapshot: ""
              template: huliu-vs424a-ttzg8-rhcos-generated-region-generated-zone
              userDataSecret:
                name: master-user-data
              workspace:
                datacenter: DEVQEdatacenter
                datastore: /DEVQEdatacenter/datastore/vsanDatastore
                folder: /DEVQEdatacenter/vm/huliu-vs424a-ttzg8
                resourcePool: /DEVQEdatacenter/host/DEVQEcluster//Resources
                server: vcenter.devqe.ibmc.devcluster.openshift.com
  status:
    conditions:
    - lastTransitionTime: "2024-04-24T04:37:31Z"
      message: ""
      observedGeneration: 2
      reason: AsExpected
      status: "False"
      type: Error
    - lastTransitionTime: "2024-04-24T04:37:31Z"
      message: ""
      observedGeneration: 2
      reason: AllReplicasAvailable
      status: "True"
      type: Available
    - lastTransitionTime: "2024-04-24T04:37:31Z"
      message: ""
      observedGeneration: 2
      reason: AsExpected
      status: "False"
      type: Degraded
    - lastTransitionTime: "2024-04-24T04:37:31Z"
      message: ""
      observedGeneration: 2
      reason: AllReplicasUpdated
      status: "False"
      type: Progressing
    observedGeneration: 2
    readyReplicas: 3
    replicas: 3
    updatedReplicas: 3
kind: List
metadata:
  resourceVersion: ""

            4. Upgrade to 4.15.6 stuck

            liuhuali@Lius-MacBook-Pro huali-test % oc adm upgrade --to-image='quay.io/openshift-release-dev/ocp-release:4.15.6-x86_64' --force --allow-explicit-upgrade
warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Requesting update to release image quay.io/openshift-release-dev/ocp-release:4.15.6-x86_64
liuhuali@Lius-MacBook-Pro huali-test % oc get  clusterversion                                     
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.18   True        True          142m    Working towards 4.15.6: 229 of 872 done (26% complete), waiting on control-plane-machine-set
liuhuali@Lius-MacBook-Pro huali-test % oc get pod                                                 
NAME                                                 READY   STATUS             RESTARTS       AGE
cluster-autoscaler-operator-76b56dbdf-gnsh4          2/2     Running            0              3h18m
cluster-baremetal-operator-5f57dbf678-v8qwm          2/2     Running            0              3h18m
control-plane-machine-set-operator-c674d9976-s2dvq   0/1     CrashLoopBackOff   19 (20s ago)   124m
machine-api-controllers-64fdc59448-rk97v             7/7     Running            0              121m
machine-api-operator-5bb76b684f-2f6zl                2/2     Running            0              124m
liuhuali@Lius-MacBook-Pro huali-test % oc logs control-plane-machine-set-operator-c674d9976-s2dvq
...
I0424 07:02:35.617844       1 panic.go:884]  "msg"="Finished reconciling control plane machine set" "controller"="controlplanemachineset" "name"="cluster" "namespace"="openshift-machine-api" "reconcileID"="394fd0b0-8f9f-4680-a2a0-00e407b3c5e6"
I0424 07:02:35.617968       1 controller.go:115]  "msg"="Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference" "controller"="controlplanemachineset" "reconcileID"="394fd0b0-8f9f-4680-a2a0-00e407b3c5e6"
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1a5911c]

goroutine 254 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116 +0x1fa
panic({0x1c8ac60, 0x31c6ea0})
	/usr/lib/golang/src/runtime/panic.go:884 +0x213
github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig.VSphereProviderConfig.ExtractFailureDomain(...)
	/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig/vsphere.go:120
github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig.providerConfig.ExtractFailureDomain({{0x1f2a71a, 0x7}, {{{{...}, {...}}, {{...}, {...}, {...}, {...}, {...}, {...}, ...}, ...}}, ...})
	/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig/providerconfig.go:212 +0x23c
github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1.(*openshiftMachineProvider).updateMachineCache(0xc000b8c500, {0x22a0920, 0xc00075e3f0}, {{0x22a3820?, 0xc000625310?}, 0x0?})
	/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/provider.go:224 +0x276
github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1.NewMachineProvider({0x22a0920, 0xc00075e3f0}, {{0x22a3820?, 0xc000625310?}, 0x36?}, {0x22a9400?, 0xc0001e14d0?}, {0x229fdf0?, 0xc000332640}, 0xc000b5d1e0)
	/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/provider.go:152 +0x752
github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers.NewMachineProvider({0x22a0920, 0xc00075e3f0}, {{0x22a3820?, 0xc000625310?}, 0xc000e8b990?}, {0x22a9400, 0xc0001e14d0}, {0x229fdf0, 0xc000332640}, 0xc000b5d1e0)
	/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/machineproviders.go:46 +0x1c5
github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset.(*ControlPlaneMachineSetReconciler).reconcile(0xc00034c580, {0x22a0920, 0xc00075e3f0}, {{0x22a3820?, 0xc000625310?}, 0xc00075e3f0?}, 0xc000b5d1e0)
	/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset/controller.go:245 +0x146
github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset.(*ControlPlaneMachineSetReconciler).Reconcile(0xc00034c580, {0x22a0920, 0xc00075e3f0}, {{{0x1f407a8?, 0x0?}, {0x1f2a855?, 0xc0008dbd48?}}})
	/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/pkg/controllers/controlplanemachineset/controller.go:194 +0x3f0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x22a3820?, {0x22a0920?, 0xc00075e3f0?}, {{{0x1f407a8?, 0xb?}, {0x1f2a855?, 0x0?}}})
	/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0003263c0, {0x22a0878, 0xc0006245a0}, {0x1d36760?, 0xc0004b8200?})
	/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316 +0x3ca
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0003263c0, {0x22a0878, 0xc0006245a0})
	/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/go/src/github.com/openshift/cluster-control-plane-machine-set-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:223 +0x587
liuhuali@Lius-MacBook-Pro huali-test %

            Huali Liu added a comment - Thank you rhn-support-zhsun for pointing there is a difference in the infrastructure, there is no .spec.platformSpec.vsphere part in the infrastructure resource pasted by Nirupma, but mine have. I can reproduce the issue after deleting .spec.platformSpec.vsphere part in the infrastructure. Steps: 1. Install a vSphere 4.14.18 cluster, we use automated template: ipi-on-vsphere/versioned-installer liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.14.18 True False 22m Cluster version is 4.14.18 2. Delete .spec.platformSpec.vsphere part in the infrastructure liuhuali@Lius-MacBook-Pro huali-test % oc edit infrastructure cluster infrastructure.config.openshift.io/cluster edited liuhuali@Lius-MacBook-Pro huali-test % oc get infrastructure cluster -oyaml apiVersion: config.openshift.io/v1 kind: Infrastructure metadata: creationTimestamp: "2024-04-24T03:42:01Z" generation: 2 name: cluster resourceVersion: "38316" uid: 7b2608b3-89cc-4eab-abe4-0ec9c44c7043 spec: cloudConfig: key: config name: cloud-provider-config platformSpec: type: VSphere status: apiServerInternalURI: https: //api- int .huliu-vs424a.qe.devcluster.openshift.com:6443 apiServerURL: https: //api.huliu-vs424a.qe.devcluster.openshift.com:6443 controlPlaneTopology: HighlyAvailable cpuPartitioning: None etcdDiscoveryDomain: "" infrastructureName: huliu-vs424a-ttzg8 infrastructureTopology: HighlyAvailable platform: VSphere platformStatus: type: VSphere vsphere: apiServerInternalIP: 192.168.221.5 apiServerInternalIPs: - 192.168.221.5 ingressIP: 192.168.221.7 ingressIPs: - 192.168.221.7 3. Create the controlplanemachineset liuhuali@Lius-MacBook-Pro huali-test % oc create -f controlplanemachineset_vsphere.yaml controlplanemachineset.machine.openshift.io/cluster created liuhuali@Lius-MacBook-Pro huali-test % oc get controlplanemachineset NAME DESIRED CURRENT READY UPDATED UNAVAILABLE STATE AGE cluster 3 3 3 3 Active 7s liuhuali@Lius-MacBook-Pro huali-test % oc get controlplanemachineset -oyaml apiVersion: v1 items: - apiVersion: machine.openshift.io/v1 kind: ControlPlaneMachineSet metadata: creationTimestamp: "2024-04-24T04:37:31Z" finalizers: - controlplanemachineset.machine.openshift.io generation: 2 name: cluster namespace: openshift-machine-api resourceVersion: "40161" uid: 940392c1-b717-4e8c-810b-5ee2a3dac1c9 spec: replicas: 3 selector: matchLabels: machine.openshift.io/cluster-api-cluster: huliu-vs424a-ttzg8 machine.openshift.io/cluster-api-machine-role: master machine.openshift.io/cluster-api-machine-type: master state: Active strategy: type: RollingUpdate template: machineType: machines_v1beta1_machine_openshift_io machines_v1beta1_machine_openshift_io: failureDomains: platform: "" metadata: labels: machine.openshift.io/cluster-api-cluster: huliu-vs424a-ttzg8 machine.openshift.io/cluster-api-machine-role: master machine.openshift.io/cluster-api-machine-type: master spec: lifecycleHooks: {} metadata: {} providerSpec: value: apiVersion: machine.openshift.io/v1beta1 credentialsSecret: name: vsphere-cloud-credentials diskGiB: 120 kind: VSphereMachineProviderSpec memoryMiB: 16384 metadata: creationTimestamp: null network: devices: - networkName: devqe-segment-221 numCPUs: 4 numCoresPerSocket: 4 snapshot: "" template: huliu-vs424a-ttzg8-rhcos-generated-region-generated-zone userDataSecret: name: master-user-data workspace: datacenter: DEVQEdatacenter datastore: /DEVQEdatacenter/datastore/vsanDatastore folder: /DEVQEdatacenter/vm/huliu-vs424a-ttzg8 resourcePool: /DEVQEdatacenter/host/DEVQEcluster //Resources server: vcenter.devqe.ibmc.devcluster.openshift.com status: conditions: - lastTransitionTime: "2024-04-24T04:37:31Z" message: "" observedGeneration: 2 reason: AsExpected status: "False" type: Error - lastTransitionTime: "2024-04-24T04:37:31Z" message: "" observedGeneration: 2 reason: AllReplicasAvailable status: "True" type: Available - lastTransitionTime: "2024-04-24T04:37:31Z" message: "" observedGeneration: 2 reason: AsExpected status: "False" type: Degraded - lastTransitionTime: "2024-04-24T04:37:31Z" message: "" observedGeneration: 2 reason: AllReplicasUpdated status: "False" type: Progressing observedGeneration: 2 readyReplicas: 3 replicas: 3 updatedReplicas: 3 kind: List metadata: resourceVersion: "" 4. Upgrade to 4.15.6 stuck liuhuali@Lius-MacBook-Pro huali-test % oc adm upgrade --to-image= 'quay.io/openshift-release-dev/ocp-release:4.15.6-x86_64' --force --allow-explicit-upgrade warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures. Requesting update to release image quay.io/openshift-release-dev/ocp-release:4.15.6-x86_64 liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.14.18 True True 142m Working towards 4.15.6: 229 of 872 done (26% complete), waiting on control-plane-machine-set liuhuali@Lius-MacBook-Pro huali-test % oc get pod NAME READY STATUS RESTARTS AGE cluster-autoscaler- operator -76b56dbdf-gnsh4 2/2 Running 0 3h18m cluster-baremetal- operator -5f57dbf678-v8qwm 2/2 Running 0 3h18m control-plane-machine-set- operator -c674d9976-s2dvq 0/1 CrashLoopBackOff 19 (20s ago) 124m machine-api-controllers-64fdc59448-rk97v 7/7 Running 0 121m machine-api- operator -5bb76b684f-2f6zl 2/2 Running 0 124m liuhuali@Lius-MacBook-Pro huali-test % oc logs control-plane-machine-set- operator -c674d9976-s2dvq ... I0424 07:02:35.617844 1 panic.go:884] "msg" = "Finished reconciling control plane machine set" "controller" = "controlplanemachineset" "name" = "cluster" "namespace" = "openshift-machine-api" "reconcileID" = "394fd0b0-8f9f-4680-a2a0-00e407b3c5e6" I0424 07:02:35.617968 1 controller.go:115] "msg" = "Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference" "controller" = "controlplanemachineset" "reconcileID" = "394fd0b0-8f9f-4680-a2a0-00e407b3c5e6" panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1a5911c] goroutine 254 [running]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() /go/src/github.com/openshift/cluster-control-plane-machine-set- operator /vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116 +0x1fa panic({0x1c8ac60, 0x31c6ea0}) /usr/lib/golang/src/runtime/panic.go:884 +0x213 github.com/openshift/cluster-control-plane-machine-set- operator /pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig.VSphereProviderConfig.ExtractFailureDomain(...) /go/src/github.com/openshift/cluster-control-plane-machine-set- operator /pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig/vsphere.go:120 github.com/openshift/cluster-control-plane-machine-set- operator /pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig.providerConfig.ExtractFailureDomain({{0x1f2a71a, 0x7}, {{{{...}, {...}}, {{...}, {...}, {...}, {...}, {...}, {...}, ...}, ...}}, ...}) /go/src/github.com/openshift/cluster-control-plane-machine-set- operator /pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig/providerconfig.go:212 +0x23c github.com/openshift/cluster-control-plane-machine-set- operator /pkg/machineproviders/providers/openshift/machine/v1beta1.(*openshiftMachineProvider).updateMachineCache(0xc000b8c500, {0x22a0920, 0xc00075e3f0}, {{0x22a3820?, 0xc000625310?}, 0x0?}) /go/src/github.com/openshift/cluster-control-plane-machine-set- operator /pkg/machineproviders/providers/openshift/machine/v1beta1/provider.go:224 +0x276 github.com/openshift/cluster-control-plane-machine-set- operator /pkg/machineproviders/providers/openshift/machine/v1beta1.NewMachineProvider({0x22a0920, 0xc00075e3f0}, {{0x22a3820?, 0xc000625310?}, 0x36?}, {0x22a9400?, 0xc0001e14d0?}, {0x229fdf0?, 0xc000332640}, 0xc000b5d1e0) /go/src/github.com/openshift/cluster-control-plane-machine-set- operator /pkg/machineproviders/providers/openshift/machine/v1beta1/provider.go:152 +0x752 github.com/openshift/cluster-control-plane-machine-set- operator /pkg/machineproviders/providers.NewMachineProvider({0x22a0920, 0xc00075e3f0}, {{0x22a3820?, 0xc000625310?}, 0xc000e8b990?}, {0x22a9400, 0xc0001e14d0}, {0x229fdf0, 0xc000332640}, 0xc000b5d1e0) /go/src/github.com/openshift/cluster-control-plane-machine-set- operator /pkg/machineproviders/providers/machineproviders.go:46 +0x1c5 github.com/openshift/cluster-control-plane-machine-set- operator /pkg/controllers/controlplanemachineset.(*ControlPlaneMachineSetReconciler).reconcile(0xc00034c580, {0x22a0920, 0xc00075e3f0}, {{0x22a3820?, 0xc000625310?}, 0xc00075e3f0?}, 0xc000b5d1e0) /go/src/github.com/openshift/cluster-control-plane-machine-set- operator /pkg/controllers/controlplanemachineset/controller.go:245 +0x146 github.com/openshift/cluster-control-plane-machine-set- operator /pkg/controllers/controlplanemachineset.(*ControlPlaneMachineSetReconciler).Reconcile(0xc00034c580, {0x22a0920, 0xc00075e3f0}, {{{0x1f407a8?, 0x0?}, {0x1f2a855?, 0xc0008dbd48?}}}) /go/src/github.com/openshift/cluster-control-plane-machine-set- operator /pkg/controllers/controlplanemachineset/controller.go:194 +0x3f0 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x22a3820?, {0x22a0920?, 0xc00075e3f0?}, {{{0x1f407a8?, 0xb?}, {0x1f2a855?, 0x0?}}}) /go/src/github.com/openshift/cluster-control-plane-machine-set- operator /vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119 +0xc8 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0003263c0, {0x22a0878, 0xc0006245a0}, {0x1d36760?, 0xc0004b8200?}) /go/src/github.com/openshift/cluster-control-plane-machine-set- operator /vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316 +0x3ca sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0003263c0, {0x22a0878, 0xc0006245a0}) /go/src/github.com/openshift/cluster-control-plane-machine-set- operator /vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 +0x1d9 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() /go/src/github.com/openshift/cluster-control-plane-machine-set- operator /vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 +0x85 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 /go/src/github.com/openshift/cluster-control-plane-machine-set- operator /vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:223 +0x587 liuhuali@Lius-MacBook-Pro huali-test %

            Huali Liu added a comment -

            I try to reproduce this issue on the upgrade from 4.14 to 4.15, but upgrade succeeded. I thought maybe I should upgrade from an older version, so I upgrade form 4.12 -> 4.13 -> 4.14 -> 4.15, but the upgrade still succeeded. rhn-support-nkashyap Can you please take a look what I missed? Thanks!
            Steps:
            1. Install a vSphere 4.12.52 cluster, we use automated template: ipi-on-vsphere/versioned-installer-fips

            liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.52   True        False         17m     Cluster version is 4.12.52
liuhuali@Lius-MacBook-Pro huali-test % oc get infrastructure cluster -oyaml
apiVersion: config.openshift.io/v1
kind: Infrastructure
metadata:
  creationTimestamp: "2024-04-23T04:14:04Z"
  generation: 1
  name: cluster
  resourceVersion: "522"
  uid: 90105825-386a-4586-a81f-3c41a6ae54ed
spec:
  cloudConfig:
    key: config
    name: cloud-provider-config
  platformSpec:
    type: VSphere
    vsphere: {}
status:
  apiServerInternalURI: https://api-int.huliu-vs423d.qe.devcluster.openshift.com:6443
  apiServerURL: https://api.huliu-vs423d.qe.devcluster.openshift.com:6443
  controlPlaneTopology: HighlyAvailable
  etcdDiscoveryDomain: ""
  infrastructureName: huliu-vs423d-vrs2s
  infrastructureTopology: HighlyAvailable
  platform: VSphere
  platformStatus:
    type: VSphere
    vsphere:
      apiServerInternalIP: 192.168.221.68
      apiServerInternalIPs:
      - 192.168.221.68
      ingressIP: 192.168.221.69
      ingressIPs:
      - 192.168.221.69
liuhuali@Lius-MacBook-Pro huali-test % oc project openshift-machine-api
Now using project "openshift-machine-api" on server "https://api.huliu-vs423d.qe.devcluster.openshift.com:6443".
liuhuali@Lius-MacBook-Pro huali-test % oc get machine
NAME                              PHASE     TYPE   REGION   ZONE   AGE
huliu-vs423d-vrs2s-master-0       Running                          48m
huliu-vs423d-vrs2s-master-1       Running                          48m
huliu-vs423d-vrs2s-master-2       Running                          48m
huliu-vs423d-vrs2s-worker-q6f98   Running                          41m
huliu-vs423d-vrs2s-worker-rjkzt   Running                          41m
liuhuali@Lius-MacBook-Pro huali-test % oc get controlplanemachineset
No resources found in openshift-machine-api namespace.
liuhuali@Lius-MacBook-Pro huali-test % oc get pod
NAME                                                  READY   STATUS    RESTARTS       AGE
cluster-autoscaler-operator-78bd9c689f-bvldc          2/2     Running   2 (37m ago)    47m
cluster-baremetal-operator-5d884f97ff-rr4wv           2/2     Running   0              47m
control-plane-machine-set-operator-68f64ff4d4-fnkz5   1/1     Running   2 (37m ago)    47m
machine-api-controllers-657b4c458-xfw9g               7/7     Running   15 (36m ago)   42m
machine-api-operator-85944db7d-4ntwz                  2/2     Running   1 (38m ago)    48m

            2. Create the controlplanemachineset

            liuhuali@Lius-MacBook-Pro huali-test % oc create -f controlplanemachineset_vsphere.yaml
controlplanemachineset.machine.openshift.io/cluster created
liuhuali@Lius-MacBook-Pro huali-test % oc get controlplanemachineset
NAME      DESIRED   CURRENT   READY   UPDATED   UNAVAILABLE   STATE    AGE
cluster   3         3         3       3                       Active   8s
liuhuali@Lius-MacBook-Pro huali-test % oc get controlplanemachineset cluster -oyaml
apiVersion: machine.openshift.io/v1
kind: ControlPlaneMachineSet
metadata:
  creationTimestamp: "2024-04-23T05:08:39Z"
  finalizers:
  - controlplanemachineset.machine.openshift.io
  generation: 2
  name: cluster
  namespace: openshift-machine-api
  resourceVersion: "40134"
  uid: 8231e7a4-a316-4343-8484-6e32121873be
spec:
  replicas: 3
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: huliu-vs423d-vrs2s
      machine.openshift.io/cluster-api-machine-role: master
      machine.openshift.io/cluster-api-machine-type: master
  state: Active
  strategy:
    type: RollingUpdate
  template:
    machineType: machines_v1beta1_machine_openshift_io
    machines_v1beta1_machine_openshift_io:
      failureDomains:
        platform: ""
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: huliu-vs423d-vrs2s
          machine.openshift.io/cluster-api-machine-role: master
          machine.openshift.io/cluster-api-machine-type: master
      spec:
        lifecycleHooks: {}
        metadata: {}
        providerSpec:
          value:
            apiVersion: machine.openshift.io/v1beta1
            credentialsSecret:
              name: vsphere-cloud-credentials
            diskGiB: 120
            kind: VSphereMachineProviderSpec
            memoryMiB: 16384
            metadata:
              creationTimestamp: null
            network:
              devices:
              - networkName: devqe-segment-221
            numCPUs: 4
            numCoresPerSocket: 4
            snapshot: ""
            template: huliu-vs423d-vrs2s-rhcos
            userDataSecret:
              name: master-user-data
            workspace:
              datacenter: DEVQEdatacenter
              datastore: vsanDatastore
              folder: /DEVQEdatacenter/vm/huliu-vs423d-vrs2s
              resourcePool: /DEVQEdatacenter/host/DEVQEcluster/Resources
              server: vcenter.devqe.ibmc.devcluster.openshift.com
status:
  conditions:
  - lastTransitionTime: "2024-04-23T05:08:39Z"
    message: ""
    observedGeneration: 2
    reason: AsExpected
    status: "False"
    type: Error
  - lastTransitionTime: "2024-04-23T05:08:40Z"
    message: ""
    observedGeneration: 2
    reason: AllReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2024-04-23T05:08:40Z"
    message: ""
    observedGeneration: 2
    reason: AsExpected
    status: "False"
    type: Degraded
  - lastTransitionTime: "2024-04-23T05:08:40Z"
    message: ""
    observedGeneration: 2
    reason: AllReplicasUpdated
    status: "False"
    type: Progressing
  observedGeneration: 2
  readyReplicas: 3
  replicas: 3
  updatedReplicas: 3

            3. Upgrade to 4.13.38 succeeded

            liuhuali@Lius-MacBook-Pro huali-test % oc adm upgrade --to-image='quay.io/openshift-release-dev/ocp-release:4.13.38-x86_64' --force --allow-explicit-upgrade
warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Requesting update to release image quay.io/openshift-release-dev/ocp-release:4.13.38-x86_64
liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.13.38   True        False         6m45s   Cluster version is 4.13.38
liuhuali@Lius-MacBook-Pro huali-test % oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.13.38   True        False         False      103m    
baremetal                                  4.13.38   True        False         False      126m    
cloud-controller-manager                   4.13.38   True        False         False      128m    
cloud-credential                           4.13.38   True        False         False      132m    
cluster-autoscaler                         4.13.38   True        False         False      126m    
config-operator                            4.13.38   True        False         False      127m    
console                                    4.13.38   True        False         False      107m    
control-plane-machine-set                  4.13.38   True        False         False      7m52s   
csi-snapshot-controller                    4.13.38   True        False         False      126m    
dns                                        4.13.38   True        False         False      126m    
etcd                                       4.13.38   True        False         False      118m    
image-registry                             4.13.38   True        False         False      25m     
ingress                                    4.13.38   True        False         False      25m     
insights                                   4.13.38   True        False         False      121m    
kube-apiserver                             4.13.38   True        False         False      111m    
kube-controller-manager                    4.13.38   True        False         False      118m    
kube-scheduler                             4.13.38   True        False         False      118m    
kube-storage-version-migrator              4.13.38   True        False         False      25m     
machine-api                                4.13.38   True        False         False      111m    
machine-approver                           4.13.38   True        False         False      126m    
machine-config                             4.13.38   True        False         False      126m    
marketplace                                4.13.38   True        False         False      126m    
monitoring                                 4.13.38   True        False         False      108m    
network                                    4.13.38   True        False         False      127m    
node-tuning                                4.13.38   True        False         False      49m     
openshift-apiserver                        4.13.38   True        False         False      112m    
openshift-controller-manager               4.13.38   True        False         False      120m    
openshift-samples                          4.13.38   True        False         False      49m     
operator-lifecycle-manager                 4.13.38   True        False         False      126m    
operator-lifecycle-manager-catalog         4.13.38   True        False         False      126m    
operator-lifecycle-manager-packageserver   4.13.38   True        False         False      115m    
service-ca                                 4.13.38   True        False         False      127m    
storage                                    4.13.38   True        False         False      12m

            4. Upgrade to 4.14.18 succeeded

            liuhuali@Lius-MacBook-Pro huali-test % oc adm upgrade --to-image='quay.io/openshift-release-dev/ocp-release:4.14.18-x86_64' --force --allow-explicit-upgrade
warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Requesting update to release image quay.io/openshift-release-dev/ocp-release:4.14.18-x86_64
liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.18   True        False         3m48s   Cluster version is 4.14.18
liuhuali@Lius-MacBook-Pro huali-test % oc get co  
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.14.18   True        False         False      174m    
baremetal                                  4.14.18   True        False         False      3h17m   
cloud-controller-manager                   4.14.18   True        False         False      3h20m   
cloud-credential                           4.14.18   True        False         False      3h23m   
cluster-autoscaler                         4.14.18   True        False         False      3h17m   
config-operator                            4.14.18   True        False         False      3h18m   
console                                    4.14.18   True        False         False      178m    
control-plane-machine-set                  4.14.18   True        False         False      6m39s   
csi-snapshot-controller                    4.14.18   True        False         False      3h18m   
dns                                        4.14.18   True        False         False      3h17m   
etcd                                       4.14.18   True        False         False      3h9m    
image-registry                             4.14.18   True        False         False      20m     
ingress                                    4.14.18   True        False         False      96m     
insights                                   4.14.18   True        False         False      3h12m   
kube-apiserver                             4.14.18   True        False         False      3h2m    
kube-controller-manager                    4.14.18   True        False         False      3h9m    
kube-scheduler                             4.14.18   True        False         False      3h9m    
kube-storage-version-migrator              4.14.18   True        False         False      21m     
machine-api                                4.14.18   True        False         False      3h2m    
machine-approver                           4.14.18   True        False         False      3h17m   
machine-config                             4.14.18   True        False         False      3h17m   
marketplace                                4.14.18   True        False         False      3h17m   
monitoring                                 4.14.18   True        False         False      179m    
network                                    4.14.18   True        False         False      3h19m   
node-tuning                                4.14.18   True        False         False      45m     
openshift-apiserver                        4.14.18   True        False         False      3h3m    
openshift-controller-manager               4.14.18   True        False         False      3h11m   
openshift-samples                          4.14.18   True        False         False      46m     
operator-lifecycle-manager                 4.14.18   True        False         False      3h18m   
operator-lifecycle-manager-catalog         4.14.18   True        False         False      3h18m   
operator-lifecycle-manager-packageserver   4.14.18   True        False         False      3h6m    
service-ca                                 4.14.18   True        False         False      3h18m   
storage                                    4.14.18   True        False         False      26m

            5. Upgrade to 4.15.6 succeeded

            liuhuali@Lius-MacBook-Pro huali-test % oc adm upgrade --to-image='quay.io/openshift-release-dev/ocp-release:4.15.6-x86_64' --force --allow-explicit-upgrade
warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead
warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Requesting update to release image quay.io/openshift-release-dev/ocp-release:4.15.6-x86_64
liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.15.6    True        False         12m     Cluster version is 4.15.6
liuhuali@Lius-MacBook-Pro huali-test % oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.15.6    True        False         False      4h18m   
baremetal                                  4.15.6    True        False         False      4h41m   
cloud-controller-manager                   4.15.6    True        False         False      4h44m   
cloud-credential                           4.15.6    True        False         False      4h47m   
cluster-autoscaler                         4.15.6    True        False         False      4h41m   
config-operator                            4.15.6    True        False         False      4h43m   
console                                    4.15.6    True        False         False      4h23m   
control-plane-machine-set                  4.15.6    True        False         False      14m     
csi-snapshot-controller                    4.15.6    True        False         False      4h42m   
dns                                        4.15.6    True        False         False      4h41m   
etcd                                       4.15.6    True        False         False      4h33m   
image-registry                             4.15.6    True        False         False      30m     
ingress                                    4.15.6    True        False         False      30m     
insights                                   4.15.6    True        False         False      4h36m   
kube-apiserver                             4.15.6    True        False         False      4h26m   
kube-controller-manager                    4.15.6    True        False         False      4h33m   
kube-scheduler                             4.15.6    True        False         False      4h34m   
kube-storage-version-migrator              4.15.6    True        False         False      30m     
machine-api                                4.15.6    True        False         False      4h26m   
machine-approver                           4.15.6    True        False         False      4h41m   
machine-config                             4.15.6    True        False         False      4h41m   
marketplace                                4.15.6    True        False         False      4h41m   
monitoring                                 4.15.6    True        False         False      4h23m   
network                                    4.15.6    True        False         False      4h43m   
node-tuning                                4.15.6    True        False         False      49m     
openshift-apiserver                        4.15.6    True        False         False      4h28m   
openshift-controller-manager               4.15.6    True        False         False      4h35m   
openshift-samples                          4.15.6    True        False         False      50m     
operator-lifecycle-manager                 4.15.6    True        False         False      4h42m   
operator-lifecycle-manager-catalog         4.15.6    True        False         False      4h42m   
operator-lifecycle-manager-packageserver   4.15.6    True        False         False      50m     
service-ca                                 4.15.6    True        False         False      4h42m   
storage                                    4.15.6    True        False         False      28m  
liuhuali@Lius-MacBook-Pro huali-test % oc get pod
NAME                                                 READY   STATUS    RESTARTS   AGE
cluster-autoscaler-operator-766578f547-rc72t         2/2     Running   0          27m
cluster-baremetal-operator-5b49c7bfc7-wfjqr          2/2     Running   0          27m
control-plane-machine-set-operator-c674d9976-kkxld   1/1     Running   0          27m
machine-api-controllers-64fdc59448-7kpgk             7/7     Running   0          27m
machine-api-operator-5bb76b684f-xq5v9                2/2     Running   0          27m
liuhuali@Lius-MacBook-Pro huali-test % oc get controlplanemachineset 
NAME      DESIRED   CURRENT   READY   UPDATED   UNAVAILABLE   STATE    AGE
cluster   3         3         3       3                       Active   3h54m
liuhuali@Lius-MacBook-Pro huali-test % oc get controlplanemachineset cluster -oyaml
apiVersion: machine.openshift.io/v1
kind: ControlPlaneMachineSet
metadata:
  creationTimestamp: "2024-04-23T05:08:39Z"
  finalizers:
  - controlplanemachineset.machine.openshift.io
  generation: 2
  name: cluster
  namespace: openshift-machine-api
  resourceVersion: "184486"
  uid: 8231e7a4-a316-4343-8484-6e32121873be
spec:
  replicas: 3
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: huliu-vs423d-vrs2s
      machine.openshift.io/cluster-api-machine-role: master
      machine.openshift.io/cluster-api-machine-type: master
  state: Active
  strategy:
    type: RollingUpdate
  template:
    machineType: machines_v1beta1_machine_openshift_io
    machines_v1beta1_machine_openshift_io:
      failureDomains:
        platform: ""
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: huliu-vs423d-vrs2s
          machine.openshift.io/cluster-api-machine-role: master
          machine.openshift.io/cluster-api-machine-type: master
      spec:
        lifecycleHooks: {}
        metadata: {}
        providerSpec:
          value:
            apiVersion: machine.openshift.io/v1beta1
            credentialsSecret:
              name: vsphere-cloud-credentials
            diskGiB: 120
            kind: VSphereMachineProviderSpec
            memoryMiB: 16384
            metadata:
              creationTimestamp: null
            network:
              devices:
              - networkName: devqe-segment-221
            numCPUs: 4
            numCoresPerSocket: 4
            snapshot: ""
            template: huliu-vs423d-vrs2s-rhcos
            userDataSecret:
              name: master-user-data
            workspace:
              datacenter: DEVQEdatacenter
              datastore: vsanDatastore
              folder: /DEVQEdatacenter/vm/huliu-vs423d-vrs2s
              resourcePool: /DEVQEdatacenter/host/DEVQEcluster/Resources
              server: vcenter.devqe.ibmc.devcluster.openshift.com
status:
  conditions:
  - lastTransitionTime: "2024-04-23T05:08:39Z"
    message: ""
    observedGeneration: 2
    reason: AsExpected
    status: "False"
    type: Error
  - lastTransitionTime: "2024-04-23T08:48:42Z"
    message: ""
    observedGeneration: 2
    reason: AllReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2024-04-23T05:08:40Z"
    message: ""
    observedGeneration: 2
    reason: AsExpected
    status: "False"
    type: Degraded
  - lastTransitionTime: "2024-04-23T08:48:42Z"
    message: ""
    observedGeneration: 2
    reason: AllReplicasUpdated
    status: "False"
    type: Progressing
  observedGeneration: 2
  readyReplicas: 3
  replicas: 3
  updatedReplicas: 3
liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion version -o='jsonpath={.status.history[*].version}'
4.15.6 4.14.18 4.13.38 4.12.52%                                                                                                 
liuhuali@Lius-MacBook-Pro huali-test %

            must gather: https://drive.google.com/file/d/1CxJf257e3i5DNbRGM0Rkp2p9oGVAIpiz/view?usp=sharing

            Huali Liu added a comment - I try to reproduce this issue on the upgrade from 4.14 to 4.15, but upgrade succeeded. I thought maybe I should upgrade from an older version, so I upgrade form 4.12 -> 4.13 -> 4.14 -> 4.15, but the upgrade still succeeded. rhn-support-nkashyap Can you please take a look what I missed? Thanks! Steps: 1. Install a vSphere 4.12.52 cluster, we use automated template: ipi-on-vsphere/versioned-installer-fips liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.12.52 True False 17m Cluster version is 4.12.52 liuhuali@Lius-MacBook-Pro huali-test % oc get infrastructure cluster -oyaml apiVersion: config.openshift.io/v1 kind: Infrastructure metadata: creationTimestamp: "2024-04-23T04:14:04Z" generation: 1 name: cluster resourceVersion: "522" uid: 90105825-386a-4586-a81f-3c41a6ae54ed spec: cloudConfig: key: config name: cloud-provider-config platformSpec: type: VSphere vsphere: {} status: apiServerInternalURI: https: //api- int .huliu-vs423d.qe.devcluster.openshift.com:6443 apiServerURL: https: //api.huliu-vs423d.qe.devcluster.openshift.com:6443 controlPlaneTopology: HighlyAvailable etcdDiscoveryDomain: "" infrastructureName: huliu-vs423d-vrs2s infrastructureTopology: HighlyAvailable platform: VSphere platformStatus: type: VSphere vsphere: apiServerInternalIP: 192.168.221.68 apiServerInternalIPs: - 192.168.221.68 ingressIP: 192.168.221.69 ingressIPs: - 192.168.221.69 liuhuali@Lius-MacBook-Pro huali-test % oc project openshift-machine-api Now using project "openshift-machine-api" on server "https: //api.huliu-vs423d.qe.devcluster.openshift.com:6443" . liuhuali@Lius-MacBook-Pro huali-test % oc get machine NAME PHASE TYPE REGION ZONE AGE huliu-vs423d-vrs2s-master-0 Running 48m huliu-vs423d-vrs2s-master-1 Running 48m huliu-vs423d-vrs2s-master-2 Running 48m huliu-vs423d-vrs2s-worker-q6f98 Running 41m huliu-vs423d-vrs2s-worker-rjkzt Running 41m liuhuali@Lius-MacBook-Pro huali-test % oc get controlplanemachineset No resources found in openshift-machine-api namespace. liuhuali@Lius-MacBook-Pro huali-test % oc get pod NAME READY STATUS RESTARTS AGE cluster-autoscaler- operator -78bd9c689f-bvldc 2/2 Running 2 (37m ago) 47m cluster-baremetal- operator -5d884f97ff-rr4wv 2/2 Running 0 47m control-plane-machine-set- operator -68f64ff4d4-fnkz5 1/1 Running 2 (37m ago) 47m machine-api-controllers-657b4c458-xfw9g 7/7 Running 15 (36m ago) 42m machine-api- operator -85944db7d-4ntwz 2/2 Running 1 (38m ago) 48m 2. Create the controlplanemachineset liuhuali@Lius-MacBook-Pro huali-test % oc create -f controlplanemachineset_vsphere.yaml controlplanemachineset.machine.openshift.io/cluster created liuhuali@Lius-MacBook-Pro huali-test % oc get controlplanemachineset NAME DESIRED CURRENT READY UPDATED UNAVAILABLE STATE AGE cluster 3 3 3 3 Active 8s liuhuali@Lius-MacBook-Pro huali-test % oc get controlplanemachineset cluster -oyaml apiVersion: machine.openshift.io/v1 kind: ControlPlaneMachineSet metadata: creationTimestamp: "2024-04-23T05:08:39Z" finalizers: - controlplanemachineset.machine.openshift.io generation: 2 name: cluster namespace: openshift-machine-api resourceVersion: "40134" uid: 8231e7a4-a316-4343-8484-6e32121873be spec: replicas: 3 selector: matchLabels: machine.openshift.io/cluster-api-cluster: huliu-vs423d-vrs2s machine.openshift.io/cluster-api-machine-role: master machine.openshift.io/cluster-api-machine-type: master state: Active strategy: type: RollingUpdate template: machineType: machines_v1beta1_machine_openshift_io machines_v1beta1_machine_openshift_io: failureDomains: platform: "" metadata: labels: machine.openshift.io/cluster-api-cluster: huliu-vs423d-vrs2s machine.openshift.io/cluster-api-machine-role: master machine.openshift.io/cluster-api-machine-type: master spec: lifecycleHooks: {} metadata: {} providerSpec: value: apiVersion: machine.openshift.io/v1beta1 credentialsSecret: name: vsphere-cloud-credentials diskGiB: 120 kind: VSphereMachineProviderSpec memoryMiB: 16384 metadata: creationTimestamp: null network: devices: - networkName: devqe-segment-221 numCPUs: 4 numCoresPerSocket: 4 snapshot: "" template: huliu-vs423d-vrs2s-rhcos userDataSecret: name: master-user-data workspace: datacenter: DEVQEdatacenter datastore: vsanDatastore folder: /DEVQEdatacenter/vm/huliu-vs423d-vrs2s resourcePool: /DEVQEdatacenter/host/DEVQEcluster/Resources server: vcenter.devqe.ibmc.devcluster.openshift.com status: conditions: - lastTransitionTime: "2024-04-23T05:08:39Z" message: "" observedGeneration: 2 reason: AsExpected status: "False" type: Error - lastTransitionTime: "2024-04-23T05:08:40Z" message: "" observedGeneration: 2 reason: AllReplicasAvailable status: "True" type: Available - lastTransitionTime: "2024-04-23T05:08:40Z" message: "" observedGeneration: 2 reason: AsExpected status: "False" type: Degraded - lastTransitionTime: "2024-04-23T05:08:40Z" message: "" observedGeneration: 2 reason: AllReplicasUpdated status: "False" type: Progressing observedGeneration: 2 readyReplicas: 3 replicas: 3 updatedReplicas: 3 3. Upgrade to 4.13.38 succeeded liuhuali@Lius-MacBook-Pro huali-test % oc adm upgrade --to-image= 'quay.io/openshift-release-dev/ocp-release:4.13.38-x86_64' --force --allow-explicit-upgrade warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures. Requesting update to release image quay.io/openshift-release-dev/ocp-release:4.13.38-x86_64 liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.13.38 True False 6m45s Cluster version is 4.13.38 liuhuali@Lius-MacBook-Pro huali-test % oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.13.38 True False False 103m baremetal 4.13.38 True False False 126m cloud-controller-manager 4.13.38 True False False 128m cloud-credential 4.13.38 True False False 132m cluster-autoscaler 4.13.38 True False False 126m config- operator 4.13.38 True False False 127m console 4.13.38 True False False 107m control-plane-machine-set 4.13.38 True False False 7m52s csi-snapshot-controller 4.13.38 True False False 126m dns 4.13.38 True False False 126m etcd 4.13.38 True False False 118m image-registry 4.13.38 True False False 25m ingress 4.13.38 True False False 25m insights 4.13.38 True False False 121m kube-apiserver 4.13.38 True False False 111m kube-controller-manager 4.13.38 True False False 118m kube-scheduler 4.13.38 True False False 118m kube-storage-version-migrator 4.13.38 True False False 25m machine-api 4.13.38 True False False 111m machine-approver 4.13.38 True False False 126m machine-config 4.13.38 True False False 126m marketplace 4.13.38 True False False 126m monitoring 4.13.38 True False False 108m network 4.13.38 True False False 127m node-tuning 4.13.38 True False False 49m openshift-apiserver 4.13.38 True False False 112m openshift-controller-manager 4.13.38 True False False 120m openshift-samples 4.13.38 True False False 49m operator -lifecycle-manager 4.13.38 True False False 126m operator -lifecycle-manager-catalog 4.13.38 True False False 126m operator -lifecycle-manager-packageserver 4.13.38 True False False 115m service-ca 4.13.38 True False False 127m storage 4.13.38 True False False 12m 4. Upgrade to 4.14.18 succeeded liuhuali@Lius-MacBook-Pro huali-test % oc adm upgrade --to-image= 'quay.io/openshift-release-dev/ocp-release:4.14.18-x86_64' --force --allow-explicit-upgrade warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures. Requesting update to release image quay.io/openshift-release-dev/ocp-release:4.14.18-x86_64 liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.14.18 True False 3m48s Cluster version is 4.14.18 liuhuali@Lius-MacBook-Pro huali-test % oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.14.18 True False False 174m baremetal 4.14.18 True False False 3h17m cloud-controller-manager 4.14.18 True False False 3h20m cloud-credential 4.14.18 True False False 3h23m cluster-autoscaler 4.14.18 True False False 3h17m config- operator 4.14.18 True False False 3h18m console 4.14.18 True False False 178m control-plane-machine-set 4.14.18 True False False 6m39s csi-snapshot-controller 4.14.18 True False False 3h18m dns 4.14.18 True False False 3h17m etcd 4.14.18 True False False 3h9m image-registry 4.14.18 True False False 20m ingress 4.14.18 True False False 96m insights 4.14.18 True False False 3h12m kube-apiserver 4.14.18 True False False 3h2m kube-controller-manager 4.14.18 True False False 3h9m kube-scheduler 4.14.18 True False False 3h9m kube-storage-version-migrator 4.14.18 True False False 21m machine-api 4.14.18 True False False 3h2m machine-approver 4.14.18 True False False 3h17m machine-config 4.14.18 True False False 3h17m marketplace 4.14.18 True False False 3h17m monitoring 4.14.18 True False False 179m network 4.14.18 True False False 3h19m node-tuning 4.14.18 True False False 45m openshift-apiserver 4.14.18 True False False 3h3m openshift-controller-manager 4.14.18 True False False 3h11m openshift-samples 4.14.18 True False False 46m operator -lifecycle-manager 4.14.18 True False False 3h18m operator -lifecycle-manager-catalog 4.14.18 True False False 3h18m operator -lifecycle-manager-packageserver 4.14.18 True False False 3h6m service-ca 4.14.18 True False False 3h18m storage 4.14.18 True False False 26m 5. Upgrade to 4.15.6 succeeded liuhuali@Lius-MacBook-Pro huali-test % oc adm upgrade --to-image= 'quay.io/openshift-release-dev/ocp-release:4.15.6-x86_64' --force --allow-explicit-upgrade warning: Using by-tag pull specs is dangerous, and while we still allow it in combination with --force for backward compatibility, it would be much safer to pass a by-digest pull spec instead warning: The requested upgrade image is not one of the available updates.You have used --allow-explicit-upgrade for the update to proceed anyway warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures. Requesting update to release image quay.io/openshift-release-dev/ocp-release:4.15.6-x86_64 liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.15.6 True False 12m Cluster version is 4.15.6 liuhuali@Lius-MacBook-Pro huali-test % oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.15.6 True False False 4h18m baremetal 4.15.6 True False False 4h41m cloud-controller-manager 4.15.6 True False False 4h44m cloud-credential 4.15.6 True False False 4h47m cluster-autoscaler 4.15.6 True False False 4h41m config- operator 4.15.6 True False False 4h43m console 4.15.6 True False False 4h23m control-plane-machine-set 4.15.6 True False False 14m csi-snapshot-controller 4.15.6 True False False 4h42m dns 4.15.6 True False False 4h41m etcd 4.15.6 True False False 4h33m image-registry 4.15.6 True False False 30m ingress 4.15.6 True False False 30m insights 4.15.6 True False False 4h36m kube-apiserver 4.15.6 True False False 4h26m kube-controller-manager 4.15.6 True False False 4h33m kube-scheduler 4.15.6 True False False 4h34m kube-storage-version-migrator 4.15.6 True False False 30m machine-api 4.15.6 True False False 4h26m machine-approver 4.15.6 True False False 4h41m machine-config 4.15.6 True False False 4h41m marketplace 4.15.6 True False False 4h41m monitoring 4.15.6 True False False 4h23m network 4.15.6 True False False 4h43m node-tuning 4.15.6 True False False 49m openshift-apiserver 4.15.6 True False False 4h28m openshift-controller-manager 4.15.6 True False False 4h35m openshift-samples 4.15.6 True False False 50m operator -lifecycle-manager 4.15.6 True False False 4h42m operator -lifecycle-manager-catalog 4.15.6 True False False 4h42m operator -lifecycle-manager-packageserver 4.15.6 True False False 50m service-ca 4.15.6 True False False 4h42m storage 4.15.6 True False False 28m liuhuali@Lius-MacBook-Pro huali-test % oc get pod NAME READY STATUS RESTARTS AGE cluster-autoscaler- operator -766578f547-rc72t 2/2 Running 0 27m cluster-baremetal- operator -5b49c7bfc7-wfjqr 2/2 Running 0 27m control-plane-machine-set- operator -c674d9976-kkxld 1/1 Running 0 27m machine-api-controllers-64fdc59448-7kpgk 7/7 Running 0 27m machine-api- operator -5bb76b684f-xq5v9 2/2 Running 0 27m liuhuali@Lius-MacBook-Pro huali-test % oc get controlplanemachineset NAME DESIRED CURRENT READY UPDATED UNAVAILABLE STATE AGE cluster 3 3 3 3 Active 3h54m liuhuali@Lius-MacBook-Pro huali-test % oc get controlplanemachineset cluster -oyaml apiVersion: machine.openshift.io/v1 kind: ControlPlaneMachineSet metadata: creationTimestamp: "2024-04-23T05:08:39Z" finalizers: - controlplanemachineset.machine.openshift.io generation: 2 name: cluster namespace: openshift-machine-api resourceVersion: "184486" uid: 8231e7a4-a316-4343-8484-6e32121873be spec: replicas: 3 selector: matchLabels: machine.openshift.io/cluster-api-cluster: huliu-vs423d-vrs2s machine.openshift.io/cluster-api-machine-role: master machine.openshift.io/cluster-api-machine-type: master state: Active strategy: type: RollingUpdate template: machineType: machines_v1beta1_machine_openshift_io machines_v1beta1_machine_openshift_io: failureDomains: platform: "" metadata: labels: machine.openshift.io/cluster-api-cluster: huliu-vs423d-vrs2s machine.openshift.io/cluster-api-machine-role: master machine.openshift.io/cluster-api-machine-type: master spec: lifecycleHooks: {} metadata: {} providerSpec: value: apiVersion: machine.openshift.io/v1beta1 credentialsSecret: name: vsphere-cloud-credentials diskGiB: 120 kind: VSphereMachineProviderSpec memoryMiB: 16384 metadata: creationTimestamp: null network: devices: - networkName: devqe-segment-221 numCPUs: 4 numCoresPerSocket: 4 snapshot: "" template: huliu-vs423d-vrs2s-rhcos userDataSecret: name: master-user-data workspace: datacenter: DEVQEdatacenter datastore: vsanDatastore folder: /DEVQEdatacenter/vm/huliu-vs423d-vrs2s resourcePool: /DEVQEdatacenter/host/DEVQEcluster/Resources server: vcenter.devqe.ibmc.devcluster.openshift.com status: conditions: - lastTransitionTime: "2024-04-23T05:08:39Z" message: "" observedGeneration: 2 reason: AsExpected status: "False" type: Error - lastTransitionTime: "2024-04-23T08:48:42Z" message: "" observedGeneration: 2 reason: AllReplicasAvailable status: "True" type: Available - lastTransitionTime: "2024-04-23T05:08:40Z" message: "" observedGeneration: 2 reason: AsExpected status: "False" type: Degraded - lastTransitionTime: "2024-04-23T08:48:42Z" message: "" observedGeneration: 2 reason: AllReplicasUpdated status: "False" type: Progressing observedGeneration: 2 readyReplicas: 3 replicas: 3 updatedReplicas: 3 liuhuali@Lius-MacBook-Pro huali-test % oc get clusterversion version -o= 'jsonpath={.status.history[*].version}' 4.15.6 4.14.18 4.13.38 4.12.52% liuhuali@Lius-MacBook-Pro huali-test % must gather: https://drive.google.com/file/d/1CxJf257e3i5DNbRGM0Rkp2p9oGVAIpiz/view?usp=sharing

            Huali Liu added a comment -

            Seems this issue only occurs on the upgrade from 4.14 to 4.15, so I cannot verify it on the upgrade from 4.15 to 4.16, because there are values for failureDomain in the controlplanemachineset CR on 4.15 and I cannot delete them, so I'd like to move this bug to verified directly and verify the issue when it backported to 4.15. Thanks!

            Huali Liu added a comment - Seems this issue only occurs on the upgrade from 4.14 to 4.15, so I cannot verify it on the upgrade from 4.15 to 4.16, because there are values for failureDomain in the controlplanemachineset CR on 4.15 and I cannot delete them, so I'd like to move this bug to verified directly and verify the issue when it backported to 4.15. Thanks!

            OpenShift Jira Bot added a comment -

            Hi rhn-support-ngirard,

            Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.

            OpenShift Jira Bot added a comment - Hi rhn-support-ngirard , Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.

            Neil Girard added a comment -

            I created PR to fix the other issue dealing with the full path related error: https://issues.redhat.com/browse/OCPBUGS-32295. 

            Neil Girard added a comment - I created PR to fix the other issue dealing with the full path related error: https://issues.redhat.com/browse/OCPBUGS-32295 . 

            Nirupma Nirupma added a comment -

            KCS

            Nirupma Nirupma added a comment - KCS

              rhn-support-ngirard Neil Girard
              rhn-support-nkashyap Nirupma Nirupma
              Huali Liu Huali Liu
              Jeana Routh Jeana Routh
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: