Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-425

OpenShift 4.10.22 Storage operator won't reconcile

XMLWordPrintable

    • Important
    • None
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      The storage operator is degraded for the OCP 4.10.22 VSphere IPI cluster. The operator isn't reconciling as the transition time is very older also the operator is functioning properly.

      $ omg get co storage -oyaml
      apiVersion: config.openshift.io/v1
      kind: ClusterOperator
      metadata:
        creationTimestamp: '2022-08-04T11:01:09Z'
        generation: 1
        managedFields:
        - apiVersion: config.openshift.io/v1
          fieldsType: FieldsV1
          fieldsV1:
            f:spec: {}
          manager: Go-http-client
          operation: Update
          time: '2022-08-04T11:01:09Z'
        - apiVersion: config.openshift.io/v1
          fieldsType: FieldsV1
          fieldsV1:
            f:status:
              .: {}
              f:conditions: {}
              f:extension: {}
              f:relatedObjects: {}
              f:versions: {}
          manager: Go-http-client
          operation: Update
          subresource: status
          time: '2022-08-04T11:01:09Z'
        name: storage
        resourceVersion: '1334721569'
        uid: 7af8a420-e896-4703-a638-83e067ccca6a
      spec: {}
      status:
        conditions:
        - lastTransitionTime: '2022-08-09T11:49:03Z'
          message: 'VSphereCSIDriverOperatorCRDegraded: VMwareVSphereControllerDegraded:
            etcdserver: request timed out'
          reason: VSphereCSIDriverOperatorCR_VMwareVSphereController_SyncError
          status: 'True'
          type: Degraded
        - lastTransitionTime: '2022-08-22T13:55:18Z'
          message: 'VSphereCSIDriverOperatorCRProgressing: All is well'
          reason: AsExpected
          status: 'False'
          type: Progressing
        - lastTransitionTime: '2022-08-17T08:05:56Z'
          message: 'VSphereCSIDriverOperatorCRAvailable: All is well'
          reason: AsExpected
          status: 'True'
          type: Available
        - lastTransitionTime: '2022-08-04T11:01:09Z'
          message: 'VSphereCSIDriverOperatorCRUpgradeable: VMwareVSphereControllerUpgradeable:
            Failed to connect to vSphere: Post "https://xyz.vmware.example.com/sdk":
            dial tcp: lookup xyz.vmware.example.com on 172.30.0.10:53: read udp 10.x.x.x:54378->172.30.0.10:53:
            read: connection refused'
          reason: VSphereCSIDriverOperatorCR_VMwareVSphereController_vsphere_connection_failed
          status: Unknown
          type: Upgradeable
        extension: null
        relatedObjects:
        - group: ''
          name: vsphere-csi-driver-operator-trusted-ca-bundle
          namespace: openshift-cluster-csi-drivers
          resource: configmaps
        - group: ''
          name: vmware-vsphere-csi-driver-operator
          namespace: openshift-cluster-csi-drivers
          resource: serviceaccounts
        - group: rbac.authorization.k8s.io
          name: vmware-vsphere-csi-driver-operator-role
          namespace: openshift-cluster-csi-drivers
          resource: roles
        - group: rbac.authorization.k8s.io
          name: vmware-vsphere-csi-driver-operator-rolebinding
          namespace: openshift-cluster-csi-drivers
          resource: rolebindings
        - group: rbac.authorization.k8s.io
          name: vmware-vsphere-csi-driver-operator-clusterrole
          resource: clusterroles
        - group: rbac.authorization.k8s.io
          name: vmware-vsphere-csi-driver-operator-clusterrolebinding
          resource: clusterrolebindings
        - group: operator.openshift.io
          name: csi.vsphere.vmware.com
          resource: clustercsidrivers
        - group: ''
          name: openshift-cluster-storage-operator
          resource: namespaces
        - group: ''
          name: openshift-cluster-csi-drivers
          resource: namespaces
        - group: operator.openshift.io
          name: cluster
          resource: storages
        - group: rbac.authorization.k8s.io
          name: cluster-storage-operator-role
          resource: clusterrolebindings
        - group: sharedresource.openshift.io
          name: ''
          resource: sharedconfigmaps
        - group: sharedresource.openshift.io
          name: ''
          resource: sharedsecrets
        versions:
        - name: VSphereProblemDetectorDeploymentController
          version: 4.10.22
        - name: operator
          version: 4.10.22
      
      • Manual nslookup and curl to the VCenter URL are working fine from the nodes and pods.
      • PV PVC getting created without any issues.
      • It seems the storage operator isn't reconciling as we tried to clear the description related to the error for the operator with the following commands but still the same error comes up.

      --> $ oc proxy &
      --> $ curl -v --header "Content-Type: application/json-patch+json" --request PATCH --data '[\{"op": "remove", "path": "/status/conditions/24"}]' http://127.0.0.1:8001/apis/operator.openshift.io/v1/storages/cluster/status

      Version-Release number of selected component (if applicable):

      4.10.22 VSphere IPI

      Actual results:

      Storage operator degraded and the errors might be false.

      Expected results:

      The storage operator must not be in a degraded state.

      Additional info:

      I will be sharing the mus-gather and curl command's output.

              hekumar@redhat.com Hemant Kumar
              rhn-support-aygarg Ayush Garg
              Wei Duan Wei Duan
              Ayush Garg
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: