Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-14922

cluster operator monitoring is not available when deploying 4.14 spoke when console operator is disabled

    XMLWordPrintable

Details

    • Important
    • No
    • MON Sprint 238
    • 1
    • Approved
    • False
    • Hide

      None

      Show
      None
    • N/A
    • Release Note Not Required

    Description

      Description of problem:

      When deploying 4.14 spoke, agentclusterinstall is stuck at finalizing stage

      clusterverions on spoke report "Unable to apply 4.14.0-0.ci-2023-06-13-083232: the cluster operator monitoring is not available"

      Please note: console operator is disabled purposely - it is needed in telco case to reduce platform resource usage

      [kni@registry.kni-qe-28 ~]$ oc get clusterversions.config.openshift.io -A
      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version             False       True          46m     Unable to apply 4.14.0-0.ci-2023-06-13-083232: the cluster operator monitoring is not available
      
      [kni@registry.kni-qe-28 ~]$ oc get clusterversions.config.openshift.io -n version -o yaml 
      apiVersion: v1
      items:
      - apiVersion: config.openshift.io/v1
        kind: ClusterVersion
        metadata:
          creationTimestamp: "2023-06-13T15:16:32Z"
          generation: 2
          name: version
          resourceVersion: "20061"
          uid: f8fc0c3e-009d-4d86-a05d-2fd0aba59528
        spec:
          capabilities:
            additionalEnabledCapabilities:
            - marketplace
            - NodeTuning
            baselineCapabilitySet: None
          channel: stable-4.14
          clusterID: 5cfc0491-5a23-4383-935b-71e3c793e875
        status:
          availableUpdates: null
          capabilities:
            enabledCapabilities:
            - NodeTuning
            - marketplace
            knownCapabilities:
            - CSISnapshot
            - Console
            - Insights
            - NodeTuning
            - Storage
            - baremetal
            - marketplace
            - openshift-samples
          conditions:
          - lastTransitionTime: "2023-06-13T15:16:33Z"
            message: 'Unable to retrieve available updates: Get "https://api.openshift.com/api/upgrades_info/v1/graph?arch=amd64&channel=stable-4.14&id=5cfc0491-5a23-4383-935b-71e3c793e875&version=4.14.0-0.ci-2023-06-13-083232":
              dial tcp 54.211.39.83:443: connect: network is unreachable'
            reason: RemoteFailed
            status: "False"
            type: RetrievedUpdates
          - lastTransitionTime: "2023-06-13T15:16:33Z"
            message: Capabilities match configured spec
            reason: AsExpected
            status: "False"
            type: ImplicitlyEnabledCapabilities
          - lastTransitionTime: "2023-06-13T15:16:33Z"
            message: Payload loaded version="4.14.0-0.ci-2023-06-13-083232" image="registry.kni-qe-28.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:826bb878c5a1469ee8bb991beebc38a4e25b8f5cef9cdf1931ef99ffe5ffbc80"
              architecture="amd64"
            reason: PayloadLoaded
            status: "True"
            type: ReleaseAccepted
          - lastTransitionTime: "2023-06-13T15:16:33Z"
            status: "False"
            type: Available
          - lastTransitionTime: "2023-06-13T15:41:36Z"
            message: Cluster operator monitoring is not available
            reason: ClusterOperatorNotAvailable
            status: "True"
            type: Failing
          - lastTransitionTime: "2023-06-13T15:16:33Z"
            message: 'Unable to apply 4.14.0-0.ci-2023-06-13-083232: the cluster operator
              monitoring is not available'
            reason: ClusterOperatorNotAvailable
            status: "True"
            type: Progressing
          desired:
            image: registry.kni-qe-28.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:826bb878c5a1469ee8bb991beebc38a4e25b8f5cef9cdf1931ef99ffe5ffbc80
            version: 4.14.0-0.ci-2023-06-13-083232
          history:
          - completionTime: null
            image: registry.kni-qe-28.ptp.lab.eng.bos.redhat.com:5000/openshift-release-dev/ocp-release@sha256:826bb878c5a1469ee8bb991beebc38a4e25b8f5cef9cdf1931ef99ffe5ffbc80
            startedTime: "2023-06-13T15:16:33Z"
            state: Partial
            verified: false
            version: 4.14.0-0.ci-2023-06-13-083232
          observedGeneration: 2
          versionHash: H6tRc6p_ZWU=
      kind: List
      metadata:
        resourceVersion: ""
      
      [kni@registry.kni-qe-28 ~]$ oc get co -A
      NAME                                       VERSION                         AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      authentication                             4.14.0-0.ci-2023-06-13-083232   True        False         False      14m     
      cloud-controller-manager                   4.14.0-0.ci-2023-06-13-083232   True        False         False      24m     
      cloud-credential                           4.14.0-0.ci-2023-06-13-083232   True        False         False      25m     
      cluster-autoscaler                         4.14.0-0.ci-2023-06-13-083232   True        False         False      24m     
      config-operator                            4.14.0-0.ci-2023-06-13-083232   True        False         False      25m     
      control-plane-machine-set                  4.14.0-0.ci-2023-06-13-083232   True        False         False      24m     
      dns                                        4.14.0-0.ci-2023-06-13-083232   True        False         False      19m     
      etcd                                       4.14.0-0.ci-2023-06-13-083232   True        False         False      22m     
      image-registry                             4.14.0-0.ci-2023-06-13-083232   True        False         False      14m     
      ingress                                    4.14.0-0.ci-2023-06-13-083232   True        False         False      25m     
      kube-apiserver                             4.14.0-0.ci-2023-06-13-083232   True        False         False      18m     
      kube-controller-manager                    4.14.0-0.ci-2023-06-13-083232   True        False         False      19m     
      kube-scheduler                             4.14.0-0.ci-2023-06-13-083232   True        False         False      17m     
      kube-storage-version-migrator              4.14.0-0.ci-2023-06-13-083232   True        False         False      25m     
      machine-api                                4.14.0-0.ci-2023-06-13-083232   True        False         False      25m     
      machine-approver                           4.14.0-0.ci-2023-06-13-083232   True        False         False      24m     
      machine-config                             4.14.0-0.ci-2023-06-13-083232   True        False         False      21m     
      marketplace                                4.14.0-0.ci-2023-06-13-083232   True        False         False      25m     
      monitoring                                                                 False       True          True       14m     reconciling Console Plugin failed: creating ConsolePlugin object failed: the server could not find the requested resource (post consoleplugins.console.openshift.io)
      network                                    4.14.0-0.ci-2023-06-13-083232   True        False         False      26m     
      node-tuning                                4.14.0-0.ci-2023-06-13-083232   True        False         False      25m     
      openshift-apiserver                        4.14.0-0.ci-2023-06-13-083232   True        False         False      14m     
      openshift-controller-manager               4.14.0-0.ci-2023-06-13-083232   True        False         False      18m     
      operator-lifecycle-manager                 4.14.0-0.ci-2023-06-13-083232   True        False         False      25m     
      operator-lifecycle-manager-catalog         4.14.0-0.ci-2023-06-13-083232   True        False         False      25m     
      operator-lifecycle-manager-packageserver   4.14.0-0.ci-2023-06-13-083232   True        False         False      19m     
      service-ca                                 4.14.0-0.ci-2023-06-13-083232   True        False         False      25m    

      Version-Release number of selected component (if applicable):
      4.14

      How reproducible:

      100%

      Steps to Reproduce:

      1. Deploy RAN DU spoke cluster via gitops ZTP approach with multiple base capabilities disabled including Console operator.
         spec:     
           capabilities:       
             additionalEnabledCapabilities:
               - marketplace       
               - NodeTuning       
           baselineCapabilitySet: None     
           channel: stable-4.14 
      2. Monitor ocp deployment on spoke.
      

      Actual results:

      Deployment fails while finalizing agentclusterinstall.  clusterverions on spoke report "the cluster operator monitoring is not available"

      Expected results:

      Successful spoke deployment

      Additional info:

      After manually enabling console in clusterversion, the monitoring operator succeeded and OCP install completed

      must-gather logs:
      https://drive.google.com/file/d/19zO21jqcVTIkAdGS2DEqQuhg2oGUmuNY/view?usp=sharing
      https://drive.google.com/file/d/1PXjZmBdMwHWNwkaXr2wE9tTtBRJWYeKP/view?usp=sharing

       

      Attachments

        Activity

          People

            sthaha@redhat.com Sunil Thaha
            bblock@redhat.com Bonnie Block
            Tai Gao Tai Gao
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: