Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-29945

CNV upgrade to 4.14.3->4.14.4->4.15.0, gets stuck with message 'install strategy failed: clusterrolebindings.rbac.authorization.k8s.io "hco-webhook-service-system:auth-delegator" already exists'

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Undefined Undefined
    • None
    • 4.15.0
    • OLM
    • None
    • No
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Upgrade starting with 4.14.3->4.14.4->4.15.0, gets stuck with 'install strategy failed: clusterrolebindings.rbac.authorization.k8s.io "hco-webhook-service-system:auth-delegator" already exists' message. My understanding is this is deployed by OLM.
          

      Version-Release number of selected component (if applicable):

      4.15.0 (upgrade involving multiple hops)
          

      How reproducible:

      2 out of 2. This issue is not seen with upgrade path of just 4.14.4->4.15.0
          

      Steps to Reproduce:

          1. Deploy a cluster with OCP 4.14.13, CNV 4.14.3
          2. Upgrade CNV to 4.14.4
          3. Upgrade OCP to 4.15.0-rc.7-x86_64
          4. Upgrade CNV to 4.15.0
          

      Actual results:

      [cnv-qe-jenkins@cnv-qe-infra-01 dbasu_upg]$ oc get clusterversion
      NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.15.0-rc.7   True        False         3d18h   Cluster version is 4.15.0-rc.7
      [cnv-qe-jenkins@cnv-qe-infra-01 dbasu_upg]$ 
      cnv-qe-jenkins@cnv-qe-infra-01 dbasu_upg]$ oc get csv -n openshift-cnv
      NAME                                       DISPLAY                    VERSION   REPLACES                                   PHASE
      kubevirt-hyperconverged-operator.v4.14.4   OpenShift Virtualization   4.14.4    kubevirt-hyperconverged-operator.v4.14.3   Failed
      kubevirt-hyperconverged-operator.v4.15.0   OpenShift Virtualization   4.15.0    kubevirt-hyperconverged-operator.v4.14.4   Pending
      [cnv-qe-jenkins@cnv-qe-infra-01 dbasu_upg]$ 
      [cnv-qe-jenkins@cnv-qe-infra-01 dbasu_upg]$ oc get csv -n openshift-cnv kubevirt-hyperconverged-operator.v4.15.0  -o json | jq ".status.conditions"
      [
        {
          "lastTransitionTime": "2024-02-26T16:06:09Z",
          "lastUpdateTime": "2024-02-26T16:06:09Z",
          "message": "requirements not yet checked",
          "phase": "Pending",
          "reason": "RequirementsUnknown"
        },
        {
          "lastTransitionTime": "2024-02-26T16:06:09Z",
          "lastUpdateTime": "2024-02-26T16:06:11Z",
          "message": "one or more requirements couldn't be found",
          "phase": "Pending",
          "reason": "RequirementsNotMet"
        }
      ]
      [cnv-qe-jenkins@cnv-qe-infra-01 dbasu_upg]$ 
      [cnv-qe-jenkins@cnv-qe-infra-01 dbasu_upg]$ oc get csv -n openshift-cnv kubevirt-hyperconverged-operator.v4.14.4  -o json | jq ".status.conditions"
      [
        {
          "lastTransitionTime": "2024-02-26T16:54:46Z",
          "lastUpdateTime": "2024-02-26T16:54:46Z",
          "message": "webhooks not installed",
          "phase": "Pending",
          "reason": "NeedsReinstall"
        },
        {
          "lastTransitionTime": "2024-02-26T16:54:48Z",
          "lastUpdateTime": "2024-02-26T16:54:48Z",
          "message": "all requirements found, attempting install",
          "phase": "InstallReady",
          "reason": "AllRequirementsMet"
        },
        {
          "lastTransitionTime": "2024-02-26T16:54:50Z",
          "lastUpdateTime": "2024-02-26T16:54:50Z",
          "message": "install strategy failed: clusterrolebindings.rbac.authorization.k8s.io \"hco-webhook-service-system:auth-delegator\" already exists",
          "phase": "Failed",
          "reason": "InstallComponentFailed"
        },
        {
          "lastTransitionTime": "2024-02-26T16:54:51Z",
          "lastUpdateTime": "2024-02-26T16:54:51Z",
          "message": "webhooks not installed",
          "phase": "Pending",
          "reason": "NeedsReinstall"
        },
        {
          "lastTransitionTime": "2024-02-26T16:54:52Z",
          "lastUpdateTime": "2024-02-26T16:54:52Z",
          "message": "all requirements found, attempting install",
          "phase": "InstallReady",
          "reason": "AllRequirementsMet"
        },
        {
          "lastTransitionTime": "2024-02-26T16:54:53Z",
          "lastUpdateTime": "2024-02-26T16:54:53Z",
          "message": "install strategy failed: clusterrolebindings.rbac.authorization.k8s.io \"hco-webhook-service-system:auth-delegator\" already exists",
          "phase": "Failed",
          "reason": "InstallComponentFailed"
        },
        {
          "lastTransitionTime": "2024-02-26T16:54:55Z",
          "lastUpdateTime": "2024-02-26T16:54:55Z",
          "message": "webhooks not installed",
          "phase": "Pending",
          "reason": "NeedsReinstall"
        },
        {
          "lastTransitionTime": "2024-02-26T16:54:56Z",
          "lastUpdateTime": "2024-02-26T16:54:56Z",
          "message": "all requirements found, attempting install",
          "phase": "InstallReady",
          "reason": "AllRequirementsMet"
        },
        {
          "lastTransitionTime": "2024-02-26T16:54:57Z",
          "lastUpdateTime": "2024-02-26T16:54:57Z",
          "message": "install strategy failed: clusterrolebindings.rbac.authorization.k8s.io \"hco-webhook-service-system:auth-delegator\" already exists",
          "phase": "Failed",
          "reason": "InstallComponentFailed"
        },
      
          

      Expected results:

      Upgrade completes successfully
          

      Additional info:

      Will attach must gather under https://drive.google.com/drive/folders/15BXQpT1k0Gnsy24uiPPhW4mFFgHIE41G?usp=drive_link
          

            krizza@redhat.com Kevin Rizza
            rhn-support-dbasunag Debarati Basu-Nag
            Jian Zhang Jian Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: