Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-17160

OpenShift on OpenStack: Password Rotation of OSP User still leads to unknown authentication failures in Keystone


    • ?
    • Important
    • No
    • ShiftStack Sprint 240, ShiftStack Sprint 241
    • 2
    • Rejected
    • False
    • Hide


    • Previously, the cloud credentials used in Manila CSI Driver Operator were cached, resulting in authentication issues if these credentials were rotated. This issue has now been addressed.
    • Bug Fix

      This is a clone of issue OCPBUGS-14049. The following is the description of the original issue:

      Description of problem:

      After all cluster operators have reconciled after the password rotation, we can still see authentication failures in keystone (attached screenshot of splunk query)

      Version-Release number of selected component (if applicable):

      - OpenShift 4.12.10 on OpenStack 16
      - The cluster is managed via RHACM, but password rotation shall be done via "regular"  OpenShift means.

      How reproducible:

      Rotated the OpenStack credentials according to the documentation [1]
      [1] https://docs.openshift.com/container-platform/4.12/authentication/managing_cloud_provider_credentials/cco-mode-passthrough.html#manually-rotating-cloud-creds_cco-mode-passthrough 

      Additional info:

      - we can't trace back where these authentication failures come from - they do disappear after a cluster upgrade (so when nodes are rebooted and all pods are restarted which indicates that there's still a component using the old credentials)
      - The relevant technical integration points _seem_ to be working though (LBaaS, CSI, Machine API, Swift)

      What is the business impact? Please also provide timeframe information.

      - We cannot rely on splunk monitoring for authentication issues since it's currently constantly showing authentication errors - We cannot be entirely sure that everything works as expected since we don't know the component that doesn't seem to use the new credentials


            sfinucan@redhat.com Stephen Finucane
            openshift-crt-jira-prow OpenShift Prow Bot
            Yaakov Khodorkovski Yaakov Khodorkovski
            Janine Eichler, Mariya Gokhool
            0 Vote for this issue
            7 Start watching this issue