Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2127

ES Operator Stuck on Quota after Upgrade [openshift-logging 5.3]

    XMLWordPrintable

Details

    • False
    • False
    • NEW
    • NEW
    • Hide
      Before this update, elasticsearch generated the error "Unable to create PersistentVolumeClaim due to forbidden: exceeded quota: infra-storage-quota." if the PVC already existed. With this update, elasticsearch checks for existing PVCs, resolving the issue.
      Show
      Before this update, elasticsearch generated the error "Unable to create PersistentVolumeClaim due to forbidden: exceeded quota: infra-storage-quota." if the PVC already existed. With this update, elasticsearch checks for existing PVCs, resolving the issue.
    • Logging (LogExp) - Sprint 213

    Description

      After upgrading to 5.2.4, we noticed the operator error with an Unable to create PersistentVolumeClaim due to forbidden: exceeded quota: infra-storage-quota.

      These PVCs already exist.

      I noticed in an old version of the code:

      // for some reason if the PVC already exists but creating it again would violate
      // quota we get an error regarding quota not that it already exists
      // so check to see if it already exists

      https://github.com/openshift/elasticsearch-operator/blob/6aeb7c0412bf6ec6f9f1c0b74369b742bdccdab0/internal/k8shandler/persistentvolumeclaims.go#L18

      I don't think we're handling that case anymore and anyone with tighter quotas could see issues during upgrades.

      This was added via: https://github.com/openshift/elasticsearch-operator/commit/9e0083e0806b1afc3e0bf58b5668a10d5cb314ee

      We tried increasing the quota to get past the issue, but the quota reported it creating over 20 PVCs despite no PVCs being created. It just then keeps looping and failing due to a lack of quota.

      {"_ts":"2022-01-07T16:57:02.671367655Z","_level":"0","_component":"elasticsearch-operator","_message":"Unable to create PersistentVolumeClaim","_error":{"cause":{"ErrStatus":{"metadata":{},"status":"Failure","message":"persistentvolumeclaims \"elasticsearch-elasticsearch-cdm-9aeuzkeh-1\" is forbidden: exceeded quota: infra-storage-quota, requested: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=2500Gi,requests.storage=2500Gi, used: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=10000Gi,requests.storage=10000Gi, limited: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=10Ti,requests.storage=10Ti","reason":"Forbidden","details":{"name":"elasticsearch-elasticsearch-cdm-9aeuzkeh-1","kind":"persistentvolumeclaims"},"code":403}},"msg":"failed to create persistentvolumeclaim","name":"elasticsearch-elasticsearch-cdm-9aeuzkeh-1","namespace":"openshift-logging"}}
      {"_ts":"2022-01-07T16:57:02.673452701Z","_level":"0","_component":"elasticsearch-operator","_message":"Unable to create PersistentVolumeClaim","_error":{"cause":{"ErrStatus":{"metadata":{},"status":"Failure","message":"persistentvolumeclaims \"elasticsearch-elasticsearch-cdm-9aeuzkeh-2\" is forbidden: exceeded quota: infra-storage-quota, requested: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=2500Gi,requests.storage=2500Gi, used: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=10000Gi,requests.storage=10000Gi, limited: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=10Ti,requests.storage=10Ti","reason":"Forbidden","details":{"name":"elasticsearch-elasticsearch-cdm-9aeuzkeh-2","kind":"persistentvolumeclaims"},"code":403}},"msg":"failed to create persistentvolumeclaim","name":"elasticsearch-elasticsearch-cdm-9aeuzkeh-2","namespace":"openshift-logging"}}
      {"_ts":"2022-01-07T16:57:02.676277283Z","_level":"0","_component":"elasticsearch-operator","_message":"Unable to create PersistentVolumeClaim","_error":{"cause":{"ErrStatus":{"metadata":{},"status":"Failure","message":"persistentvolumeclaims \"elasticsearch-elasticsearch-cdm-9aeuzkeh-3\" is forbidden: exceeded quota: infra-storage-quota, requested: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=2500Gi,requests.storage=2500Gi, used: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=10000Gi,requests.storage=10000Gi, limited: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=10Ti,requests.storage=10Ti","reason":"Forbidden","details":{"name":"elasticsearch-elasticsearch-cdm-9aeuzkeh-3","kind":"persistentvolumeclaims"},"code":403}},"msg":"failed to create persistentvolumeclaim","name":"elasticsearch-elasticsearch-cdm-9aeuzkeh-3","namespace":"openshift-logging"}}
      

      Attachments

        Issue Links

          Activity

            People

              gvanloo Gerard Vanloo
              gvanloo Gerard Vanloo
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: