Details
-
Bug
-
Resolution: Done
-
Critical
-
Logging 5.2.4
-
False
-
False
-
NEW
-
NEW
-
-
Logging (LogExp) - Sprint 213
Description
After upgrading to 5.2.4, we noticed the operator error with an Unable to create PersistentVolumeClaim due to forbidden: exceeded quota: infra-storage-quota.
These PVCs already exist.
I noticed in an old version of the code:
// for some reason if the PVC already exists but creating it again would violate
// quota we get an error regarding quota not that it already exists
// so check to see if it already exists
I don't think we're handling that case anymore and anyone with tighter quotas could see issues during upgrades.
This was added via: https://github.com/openshift/elasticsearch-operator/commit/9e0083e0806b1afc3e0bf58b5668a10d5cb314ee
We tried increasing the quota to get past the issue, but the quota reported it creating over 20 PVCs despite no PVCs being created. It just then keeps looping and failing due to a lack of quota.
{"_ts":"2022-01-07T16:57:02.671367655Z","_level":"0","_component":"elasticsearch-operator","_message":"Unable to create PersistentVolumeClaim","_error":{"cause":{"ErrStatus":{"metadata":{},"status":"Failure","message":"persistentvolumeclaims \"elasticsearch-elasticsearch-cdm-9aeuzkeh-1\" is forbidden: exceeded quota: infra-storage-quota, requested: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=2500Gi,requests.storage=2500Gi, used: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=10000Gi,requests.storage=10000Gi, limited: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=10Ti,requests.storage=10Ti","reason":"Forbidden","details":{"name":"elasticsearch-elasticsearch-cdm-9aeuzkeh-1","kind":"persistentvolumeclaims"},"code":403}},"msg":"failed to create persistentvolumeclaim","name":"elasticsearch-elasticsearch-cdm-9aeuzkeh-1","namespace":"openshift-logging"}}
{"_ts":"2022-01-07T16:57:02.673452701Z","_level":"0","_component":"elasticsearch-operator","_message":"Unable to create PersistentVolumeClaim","_error":{"cause":{"ErrStatus":{"metadata":{},"status":"Failure","message":"persistentvolumeclaims \"elasticsearch-elasticsearch-cdm-9aeuzkeh-2\" is forbidden: exceeded quota: infra-storage-quota, requested: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=2500Gi,requests.storage=2500Gi, used: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=10000Gi,requests.storage=10000Gi, limited: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=10Ti,requests.storage=10Ti","reason":"Forbidden","details":{"name":"elasticsearch-elasticsearch-cdm-9aeuzkeh-2","kind":"persistentvolumeclaims"},"code":403}},"msg":"failed to create persistentvolumeclaim","name":"elasticsearch-elasticsearch-cdm-9aeuzkeh-2","namespace":"openshift-logging"}}
{"_ts":"2022-01-07T16:57:02.676277283Z","_level":"0","_component":"elasticsearch-operator","_message":"Unable to create PersistentVolumeClaim","_error":{"cause":{"ErrStatus":{"metadata":{},"status":"Failure","message":"persistentvolumeclaims \"elasticsearch-elasticsearch-cdm-9aeuzkeh-3\" is forbidden: exceeded quota: infra-storage-quota, requested: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=2500Gi,requests.storage=2500Gi, used: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=10000Gi,requests.storage=10000Gi, limited: netapp-block-extended.storageclass.storage.k8s.io/requests.storage=10Ti,requests.storage=10Ti","reason":"Forbidden","details":{"name":"elasticsearch-elasticsearch-cdm-9aeuzkeh-3","kind":"persistentvolumeclaims"},"code":403}},"msg":"failed to create persistentvolumeclaim","name":"elasticsearch-elasticsearch-cdm-9aeuzkeh-3","namespace":"openshift-logging"}}
