-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
Pipelines 1.8.2
-
None
-
False
-
None
-
False
-
-
Description of problem:
Creation of `pipeline` ServiceAccount after new namespace creation sometimes takes too long (more than 30 seconds)
Prerequisites (if any, like setup, operators/versions):
Openshift Pipelines 1.8.2 installed as part of Stonesoup.
Steps to Reproduce
This issue started randomly appearing in PR check for Stonesoup. Our e2e-tests create test namespace and part of the method is waiting of 30 seconds for `pipeline` service account to be present there.
Expected results:
`pipeline` service account is created faster than 30 seconds after namesapce creation
Reproducibility (Always/Intermittent/Only Once):
Recently it's happening quite often in our CI
Build Details:
Additional info (Such as Logs, Screenshots, etc):
You can find lot of logs in artifacts of the failed jobs in openshift-ci I linked in "steps to reproduce" section.
Most notably you can find
- all pod logs in artifacts/appstudio-e2e-tests/gather-extra/artifacts/pods/
- events, namespaces and other typical openshift resources in artifacts/appstudio-e2e-tests/gather-extra/artifacts/
- etcd, kube-apiserver etc. logs in artifacts/appstudio-e2e-tests/gather-extra/artifacts/audit_logs/
- other CRs like Applications, Components, PipelineRuns etc. in artifacts/appstudio-e2e-tests/redhat-appstudio-gather/artifacts/
I couldn't find anything suspicious in pipelines operators, but when I was looking at kubeapi logs I found out this: https://pastebin.com/uR0BQD9x This is output of :
$ cat artifacts/audit_logs/kube-apiserver/ci-op-vff0liqj-a3e7f-gd9qk-master-2-audit-2023-01-25T16-35-45.442.log|grep 'resource":"serviceaccounts","namespace":"integ-e2e-qmfi","name":"pipeline"'
There we can see lot of GET requests to kubeapi looking for the service account in one of the affected namespaces between times 2023-01-25T16:25:14.580484Z and 2023-01-25T16:25:50.208224Z which roughly corresponds to the 30sec timeout where our tests are trying to get the serviceaccount.
But the CREATE request comes AFTER that - at 2023-01-25T16:25:50.228554Z
- is duplicated by
-
SRVKP-2814 Delay of Pipelines Service Account Rolebindings Exceeds 20 Seconds at Scale
- To Do