Uploaded image for project: 'OpenShift Pipelines'
  1. OpenShift Pipelines
  2. SRVKP-2793

`pipeline` SA sometimes takes too long to be created in new namespace

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • None
    • Pipelines 1.8.2
    • None
    • False
    • None
    • False

      Description of problem:

      Creation of `pipeline` ServiceAccount after new namespace creation sometimes takes too long (more than 30 seconds)

      Prerequisites (if any, like setup, operators/versions):

      Openshift Pipelines 1.8.2 installed as part of Stonesoup.

      Steps to Reproduce

      This issue started randomly appearing in PR check for Stonesoup. Our e2e-tests create test namespace and part of the method is waiting of 30 seconds for `pipeline` service account to be present there.

      Example of faield test run #1

      Example of failed test run #2

      Example of failed test run #3

      Expected results:

      `pipeline` service account is created faster than 30 seconds after namesapce creation

      Reproducibility (Always/Intermittent/Only Once):

      Recently it's happening quite often in our CI

      Build Details:

      Additional info (Such as Logs, Screenshots, etc):

      You can find lot of logs in artifacts of the failed jobs in openshift-ci I linked in "steps to reproduce" section.

      Most notably you can find

      • all pod logs in artifacts/appstudio-e2e-tests/gather-extra/artifacts/pods/
      • events, namespaces and other typical openshift resources in artifacts/appstudio-e2e-tests/gather-extra/artifacts/
      • etcd, kube-apiserver etc. logs in artifacts/appstudio-e2e-tests/gather-extra/artifacts/audit_logs/
      • other CRs like Applications, Components, PipelineRuns etc. in artifacts/appstudio-e2e-tests/redhat-appstudio-gather/artifacts/

      I couldn't find anything suspicious in pipelines operators, but when I was looking at kubeapi logs I found out this: https://pastebin.com/uR0BQD9x This is output of :

      $ cat artifacts/audit_logs/kube-apiserver/ci-op-vff0liqj-a3e7f-gd9qk-master-2-audit-2023-01-25T16-35-45.442.log|grep 'resource":"serviceaccounts","namespace":"integ-e2e-qmfi","name":"pipeline"' 

      There we can see lot of GET requests to kubeapi looking for the service account in one of the affected namespaces between times 2023-01-25T16:25:14.580484Z and 2023-01-25T16:25:50.208224Z which roughly corresponds to the 30sec timeout where our tests are trying to get the serviceaccount. 
      But the CREATE request comes AFTER that - at 2023-01-25T16:25:50.228554Z

              rhopp@redhat.com Radim Hopp
              rhopp@redhat.com Radim Hopp
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: