Uploaded image for project: 'OpenShift Pipelines'
  1. OpenShift Pipelines
  2. SRVKP-7302

Anonymous Git Clone Resulting in Extreme Memory Usage, OOMs Resolver Pods

XMLWordPrintable

    • False
    • None
    • False
    • Pipelines Sprint Tekshift 26, Pipelines Sprint Tekshift 27
    • Customer Facing, Customer Reported

      Description of problem:

      When referencing a Remote pipeline using the following parameters, the git resolver uses an excessive amount of memory resulting in the Remote Resolver pod getting OOM killed, and the Remote Resolution Request timing out.

       

      ```yaml

        pipelineRef:
          resolver: git
          params:
          - name: url
            value: https://github.com/red-hat-data-services/RHOAI-Build-Config.git
          - name: revision
            value: pipeline
          - name: pathInRepo
            value: pipeline/fbc-fragment-build-pipeline.yaml
      ```

      The repository referenced is only ~145M when cloned with `git clone`, however in my local tests this remote resolution caused a pod with a memory limit of 4G to be OOM killed, and when I wrote a unit test to run `gitresolution.ResolveAnonymousGit()` using these parameters (with -memoryprof, which may add some memory overhead) the unit test consumed over 25G of memory before my computer froze.

       

      The customer was provided the workaround of using the http resolver instead specifying the URL as `https://raw.githubusercontent.com/red-hat-data-services/RHOAI-Build-Config/refs/heads/pipeline/pipeline/fbc-fragment-build-pipeline.yaml`

       

      Upstream Issue, since the direction here requires some community discussion: https://github.com/tektoncd/pipeline/issues/8652

      Prerequisites (if any, like setup, operators/versions):

      Steps to Reproduce

      • Download the attached patch `memory-usage-test.diff`
      • In a copy of the tektoncd/pipeline repository clean on origin/main, apply the attached diff memory-usage-test.diff with `git apply /path/to/memory-usage-test.diff`
      • Run the test with `go test -v -timeout 20m ./pkg/remoteresolution/resolver/git/... -run TestCrazyMemoryUse`
      • Separately, monitor the memory usage of the test using `top` or your preferred memory monitoring utility

      Actual results:

      The test will use an obscene amount of memory (and it remains to be seen if it even completes)

      Expected results:

      The test should complete in a timely manner and should not use much memory given the size of the repo is less than 200M

      Reproducibility (Always/Intermittent/Only Once):

      Always

      Acceptance criteria: 

       

      Definition of Done:

      Build Details:

      Additional info (Such as Logs, Screenshots, etc):

       

       *

              rh-ee-athorp Andrew Thorp
              rh-ee-athorp Andrew Thorp
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated: