Uploaded image for project: 'OpenShift Pipelines'
  1. OpenShift Pipelines
  2. SRVKP-7855

Pruner randomly fails to remove PipelineRuns when there is change to configmap settings

XMLWordPrintable

    • Pipelines Sprint pioneers 31

      Description of problem:

      When history limit is set to 0, successful pipeline-run is not pruned. I had 3 pending pipelinerun running for a longer time in running state, before starting the below scenario in same namespace. When the failure task has exit 1 to fail pipelinerun this issue is observed

      Prerequisites (if any, like setup, operators/versions):

      1.19.0

      Steps to Reproduce

      1. Set below values in global config and create 2 pass and 2 failed pipelinerun

       

        global-config: |-
    enforcedConfigLevel: global
    # ttlSecondsAfterFinished: 60
    historyLimit: 0
    successfulHistoryLimit: 2
    failedHistoryLimit: 2

       

       

       tkn pr ls
NAME                STARTED          DURATION   STATUS
fail-run-zrvx8      14 seconds ago   4s         Failed
fail-run-x2z5m      16 seconds ago   4s         Failed
hello-pr-zw299      20 seconds ago   5s         Succeeded
hello-pr-jrwgw      21 seconds ago   6s         Succeeded

       

      3. Now comment out the  successfulHistoryLimit: 2 and ** failedHistoryLimit: 2

       

        global-config: |-
    enforcedConfigLevel: global
    # ttlSecondsAfterFinished: 60
    historyLimit: 0
    # successfulHistoryLimit: 2
    # failedHistoryLimit: 2

      Actual results:

      NAME                             STARTED          DURATION   STATUS
hello-pr-zw299-run-hello-world   3 minutes ago    5s         Succeeded
hello-pr-jrwgw-run-hello-world   3 minutes ago    6s         Succeeded

      Expected results:

      No pipelineruns found

      Reproducibility (Always/Intermittent/Only Once):

      Always

      Additional info (Such as Logs, Screenshots, etc):

      Timestamp is added to the pipelinerun:

       

      {"severity":"INFO","timestamp":"2025-06-11T15:36:46.396140649Z","logger":"tekton-pruner-controller","caller":"controller/controller.go:548","message":"Reconcile succeeded","commit":"d355488-dirty","knative.dev/controller":"github.com.openshift-pipelines.tektoncd-pruner.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"bb63e380-147e-4bd5-8cd1-e58c8f02114f","knative.dev/key":"tc1-ns/hello-pr-gqx29","duration":"37.566µs"} 
{"severity":"INFO","timestamp":"2025-06-11T15:41:59.229204913Z","logger":"tekton-pruner-controller","caller":"controller/controller.go:548","message":"Reconcile succeeded","commit":"d355488-dirty","knative.dev/controller":"github.com.openshift-pipelines.tektoncd-pruner.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"6a726823-87bf-475c-9bb4-271fdc668509","knative.dev/key":"tc1-ns/hello-pr-gqx29","duration":"28.967µs"}

      **

       

      Controller logs:

      {"severity":"INFO","timestamp":"2025-06-11T13:03:20.057536808Z","logger":"tekton-pruner-controller","caller":"tektonpruner/controller.go:103","message":"Namespaces selected for garbage collection","commit":"d355488-dirty","namespaces":["default","knative-eventing","knative-serving","knative-serving-ingress","minio","namespace-one","releasetest-2c55m","releasetest-466pp","releasetest-49zfl","releasetest-4l677","releasetest-4n7t5","releasetest-4plzj","releasetest-59cz7","releasetest-5m54h","releasetest-5mjcd","releasetest-5rs46","releasetest-6c879","releasetest-6kbcz","releasetest-6s2bg","releasetest-78fz5","releasetest-7z49s","releasetest-8tcbj","releasetest-95995","releasetest-97dmj","releasetest-97kkz","releasetest-9lzrz","releasetest-9m9pr","releasetest-9mlh9","releasetest-b654w","releasetest-b7242","releasetest-bf8k6","releasetest-brh9w","releasetest-bsgj6","releasetest-btdxz","releasetest-c49tv","releasetest-ckv9m","releasetest-d4btf","releasetest-dfklm","releasetest-dgwjw","releasetest-dr5kb","releasetest-gnl2s","releasetest-h4j4h","releasetest-hfbch","releasetest-hpq5k","releasetest-hq78d","releasetest-hvb88","releasetest-hxs7c","releasetest-jq9l9","releasetest-kb2hj","releasetest-ktjbt","releasetest-kv5xl","releasetest-lhsj2","releasetest-lsxmv","releasetest-lvnj7","releasetest-ml7l9","releasetest-mmw7h","releasetest-mvs4j","releasetest-mwnvf","releasetest-nfj65","releasetest-nfqxb","releasetest-nl8n6","releasetest-p4btm","releasetest-p8kth","releasetest-pq4jh","releasetest-pswmz","releasetest-qcvdt","releasetest-qmhpx","releasetest-qp2cq","releasetest-qsh2l","releasetest-r64mg","releasetest-rvqh7","releasetest-sdk9x","releasetest-skpfl","releasetest-snk4b","releasetest-svm6f","releasetest-t8v82","releasetest-tc2l5","releasetest-tmf6d","releasetest-vgqnx","releasetest-vkwhz","releasetest-vtdrs","releasetest-xvtsq","releasetest-xwz2l","releasetest-z58s4","releasetest-zllxl","releasetest-ztc56","releasetest-zzszs","tc1-ns","test-project-1","test-project-2"]}
{"severity":"INFO","timestamp":"2025-06-11T13:03:27.464096567Z","logger":"tekton-pruner-controller","caller":"tektonpruner/controller.go:122","message":"Worker processing namespace","commit":"d355488-dirty","worker":2,"namespace":"tc1-ns"}
{"severity":"INFO","timestamp":"2025-06-11T13:03:28.171594573Z","logger":"tekton-pruner-controller","caller":"controller/controller.go:548","message":"Reconcile succeeded","commit":"d355488-dirty","knative.dev/controller":"github.com.openshift-pipelines.tektoncd-pruner.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"59e65c6d-0e42-4b3e-bef5-d28cecaa384b","knative.dev/key":"tc1-ns/fail-run-grwwm","duration":"60.974µs"}
{"severity":"INFO","timestamp":"2025-06-11T13:03:28.191015071Z","logger":"tekton-pruner-controller","caller":"controller/controller.go:548","message":"Reconcile succeeded","commit":"d355488-dirty","knative.dev/controller":"github.com.openshift-pipelines.tektoncd-pruner.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"72eb3727-00c0-44ea-b5c5-525ee9907280","knative.dev/key":"tc1-ns/fail-run-grwwm","duration":"61.93µs"}
{"severity":"INFO","timestamp":"2025-06-11T13:03:28.219788523Z","logger":"tekton-pruner-controller","caller":"controller/controller.go:548","message":"Reconcile succeeded","commit":"d355488-dirty","knative.dev/controller":"github.com.openshift-pipelines.tektoncd-pruner.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"66ac55ba-905c-4888-81ef-a649987be47c","knative.dev/key":"tc1-ns/fail-run-g9tg8","duration":"95.158µs"}
{"severity":"INFO","timestamp":"2025-06-11T13:03:28.243100384Z","logger":"tekton-pruner-controller","caller":"controller/controller.go:548","message":"Reconcile succeeded","commit":"d355488-dirty","knative.dev/controller":"github.com.openshift-pipelines.tektoncd-pruner.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"731705d3-8797-4164-b8ae-7b54f6bbd27e","knative.dev/key":"tc1-ns/fail-run-g9tg8","duration":"95.601µs"}
{"severity":"INFO","timestamp":"2025-06-11T13:03:28.274788578Z","logger":"tekton-pruner-controller","caller":"controller/controller.go:548","message":"Reconcile succeeded","commit":"d355488-dirty","knative.dev/controller":"github.com.openshift-pipelines.tektoncd-pruner.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"3f664065-9a38-40f3-9d8b-aca9039fd883","knative.dev/key":"tc1-ns/fail-run-grwwm","duration":"24.09µs"}
{"severity":"INFO","timestamp":"2025-06-11T13:03:28.325459529Z","logger":"tekton-pruner-controller","caller":"controller/controller.go:548","message":"Reconcile succeeded","commit":"d355488-dirty","knative.dev/controller":"github.com.openshift-pipelines.tektoncd-pruner.pkg.reconciler.pipelinerun.Reconciler","knative.dev/kind":"tekton.dev.PipelineRun","knative.dev/traceid":"c054065c-b922-4aec-ab30-f8972c3132e2","knative.dev/key":"tc1-ns/fail-run-g9tg8","duration":"24.396µs"}
{"severity":"ERROR","timestamp":"2025-06-11T13:03:28.32869519Z","logger":"tekton-pruner-controller","caller":"config/history_limiter.go:159","message":"error patching resource with 'mark as processed' annotation","commit":"d355488-dirty","resource":"PipelineRun","namespace":"tc1-ns","name":"fail-run-g9tg8","error":"failed to patch PipelineRun tc1-ns/fail-run-g9tg8: pipelineruns.tekton.dev \"fail-run-g9tg8\" not found","stacktrace":"github.com/openshift-pipelines/tektoncd-pruner/pkg/config.(*HistoryLimiter).markAsProcessed\n\t/go/src/github.com/openshift-pipelines/tektoncd-pruner/pkg/config/history_limiter.go:159\ngithub.com/openshift-pipelines/tektoncd-pruner/pkg/config.(*HistoryLimiter).ProcessEvent\n\t/go/src/github.com/openshift-pipelines/tektoncd-pruner/pkg/config/history_limiter.go:109\ngithub.com/openshift-pipelines/tektoncd-pruner/pkg/reconciler/tektonpruner.cleanupPRs\n\t/go/src/github.com/openshift-pipelines/tektoncd-pruner/pkg/reconciler/tektonpruner/controller.go:242\ngithub.com/openshift-pipelines/tektoncd-pruner/pkg/reconciler/tektonpruner.runGarbageCollector.func1\n\t/go/src/github.com/openshift-pipelines/tektoncd-pruner/pkg/reconciler/tektonpruner/controller.go:124"}
{"severity":"ERROR","timestamp":"2025-06-11T13:03:28.414066842Z","logger":"tekton-pruner-controller","caller":"tektonpruner/controller.go:236","message":"error patching PipelineRun to remove history limit check processed annotation","commit":"d355488-dirty","namespace":"tc1-ns","name":"fail-run-grwwm","error":"pipelineruns.tekton.dev \"fail-run-grwwm\" not found","stacktrace":"github.com/openshift-pipelines/tektoncd-pruner/pkg/reconciler/tektonpruner.cleanupPRs\n\t/go/src/github.com/openshift-pipelines/tektoncd-pruner/pkg/reconciler/tektonpruner/controller.go:236\ngithub.com/openshift-pipelines/tektoncd-pruner/pkg/reconciler/tektonpruner.runGarbageCollector.func1\n\t/go/src/github.com/openshift-pipelines/tektoncd-pruner/pkg/reconciler/tektonpruner/controller.go:124"}
{"severity":"ERROR","timestamp":"2025-06-11T13:03:28.414209055Z","logger":"tekton-pruner-controller","caller":"tektonpruner/controller.go:125","message":"Error collecting PipelineRuns","commit":"d355488-dirty","namespace":"tc1-ns","error":"pipelineruns.tekton.dev \"fail-run-grwwm\" not found","stacktrace":"github.com/openshift-pipelines/tektoncd-pruner/pkg/reconciler/tektonpruner.runGarbageCollector.func1\n\t/go/src/github.com/openshift-pipelines/tektoncd-pruner/pkg/reconciler/tektonpruner/controller.go:125"}
{"severity":"INFO","timestamp":"2025-06-11T13:03:28.414253549Z","logger":"tekton-pruner-controller","caller":"tektonpruner/controller.go:143","message":"Garbage collection completed","commit":"d355488-dirty"}
{"severity":"INFO","timestamp":"2025-06-11T13:03:28.41429724Z","logger":"tekton-pruner-controller","caller":"tektonpruner/controller.go:73","message":"Cleanup thread completed","commit":"d355488-dirty"}

              rh-ee-anataraj Anitha Natarajan
              rhn-support-sselvan Sri Vignesh Selvan
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: