Details
-
Bug
-
Resolution: Done
-
Blocker
-
None
-
Pipelines 1.14.1
-
None
-
False
-
None
-
False
-
2
-
Pipelines Sprint 259, (temp)Backlog from 260 and 259
Description
Description of problem:
Pipelines operator was updated from 1.14.0 to 1.14.1 immediately after update was available. All installer sets were ready and TektonConfig reported its status as ready as well.
After 12h I noticed that tekton-results-watcher pod is in crashloopbackoff state and installer set result-xxx is failing
Prerequisites (if any, like setup, operators/versions):
cluster https://console-openshift-console.apps.cicd.ospqa.com
OpenShift 4.14.15
Pipelines 1.14.1
Steps to Reproduce
# install 1.14.0 and upgrade to 1.14.1
Actual results:
results pod failing
Expected results:
all pods running
Reproducibility (Always/Intermittent/Only Once):
unknown
Additional info (Such as Logs, Screenshots, etc):
{"level":"info","time":"2024-03-15T11:47:59.995Z","logger":"watcher","caller":"pipelinerun/reconciler.go:74","msg":"Initiating reconciliation for PipelineRun 'svasisht/uninstall-cluster-cronjob-wpk22'","commit":"5619404","knative.dev/traceid":"6993f75e-23ae-4dab-b1d2-08a796cebcf2","knative.dev/key":"svasisht/uninstall-cluster-cronjob-wpk22","results.tekton.dev/kind":"PipelineRun"}
{"level":"info","time":"2024-03-15T11:48:00.029Z","logger":"watcher","caller":"dynamic/dynamic.go:176","msg":"Skipping resource deletion: deletion is disabled","commit":"5619404","knative.dev/traceid":"f4ab1ebb-14bf-4a50-b3d5-9180ca768686","knative.dev/key":"svasisht/uninstall-cluster-cronjob-dg69l-uninstall","results.tekton.dev/kind":"TaskRun","results.tekton.dev/result":"svasisht/results/ff93e714-ab88-48ae-8577-b2b38ed290a2","results.tekton.dev/record":"svasisht/results/ff93e714-ab88-48ae-8577-b2b38ed290a2/records/90936994-06ec-4380-9c46-f1451b3979b3","results.tekton.dev/gracePeriod":"0s"}
{"level":"info","time":"2024-03-15T11:48:00.029Z","logger":"watcher","caller":"controller/controller.go:550","msg":"Reconcile succeeded","commit":"5619404","knative.dev/traceid":"f4ab1ebb-14bf-4a50-b3d5-9180ca768686","knative.dev/key":"svasisht/uninstall-cluster-cronjob-dg69l-uninstall","duration":"101.700612ms"}
{"level":"info","time":"2024-03-15T11:48:00.029Z","logger":"watcher","caller":"taskrun/reconciler.go:59","msg":"Initiating reconciliation for TaskRun 'pipelines-ci/accepta2a27748d93f44880b997ce65952c926-release-tests-auto-prune'","commit":"5619404","knative.dev/traceid":"e56db1ea-b897-4e12-9366-32f92fc58d5e","knative.dev/key":"pipelines-ci/accepta2a27748d93f44880b997ce65952c926-release-tests-auto-prune","results.tekton.dev/kind":"TaskRun"}
{"level":"info","time":"2024-03-15T11:48:00.044Z","logger":"watcher","caller":"dynamic/dynamic.go:176","msg":"Skipping resource deletion: deletion is disabled","commit":"5619404","knative.dev/traceid":"d1efdadc-82c4-4e26-95ee-c2622c017587","knative.dev/key":"svasisht/uninstall-cluster-cronjob-sql2g","results.tekton.dev/kind":"PipelineRun","results.tekton.dev/result":"svasisht/results/20928fdc-d466-41c1-86b1-4bb03c30f237","results.tekton.dev/record":"svasisht/results/20928fdc-d466-41c1-86b1-4bb03c30f237/records/20928fdc-d466-41c1-86b1-4bb03c30f237","results.tekton.dev/gracePeriod":"0s"}
{"level":"info","time":"2024-03-15T11:48:00.044Z","logger":"watcher","caller":"controller/controller.go:550","msg":"Reconcile succeeded","commit":"5619404","knative.dev/traceid":"d1efdadc-82c4-4e26-95ee-c2622c017587","knative.dev/key":"svasisht/uninstall-cluster-cronjob-sql2g","duration":"57.767065ms"}
{"level":"info","time":"2024-03-15T11:48:00.044Z","logger":"watcher","caller":"pipelinerun/reconciler.go:74","msg":"Initiating reconciliation for PipelineRun 'pipelines-ci/create-push-ci-image-gbhpn'","commit":"5619404","knative.dev/traceid":"0d272897-66a1-424b-b794-b8ef62c3bdd8","knative.dev/key":"pipelines-ci/create-push-ci-image-gbhpn","results.tekton.dev/kind":"PipelineRun"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x3105631]
goroutine 195 [running]:
github.com/tektoncd/results/pkg/watcher/results.(*Client).ensureResult(0xc00ec6a4a0, {0x3d4bb20, 0xc011120780}, {0x3d77040?, 0xc00400afc8}, {0x0, 0x0, 0x0})
/go/src/github.com/tektoncd/results/pkg/watcher/results/results.go:149 +0x871
github.com/tektoncd/results/pkg/watcher/results.(*Client).Put(0xc011120780?, {0x3d4bb20, 0xc011120780}, {0x3d77040, 0xc00400afc8}, {0x0, 0x0, 0x0})
/go/src/github.com/tektoncd/results/pkg/watcher/results/results.go:81 +0x57
github.com/tektoncd/results/pkg/watcher/reconciler/dynamic.(*Reconciler).Reconcile(0xc011120750, {0x3d4bb20, 0xc011120780}, {0x3d77040?, 0xc00400afc8})
/go/src/github.com/tektoncd/results/pkg/watcher/reconciler/dynamic/dynamic.go:107 +0x41d
github.com/tektoncd/results/pkg/watcher/reconciler/pipelinerun.(*Reconciler).Reconcile(0xc0008a4dc0, {0x3d4bb20, 0xc0111206f0}, {0xc00c600180, 0x27})
/go/src/github.com/tektoncd/results/pkg/watcher/reconciler/pipelinerun/reconciler.go:100 +0x69b
knative.dev/pkg/controller.(*Impl).processNextWorkItem(0xc000411380)
/go/src/github.com/tektoncd/results/vendor/knative.dev/pkg/controller/controller.go:542 +0x4cd
knative.dev/pkg/controller.(*Impl).RunContext.func3()
/go/src/github.com/tektoncd/results/vendor/knative.dev/pkg/controller/controller.go:491 +0x68
created by knative.dev/pkg/controller.(*Impl).RunContext
/go/src/github.com/tektoncd/results/vendor/knative.dev/pkg/controller/controller.go:489 +0x354
When pod is recreated, it fails again with the same stack trace but the last info message is again about the same pipeline run, I'm not sure if that is significant
{"level":"info","time":"2024-03-15T11:54:27.492Z","logger":"watcher","caller":"pipelinerun/reconciler.go:74","msg":"Initiating reconciliation for PipelineRun 'pipelines-ci/create-push-ci-image-gbhpn'","commit":"5619404","knative.dev/traceid":"98e39bb2-b1e9-4cf5-9653-7ac459e99bdd","knative.dev/key":"pipelines-ci/create-push-ci-image-gbhpn","results.tekton.dev/kind":"PipelineRun"}
