-
Bug
-
Resolution: Done
-
Critical
-
Pipelines 1.16.1
-
None
-
False
-
-
False
-
-
-
Important
-
Red Hat OpenShift Pipelines
Description of problem:
Customer reports that their Tekton Chains Controller is in a crash loop, it reports SIGSEGV continuously. It takes the leader lease, does some reconciling after start, and runs into SIGSEGV almost instantly. The container log shows:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x29586ed]
goroutine 156 [running]:
github.com/tektoncd/chains/pkg/chains/formats/slsa/v1/pipelinerun.buildConfig({0x3d77928?, 0xc044e8c870?}, 0xc044d68dc0)
/go/src/github.com/tektoncd/chains/pkg/chains/formats/slsa/v1/pipelinerun/pipelinerun.go:122 +0x5ad
github.com/tektoncd/chains/pkg/chains/formats/slsa/v1/pipelinerun.GenerateAttestation({0x3d77928, 0xc044e8c870}, 0xc044d68dc0, 0xc044d61980)
/go/src/github.com/tektoncd/chains/pkg/chains/formats/slsa/v1/pipelinerun/pipelinerun.go:69 +0x17a
github.com/tektoncd/chains/pkg/chains/formats/slsa/v1.(*InTotoIte6).CreatePayload(0xc044ae72d8, {0x3d77928, 0xc044e8c870}, {0x341ab60?, 0xc044e8a0c0})
/go/src/github.com/tektoncd/chains/pkg/chains/formats/slsa/v1/intotoite6.go:85 +0x4bb
github.com/tektoncd/chains/pkg/chains.(*ObjectSigner).Sign(0xc000c4b040, {0x3d77928, 0xc044e8c870}, {0x3db12a0, 0xc044e8a0c0})
/go/src/github.com/tektoncd/chains/pkg/chains/signing.go:150 +0x597
github.com/tektoncd/chains/pkg/reconciler/pipelinerun.(*Reconciler).FinalizeKind(0xc000c4b080, {0x3d77928, 0xc044e8c870}, 0xc019421208)
/go/src/github.com/tektoncd/chains/pkg/reconciler/pipelinerun/pipelinerun.go:109 +0x80e
github.com/tektoncd/chains/pkg/reconciler/pipelinerun.(*Reconciler).ReconcileKind(0xc000c4b080, {0x3d77928, 0xc044e8c090}, 0xc019421208)
/go/src/github.com/tektoncd/chains/pkg/reconciler/pipelinerun/pipelinerun.go:52 +0x175
github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1/pipelinerun.(*reconcilerImpl).Reconcile(0xc0004d3d60, {0x3d77928, 0xc044bf3f50}, {0xc044bf5180, 0x49})
/go/src/github.com/tektoncd/chains/vendor/github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1/pipelinerun/reconciler.go:236 +0x4f2
knative.dev/pkg/controller.(*Impl).processNextWorkItem(0xc000c6c480)
/go/src/github.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:542 +0x508
knative.dev/pkg/controller.(*Impl).RunContext.func3()
/go/src/github.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:491 +0x59
created by knative.dev/pkg/controller.(*Impl).RunContext in goroutine 154
/go/src/github.com/tektoncd/chains/vendor/knative.dev/pkg/controller/controller.go:489 +0x325
The cluster is filling up with PipelineRuns which cannot be reconciled due to the controlller being in a crash loop.
This issue is described in the following upstream issue: https://github.com/tektoncd/chains/issues/1243
Workaround
Current workaround is to disable Tekton Chains.
Prerequisites (if any, like setup, operators/versions):
OpenShift Container Platform 4.16.21
openshift-pipelines-operator-rh.v1.16.1
Steps to Reproduce
Only reproducible on customer cluster
Actual results:
Tekton Chains Controller is crashing with SIGSEGV mentioned above.
Expected results:
Tekton Chains Controller is reporting an issue when encountering an unexpected state, but does not crash.
Reproducibility (Always/Intermittent/Only Once):
Only on customer cluster