Expected Behavior

When tektoncd/pipeline is installed, the webhook will register itself, generate certificates (if need be) and add them to the CRDs (so that everything works).

Actual Behavior

On 1.14.x it fails at some point (with StepActions) and thus does not completely register all the objects. ClusterTask, CustomRun, VerificationPolicy (didn't know this one existed 😅 ) and StepAction CRDs are not correctly setup.

This generates "log" spam in the webhook log, as well as in the kube apiserver operator (at least in OpenShift).

From kube apiserver operator:
 {{[...]
W0322 14:51:48.986682 1 degraded_webhook.go:135] failed to connect to webhook "clustertasks.tekton.dev" via service "tekton-pipelines-webhook.openshift-pipelines.svc:443": tls: failed to verify certificate: x509: certificate signed by unknown authority}}
 
From the webhook:
 {{

{"severity":"info","timestamp":"2024-03-22T14:22:49.905Z","logger":"tekton-pipelines-webhook.ConversionWebhook","caller":"controller/controller.go:550","message":"Reconcile succeeded","commit":"f3dd478","knative.dev/traceid":"48ea1f50-72b7-4ea7-b88d-20e73b231adb","knative.dev/key":"tasks.tekton.dev","duration":0.013954525}

{"severity":"info","timestamp":"2024-03-22T14:22:49.905Z","logger":"tekton-pipelines-webhook.ConversionWebhook","caller":"controller/controller.go:550","message":"Reconcile succeeded","commit":"f3dd478","knative.dev/traceid":"67b5f588-7dc8-4c28-970f-2f2caba5c43c","knative.dev/key":"pipelines.tekton.dev","duration":0.046292156}
{"severity":"info","timestamp":"2024-03-22T14:22:49.909Z","logger":"tekton-pipelines-webhook.ConversionWebhook","caller":"conversion/reconciler.go:115","message":"CRD is up to date","commit":"f3dd478","knative.dev/traceid":"6b81f1a9-864d-4f5d-8bb5-de276202aa49","knative.dev/key":"taskruns.tekton.dev"}
{"severity":"info","timestamp":"2024-03-22T14:22:49.909Z","logger":"tekton-pipelines-webhook.ConversionWebhook","caller":"controller/controller.go:550","message":"Reconcile succeeded","commit":"f3dd478","knative.dev/traceid":"6b81f1a9-864d-4f5d-8bb5-de276202aa49","knative.dev/key":"taskruns.tekton.dev","duration":0.004403825}
{"severity":"info","timestamp":"2024-03-22T14:22:49.910Z","logger":"tekton-pipelines-webhook.ConversionWebhook","caller":"conversion/reconciler.go:115","message":"CRD is up to date","commit":"f3dd478","knative.dev/traceid":"19e1ba34-9371-490e-a25d-9fef1c2e1971","knative.dev/key":"pipelineruns.tekton.dev"}
{"severity":"info","timestamp":"2024-03-22T14:22:49.910Z","logger":"tekton-pipelines-webhook.ConversionWebhook","caller":"controller/controller.go:550","message":"Reconcile succeeded","commit":"f3dd478","knative.dev/traceid":"19e1ba34-9371-490e-a25d-9fef1c2e1971","knative.dev/key":"pipelineruns.tekton.dev","duration":0.00444386}
{"severity":"info","timestamp":"2024-03-22T14:22:49.913Z","logger":"tekton-pipelines-webhook.ConversionWebhook","caller":"conversion/reconciler.go:115","message":"CRD is up to date","commit":"f3dd478","knative.dev/traceid":"2f587e53-d40b-4ea9-a6b5-ebb734ed3fda","knative.dev/key":"resolutionrequests.resolution.tekton.dev"}
{"severity":"info","timestamp":"2024-03-22T14:22:49.913Z","logger":"tekton-pipelines-webhook.ConversionWebhook","caller":"controller/controller.go:550","message":"Reconcile succeeded","commit":"f3dd478","knative.dev/traceid":"2f587e53-d40b-4ea9-a6b5-ebb734ed3fda","knative.dev/key":"resolutionrequests.resolution.tekton.dev","duration":0.004400819}
{"severity":"error","timestamp":"2024-03-22T14:22:49.914Z","logger":"tekton-pipelines-webhook.ConversionWebhook","caller":"controller/controller.go:566","message":"Reconcile error","commit":"f3dd478","knative.dev/traceid":"fe498f2a-b626-41ce-9ced-6be4871e38db","knative.dev/key":"stepactions.tekton.dev","duration":0.000032431,"error":"custom resource \"stepactions.tekton.dev\" isn't configured for webhook conversion","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\t/go/src/github.com/tektoncd/pipeline/vendor/knative.dev/pkg/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/go/src/github.com/tektoncd/pipeline/vendor/knative.dev/pkg/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\t/go/src/github.com/tektoncd/pipeline/vendor/knative.dev/pkg/controller/controller.go:491"}
{"severity":"error","timestamp":"2024-03-22T14:22:49.914Z","logger":"tekton-pipelines-webhook.ConfigMapWebhook","caller":"controller/controller.go:566","message":"Reconcile error","commit":"f3dd478","knative.dev/traceid":"e8f465b8-efbe-4d42-86d5-153bcaea0278","knative.dev/key":"config.webhook.pipeline.tekton.dev","duration":0.064861311,"error":"failed to update webhook: Operation cannot be fulfilled on validatingwebhookconfigurations.admissionregistration.k8s.io \"config.webhook.pipeline.tekton.dev\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\t/go/src/github.com/tektoncd/pipeline/vendor/knative.dev/pkg/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/go/src/github.com/tektoncd/pipeline/vendor/knative.dev/pkg/controller/controller.go:543\nknative}}

See https://github.com/tektoncd/pipeline/issues/7794