Uploaded image for project: 'Red Hat Data Grid'
  1. Red Hat Data Grid
  2. JDG-5502

[Operator] Operator webhook server fails to start on Openshift 4.6

XMLWordPrintable

      The Operator pod will fail to start as the CA certs required by the webhook server cannot be found:

      I0726 14:32:05.675675       1 request.go:645] Throttling request took 1.035736567s, request: GET:https://172.30.0.1:443/apis/imageregistry.operator.openshift.io/v1?timeout=32s
      2022-07-26T14:32:07.082Z        INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
      2022-07-26T14:32:07.088Z        INFO    controllers.Infinispan  Defaults:       {"Annotations": {"infinispan.org/operatorPodTargetLabels":"com.redhat.component-name,com.redhat.component-type,com.redhat.component-version,com.redhat.product-name,com.redhat.product-version"}, "Labels": {"com.redhat.component-name":"Data_Grid","com.redhat.component-type":"application","com.redhat.component-version":"8.3.1","com.redhat.product-name":"Red_Hat_Runtimes","com.redhat.product-version":"2022-Q2"}}
      2022-07-26T14:32:07.098Z        DEBUG   k8sutil Found namespace {"Namespace": "openshift-operators"}
      2022-07-26T14:32:07.101Z        INFO    controller-runtime.builder      Registering a mutating webhook  {"GVK": "infinispan.org/v1, Kind=Infinispan", "path": "/mutate-infinispan-org-v1-infinispan"}
      2022-07-26T14:32:07.101Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/mutate-infinispan-org-v1-infinispan"}
      2022-07-26T14:32:07.101Z        INFO    controller-runtime.builder      Registering a validating webhook        {"GVK": "infinispan.org/v1, Kind=Infinispan", "path": "/validate-infinispan-org-v1-infinispan"}
      2022-07-26T14:32:07.101Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/validate-infinispan-org-v1-infinispan"}
      2022-07-26T14:32:07.101Z        INFO    controller-runtime.builder      skip registering a mutating webhook, admission.Defaulter interface is not implemented   {"GVK": "infinispan.org/v2alpha1, Kind=Batch"}
      2022-07-26T14:32:07.101Z        INFO    controller-runtime.builder      Registering a validating webhook        {"GVK": "infinispan.org/v2alpha1, Kind=Batch", "path": "/validate-infinispan-org-v2alpha1-batch"}
      2022-07-26T14:32:07.101Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/validate-infinispan-org-v2alpha1-batch"}
      2022-07-26T14:32:07.102Z        INFO    controller-runtime.builder      Registering a mutating webhook  {"GVK": "infinispan.org/v2alpha1, Kind=Cache", "path": "/mutate-infinispan-org-v2alpha1-cache"}
      2022-07-26T14:32:07.102Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/mutate-infinispan-org-v2alpha1-cache"}
      2022-07-26T14:32:07.102Z        INFO    controller-runtime.builder      skip registering a validating webhook, admission.Validator interface is not implemented {"GVK": "infinispan.org/v2alpha1, Kind=Cache"}
      2022-07-26T14:32:07.102Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/validate-infinispan-org-v2alpha1-cache"}
      2022-07-26T14:32:07.102Z        INFO    controller-runtime.builder      Registering a mutating webhook  {"GVK": "infinispan.org/v2alpha1, Kind=Backup", "path": "/mutate-infinispan-org-v2alpha1-backup"}
      2022-07-26T14:32:07.102Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/mutate-infinispan-org-v2alpha1-backup"}
      2022-07-26T14:32:07.102Z        INFO    controller-runtime.builder      Registering a validating webhook        {"GVK": "infinispan.org/v2alpha1, Kind=Backup", "path": "/validate-infinispan-org-v2alpha1-backup"}
      2022-07-26T14:32:07.102Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/validate-infinispan-org-v2alpha1-backup"}
      2022-07-26T14:32:07.102Z        INFO    controller-runtime.builder      Registering a mutating webhook  {"GVK": "infinispan.org/v2alpha1, Kind=Restore", "path": "/mutate-infinispan-org-v2alpha1-restore"}
      2022-07-26T14:32:07.102Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/mutate-infinispan-org-v2alpha1-restore"}
      2022-07-26T14:32:07.102Z        INFO    controller-runtime.builder      Registering a validating webhook        {"GVK": "infinispan.org/v2alpha1, Kind=Restore", "path": "/validate-infinispan-org-v2alpha1-restore"}
      2022-07-26T14:32:07.102Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/validate-infinispan-org-v2alpha1-restore"}
      2022-07-26T14:32:07.102Z        INFO    setup   Starting Infinispan Operator Version: RHDG-8.3.7.CD20220725
      I0726 14:32:07.102920       1 leaderelection.go:243] attempting to acquire leader lease  openshift-operators/632512e4.infinispan.org...
      2022-07-26T14:32:07.103Z        INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
      2022-07-26T14:32:07.203Z        INFO    controller-runtime.webhook.webhooks     starting webhook server
      2022-07-26T14:32:07.204Z        ERROR   setup   problem running manager {"error": "open /tmp/k8s-webhook-server/serving-certs/tls.crt: no such file or directory"}
      github.com/go-logr/zapr.(*zapLogger).Error
              /root/go/src/github.com/infinispan/infinispan-operator/vendor/github.com/go-logr/zapr/zapr.go:132
      github.com/infinispan/infinispan-operator/launcher/operator.NewWithContext
              /root/go/src/github.com/infinispan/infinispan-operator/launcher/operator/operator.go:183
      github.com/infinispan/infinispan-operator/launcher/operator.New
              /root/go/src/github.com/infinispan/infinispan-operator/launcher/operator/operator.go:58
      main.main
              /root/go/src/github.com/infinispan/infinispan-operator/main.go:41
      runtime.main
              /usr/lib/golang/src/runtime/proc.go:255
      

      The root cause is detailed here. The solution is to add an additional check to Operator startup that forces the old path /apiserver.local.config/certificates/ to be used if /tmp/k8s-webhook-server/serving-certs/ is not mounted in the pod.

              remerson@redhat.com Ryan Emerson
              remerson@redhat.com Ryan Emerson
              Pavel Drobek Pavel Drobek (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: