Uploaded image for project: 'Knative Serving'
  1. Knative Serving
  2. SRVKS-390

Cert rotation is broken for internal registry somewhere

XMLWordPrintable

      Issue:

      Get https://image-registry.openshift-image-registry.svc:5000/v2/: x509: certificate signed by unknown authority
      

      when you try to create a knative service using an image from internal registry. Example

      % kn service create quarkusbackend-sl --image=image-registry.openshift-image-registry.svc:5000/quarkusbackend/quarkusbackend \
       -e QUARKUS_DATASOURCE_URL=jdbc:postgresql://quarkusbackend-database/my_data \
       -e QUARKUS_DATASOURCE_USERNAME=dbuser \
       -e QUARKUS_DATASOURCE_PASSWORD=secret
      Creating service 'quarkusbackend-sl' in namespace 'quarkusbackend':
        0.201s The Route is still working to reflect the latest desired specification.
        0.272s Configuration "quarkusbackend-sl" is waiting for a Revision to become ready.
      RevisionFailed: Revision "quarkusbackend-sl-kbvjs-1" failed with message: Unable to fetch image "image-registry.openshift-image-registry.svc:5000/quarkusbackend/quarkusbackend": Get https://image-registry.openshift-image-registry.svc:5000/v2/: x509: certificate signed by unknown authority.
      

      Controller shows these errors

      {"level":"info","ts":"2020-01-10T21:00:03.413Z","logger":"controller.revision-controller","caller":"revision/revision.go:83","msg":"Running reconcile Revision","knative.dev/controller":"revision-controller","knative.dev/traceid":"33ce379b-cff0-41d4-87ea-c4758aa0eadb","knative.dev/key":"quarkusbackend/quarkus-sl-bflhx-1"}
      {"level":"error","ts":"2020-01-10T21:00:03.529Z","logger":"controller.revision-controller","caller":"controller/controller.go:357","msg":"Reconcile error","knative.dev/controller":"revision-controller","error":"Get https://image-registry.openshift-image-registry.svc:5000/v2/: x509: certificate signed by unknown authority","stacktrace":"knative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).handleErr\n\t/opt/app-root/src/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:357\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).processNextWorkItem\n\t/opt/app-root/src/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:343\nknative.dev/serving/vendor/knative.dev/pkg/controller.(*Impl).Run.func2\n\t/opt/app-root/src/go/src/knative.dev/serving/vendor/knative.dev/pkg/controller/controller.go:291"}
      {"level":"info","ts":"2020-01-10T21:00:03.529Z","logger":"controller.revision-controller","caller":"controller/controller.go:344","msg":"Reconcile failed. Time taken: 116.853769ms.","knative.dev/controller":"revision-controller","knative.dev/traceid":"33ce379b-cff0-41d4-87ea-c4758aa0eadb","knative.dev/key":"quarkusbackend/quarkus-sl-bflhx-1"}
      {"level":"info","ts":"2020-01-10T21:00:03.530Z","logger":"controller.revision-controller.event-broadcaster","caller":"record/event.go:258","msg":"Event(v1.ObjectReference{Kind:\"Revision\", Namespace:\"quarkusbackend\", Name:\"quarkus-sl-bflhx-1\", UID:\"68b0269b-33eb-11ea-a975-52540017cd81\", APIVersion:\"serving.knative.dev/v1alpha1\", ResourceVersion:\"199521689\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' Get https://image-registry.openshift-image-registry.svc:5000/v2/: x509: certificate signed by unknown authority","knative.dev/controller":"revision-controller"}
      

      Workaround:
      I had to remove the configmap config-service-ca and bounce knative-serving-operator pod to recreate a new configmap and then delete the controller pod. Then the new controller that got created had the right certificate to access internal registry.

              rhn-support-knakayam Kenjiro Nakayama (Inactive)
              vmuchand Veer Muchandi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: