Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-33958

Router should continuously monitor the existence of the secret(spec.tls.externalCertificate)

XMLWordPrintable

    • ?
    • Important
    • No
    • CFE Sprint 254, CFE Sprint 255, CFE Sprint 256, CFE Sprint 257, CFE Sprint 258, NE Sprint 259, NE Sprint 260, NE Sprint 261, NE Sprint 262
    • 9
    • Rejected
    • True
    • Show
      https://issues.redhat.com/browse/NE-1787
    • 08/30 work in progress; no PxE inputs for now.

      Description of problem:

      If deleted the route referenced tls secret, route resource would failed with "spec.tls.externalCertificate: Not found: "secrets \"myroute-tls\" not found"" as expected.
      
      But if we re-create the secret with the same name, the route will not monitor the existence of the secret again.
      
      It will keep showing 'ExternalCertificateValidationFailed' and never return to normal, which is less user-friendly.

      Version-Release number of selected component (if applicable):

      OCP 4.16.0-0.nightly-2024-05-16-092402  

      How reproducible:

          Always

      Steps to Reproduce:

          1. Enable the "ExternalRouteCertificate" feature gate
      $ oc patch featuregate/cluster --type=merge -p '{"spec":{"featureSet":"TechPreviewNoUpgrade"}}'
      
          2. Create a route object from the "hello-openshift" demo app
      $ oc new-project test
      $ oc new-app openshift/hello-openshift
      $ oc create route edge myroute --service hello-openshift
      
          3. Create the TLS secret for route to reference
      $ oc create secret tls myroute-tls --cert=tls.crt --key=tls.key
      
          4. Patch the route object with created secret
      $ oc patch route myroute --type=merge --patch='{"spec":{"tls":{"externalCertificate":{"name":"myroute-tls"}}}}'
      
          5. Check if the route is ready, and serve the TLS certs
      $ oc get route
      NAME      HOST/PORT                                                    PATH   SERVICES          PORT       TERMINATION   WILDCARD
      myroute   myroute-test.apps.ptalgulk-bug.qe.devcluster.openshift.com          hello-openshift   8080-tcp   edge          None
      
      $ curl --cacert ./ca.crt https://myroute-test.apps.ptalgulk-bug.qe.devcluster.openshift.com
      Hello OpenShift!
      
           6. Delete the TLS secret. Check the route status
      $ oc delete secret myroute-tls
      secret "myroute-tls" deleted
      
      $ oc describe route myroute
      ...
      Requested Host:		myroute-test.apps.ptalgulk-bug.qe.devcluster.openshift.com			rejected by router default:  (host router-default.apps.ptalgulk-bug.qe.devcluster.openshift.com)ExternalCertificateValidationFailed (57 minutes ago)			  spec.tls.externalCertificate: Not found: "secrets \"myroute-tls\" not found"
      ...
      
      $ oc get route myroute -o yaml
      ...
      status:
        ingress:
        - conditions:
          - lastTransitionTime: "2024-05-20T09:37:17Z"
            message: 'spec.tls.externalCertificate: Not found: "secrets \"myroute-tls\"
              not found"'
            reason: ExternalCertificateValidationFailed
            status: "False"
            type: Admitted
      ...
      
           7. Re create the TLS secret manually
      $ oc create secret tls myroute-tls --cert=tls.crt --key=tls.key
      
           8. Check the route status, it still failed with "ExternalCertificateValidationFailed" message after ~30m
      $ oc get route
      NAME      HOST/PORT                             PATH   SERVICES          PORT       TERMINATION   WILDCARD
      myroute   ExternalCertificateValidationFailed          hello-openshift   8080-tcp   edge          None

      Actual results:

          route "myroute" keep failed with "ExternalCertificateValidationFailed"

      Expected results:

          route "myroute" should back to normal because the secret is re-created.

      Additional info:

          router pod logs:
      ...
      I0520 09:28:03.398627       1 monitor.go:76] starting informer
      I0520 09:28:03.399864       1 reflector.go:351] Caches populated for *v1.Secret from github.com/openshift/library-go/pkg/secret/monitor.go:79
      I0520 09:28:03.498762       1 secret_monitor.go:122] secret informer started item key {test myroute-tls}
      I0520 09:28:03.498790       1 secret_monitor.go:135] secret handler added item key {test myroute-tls}
      I0520 09:28:03.498798       1 manager.go:80] secret manager registered route for key test/myroute with secret myroute-tls
      I0520 09:28:03.638120       1 router.go:669] "msg"="router reloaded" "logger"="template" "output"=" - Checking http://localhost:80 using PROXY protocol ...\n - Health check ok : 0 retry attempt(s).\n"
      I0520 09:33:00.162473       1 router.go:669] "msg"="router reloaded" "logger"="template" "output"=" - Checking http://localhost:80 using PROXY protocol ...\n - Health check ok : 0 retry attempt(s).\n"
      I0520 09:37:17.820979       1 secret_monitor.go:165] secret handler removed item key{test myroute-tls}
      I0520 09:37:17.821018       1 monitor.go:93] informer stopped
      I0520 09:37:17.821023       1 secret_monitor.go:174] secret informer stopped item key {test myroute-tls}
      I0520 09:37:17.821028       1 manager.go:108] secret manager unregistered route for key test/myroute
      E0520 09:37:17.845149       1 route_secret_manager.go:124] "msg"="skipping route due to invalid externalCertificate configuration" "error"="spec.tls.externalCertificate: Not found: \"secrets \\\"myroute-tls\\\" not found\"" "logger"="controller" "namespace"="test" "route"="myroute"
      E0520 09:37:17.845191       1 router_controller.go:273] spec.tls.externalCertificate: Not found: "secrets \"myroute-tls\" not found"
      E0520 09:37:17.862319       1 route_secret_manager.go:124] "msg"="skipping route due to invalid externalCertificate configuration" "error"="spec.tls.externalCertificate: Not found: \"secrets \\\"myroute-tls\\\" not found\"" "logger"="controller" "namespace"="test" "route"="myroute"
      E0520 09:37:17.862358       1 router_controller.go:273] spec.tls.externalCertificate: Not found: "secrets \"myroute-tls\" not found"
      ...
      
      It seems that the informer stopped once the secret was deleted and didn't start again when the secret was re-created.

       

      Workaround:

      Recreate the route when the secret is recreated.

            rh-ee-ckyal Chirag Kyal
            rh-ee-yuewu Yuedong Wu
            Hongan Li Hongan Li
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: