Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-32887

OCP upgrade from 4.13 to 4.14 triggers the error "failed to update canary route openshift-ingress-canary/canary"

XMLWordPrintable

    • Moderate
    • No
    • 1
    • Sprint 252, Sprint 253, Sprint 254, NE Sprint 255
    • 4
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      Cause: A previous change to the Ingress Operator added logic to clear spec.host and instead set spec.subdomain on the canary route. However, the operator did not have permission to update spec.host or spec.subdomain on an existing route as the operator's serviceaccount did not have the necessary "routes/custom-host" permission.

      Consequence: Without the required permission, the update would fail with the following error message:

          ERROR operator.init controller/controller.go:265 Reconciler error
      {"controller": "canary_controller", "object": {"name":"default","namespace":"openshift-ingress-operator"},
      "namespace": "openshift-ingress-operator", "name": "default", "reconcileID": "463061e3-93a1-4067-802e-03e3f1f8cdd0",
      "error": "failed to ensure canary route: failed to update canary route openshift-ingress-canary/canary:
      Route.route.openshift.io \"canary\" is invalid: spec.subdomain: Invalid value: \"canary-openshift-ingress-canary\": field is immutable"}

      Fix: The needed permission was added to the clusterrole for the operator's serviceaccount.

      Result: The Ingress Operator can successfully update the canary route.
      Show
      Cause: A previous change to the Ingress Operator added logic to clear spec.host and instead set spec.subdomain on the canary route. However, the operator did not have permission to update spec.host or spec.subdomain on an existing route as the operator's serviceaccount did not have the necessary "routes/custom-host" permission. Consequence: Without the required permission, the update would fail with the following error message:     ERROR operator.init controller/controller.go:265 Reconciler error {"controller": "canary_controller", "object": {"name":"default","namespace":"openshift-ingress-operator"}, "namespace": "openshift-ingress-operator", "name": "default", "reconcileID": "463061e3-93a1-4067-802e-03e3f1f8cdd0", "error": "failed to ensure canary route: failed to update canary route openshift-ingress-canary/canary: Route.route.openshift.io \"canary\" is invalid: spec.subdomain: Invalid value: \"canary-openshift-ingress-canary\": field is immutable"} Fix: The needed permission was added to the clusterrole for the operator's serviceaccount. Result: The Ingress Operator can successfully update the canary route.
    • Bug Fix
    • In Progress

      Description of problem:

      In the OCP upgrades from 4.13 to 4.14, the canary route configuration is changed as below: 

       

      Canary route configuration in OCP 4.13
      $ oc get route -n openshift-ingress-canary canary -oyaml
      apiVersion: route.openshift.io/v1
      kind: Route
      metadata:
      labels:
      ingress.openshift.io/canary: canary_controller
      name: canary
      namespace: openshift-ingress-canary
      spec:
      host: canary-openshift-ingress-canary.apps.<cluster-domain>.com <---- canary route configured with .spec.host
      Canary route configuration in OCP 4.14:
      $ oc get route -n openshift-ingress-canary canary -oyaml
      apiVersion: route.openshift.io/v1
      kind: Route
      labels:
      ingress.openshift.io/canary: canary_controller
      name: canary
      namespace: openshift-ingress-canary
      spec:
      port:
      targetPort: 8080
      subdomain: canary-openshift-ingress-canary <---- canary route configured with .spec.subdomain
      

       

      After the upgrade, the following messages are printed in the ingress-operator pod: 

      2024-04-24T13:16:34.637Z        ERROR   operator.init   controller/controller.go:265    Reconciler error        {"controller": "canary_controller", "object": {"name":"default","namespace":"openshift-ingress-operator"}, "namespace": "openshift-ingress-operator", "name": "default", "reconcileID": "46290893-d755-4735-bb01-e8b707be4053", "error": "failed to ensure canary route: failed to update canary route openshift-ingress-canary/canary: Route.route.openshift.io \"canary\" is invalid: spec.subdomain: Invalid value: \"canary-openshift-ingress-canary\": field is immutable"}
       

      The issue is resolved when the canary route is deleted. 

      See below the audit logs from the process: 

      # The route can't be updated with error 422: 
      
      {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4e8bfb36-21cc-422b-9391-ef8ff42970ca","stage":"ResponseComplete","requestURI":"/apis/route.openshift.io/v1/namespaces/openshift-ingress-canary/routes/canary","verb":"update","user":{"username":"system:serviceaccount:openshift-ingress-operator:ingress-operator","groups":["system:serviceaccounts","system:serviceaccounts:openshift-ingress-operator","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["ingress-operator-746cd8598-hq2st"],"authentication.kubernetes.io/pod-uid":["f3ebccdf-f3b3-420d-8ea5-e33d98945403"]}},"sourceIPs":["10.128.0.93","10.128.0.2"],"userAgent":"Go-http-client/2.0","objectRef":{"resource":"routes","namespace":"openshift-ingress-canary","name":"canary","uid":"3e179946-d4e3-45ad-9380-c305baefd14e","apiGroup":"route.openshift.io","apiVersion":"v1","resourceVersion":"297888"},"responseStatus":{"metadata":{},"status":"Failure","message":"Route.route.openshift.io \"canary\" is invalid: spec.subdomain: Invalid value: \"canary-openshift-ingress-canary\": field is immutable","reason":"Invalid","details":{"name":"canary","group":"route.openshift.io","kind":"Route","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: \"canary-openshift-ingress-canary\": field is immutable","field":"spec.subdomain"}]},"code":422},"requestReceivedTimestamp":"2024-04-24T13:16:34.630249Z","stageTimestamp":"2024-04-24T13:16:34.636869Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"openshift-ingress-operator\" of ClusterRole \"openshift-ingress-operator\" to ServiceAccount \"ingress-operator/openshift-ingress-operator\""}}
      
      # Route is deleted manually
      
      "kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"70821b58-dabc-4593-ba6d-5e81e5d27d21","stage":"ResponseComplete","requestURI":"/aps/route.openshift.io/v1/namespaces/openshift-ingress-canary/routes/canary","verb":"delete","user":{"username":"system:admin","groups":["system:masters","syste:authenticated"]},"sourceIPs":["10.0.91.78","10.128.0.2"],"userAgent":"oc/4.13.0 (linux/amd64) kubernetes/7780c37","objectRef":{"resource":"routes","namespace:"openshift-ingress-canary","name":"canary","apiGroup":"route.openshift.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Success","details":{"ame":"canary","group":"route.openshift.io","kind":"routes","uid":"3e179946-d4e3-45ad-9380-c305baefd14e"},"code":200},"requestReceivedTimestamp":"2024-04-24T1324:39.558620Z","stageTimestamp":"2024-04-24T13:24:39.561267Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
      
      # Route is created again
      
      {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"92e6132a-aa1d-482d-a1dc-9ce021ae4c37","stage":"ResponseComplete","requestURI":"/aps/route.openshift.io/v1/namespaces/openshift-ingress-canary/routes","verb":"create","user":{"username":"system:serviceaccount:openshift-ingress-operator:ingres-operator","groups":["system:serviceaccounts","system:serviceaccounts:openshift-ingress-operator","system:authenticated"],"extra":{"authentication.kubernetesio/pod-name":["ingress-operator-746cd8598-hq2st"],"authentication.kubernetes.io/pod-uid":["f3ebccdf-f3b3-420d-8ea5-e33d98945403"]}},"sourceIPs":["10.128.0.93""10.128.0.2"],"userAgent":"Go-http-client/2.0","objectRef":{"resource":"routes","namespace":"openshift-ingress-canary","name":"canary","apiGroup":"route.opensift.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2024-04-24T13:24:39.577255Z","stageTimestamp":"2024-04-24T1:24:39.584371Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"openshift-ingress-perator\" of ClusterRole \"openshift-ingress-operator\" to ServiceAccount \"ingress-operator/openshift-ingress-operator\""}}
      
      

       

      Version-Release number of selected component (if applicable):

          Ocp upgrade between 4.13 and 4.14

      How reproducible:

          Upgrade the cluster from OCP 4.13 to 4.14 and check the ingress operator pod logs

      Steps to Reproduce:

          1. Install cluster in OCP 4.13
          2. Upgrade to OCP 4.14
          3. Check the ingress operator logs
          

      Actual results:

          Reported errors above

      Expected results:

          The ingress canary route should be update without isssues

      Additional info:

          

            mmasters1@redhat.com Miciah Masters
            rhn-support-bgomes Bruno Gomes
            Melvin Joseph Melvin Joseph
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: