Uploaded image for project: 'OpenShift Service Mesh'
  1. OpenShift Service Mesh
  2. OSSM-1974

All IOR routes are recreated when changes are made in the SMCP

XMLWordPrintable

    • Icon: Ticket Ticket
    • Resolution: Done
    • Icon: Critical Critical
    • OSSM 2.4.0
    • OSSM 2.2.2
    • Maistra
    • None

      When making changes to the SMCP like adding ingress/egress gateways to a namespace, all the routes created by IOR are deleted and crated again. 

      Depending on the number of routes it takes several seconds for the routes to be crated, which can cause interruptions in the service. 

      Moreover, customers have reported that when the new configuration in the SMCP is not fully applied (the new ingress/egress gateways have any issue), some of the routes are not created again. 

      I have tried to reproduce this behaviour, but I haven't succeeded, as in my case all the routes are always created. 

      I'm using OCP 4.11.3, and SM 2.2.2. 

       

      Steps to reproduce the issue:

      • Create a bunch of namespaces:
      # for i in {1..101}; do oc new-project mesh-routes$i; done
      • Add those namespaces to the SMMR
      • Create a Gateway in each of those namespaces
      # for i in {1..101}; do
oc apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  annotations:
  name: gw-http-$i
  namespace: mesh-routes$i
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - 'http$i.test.ocp'
    port:
      name: http
      number: 80
      protocol: HTTP
EOF
done

       

      • Wait for all the routes to be created, and check the creation time:

       

      # oc get -o yaml -n istio-system route  | egrep '(name: mesh-routes|creationTime)'
    creationTimestamp: "2022-09-08T14:53:57Z"
    name: mesh-routes1-gw-http-1-d6539c0787dce9d9
    creationTimestamp: "2022-09-08T14:54:00Z"
    name: mesh-routes10-gw-http-10-4de5a53b03f8fab2
    creationTimestamp: "2022-09-08T14:54:00Z"
    name: mesh-routes11-gw-http-11-e2d8b8f84c1dab08
    creationTimestamp: "2022-09-08T14:54:01Z"
    name: mesh-routes12-gw-http-12-ce1dfeb24371c87c
    creationTimestamp: "2022-09-08T14:54:05Z"
    name: mesh-routes13-gw-http-13-14b1c74f39fb88ca
    creationTimestamp: "2022-09-08T14:54:06Z"
    name: mesh-routes14-gw-http-14-edba1bb1c287a307
    creationTimestamp: "2022-09-08T14:53:59Z"
    name: mesh-routes15-gw-http-15-d25ccdbce5f252a0
    creationTimestamp: "2022-09-08T14:53:56Z"
    name: mesh-routes16-gw-http-16-cf19d89ed9a11e25
    creationTimestamp: "2022-09-08T14:54:06Z"
[...]

       

       

      • Add an ingressgateway and an egressgateway to a namespace in the mesh (or delete the istiod pod).
      spec:
  gateways:
    additionalEgress:
      eg0001:
        enabled: true
        namespace: <namespace>
        runtime:
          container:
            resources:
              limits:
                cpu: 1000m
                memory: 1028Mi
              requests:
                cpu: 300m
                memory: 250Mi
          deployment:
            autoScaling:
              enabled: false
            replicas: 1
    additionalIngress:
      ig0001:
        enabled: true
        namespace: <namespace>
        runtime:
          container:
            resources:
              limits:
                cpu: 1000m
                memory: 1024Mi
              requests:
                cpu: 200m
                memory: 150Mi
          deployment:
            autoScaling:
              enabled: false

       

       

      • Wait for the SMCP to reconcile.
      • Check the creation time of the routes:

       

      # oc get -o yaml -n istio-system route | egrep '(name: mesh-routes|creationTime)'
[...]
    creationTimestamp: "2022-09-09T08:47:36Z"
    name: mesh-routes1-gw-http-1-d6539c0787dce9d9
    creationTimestamp: "2022-09-09T08:47:41Z"
    name: mesh-routes10-gw-http-10-4de5a53b03f8fab2
    creationTimestamp: "2022-09-09T08:47:43Z"
    name: mesh-routes100-gw-http-100-241da8d56f9aca1d
    creationTimestamp: "2022-09-09T08:47:50Z"
    name: mesh-routes101-gw-http-101-830ca5627eed0ad8
    creationTimestamp: "2022-09-09T08:47:48Z"
    name: mesh-routes11-gw-http-11-e2d8b8f84c1dab08
    creationTimestamp: "2022-09-09T08:47:36Z"
[...]
    name: mesh-routes94-gw-http-94-7ad3a4dc27392063
    creationTimestamp: "2022-09-09T08:47:32Z"
    name: mesh-routes95-gw-http-95-e2a9f24b74f02500
    creationTimestamp: "2022-09-09T08:47:38Z"
    name: mesh-routes96-gw-http-96-4bd59e5c5378431a
    creationTimestamp: "2022-09-09T08:47:39Z"
    name: mesh-routes97-gw-http-97-71d32b145ad7d5ff
    creationTimestamp: "2022-09-09T08:47:35Z"
    name: mesh-routes98-gw-http-98-d754ec7dcb7cdae1
    creationTimestamp: "2022-09-09T08:47:38Z"
    name: mesh-routes99-gw-http-99-dc76dc55019e97fe
    creationTimestamp: "2022-09-06T12:00:27Z"

       

      It takes 16+ seconds to create all 101 routes in an idle cluster. 

      This will get worse the more routes are created by IOR and the busier the cluster gets. 

       

      Also, if routes were not deleted in the first place customers would not get a situation where some of the routes are not recreated. 

       

      This issue looks like a mix of these two:

      https://issues.redhat.com/browse/OSSM-1442

      https://issues.redhat.com/browse/OSSM-1693

       

       

       

              rh-ee-yannliu Yann Liu
              rhn-support-asolanas Alexis Solanas
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: