Uploaded image for project: 'Knative Serving'
  1. Knative Serving
  2. SRVKS-1334

3scale-kourier-gateway losing Readiness/Liveness during LDS updates

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • 1.38.0
    • 1.37.0, 1.38.0
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      When new ksvcs are modified (or added/removed), with larger number of ksvcs on the cluster (~500ksvcs),
      the LDS updates in 3scale-kourier-gateway take longer than 1s, the gateway fail readiness/liveness probes (and possible also blocks serving traffic), causing restarts of 3scale-kourier-gateway due to liveness failures.

      With internal-tls-encryption enabled, during a script creating 100 namespaces with 10 ksvcs each (10ksvcs at a time, then waiting for their readiness)

      Events:
        Type     Reason          Age                   From               Message
        ----     ------          ----                  ----               -------
        Normal   Scheduled       13m                   default-scheduler  Successfully assigned knative-serving-ingress/3scale-kourier-gateway-6cb499dc6d-s8cl9 to maschmid-1178-xwr8p-worker-0-j7mjj
        Normal   AddedInterface  13m                   multus             Add eth0 [10.129.5.142/23] from ovn-kubernetes
        Warning  Unhealthy       11m (x11 over 13m)    kubelet            Liveness probe failed: Get "http://10.129.5.142:8081/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
        Warning  Unhealthy       3m32s (x45 over 13m)  kubelet            Readiness probe failed: Get "http://10.129.5.142:8081/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
        Normal   Killing         113s                  kubelet            Container kourier-gateway failed liveness probe, will be restarted
        Normal   Pulled          96s (x2 over 13m)     kubelet            Container image "registry.redhat.io/openshift-service-mesh/proxyv2-rhel9@sha256:fe1e78971352ddd76ee6f88aa0f4e4ac6d1298da8c960f877ae9f883c291dfd6" already present on machine
        Normal   Created         96s (x2 over 13m)     kubelet            Created container: kourier-gateway
        Normal   Started         96s (x2 over 13m)     kubelet            Started container kourier-gateway
      

        1. create-1000-ksvcs.sh
          0.5 kB
          Marek Schmidt
        2. create-and-delete-ksvc.sh
          0.7 kB
          Marek Schmidt

              dsimansk@redhat.com David Simansky
              maschmid@redhat.com Marek Schmidt
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: