Uploaded image for project: 'Knative Serving'
  1. Knative Serving
  2. SRVKS-609

knative-serving memory usage increases even with total number of ksvcs/revisions kept constant

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • 1.8.0
    • 1.7.2, 1.8.0
    • None
    • Serverless Sprint 188

      Having a soak test that repeatedly creates and deletes ksvcs. Even though the total number of ksvcs/revisions is kept constant, the knative-serving pods increases their memory usage. This is noticable especially on the webhook.

      Eventually the overhead causes the serving webhook to become CPU throttled and requests start failing with

      Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s

        1. knative_serving_cpu_throttled_periods.png
          116 kB
          Marek Schmidt
        2. soak_apiserver_latencies_alerts.png
          158 kB
          Marek Schmidt
        3. soak_knative_serving_memory_usage.png
          209 kB
          Marek Schmidt
        4. SRVKS-609.tar
          1.14 MB
          Marek Schmidt
        5. SRVKS-609-reproducer.sh
          1 kB
          Marek Schmidt
        6. webhook_cpu_mem.png
          52 kB
          Martin Gencur

              markusthoemmes Markus Thömmes (Inactive)
              maschmid@redhat.com Marek Schmidt
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: