Uploaded image for project: 'Knative Serving'
  1. Knative Serving
  2. SRVKS-609

knative-serving memory usage increases even with total number of ksvcs/revisions kept constant

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • 1.8.0
    • 1.7.2, 1.8.0
    • None
    • Serverless Sprint 188

      Having a soak test that repeatedly creates and deletes ksvcs. Even though the total number of ksvcs/revisions is kept constant, the knative-serving pods increases their memory usage. This is noticable especially on the webhook.

      Eventually the overhead causes the serving webhook to become CPU throttled and requests start failing with

      Internal error occurred: admission plugin "MutatingAdmissionWebhook" failed to complete mutation in 13s

        1. knative_serving_cpu_throttled_periods.png
          knative_serving_cpu_throttled_periods.png
          116 kB
        2. soak_apiserver_latencies_alerts.png
          soak_apiserver_latencies_alerts.png
          158 kB
        3. soak_knative_serving_memory_usage.png
          soak_knative_serving_memory_usage.png
          209 kB
        4. SRVKS-609.tar
          1.14 MB
        5. SRVKS-609-reproducer.sh
          1 kB
        6. webhook_cpu_mem.png
          webhook_cpu_mem.png
          52 kB

              markusthoemmes Markus Thömmes (Inactive)
              maschmid@redhat.com Marek Schmidt
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: