Uploaded image for project: 'Knative Serving'
  1. Knative Serving
  2. SRVKS-679

autoscaler memory usage increases even with total number of ksvcs/revisions kept constant

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • 1.12.0
    • 1.12.0
    • None

      Possibly a https://issues.redhat.com/browse/SRVKS-609 regression.

       

      Running the SRVKS-609 reproducer ( https://issues.redhat.com/secure/attachment/12483247/SRVKS-609-reproducer.sh ) (a soak test that repeatedly creates and deletes ksvcs, keeping the total number of ksvcs constant) on Serverless 1.12, autoscaler gets OOMKilled after 4 hours.

       

      [root@ocp-dynamic-6653 ~]# go tool pprof --base "knative-serving.autoscaler-699dff8cff-hk27b.heap.2021-01-10_09:24:04-05:00.pb.gz" "knative-serving.autoscaler-699dff8cff-hk27b.heap.2021-01-10_10:47:41-05:00.pb.gz"                        
      File: autoscaler
      Type: inuse_space
      Time: Jan 10, 2021 at 9:24am (EST)
      Entering interactive mode (type "help" for commands, "o" for options)
      (pprof) top
      Showing nodes accounting for 47.90MB, 85.65% of 55.93MB total
      Dropped 5 nodes (cum <= 0.28MB)
      Showing top 10 nodes out of 130
            flat  flat%   sum%        cum   cum%
         26.89MB 48.08% 48.08%    26.89MB 48.08%  go.opencensus.io/stats/view.NewMeter
             6MB 10.73% 58.81%        6MB 10.73%  go.opencensus.io/stats/view.(*collector).addSample
          3.50MB  6.26% 65.07%        9MB 16.10%  go.opencensus.io/stats/view.(*worker).tryRegisterView
             3MB  5.36% 70.44%        3MB  5.36%  go.opencensus.io/stats/view.viewToMetricDescriptor
          2.50MB  4.47% 74.91%     2.50MB  4.47%  knative.dev/pkg/metrics.copyViews
          1.51MB  2.69% 77.60%     1.51MB  2.69%  k8s.io/apimachinery/pkg/apis/meta/v1.(*FieldsV1).UnmarshalJSON
          1.50MB  2.68% 80.28%     1.50MB  2.68%  go.opencensus.io/stats/view.(*worker).getMeasureRef (inline)
             1MB  1.79% 82.07%        1MB  1.79%  go.opencensus.io/stats/view.(*worker).RegisterExporter
             1MB  1.79% 83.86%        2MB  3.58%  go.opencensus.io/stats/view.(*collector).collectedRows
             1MB  1.79% 85.65%        4MB  7.15%  go.opencensus.io/stats/view.newViewInternal (inline)
      

        1. autoscaler_oomkilled.png
          48 kB
          Marek Schmidt
        2. autoscaler.heap.tar
          250 kB
          Marek Schmidt

              skontopo@redhat.com Stavros Kontopoulos
              maschmid@redhat.com Marek Schmidt
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: