Uploaded image for project: 'Knative Serving'
  1. Knative Serving
  2. SRVKS-513

knative-serving webhook CPU throttled, asked for AdmissionReview for knative-serving/config-service-ca 10 times a second

XMLWordPrintable

    • Serverless Sprint 182

      With Serverless 1.7.0 (current CI builds), on OCP 4.3.9, after ~11h runtime

      an alert is shown for knative-serving webhook

      CPUThrottlingHigh
      39.03% throttling of CPU in namespace knative-serving for container webhook in pod webhook-67fb8b566d-lf7fq.

      It seems it is being repeatedly hammered by admission reviews for UPDATE on knative-serving/config-service-ca (at rate of about 10 per second)

      {"level":"info","ts":"2020-04-08T19:30:58.021Z","logger":"webhook","caller":"webhook/admission.go:76","msg":"AdmissionReview for v1.GroupVersionKind{Group:\"\", Version:\"v1\", Kind:\"ConfigMap\"}: knative-serving/config-service-ca response=&v1beta1.AdmissionResponse{UID:\"\", Allowed:true, Result:(*v1.Status)(nil), Patch:[]uint8(nil), PatchType:(*v1beta1.PatchType)(nil), AuditAnnotations:map[string]string(nil)}","knative.dev/kind":"/v1, Kind=ConfigMap","knative.dev/namespace":"knative-serving","knative.dev/name":"config-service-ca","knative.dev/operation":"UPDATE","knative.dev/resource":"/v1, Resource=configmaps","knative.dev/subresource":"","knative.dev/userinfo":"{system:serviceaccount:openshift-network-operator:default 85f8eb21-4ac6-47e0-93e6-eefdb578a712 [system:serviceaccounts system:serviceaccounts:openshift-network-operator system:authenticated] map[]}"}
{"level":"info","ts":"2020-04-08T19:30:58.193Z","logger":"webhook","caller":"webhook/admission.go:76","msg":"AdmissionReview for v1.GroupVersionKind{Group:\"\", Version:\"v1\", Kind:\"ConfigMap\"}: knative-serving/config-service-ca response=&v1beta1.AdmissionResponse{UID:\"\", Allowed:true, Result:(*v1.Status)(nil), Patch:[]uint8(nil), PatchType:(*v1beta1.PatchType)(nil), AuditAnnotations:map[string]string(nil)}","knative.dev/kind":"/v1, Kind=ConfigMap","knative.dev/namespace":"knative-serving","knative.dev/name":"config-service-ca","knative.dev/operation":"UPDATE","knative.dev/resource":"/v1, Resource=configmaps","knative.dev/subresource":"","knative.dev/userinfo":"{system:serviceaccount:openshift-service-ca:configmap-cabundle-injector-sa 146755c7-fbee-4bf1-b715-96f3d747f703 [system:serviceaccounts system:serviceaccounts:openshift-service-ca system:authenticated] map[]}"}
{"level":"info","ts":"2020-04-08T19:30:58.225Z","logger":"webhook","caller":"webhook/admission.go:76","msg":"AdmissionReview for v1.GroupVersionKind{Group:\"\", Version:\"v1\", Kind:\"ConfigMap\"}: knative-serving/config-service-ca response=&v1beta1.AdmissionResponse{UID:\"\", Allowed:true, Result:(*v1.Status)(nil), Patch:[]uint8(nil), PatchType:(*v1beta1.PatchType)(nil), AuditAnnotations:map[string]string(nil)}","knative.dev/kind":"/v1, Kind=ConfigMap","knative.dev/namespace":"knative-serving","knative.dev/name":"config-service-ca","knative.dev/operation":"UPDATE","knative.dev/resource":"/v1, Resource=configmaps","knative.dev/subresource":"","knative.dev/userinfo":"{system:serviceaccount:openshift-network-operator:default 85f8eb21-4ac6-47e0-93e6-eefdb578a712 [system:serviceaccounts system:serviceaccounts:openshift-network-operator system:authenticated] map[]}"}
{"level":"info","ts":"2020-04-08T19:30:58.396Z","logger":"webhook","caller":"webhook/admission.go:76","msg":"AdmissionReview for v1.GroupVersionKind{Group:\"\", Version:\"v1\", Kind:\"ConfigMap\"}: knative-serving/config-service-ca response=&v1beta1.AdmissionResponse{UID:\"\", Allowed:true, Result:(*v1.Status)(nil), Patch:[]uint8(nil), PatchType:(*v1beta1.PatchType)(nil), AuditAnnotations:map[string]string(nil)}","knative.dev/kind":"/v1, Kind=ConfigMap","knative.dev/namespace":"knative-serving","knative.dev/name":"config-service-ca","knative.dev/operation":"UPDATE","knative.dev/resource":"/v1, Resource=configmaps","knative.dev/subresource":"","knative.dev/userinfo":"{system:serviceaccount:openshift-service-ca:configmap-cabundle-injector-sa 146755c7-fbee-4bf1-b715-96f3d747f703 [system:serviceaccounts system:serviceaccounts:openshift-service-ca system:authenticated] map[]}"}
{"level":"info","ts":"2020-04-08T19:30:58.426Z","logger":"webhook","caller":"webhook/admission.go:76","msg":"AdmissionReview for v1.GroupVersionKind{Group:\"\", Version:\"v1\", Kind:\"ConfigMap\"}: knative-serving/config-service-ca response=&v1beta1.AdmissionResponse{UID:\"\", Allowed:true, Result:(*v1.Status)(nil), Patch:[]uint8(nil), PatchType:(*v1beta1.PatchType)(nil), AuditAnnotations:map[string]string(nil)}","knative.dev/kind":"/v1, Kind=ConfigMap","knative.dev/namespace":"knative-serving","knative.dev/name":"config-service-ca","knative.dev/operation":"UPDATE","knative.dev/resource":"/v1, Resource=configmaps","knative.dev/subresource":"","knative.dev/userinfo":"{system:serviceaccount:openshift-network-operator:default 85f8eb21-4ac6-47e0-93e6-eefdb578a712 [system:serviceaccounts system:serviceaccounts:openshift-network-operator system:authenticated] map[]}"}
{"level":"info","ts":"2020-04-08T19:30:58.593Z","logger":"webhook","caller":"webhook/admission.go:76","msg":"AdmissionReview for v1.GroupVersionKind{Group:\"\", Version:\"v1\", Kind:\"ConfigMap\"}: knative-serving/config-service-ca response=&v1beta1.AdmissionResponse{UID:\"\", Allowed:true, Result:(*v1.Status)(nil), Patch:[]uint8(nil), PatchType:(*v1beta1.PatchType)(nil), AuditAnnotations:map[string]string(nil)}","knative.dev/kind":"/v1, Kind=ConfigMap","knative.dev/namespace":"knative-serving","knative.dev/name":"config-service-ca","knative.dev/operation":"UPDATE","knative.dev/resource":"/v1, Resource=configmaps","knative.dev/subresource":"","knative.dev/userinfo":"{system:serviceaccount:openshift-service-ca:configmap-cabundle-injector-sa 146755c7-fbee-4bf1-b715-96f3d747f703 [system:serviceaccounts system:serviceaccounts:openshift-service-ca system:authenticated] map[]}"}
{"level":"info","ts":"2020-04-08T19:30:58.622Z","logger":"webhook","caller":"webhook/admission.go:76","msg":"AdmissionReview for v1.GroupVersionKind{Group:\"\", Version:\"v1\", Kind:\"ConfigMap\"}: knative-serving/config-service-ca response=&v1beta1.AdmissionResponse{UID:\"\", Allowed:true, Result:(*v1.Status)(nil), Patch:[]uint8(nil), PatchType:(*v1beta1.PatchType)(nil), AuditAnnotations:map[string]string(nil)}","knative.dev/kind":"/v1, Kind=ConfigMap","knative.dev/namespace":"knative-serving","knative.dev/name":"config-service-ca","knative.dev/operation":"UPDATE","knative.dev/resource":"/v1, Resource=configmaps","knative.dev/subresource":"","knative.dev/userinfo":"{system:serviceaccount:openshift-network-operator:default 85f8eb21-4ac6-47e0-93e6-eefdb578a712 [system:serviceaccounts system:serviceaccounts:openshift-network-operator system:authenticated] map[]}"}
{"level":"info","ts":"2020-04-08T19:30:58.793Z","logger":"webhook","caller":"webhook/admission.go:76","msg":"AdmissionReview for v1.GroupVersionKind{Group:\"\", Version:\"v1\", Kind:\"ConfigMap\"}: knative-serving/config-service-ca response=&v1beta1.AdmissionResponse{UID:\"\", Allowed:true, Result:(*v1.Status)(nil), Patch:[]uint8(nil), PatchType:(*v1beta1.PatchType)(nil), AuditAnnotations:map[string]string(nil)}","knative.dev/kind":"/v1, Kind=ConfigMap","knative.dev/namespace":"knative-serving","knative.dev/name":"config-service-ca","knative.dev/operation":"UPDATE","knative.dev/resource":"/v1, Resource=configmaps","knative.dev/subresource":"","knative.dev/userinfo":"{system:serviceaccount:openshift-service-ca:configmap-cabundle-injector-sa 146755c7-fbee-4bf1-b715-96f3d747f703 [system:serviceaccounts system:serviceaccounts:openshift-service-ca system:authenticated] map[]}"}
{"level":"info","ts":"2020-04-08T19:30:58.823Z","logger":"webhook","caller":"webhook/admission.go:76","msg":"AdmissionReview for v1.GroupVersionKind{Group:\"\", Version:\"v1\", Kind:\"ConfigMap\"}: knative-serving/config-service-ca response=&v1beta1.AdmissionResponse{UID:\"\", Allowed:true, Result:(*v1.Status)(nil), Patch:[]uint8(nil), PatchType:(*v1beta1.PatchType)(nil), AuditAnnotations:map[string]string(nil)}","knative.dev/kind":"/v1, Kind=ConfigMap","knative.dev/namespace":"knative-serving","knative.dev/name":"config-service-ca","knative.dev/operation":"UPDATE","knative.dev/resource":"/v1, Resource=configmaps","knative.dev/subresource":"","knative.dev/userinfo":"{system:serviceaccount:openshift-network-operator:default 85f8eb21-4ac6-47e0-93e6-eefdb578a712 [system:serviceaccounts system:serviceaccounts:openshift-network-operator system:authenticated] map[]}"}
{"level":"info","ts":"2020-04-08T19:30:58.993Z","logger":"webhook","caller":"webhook/admission.go:76","msg":"AdmissionReview for v1.GroupVersionKind{Group:\"\", Version:\"v1\", Kind:\"ConfigMap\"}: knative-serving/config-service-ca response=&v1beta1.AdmissionResponse{UID:\"\", Allowed:true, Result:(*v1.Status)(nil), Patch:[]uint8(nil), PatchType:(*v1beta1.PatchType)(nil), AuditAnnotations:map[string]string(nil)}","knative.dev/kind":"/v1, Kind=ConfigMap","knative.dev/namespace":"knative-serving","knative.dev/name":"config-service-ca","knative.dev/operation":"UPDATE","knative.dev/resource":"/v1, Resource=configmaps","knative.dev/subresource":"","knative.dev/userinfo":"{system:serviceaccount:openshift-service-ca:configmap-cabundle-injector-sa 146755c7-fbee-4bf1-b715-96f3d747f703 [system:serviceaccounts system:serviceaccounts:openshift-service-ca system:authenticated] map[]}"}
{"level":"info","ts":"2020-04-08T19:30:59.033Z","logger":"webhook","caller":"webhook/admission.go:76","msg":"AdmissionReview for v1.GroupVersionKind{Group:\"\", Version:\"v1\", Kind:\"ConfigMap\"}: knative-serving/config-service-ca response=&v1beta1.AdmissionResponse{UID:\"\", Allowed:true, Result:(*v1.Status)(nil), Patch:[]uint8(nil), PatchType:(*v1beta1.PatchType)(nil), AuditAnnotations:map[string]string(nil)}","knative.dev/kind":"/v1, Kind=ConfigMap","knative.dev/namespace":"knative-serving","knative.dev/name":"config-service-ca","knative.dev/operation":"UPDATE","knative.dev/resource":"/v1, Resource=configmaps","knative.dev/subresource":"","knative.dev/userinfo":"{system:serviceaccount:openshift-network-operator:default 85f8eb21-4ac6-47e0-93e6-eefdb578a712 [system:serviceaccounts system:serviceaccounts:openshift-network-operator system:authenticated] map[]}"}

      [maschmid@archie ocf-qe-testsuite]$ oc logs -n knative-serving webhook-67fb8b566d-lf7fq | grep knative-serving/config-service-ca | wc -l && date
6778
Wed 08 Apr 2020 09:33:26 PM CEST
[maschmid@archie ocf-qe-testsuite]$ oc logs -n knative-serving webhook-67fb8b566d-lf7fq | grep knative-serving/config-service-ca | wc -l && date
7156
Wed 08 Apr 2020 09:34:04 PM CEST

        1. SRVKS-513.tar.bz2
          1.04 MB
          Marek Schmidt

              markusthoemmes Markus Thömmes (Inactive)
              maschmid@redhat.com Marek Schmidt
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: