Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-5991

Kube APIServer panics in admission controller

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • None
    • 4.13.0
    • kube-apiserver
    • None
    • Auth - Sprint 230, Auth - Sprint 231
    • 2
    • Approved
    • False
    • Hide

      It is crashing the apiserver, which is not acceptable

      Show
      It is crashing the apiserver, which is not acceptable
    • NA

      Description of problem:

      kubernetes-apiserver panics when trying to warn about pod security

      Version-Release number of selected component (if applicable):

      registry.ci.openshift.org/ocp/release:4.13.0-0.ci-2023-01-17-112321

      How reproducible:

      Often

      Steps to Reproduce:

      1. Create HyperShift cluster with a recent OCP release payload
      2. Watch control plane operator pods
      

      Actual results:

      Expected results:

      kube-apiserver crashes and restarts after it panics

      Additional info:

      Log from panic:
      E0117 13:52:17.183773       1 patch_podspecextractor.go:96] "failed to mutate object for PSA using SCC" err="pods \"pod-for-container-named-konnectivity-agent\" is forbidden: unable to validate against any security context constraint: [provider \"anyuid\": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used, spec.containers[0].securityContext.runAsUser: Invalid value: 1000: must be in the ranges: [1000140000, 1000149999], spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used, provider \"restricted\": Forbidden: not usable by user or serviceaccount, provider \"nonroot-v2\": Forbidden: not usable by user or serviceaccount, provider \"nonroot\": Forbidden: not usable by user or serviceaccount, provider \"hostmount-anyuid\": Forbidden: not usable by user or serviceaccount, provider \"hostnetwork-v2\": Forbidden: not usable by user or serviceaccount, provider \"hostnetwork\": Forbidden: not usable by user or serviceaccount, provider \"hostaccess\": Forbidden: not usable by user or serviceaccount, provider \"privileged\": Forbidden: not usable by user or serviceaccount]"
      E0117 13:52:17.183796       1 patch_podspecextractor.go:97] failed to mutate object for PSA using SCC: pods "pod-for-container-named-konnectivity-agent" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used, spec.containers[0].securityContext.runAsUser: Invalid value: 1000: must be in the ranges: [1000140000, 1000149999], spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]
      E0117 13:52:17.183911       1 runtime.go:77] Observed a panic: Header called after Handler finished
      goroutine 154006 [running]:
      k8s.io/apimachinery/pkg/util/runtime.logPanic({0x450dce0?, 0x5cb8a40})
      	vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0209bad80?})
      	vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
      panic({0x450dce0, 0x5cb8a40})
      	/usr/lib/golang/src/runtime/panic.go:884 +0x212
      golang.org/x/net/http2.(*responseWriter).Header(0x450dce0?)
      	vendor/golang.org/x/net/http2/server.go:2693 +0x7a
      k8s.io/apiserver/pkg/endpoints/filters.(*recorder).AddWarning(0xc01ba662d0, {0x0, 0x0}, {0xc0325a3d40, 0x22d})
      	vendor/k8s.io/apiserver/pkg/endpoints/filters/warning.go:110 +0x43b
      k8s.io/apiserver/pkg/warning.AddWarning({0x5cf9038?, 0xc01ba744b0?}, {0x0, 0x0}, {0xc0325a3d40, 0x22d})
      	vendor/k8s.io/apiserver/pkg/warning/context.go:58 +0x82
      k8s.io/kubernetes/plugin/pkg/admission/security/podsecurity.(*Plugin).Validate(0xc000e6ec30, {0x5cf9038, 0xc01ba744b0}, {0x5d15970?, 0xc0209b2c60?}, {0x5cfde80?, 0xc003263ba0?})
      	plugin/pkg/admission/security/podsecurity/admission.go:185 +0x555
      k8s.io/apiserver/pkg/admission/metrics.pluginHandlerWithMetrics.Validate({{0x5cc4800, 0xc000e6ec30}, 0xc000b55690, {0xc000b556a0, 0x1, 0x1}}, {0x5cf9038, 0xc01ba744b0}, {0x5d15970, 0xc0209b2c60}, ...)
      	vendor/k8s.io/apiserver/pkg/admission/metrics/metrics.go:108 +0xef
      github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout.pluginHandlerWithTimeout.Validate.func1()
      	vendor/github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout/timeoutadmission.go:59 +0x8d
      created by github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout.pluginHandlerWithTimeout.Validate
      	vendor/github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout/timeoutadmission.go:56 +0x22a
      panic: Header called after Handler finished [recovered]
      	panic: Header called after Handler finished
      
      goroutine 154006 [running]:
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0209bad80?})
      	vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xd7
      panic({0x450dce0, 0x5cb8a40})
      	/usr/lib/golang/src/runtime/panic.go:884 +0x212
      golang.org/x/net/http2.(*responseWriter).Header(0x450dce0?)
      	vendor/golang.org/x/net/http2/server.go:2693 +0x7a
      k8s.io/apiserver/pkg/endpoints/filters.(*recorder).AddWarning(0xc01ba662d0, {0x0, 0x0}, {0xc0325a3d40, 0x22d})
      	vendor/k8s.io/apiserver/pkg/endpoints/filters/warning.go:110 +0x43b
      k8s.io/apiserver/pkg/warning.AddWarning({0x5cf9038?, 0xc01ba744b0?}, {0x0, 0x0}, {0xc0325a3d40, 0x22d})
      	vendor/k8s.io/apiserver/pkg/warning/context.go:58 +0x82
      k8s.io/kubernetes/plugin/pkg/admission/security/podsecurity.(*Plugin).Validate(0xc000e6ec30, {0x5cf9038, 0xc01ba744b0}, {0x5d15970?, 0xc0209b2c60?}, {0x5cfde80?, 0xc003263ba0?})
      	plugin/pkg/admission/security/podsecurity/admission.go:185 +0x555
      k8s.io/apiserver/pkg/admission/metrics.pluginHandlerWithMetrics.Validate({{0x5cc4800, 0xc000e6ec30}, 0xc000b55690, {0xc000b556a0, 0x1, 0x1}}, {0x5cf9038, 0xc01ba744b0}, {0x5d15970, 0xc0209b2c60}, ...)
      	vendor/k8s.io/apiserver/pkg/admission/metrics/metrics.go:108 +0xef
      github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout.pluginHandlerWithTimeout.Validate.func1()
      	vendor/github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout/timeoutadmission.go:59 +0x8d
      created by github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout.pluginHandlerWithTimeout.Validate
      	vendor/github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout/timeoutadmission.go:56 +0x22a

      This started happening after this PR merged:
      https://github.com/openshift/kubernetes/pull/1453 

              akashem@redhat.com Abu H Kashem
              cewong@redhat.com Cesar Wong
              Rahul Gangwar Rahul Gangwar
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: