apiserver panics in admission controller

kubernetes-apiserver panics when trying to warn about pod security

registry.ci.openshift.org/ocp/release:4.13.0-0.ci-2023-01-17-112321

Often

1. Create HyperShift cluster with a recent OCP release payload
2. Watch control plane operator pods

kube-apiserver crashes and restarts after it panics

Log from panic:
E0117 13:52:17.183773       1 patch_podspecextractor.go:96] "failed to mutate object for PSA using SCC" err="pods \"pod-for-container-named-konnectivity-agent\" is forbidden: unable to validate against any security context constraint: [provider \"anyuid\": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used, spec.containers[0].securityContext.runAsUser: Invalid value: 1000: must be in the ranges: [1000140000, 1000149999], spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used, provider \"restricted\": Forbidden: not usable by user or serviceaccount, provider \"nonroot-v2\": Forbidden: not usable by user or serviceaccount, provider \"nonroot\": Forbidden: not usable by user or serviceaccount, provider \"hostmount-anyuid\": Forbidden: not usable by user or serviceaccount, provider \"hostnetwork-v2\": Forbidden: not usable by user or serviceaccount, provider \"hostnetwork\": Forbidden: not usable by user or serviceaccount, provider \"hostaccess\": Forbidden: not usable by user or serviceaccount, provider \"privileged\": Forbidden: not usable by user or serviceaccount]"
E0117 13:52:17.183796       1 patch_podspecextractor.go:97] failed to mutate object for PSA using SCC: pods "pod-for-container-named-konnectivity-agent" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used, spec.containers[0].securityContext.runAsUser: Invalid value: 1000: must be in the ranges: [1000140000, 1000149999], spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]
E0117 13:52:17.183911       1 runtime.go:77] Observed a panic: Header called after Handler finished
goroutine 154006 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x450dce0?, 0x5cb8a40})
	vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0209bad80?})
	vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
panic({0x450dce0, 0x5cb8a40})
	/usr/lib/golang/src/runtime/panic.go:884 +0x212
golang.org/x/net/http2.(*responseWriter).Header(0x450dce0?)
	vendor/golang.org/x/net/http2/server.go:2693 +0x7a
k8s.io/apiserver/pkg/endpoints/filters.(*recorder).AddWarning(0xc01ba662d0, {0x0, 0x0}, {0xc0325a3d40, 0x22d})
	vendor/k8s.io/apiserver/pkg/endpoints/filters/warning.go:110 +0x43b
k8s.io/apiserver/pkg/warning.AddWarning({0x5cf9038?, 0xc01ba744b0?}, {0x0, 0x0}, {0xc0325a3d40, 0x22d})
	vendor/k8s.io/apiserver/pkg/warning/context.go:58 +0x82
k8s.io/kubernetes/plugin/pkg/admission/security/podsecurity.(*Plugin).Validate(0xc000e6ec30, {0x5cf9038, 0xc01ba744b0}, {0x5d15970?, 0xc0209b2c60?}, {0x5cfde80?, 0xc003263ba0?})
	plugin/pkg/admission/security/podsecurity/admission.go:185 +0x555
k8s.io/apiserver/pkg/admission/metrics.pluginHandlerWithMetrics.Validate({{0x5cc4800, 0xc000e6ec30}, 0xc000b55690, {0xc000b556a0, 0x1, 0x1}}, {0x5cf9038, 0xc01ba744b0}, {0x5d15970, 0xc0209b2c60}, ...)
	vendor/k8s.io/apiserver/pkg/admission/metrics/metrics.go:108 +0xef
github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout.pluginHandlerWithTimeout.Validate.func1()
	vendor/github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout/timeoutadmission.go:59 +0x8d
created by github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout.pluginHandlerWithTimeout.Validate
	vendor/github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout/timeoutadmission.go:56 +0x22a
panic: Header called after Handler finished [recovered]
	panic: Header called after Handler finished

goroutine 154006 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0209bad80?})
	vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xd7
panic({0x450dce0, 0x5cb8a40})
	/usr/lib/golang/src/runtime/panic.go:884 +0x212
golang.org/x/net/http2.(*responseWriter).Header(0x450dce0?)
	vendor/golang.org/x/net/http2/server.go:2693 +0x7a
k8s.io/apiserver/pkg/endpoints/filters.(*recorder).AddWarning(0xc01ba662d0, {0x0, 0x0}, {0xc0325a3d40, 0x22d})
	vendor/k8s.io/apiserver/pkg/endpoints/filters/warning.go:110 +0x43b
k8s.io/apiserver/pkg/warning.AddWarning({0x5cf9038?, 0xc01ba744b0?}, {0x0, 0x0}, {0xc0325a3d40, 0x22d})
	vendor/k8s.io/apiserver/pkg/warning/context.go:58 +0x82
k8s.io/kubernetes/plugin/pkg/admission/security/podsecurity.(*Plugin).Validate(0xc000e6ec30, {0x5cf9038, 0xc01ba744b0}, {0x5d15970?, 0xc0209b2c60?}, {0x5cfde80?, 0xc003263ba0?})
	plugin/pkg/admission/security/podsecurity/admission.go:185 +0x555
k8s.io/apiserver/pkg/admission/metrics.pluginHandlerWithMetrics.Validate({{0x5cc4800, 0xc000e6ec30}, 0xc000b55690, {0xc000b556a0, 0x1, 0x1}}, {0x5cf9038, 0xc01ba744b0}, {0x5d15970, 0xc0209b2c60}, ...)
	vendor/k8s.io/apiserver/pkg/admission/metrics/metrics.go:108 +0xef
github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout.pluginHandlerWithTimeout.Validate.func1()
	vendor/github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout/timeoutadmission.go:59 +0x8d
created by github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout.pluginHandlerWithTimeout.Validate
	vendor/github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout/timeoutadmission.go:56 +0x22a