-
Bug
-
Resolution: Done
-
Critical
-
None
-
4.13.0
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
None
-
None
-
Approved
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
The following is the description from the original bug for kube-apiserver. For openshift-apiserver we have not had any report of panic and server crashing. But the gap still exists, the panic (if it is raised for some reason) must be handled so that the openshift-apiserver does not crash
<<<<<<<<<<<<<<<<<<<<<<---------------------------------------------------------------------->>>>>>>>>>>>>>>>>>>>>>>>>>
Description of problem:
kubernetes-apiserver panics when trying to warn about pod security
Version-Release number of selected component (if applicable):
registry.ci.openshift.org/ocp/release:4.13.0-0.ci-2023-01-17-112321
How reproducible:
Often
Steps to Reproduce:
1. Create HyperShift cluster with a recent OCP release payload 2. Watch control plane operator pods
Actual results:
Expected results:
kube-apiserver crashes and restarts after it panics
Additional info:
Log from panic:
E0117 13:52:17.183773 1 patch_podspecextractor.go:96] "failed to mutate object for PSA using SCC" err="pods \"pod-for-container-named-konnectivity-agent\" is forbidden: unable to validate against any security context constraint: [provider \"anyuid\": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used, spec.containers[0].securityContext.runAsUser: Invalid value: 1000: must be in the ranges: [1000140000, 1000149999], spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used, provider \"restricted\": Forbidden: not usable by user or serviceaccount, provider \"nonroot-v2\": Forbidden: not usable by user or serviceaccount, provider \"nonroot\": Forbidden: not usable by user or serviceaccount, provider \"hostmount-anyuid\": Forbidden: not usable by user or serviceaccount, provider \"hostnetwork-v2\": Forbidden: not usable by user or serviceaccount, provider \"hostnetwork\": Forbidden: not usable by user or serviceaccount, provider \"hostaccess\": Forbidden: not usable by user or serviceaccount, provider \"privileged\": Forbidden: not usable by user or serviceaccount]"
E0117 13:52:17.183796 1 patch_podspecextractor.go:97] failed to mutate object for PSA using SCC: pods "pod-for-container-named-konnectivity-agent" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used, spec.containers[0].securityContext.runAsUser: Invalid value: 1000: must be in the ranges: [1000140000, 1000149999], spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]
E0117 13:52:17.183911 1 runtime.go:77] Observed a panic: Header called after Handler finished
goroutine 154006 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x450dce0?, 0x5cb8a40})
vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0209bad80?})
vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x75
panic({0x450dce0, 0x5cb8a40})
/usr/lib/golang/src/runtime/panic.go:884 +0x212
golang.org/x/net/http2.(*responseWriter).Header(0x450dce0?)
vendor/golang.org/x/net/http2/server.go:2693 +0x7a
k8s.io/apiserver/pkg/endpoints/filters.(*recorder).AddWarning(0xc01ba662d0, {0x0, 0x0}, {0xc0325a3d40, 0x22d})
vendor/k8s.io/apiserver/pkg/endpoints/filters/warning.go:110 +0x43b
k8s.io/apiserver/pkg/warning.AddWarning({0x5cf9038?, 0xc01ba744b0?}, {0x0, 0x0}, {0xc0325a3d40, 0x22d})
vendor/k8s.io/apiserver/pkg/warning/context.go:58 +0x82
k8s.io/kubernetes/plugin/pkg/admission/security/podsecurity.(*Plugin).Validate(0xc000e6ec30, {0x5cf9038, 0xc01ba744b0}, {0x5d15970?, 0xc0209b2c60?}, {0x5cfde80?, 0xc003263ba0?})
plugin/pkg/admission/security/podsecurity/admission.go:185 +0x555
k8s.io/apiserver/pkg/admission/metrics.pluginHandlerWithMetrics.Validate({{0x5cc4800, 0xc000e6ec30}, 0xc000b55690, {0xc000b556a0, 0x1, 0x1}}, {0x5cf9038, 0xc01ba744b0}, {0x5d15970, 0xc0209b2c60}, ...)
vendor/k8s.io/apiserver/pkg/admission/metrics/metrics.go:108 +0xef
github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout.pluginHandlerWithTimeout.Validate.func1()
vendor/github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout/timeoutadmission.go:59 +0x8d
created by github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout.pluginHandlerWithTimeout.Validate
vendor/github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout/timeoutadmission.go:56 +0x22a
panic: Header called after Handler finished [recovered]
panic: Header called after Handler finished
goroutine 154006 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0209bad80?})
vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xd7
panic({0x450dce0, 0x5cb8a40})
/usr/lib/golang/src/runtime/panic.go:884 +0x212
golang.org/x/net/http2.(*responseWriter).Header(0x450dce0?)
vendor/golang.org/x/net/http2/server.go:2693 +0x7a
k8s.io/apiserver/pkg/endpoints/filters.(*recorder).AddWarning(0xc01ba662d0, {0x0, 0x0}, {0xc0325a3d40, 0x22d})
vendor/k8s.io/apiserver/pkg/endpoints/filters/warning.go:110 +0x43b
k8s.io/apiserver/pkg/warning.AddWarning({0x5cf9038?, 0xc01ba744b0?}, {0x0, 0x0}, {0xc0325a3d40, 0x22d})
vendor/k8s.io/apiserver/pkg/warning/context.go:58 +0x82
k8s.io/kubernetes/plugin/pkg/admission/security/podsecurity.(*Plugin).Validate(0xc000e6ec30, {0x5cf9038, 0xc01ba744b0}, {0x5d15970?, 0xc0209b2c60?}, {0x5cfde80?, 0xc003263ba0?})
plugin/pkg/admission/security/podsecurity/admission.go:185 +0x555
k8s.io/apiserver/pkg/admission/metrics.pluginHandlerWithMetrics.Validate({{0x5cc4800, 0xc000e6ec30}, 0xc000b55690, {0xc000b556a0, 0x1, 0x1}}, {0x5cf9038, 0xc01ba744b0}, {0x5d15970, 0xc0209b2c60}, ...)
vendor/k8s.io/apiserver/pkg/admission/metrics/metrics.go:108 +0xef
github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout.pluginHandlerWithTimeout.Validate.func1()
vendor/github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout/timeoutadmission.go:59 +0x8d
created by github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout.pluginHandlerWithTimeout.Validate
vendor/github.com/openshift/library-go/pkg/apiserver/admission/admissiontimeout/timeoutadmission.go:56 +0x22a
This started happening after this PR merged:
https://github.com/openshift/kubernetes/pull/1453
- clones
-
OCPBUGS-5991 Kube APIServer panics in admission controller
-
- Closed
-
- is blocked by
-
API-1537 rebase openshift/apiserver
-
- Closed
-
- links to