-
Story
-
Resolution: Done
-
Normal
-
None
-
None
-
None
-
None
-
False
-
-
False
-
None
-
None
-
None
-
None
-
None
Recent PodSecurityViolation failures were hard to pin down where the failure occurred
The tests showed:
[bz-apiserver-auth][invariant] alert/PodSecurityViolation should not be at or above info expand_less 0s
{ PodSecurityViolation was at or above info for at least 40m36s on platformidentification.JobType{Release:"4.18", FromRelease:"4.18", Platform:"gcp", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 0s, firing for 40m36s:
Nov 21 22:34:16.134 - 1218s I namespace/openshift-kube-apiserver alert/PodSecurityViolation alertstate/firing severity/info ALERTS{alertname="PodSecurityViolation", alertstate="firing", namespace="openshift-kube-apiserver", ocp_namespace="openshift-catalogd", policy_level="restricted", prometheus="openshift-monitoring/k8s", severity="info"}
Nov 21 22:34:16.134 - 1218s I namespace/openshift-kube-apiserver alert/PodSecurityViolation alertstate/firing severity/info ALERTS{alertname="PodSecurityViolation", alertstate="firing", namespace="openshift-kube-apiserver", ocp_namespace="openshift-operator-controller", policy_level="restricted", prometheus="openshift-monitoring/k8s", severity="info"}}
Documentation indicates we should review the audit logs
Which contains entries like:
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "Metadata",
"auditID": "6d8f3eda-2ad6-41d4-a158-c0474bc2eba5",
"stage": "ResponseComplete",
"requestURI": "/apis/apps/v1/namespaces/openshift-operator-controller/deployments",
"verb": "create",
"user": {
"username": "system:serviceaccount:openshift-cluster-olm-operator:cluster-olm-operator",
"uid": "ebda9d89-6c84-4e46-8919-c622af7922f3",
"groups": [
"system:serviceaccounts",
"system:serviceaccounts:openshift-cluster-olm-operator",
"system:authenticated"
],
"extra": {
"authentication.kubernetes.io/credential-id": [
"JTI=4813932a-d08d-4735-bca1-2a2bea50b6c4"
],
"authentication.kubernetes.io/node-name": [
"ci-op-hs76pwwz-1e160-l6xvj-master-0"
],
"authentication.kubernetes.io/node-uid": [
"45dd8e53-80fc-499c-bdad-0cc9b24a06ba"
],
"authentication.kubernetes.io/pod-name": [
"cluster-olm-operator-5b7c88f447-54phc"
],
"authentication.kubernetes.io/pod-uid": [
"86600c7f-612f-492b-83b3-498648da225e"
]
}
},
"sourceIPs": [
"10.0.0.3"
],
"userAgent": "cluster-olm-operator/v0.0.0 (linux/amd64) kubernetes/$Format",
"objectRef": {
"resource": "deployments",
"namespace": "openshift-operator-controller",
"name": "operator-controller-controller-manager",
"apiGroup": "apps",
"apiVersion": "v1"
},
"responseStatus": {
"metadata": {},
"code": 201
},
"requestReceivedTimestamp": "2024-11-21T22:26:32.795959Z",
"stageTimestamp": "2024-11-21T22:26:33.021455Z",
"annotations": {
"authorization.k8s.io/decision": "allow",
"authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"cluster-olm-operator-role\" of ClusterRole \"cluster-olm-operator\" to ServiceAccount \"cluster-olm-operator/openshift-cluster-olm-operator\"",
"pod-security.kubernetes.io/audit-violations": "would violate PodSecurity \"restricted:latest\": restricted volume types (volume \"etc-containers\" uses restricted volume type \"hostPath\")"
}
}
Adding auditloganalyzer test to extract more detail on the cause