-
Bug
-
Resolution: Done
-
Major
-
None
-
4.12
-
Quality / Stability / Reliability
-
False
-
-
None
-
Important
-
None
-
None
-
Rejected
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
In kube-apiserver audit logs shows various concerning messages with "APIServer panic'd: context deadline exceeded - InternalError" error
Version-Release number of selected component (if applicable): 4.12.0-0.nightly-2022-08-31-101631
How reproducible:
Appears to be there always without needing to do specific things to reproduce it.
Steps to Reproduce:
1. On all master nodes, check kas audit logs for abnormal (panic/fatal/SHOULD NOT HAPPEN) logs.
2. grep -ihE '(panic|fatal|SHOULD NOT HAPPEN)' /var/log/kube-apiserver/audit*.log
Actual results:
Sep 1 02:48:43.253: INFO: KAS audit abnormal Logs --------------------->
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "Metadata",
"auditID": "7706330b-327a-4b5a-b168-1e81ab8a6f16",
"stage": "Panic",
"requestURI": "/apis/quota.openshift.io/v1/clusterresourcequotas?resourceVersion=17572",
"verb": "list",
"user": {
"username": "system:kube-controller-manager",
"groups": [
"system:authenticated"
]
},
"sourceIPs": [
"10.0.78.3"
],
"userAgent": "Go-http-client/2.0",
"objectRef": {
"resource": "clusterresourcequotas",
"apiGroup": "quota.openshift.io",
"apiVersion": "v1"
},
"responseStatus": {
"metadata": {},
"status": "Failure",
"message": "APIServer panic'd: context deadline exceeded",
"reason": "InternalError",
"code": 500
},
"requestReceivedTimestamp": "2022-09-01T01:49:25.928615Z",
"stageTimestamp": "2022-09-01T01:49:46.654102Z",
"annotations": {
"authorization.k8s.io/decision": "allow",
"authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\"",
"openshift.io/unready": "loopback=false,,readyz=false"
}
}
}
Expected results:
Not show any panic error
Additional info:
Must-Gather Logs: Attached
Automation Test execution logs : Attached