-
Bug
-
Resolution: Cannot Reproduce
-
Normal
-
None
-
4.15
-
Important
-
No
-
False
-
Description of problem:
In Reliability (loaded longrun, the load is stable) test, the 'console-operator' container's cpu memory increased.
Version-Release number of selected component (if applicable):
4.15.0-0.nightly-2023-12-09-012410
How reproducible:
I didn't see this issue in 4.14 testing
Steps to Reproduce:
1. Install a AWS cluster with 3 masters, 3 workers. vm_type: m5.xlarge 2. Run reliability-v2 test https://github.com/openshift/svt/tree/master/reliability-v2. The test will long run and simulate multiple customers usage on the cluster. config: 1 admin, 10 dev-test, 5 dev-prod, 1 dev-cron(10cronjobs)
Actual results:
console-operator container's cpu and memory increased during the test
Expected results:
console-operator container's cpu and memory usage should be stable
Additional info:
% oc get po -n openshift-console-operator
NAME READY STATUS RESTARTS AGE
console-operator-5f8ff488c7-ccnz5 2/2 Running 6 (11d ago) 11d
% oc describe po console-operator-5f8ff488c7-ccnz5 -n openshift-console-operator
Name: console-operator-5f8ff488c7-ccnz5
Namespace: openshift-console-operator
Priority: 2000000000
Priority Class Name: system-cluster-critical
Service Account: console-operator
Node: ip-10-0-67-9.us-east-2.compute.internal/10.0.67.9
Start Time: Mon, 11 Dec 2023 09:49:15 +0800
Labels: name=console-operator
pod-template-hash=5f8ff488c7
Annotations: k8s.ovn.org/pod-networks:
{"default":{"ip_addresses":["10.130.0.41/23"],"mac_address":"0a:58:0a:82:00:29","gateway_ips":["10.130.0.1"],"routes":[{"dest":"10.128.0.0...
k8s.v1.cni.cncf.io/network-status:
[{
"name": "ovn-kubernetes",
"interface": "eth0",
"ips": [
"10.130.0.41"
],
"mac": "0a:58:0a:82:00:29",
"default": true,
"dns": {}
}]
openshift.io/scc: restricted-v2
seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status: Running
SeccompProfile: RuntimeDefault
IP: 10.130.0.41
IPs:
IP: 10.130.0.41
Controlled By: ReplicaSet/console-operator-5f8ff488c7
Containers:
console-operator:
Container ID: cri-o://808f374466c03357a42433fb496378996180a694d2ee6b99f5431e028fda8834
Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:10d91d35b86a547ca0e1b4f561a94a39b66af642fc0e701028ce3dca7b5f5966
Image ID: c03dd739e4449beb429b76a880636bad94c4085ef7e60f818edc1d9cd594ea78
Port: 60000/TCP
Host Port: 0/TCP
Command:
console
operator
Args:
-v=2
--config=/var/run/configmaps/config/controller-config.yaml
State: Running
Started: Mon, 11 Dec 2023 09:50:02 +0800
Last State: Terminated
Reason: Error
Message: W1211 01:49:39.641371 1 cmd.go:234] Using insecure, self-signed certificates
F1211 01:49:39.641644 1 cmd.go:159] mkdir /tmp/serving-cert-3011261756: read-only file system
Exit Code: 255
Started: Mon, 11 Dec 2023 09:49:39 +0800
Finished: Mon, 11 Dec 2023 09:49:39 +0800
Ready: True
Restart Count: 3
Requests:
cpu: 10m
memory: 100Mi
Liveness: http-get https://:8443/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get https://:8443/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
CONSOLE_IMAGE: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c3445cf238b78f6a603c595218f092df6f7eb68f96e5293e11e11821b51e68eb
DOWNLOADS_IMAGE: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ae7550b0f6905d47acfd2529718d697a85de55a9ee8a256591361693f931bea
RELEASE_VERSION: 4.15.0-0.nightly-2023-12-09-012410
OPERATOR_NAME: console-operator
POD_NAME: console-operator-5f8ff488c7-ccnz5 (v1:metadata.name)
Mounts:
/var/run/configmaps/config from config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwssw (ro)
/var/run/secrets/serving-cert from serving-cert (rw)
conversion-webhook-server:
Container ID: cri-o://28638bc6dbfa3fed0e4983ef1f94105015240d91357e6a7fbef8e8169213338c
Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:10d91d35b86a547ca0e1b4f561a94a39b66af642fc0e701028ce3dca7b5f5966
Image ID: c03dd739e4449beb429b76a880636bad94c4085ef7e60f818edc1d9cd594ea78
Port: 9443/TCP
Host Port: 0/TCP
Command:
console
crdconvert
Args:
--port=9443
--tls-cert-file=/var/run/secrets/webhook-serving-cert/tls.crt
--tls-private-key-file=/var/run/secrets/webhook-serving-cert/tls.key
State: Running
Started: Mon, 11 Dec 2023 09:50:02 +0800
Last State: Terminated
Reason: Error
Message: F1211 01:49:39.820093 1 config.go:27] open /var/run/secrets/webhook-serving-cert/tls.crt: no such file or directory
Exit Code: 255
Started: Mon, 11 Dec 2023 09:49:39 +0800
Finished: Mon, 11 Dec 2023 09:49:39 +0800
Ready: True
Restart Count: 3
Requests:
cpu: 10m
memory: 100Mi
Liveness: http-get https://:9443/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get https://:9443/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/var/run/configmaps/config from config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zwssw (ro)
/var/run/secrets/webhook-serving-cert from webhook-serving-cert (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: console-operator-config
Optional: false
serving-cert:
Type: Secret (a volume populated by a Secret)
SecretName: serving-cert
Optional: true
webhook-serving-cert:
Type: Secret (a volume populated by a Secret)
SecretName: webhook-serving-cert
Optional: true
kube-api-access-zwssw:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
ConfigMapName: openshift-service-ca.crt
ConfigMapOptional: <nil>
QoS Class: Burstable
Node-Selectors: node-role.kubernetes.io/master=
Tolerations: node-role.kubernetes.io/master:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 120s
node.kubernetes.io/unreachable:NoExecute op=Exists for 120s
Events: <none>