Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-12023

Pods in open-cluster-management-addon-observability are being frequently replaced unnecessarily

XMLWordPrintable

    • 1
    • False
    • None
    • False
    • MCO Sprint 24, MCO Sprint 25
    • No

      Description of problem:

      While deploying many SNO clusters it has been observed that pods within "open-cluster-management-addon-observability" are being replaced frequently.

       

      # oc --kubeconfig /root/hv-vm/kc/vm00030/kubeconfig get po -n open-cluster-management-addon-observability 
      NAME                                               READY   STATUS    RESTARTS   AGE
      endpoint-observability-operator-758886b9cf-p6z9k   1/1     Running   0          17m
      metrics-collector-deployment-8556767ff8-zglcw      1/1     Running   0          17m
      

      It appears the pods are being replaced (not restarted) and are cycling every ~20 minutes, this is especially noticeable when viewing the other open-cluster-management pods: 

       

      # oc --kubeconfig /root/hv-vm/kc/vm00030/kubeconfig get po -A | grep open-cluster-management
      open-cluster-management-addon-observability        endpoint-observability-operator-758886b9cf-p6z9k                  1/1     Running            0               17m
      open-cluster-management-addon-observability        metrics-collector-deployment-8556767ff8-zglcw                     1/1     Running            0               17m
      open-cluster-management-agent-addon                cluster-proxy-proxy-agent-65557f7dff-chqwf                        3/3     Running            0               24h
      open-cluster-management-agent-addon                config-policy-controller-d55f668c4-nwwqv                          2/2     Running            0               24h
      open-cluster-management-agent-addon                governance-policy-framework-c6d67bbff-dvnkl                       2/2     Running            2 (3h18m ago)   24h
      open-cluster-management-agent-addon                klusterlet-addon-search-57cb7d8f8b-5rgwd                          1/1     Running            0               24h
      open-cluster-management-agent-addon                klusterlet-addon-workmgr-7fc66d96c6-j4sbw                         1/1     Running            0               24h
      open-cluster-management-agent-addon                managed-serviceaccount-addon-agent-7fd5986d54-bdzdm               1/1     Running            0               24h
      open-cluster-management-agent                      klusterlet-7467d8c57d-t47lg                                       1/1     Running            0               24h
      open-cluster-management-agent                      klusterlet-agent-77f4d8bb7d-kgm2p                                 1/1     Running            0               24h

      Version-Release number of selected component (if applicable):

      ACM - 2.11.0-DOWNSTREAM-2024-05-23-15-16-26

      Hub OCP  - 4.16.0-rc.3

      Deployed SNOs - 4.14.16 (Though this is also occurring in the large scale environment with 4.15.16 and 4.16.0-rc.3)

      How reproducible:

      Steps to Reproduce:

      1.  
      2.  
      3. ...

      Actual results:

      Expected results:

      Additional info:

      It seems the deploys are being scaled up and down frequently which seems unnecessary:

      # oc --kubeconfig /root/hv-vm/kc/vm00030/kubeconfig describe deploy -n open-cluster-management-addon-observability endpoint-observability-operator
      Name:                   endpoint-observability-operator
      Namespace:              open-cluster-management-addon-observability
      CreationTimestamp:      Tue, 04 Jun 2024 18:55:04 +0000
      Labels:                 <none>
      Annotations:            deployment.kubernetes.io/revision: 224
      Selector:               name=endpoint-observability-operator
      Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
      StrategyType:           RollingUpdate
      MinReadySeconds:        0
      RollingUpdateStrategy:  25% max unavailable, 25% max surge
      Pod Template:
        Labels:           name=endpoint-observability-operator
        Annotations:      target.workload.openshift.io/management: {"effect":"PreferredDuringScheduling"}
        Service Account:  endpoint-observability-operator-sa
        Containers:
         endpoint-observability-operator:
          Image:      e38-h01-000-r650.rdu2.scalelab.redhat.com:5000/acm-d/endpoint-monitoring-rhel9-operator@sha256:06e3c7b67ff801873e951f47171e2ee2cba9603f0363e646e0c313a7f1354ee4
          Port:       8383/TCP
          Host Port:  0/TCP
          Command:
            endpoint-monitoring-operator
          Requests:
            cpu:     2m
            memory:  50Mi
          Environment:
            HUB_NAMESPACE:    vm00030
            WATCH_NAMESPACE:   (v1:metadata.namespace)
            POD_NAME:          (v1:metadata.name)
            SERVICE_ACCOUNT:   (v1:spec.serviceAccountName)
            OPERATOR_NAME:    endpoint-monitoring-operator
            HUB_KUBECONFIG:   /spoke/hub-kubeconfig/kubeconfig
            INSTALL_PROM:     false
            PULL_SECRET:      multiclusterhub-operator-pull-secret
          Mounts:
            /spoke/hub-kubeconfig from hub-kubeconfig-secret (ro)
        Volumes:
         hub-kubeconfig-secret:
          Type:        Secret (a volume populated by a Secret)
          SecretName:  observability-controller-hub-kubeconfig
          Optional:    false
      Conditions:
        Type           Status  Reason
        ----           ------  ------
        Available      True    MinimumReplicasAvailable
        Progressing    True    NewReplicaSetAvailable
      OldReplicaSets:  endpoint-observability-operator-66f5bb4db6 (0/0 replicas created), endpoint-observability-operator-864b5db55f (0/0 replicas created)
      NewReplicaSet:   endpoint-observability-operator-758886b9cf (1/1 replicas created)
      Events:
        Type    Reason             Age                    From                   Message
        ----    ------             ----                   ----                   -------
        Normal  ScalingReplicaSet  150m (x30 over 5h30m)  deployment-controller  Scaled down replica set endpoint-observability-operator-758886b9cf to 0 from 1
        Normal  ScalingReplicaSet  141m (x33 over 5h30m)  deployment-controller  Scaled up replica set endpoint-observability-operator-864b5db55f to 1 from 0
        Normal  ScalingReplicaSet  115m (x7 over 135m)    deployment-controller  Scaled down replica set endpoint-observability-operator-864b5db55f to 0 from 1
        Normal  ScalingReplicaSet  71m (x16 over 135m)    deployment-controller  Scaled up replica set endpoint-observability-operator-758886b9cf to 1 from 0
        Normal  ScalingReplicaSet  64m (x17 over 135m)    deployment-controller  Scaled down replica set endpoint-observability-operator-758886b9cf to 0 from 1
        Normal  ScalingReplicaSet  58m (x18 over 135m)    deployment-controller  Scaled up replica set endpoint-observability-operator-864b5db55f to 1 from 0
        Normal  ScalingReplicaSet  10m (x6 over 50m)      deployment-controller  Scaled up replica set endpoint-observability-operator-864b5db55f to 1 from 0
        Normal  ScalingReplicaSet  10m (x6 over 50m)      deployment-controller  Scaled down replica set endpoint-observability-operator-758886b9cf to 0 from 1
        Normal  ScalingReplicaSet  10m (x6 over 50m)      deployment-controller  Scaled up replica set endpoint-observability-operator-758886b9cf to 1 from 0
        Normal  ScalingReplicaSet  10m (x6 over 50m)      deployment-controller  Scaled down replica set endpoint-observability-operator-864b5db55f to 0 from 1
      

      The other deploy:

      Name:                   metrics-collector-deployment
      Namespace:              open-cluster-management-addon-observability
      CreationTimestamp:      Tue, 04 Jun 2024 18:56:00 +0000
      Labels:                 <none>
      Annotations:            deployment.kubernetes.io/revision: 346
                              owner: observabilityaddon
      Selector:               component=metrics-collector
      Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
      StrategyType:           RollingUpdate
      MinReadySeconds:        0
      RollingUpdateStrategy:  25% max unavailable, 25% max surge
      Pod Template:
        Labels:           cert/time-restarted=2024-6-5.1904
                          component=metrics-collector
        Annotations:      owner: observabilityaddon
                          target.workload.openshift.io/management: {"effect":"PreferredDuringScheduling"}
        Service Account:  endpoint-observability-operator-sa
        Containers:
         metrics-collector:
          Image:      e38-h01-000-r650.rdu2.scalelab.redhat.com:5000/acm-d/metrics-collector-rhel9@sha256:8e37489021df05d276d92b35d0fd329e62ca691797ebcf0415f60211de8bc036
          Port:       8080/TCP
          Host Port:  0/TCP
          Command:
            /usr/bin/metrics-collector
            --listen=:8080
            --from=$(FROM)
            --from-query=$(FROM_QUERY)
            --to-upload=$(TO)
            --to-upload-ca=/tlscerts/ca/ca.crt
            --to-upload-cert=/tlscerts/certs/tls.crt
            --to-upload-key=/tlscerts/certs/tls.key
            --interval=300s
            --evaluate-interval=30s
            --limit-bytes=1073741824
            --label="cluster=vm00030"
            --label="clusterID=a36f30ba-21de-43cb-b454-139ad3f2f5cb"
            --from-token-file=/var/run/secrets/kubernetes.io/serviceaccount/token
            --from-ca-file=/etc/serving-certs-ca-bundle/service-ca.crt
            --label="clusterType=SNO"
            --collectrule={"name":"SNOHighCPUUsage","expr":"(1 - avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m]))) * 100 > 70","for":"2m","names":["container_cpu_cfs_periods_total","container_cpu_cfs_throttled_periods_total","kube_pod_container_resource_limits","kube_pod_container_resource_requests","namespace_workload_pod:kube_pod_owner:relabel","node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate","node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate"],"matches":[]}
            --collectrule={"name":"SNOHighMemoryUsage","expr":"(1 - sum(:node_memory_MemAvailable_bytes:sum) / sum(kube_node_status_allocatable{resource=\"memory\"})) * 100 > 70","for":"2m","names":["kube_pod_container_resource_limits","kube_pod_container_resource_requests","namespace_workload_pod:kube_pod_owner:relabel"],"matches":["__name__=\"container_memory_cache\",container!=\"\"","__name__=\"container_memory_rss\",container!=\"\"","__name__=\"container_memory_swap\",container!=\"\"","__name__=\"container_memory_working_set_bytes\",container!=\"\""]}
            --match={__name__=":node_memory_MemAvailable_bytes:sum"}
            --match={__name__="ALERTS"}
            --match={__name__="acm_managed_cluster_labels"}
            --match={__name__="authenticated_user_requests"}
            --match={__name__="authentication_attempts"}
            --match={__name__="cluster:capacity_cpu_cores:sum"}
            --match={__name__="cluster:capacity_memory_bytes:sum"}
            --match={__name__="cluster:container_cpu_usage:ratio"}
            --match={__name__="cluster:container_spec_cpu_shares:ratio"}
            --match={__name__="cluster:cpu_usage_cores:sum"}
            --match={__name__="cluster:memory_usage:ratio"}
            --match={__name__="cluster:memory_usage_bytes:sum"}
            --match={__name__="cluster:node_cpu:ratio"}
            --match={__name__="cluster:usage:resources:sum"}
            --match={__name__="cluster_infrastructure_provider"}
            --match={__name__="cluster_version"}
            --match={__name__="cluster_version_payload"}
            --match={__name__="container_spec_cpu_quota"}
            --match={__name__="coredns_dns_requests_total"}
            --match={__name__="coredns_dns_request_duration_seconds_sum"}
            --match={__name__="coredns_forward_responses_total"}
            --match={__name__="csv_succeeded"}
            --match={__name__="csv_abnormal"}
            --match={__name__="etcd_debugging_mvcc_db_total_size_in_bytes"}
            --match={__name__="etcd_mvcc_db_total_size_in_bytes"}
            --match={__name__="etcd_debugging_snap_save_total_duration_seconds_sum"}
            --match={__name__="etcd_disk_backend_commit_duration_seconds_bucket"}
            --match={__name__="etcd_disk_backend_commit_duration_seconds_sum"}
            --match={__name__="etcd_disk_wal_fsync_duration_seconds_bucket"}
            --match={__name__="etcd_disk_wal_fsync_duration_seconds_sum"}
            --match={__name__="etcd_object_counts"}
            --match={__name__="etcd_network_client_grpc_received_bytes_total"}
            --match={__name__="etcd_network_client_grpc_sent_bytes_total"}
            --match={__name__="etcd_network_peer_received_bytes_total"}
            --match={__name__="etcd_network_peer_sent_bytes_total"}
            --match={__name__="etcd_server_client_requests_total"}
            --match={__name__="etcd_server_has_leader"}
            --match={__name__="etcd_server_health_failures"}
            --match={__name__="etcd_server_leader_changes_seen_total"}
            --match={__name__="etcd_server_proposals_failed_total"}
            --match={__name__="etcd_server_proposals_pending"}
            --match={__name__="etcd_server_proposals_committed_total"}
            --match={__name__="etcd_server_proposals_applied_total"}
            --match={__name__="etcd_server_quota_backend_bytes"}
            --match={__name__="grpc_server_started_total"}
            --match={__name__="haproxy_backend_connection_errors_total"}
            --match={__name__="haproxy_backend_connections_total"}
            --match={__name__="haproxy_backend_current_queue"}
            --match={__name__="haproxy_backend_http_average_response_latency_milliseconds"}
            --match={__name__="haproxy_backend_max_sessions"}
            --match={__name__="haproxy_backend_response_errors_total"}
            --match={__name__="haproxy_backend_up"}
            --match={__name__="http_requests_total"}
            --match={__name__="instance:node_filesystem_usage:sum"}
            --match={__name__="instance:node_cpu_utilisation:rate1m"}
            --match={__name__="instance:node_load1_per_cpu:ratio"}
            --match={__name__="instance:node_memory_utilisation:ratio"}
            --match={__name__="instance:node_network_receive_bytes_excluding_lo:rate1m"}
            --match={__name__="instance:node_network_receive_drop_excluding_lo:rate1m"}
            --match={__name__="instance:node_network_transmit_bytes_excluding_lo:rate1m"}
            --match={__name__="instance:node_network_transmit_drop_excluding_lo:rate1m"}
            --match={__name__="instance:node_num_cpu:sum"}
            --match={__name__="instance:node_vmstat_pgmajfault:rate1m"}
            --match={__name__="instance_device:node_disk_io_time_seconds:rate1m"}
            --match={__name__="instance_device:node_disk_io_time_weighted_seconds:rate1m"}
            --match={__name__="kube_daemonset_status_desired_number_scheduled"}
            --match={__name__="kube_daemonset_status_number_unavailable"}
            --match={__name__="kube_node_spec_unschedulable"}
            --match={__name__="kube_node_status_allocatable"}
            --match={__name__="kube_node_status_allocatable_cpu_cores"}
            --match={__name__="kube_node_status_allocatable_memory_bytes"}
            --match={__name__="kube_node_status_capacity"}
            --match={__name__="kube_node_status_capacity_pods"}
            --match={__name__="kube_node_status_capacity_cpu_cores"}
            --match={__name__="kube_node_status_condition"}
            --match={__name__="kube_pod_container_resource_limits_cpu_cores"}
            --match={__name__="kube_pod_container_resource_limits_memory_bytes"}
            --match={__name__="kube_pod_container_resource_requests_cpu_cores"}
            --match={__name__="kube_pod_container_resource_requests_memory_bytes"}
            --match={__name__="kube_pod_info"}
            --match={__name__="kube_pod_owner"}
            --match={__name__="kube_resourcequota"}
            --match={__name__="kubelet_running_container_count"}
            --match={__name__="kubelet_runtime_operations"}
            --match={__name__="kubelet_runtime_operations_duration_seconds_sum"}
            --match={__name__="kubelet_volume_stats_available_bytes"}
            --match={__name__="kubelet_volume_stats_capacity_bytes"}
            --match={__name__="kube_persistentvolume_status_phase"}
            --match={__name__="machine_cpu_cores"}
            --match={__name__="machine_memory_bytes"}
            --match={__name__="mce_hs_addon_request_based_hcp_capacity_gauge"}
            --match={__name__="mce_hs_addon_low_qps_based_hcp_capacity_gauge"}
            --match={__name__="mce_hs_addon_medium_qps_based_hcp_capacity_gauge"}
            --match={__name__="mce_hs_addon_high_qps_based_hcp_capacity_gauge"}
            --match={__name__="mce_hs_addon_average_qps_based_hcp_capacity_gauge"}
            --match={__name__="mce_hs_addon_total_hosted_control_planes_gauge"}
            --match={__name__="mce_hs_addon_available_hosted_control_planes_gauge"}
            --match={__name__="mce_hs_addon_available_hosted_clusters_gauge"}
            --match={__name__="mce_hs_addon_deleted_hosted_clusters_gauge"}
            --match={__name__="mce_hs_addon_hypershift_operator_degraded_bool"}
            --match={__name__="mce_hs_addon_hosted_control_planes_status_gauge"}
            --match={__name__="mce_hs_addon_qps_based_hcp_capacity_gauge"}
            --match={__name__="mce_hs_addon_worker_node_resource_capacities_gauge"}
            --match={__name__="mce_hs_addon_qps_gauge"}
            --match={__name__="mce_hs_addon_request_based_hcp_capacity_current_gauge"}
            --match={__name__="mixin_pod_workload"}
            --match={__name__="namespace:kube_pod_container_resource_requests_cpu_cores:sum"}
            --match={__name__="namespace_memory:kube_pod_container_resource_requests:sum"}
            --match={__name__="namespace:container_memory_usage_bytes:sum"}
            --match={__name__="namespace_cpu:kube_pod_container_resource_requests:sum"}
            --match={__name__="node_cpu_seconds_total"}
            --match={__name__="node_filesystem_avail_bytes"}
            --match={__name__="node_filesystem_free_bytes"}
            --match={__name__="node_filesystem_size_bytes"}
            --match={__name__="node_memory_MemAvailable_bytes"}
            --match={__name__="node_netstat_Tcp_OutSegs"}
            --match={__name__="node_netstat_Tcp_RetransSegs"}
            --match={__name__="node_netstat_TcpExt_TCPSynRetrans"}
            --match={__name__="policyreport_info"}
            --match={__name__="up"}
            --match={__name__="prometheus_operator_reconcile_errors_total"}
            --match={__name__="prometheus_operator_reconcile_operations_total"}
            --match={__name__="cluster_operator_conditions"}
            --match={__name__="cluster_operator_up"}
            --match={__name__="cluster:policy_governance_info:propagated_count"}
            --match={__name__="cluster:policy_governance_info:propagated_noncompliant_count"}
            --match={__name__="policy:policy_governance_info:propagated_count"}
            --match={__name__="policy:policy_governance_info:propagated_noncompliant_count"}
            --match={__name__="cnv:vmi_status_running:count"}
            --match={__name__="kubevirt_hyperconverged_operator_health_status"}
            --match={__name__="workqueue_queue_duration_seconds_bucket",job="apiserver"}
            --match={__name__="workqueue_adds_total",job="apiserver"}
            --match={__name__="workqueue_depth",job="apiserver"}
            --match={__name__="go_goroutines",job="apiserver"}
            --match={__name__="process_cpu_seconds_total",job="apiserver"}
            --match={__name__="process_resident_memory_bytes",job="apiserver"}
            --rename="etcd_mvcc_db_total_size_in_bytes=etcd_debugging_mvcc_db_total_size_in_bytes"
            --rename="mixin_pod_workload=namespace_workload_pod:kube_pod_owner:relabel"
            --rename="namespace:kube_pod_container_resource_requests_cpu_cores:sum=namespace_cpu:kube_pod_container_resource_requests:sum"
            --rename="node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate=node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate"
            --recordingrule={"name":"apiserver_request_duration_seconds:histogram_quantile_99","query":"histogram_quantile(0.99,sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\", verb!=\"WATCH\"}[5m])) by (le))"}
            --recordingrule={"name":"apiserver_request_duration_seconds:histogram_quantile_99:instance","query":"histogram_quantile(0.99, sum(rate(apiserver_request_duration_seconds_bucket{job=\"apiserver\", verb!=\"WATCH\"}[5m])) by (le, verb, instance))"}
            --recordingrule={"name":"sum:apiserver_request_total:1h","query":"sum(rate(apiserver_request_total{job=\"apiserver\"}[1h])) by(code, instance)"}
            --recordingrule={"name":"sum:apiserver_request_total:5m","query":"sum(rate(apiserver_request_total{job=\"apiserver\"}[5m])) by(code, instance)"}
            --recordingrule={"name":"rpc_rate:grpc_server_handled_total:sum_rate","query":"sum(rate(grpc_server_handled_total{job=\"etcd\",grpc_type=\"unary\",grpc_code!=\"OK\"}[5m]))"}
            --recordingrule={"name":"active_streams_watch:grpc_server_handled_total:sum","query":"sum(grpc_server_started_total{job=\"etcd\",grpc_service=\"etcdserverpb.Watch\",grpc_type=\"bidi_stream\"}) - sum(grpc_server_handled_total{job=\"etcd\",grpc_service=\"etcdserverpb.Watch\",grpc_type=\"bidi_stream\"})"}
            --recordingrule={"name":"active_streams_lease:grpc_server_handled_total:sum","query":"sum(grpc_server_started_total{job=\"etcd\",grpc_service=\"etcdserverpb.Lease\",grpc_type=\"bidi_stream\"}) - sum(grpc_server_handled_total{job=\"etcd\",grpc_service=\"etcdserverpb.Lease\",grpc_type=\"bidi_stream\"})"}
            --recordingrule={"name":"cluster:kube_pod_container_resource_requests:cpu:sum","query":"sum(sum(sum(kube_pod_container_resource_requests{resource=\"cpu\"}) by (pod,namespace,container) * on(pod,namespace) group_left(phase) max(kube_pod_status_phase{phase=~\"Running|Pending|Unknown\"} >0) by (pod,namespace,phase)) by (pod,namespace,phase))"}
            --recordingrule={"name":"cluster:kube_pod_container_resource_requests:memory:sum","query":"sum(sum(sum(kube_pod_container_resource_requests{resource=\"memory\"}) by (pod,namespace,container) * on(pod,namespace) group_left(phase) max(kube_pod_status_phase{phase=~\"Running|Pending|Unknown\"} >0) by (pod,namespace,phase)) by (pod,namespace,phase))"}
            --recordingrule={"name":"sli:apiserver_request_duration_seconds:trend:1m","query":"sum(increase(apiserver_request_duration_seconds_bucket{job=\"apiserver\",service=\"kubernetes\",le=\"1\",verb=~\"POST|PUT|DELETE|PATCH\"}[1m])) / sum(increase(apiserver_request_duration_seconds_count{job=\"apiserver\",service=\"kubernetes\",verb=~\"POST|PUT|DELETE|PATCH\"}[1m]))"}
            --recordingrule={"name":"container_memory_rss:sum","query":"sum(container_memory_rss) by (container, namespace)"}
            --recordingrule={"name":"kube_pod_container_resource_limits:sum","query":"sum(kube_pod_container_resource_limits) by (resource, namespace)"}
            --recordingrule={"name":"kube_pod_container_resource_requests:sum","query":"sum(kube_pod_container_resource_requests{container!=\"\"}) by (resource, namespace)"}
            --recordingrule={"name":"namespace_workload_pod:kube_pod_owner:relabel:avg","query":"count(avg(namespace_workload_pod:kube_pod_owner:relabel{pod!=\"\"}) by (workload, namespace)) by (namespace)"}
            --recordingrule={"name":"node_namespace_pod_container:container_cpu_usage_seconds_total:sum","query":"sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{container!=\"\"}) by (namespace) or sum(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate{container!=\"\"}) by (namespace)"}
          Requests:
            cpu:     10m
            memory:  100Mi
          Environment:
            FROM:        https://prometheus-k8s.openshift-monitoring.svc:9091
            FROM_QUERY:  https://prometheus-k8s.openshift-monitoring.svc:9091
            TO:          https://observatorium-api-open-cluster-management-observability.apps.acm-lta.rdu2.scalelab.redhat.com/api/metrics/v1/default/api/v1/receive
          Mounts:
            /etc/serving-certs-ca-bundle from serving-certs-ca-bundle (rw)
            /tlscerts/ca from mtlsca (rw)
            /tlscerts/certs from mtlscerts (rw)
        Volumes:
         mtlscerts:
          Type:        Secret (a volume populated by a Secret)
          SecretName:  observability-controller-open-cluster-management.io-observability-signer-client-cert
          Optional:    false
         mtlsca:
          Type:        Secret (a volume populated by a Secret)
          SecretName:  observability-managed-cluster-certs
          Optional:    false
         secret-kube-rbac-proxy-tls:
          Type:        Secret (a volume populated by a Secret)
          SecretName:  metrics-collector-kube-rbac-tls
          Optional:    false
         secret-kube-rbac-proxy-metric:
          Type:        Secret (a volume populated by a Secret)
          SecretName:  metrics-collector-kube-rbac-proxy-metric
          Optional:    false
         metrics-client-ca:
          Type:      ConfigMap (a volume populated by a ConfigMap)
          Name:      metrics-collector-clientca-metric
          Optional:  false
         serving-certs-ca-bundle:
          Type:      ConfigMap (a volume populated by a ConfigMap)
          Name:      metrics-collector-serving-certs-ca-bundle
          Optional:  false
      Conditions:
        Type           Status  Reason
        ----           ------  ------
        Available      True    MinimumReplicasAvailable
        Progressing    True    NewReplicaSetAvailable
      OldReplicaSets:  metrics-collector-deployment-5975cf4dd5 (0/0 replicas created), metrics-collector-deployment-9b7cf969f (0/0 replicas created), metrics-collector-deployment-676d87bd4f (0/0 replicas created), metrics-collector-deployment-845f49d8f7 (0/0 replicas created), metrics-collector-deployment-74476d9897 (0/0 replicas created), metrics-collector-deployment-b75df99cf (0/0 replicas created), metrics-collector-deployment-8556767ff8 (0/0 replicas created), metrics-collector-deployment-659499dd4f (0/0 replicas created), metrics-collector-deployment-d76fff759 (0/0 replicas created), metrics-collector-deployment-cc84469cb (0/0 replicas created)
      NewReplicaSet:   metrics-collector-deployment-68c55d6849 (1/1 replicas created)
      Events:
        Type    Reason             Age                    From                   Message
        ----    ------             ----                   ----                   -------
        Normal  ScalingReplicaSet  172m                   deployment-controller  Scaled up replica set metrics-collector-deployment-59dd6689f9 to 1
        Normal  ScalingReplicaSet  172m (x15 over 5h2m)   deployment-controller  Scaled up replica set metrics-collector-deployment-5975cf4dd5 to 1 from 0
        Normal  ScalingReplicaSet  172m                   deployment-controller  Scaled down replica set metrics-collector-deployment-59dd6689f9 to 0 from 1
        Normal  ScalingReplicaSet  168m                   deployment-controller  Scaled up replica set metrics-collector-deployment-7bc96c45f5 to 1
        Normal  ScalingReplicaSet  142m (x67 over 4h52m)  deployment-controller  (combined from similar events): Scaled up replica set metrics-collector-deployment-8b9849768 to 1
        Normal  ScalingReplicaSet  137m                   deployment-controller  Scaled up replica set metrics-collector-deployment-755c8988b5 to 1
        Normal  ScalingReplicaSet  137m                   deployment-controller  Scaled up replica set metrics-collector-deployment-755c8988b5 to 1 from 0
        Normal  ScalingReplicaSet  137m (x2 over 137m)    deployment-controller  Scaled down replica set metrics-collector-deployment-755c8988b5 to 0 from 1
        Normal  ScalingReplicaSet  135m                   deployment-controller  Scaled up replica set metrics-collector-deployment-59f88b7488 to 1
        Normal  ScalingReplicaSet  135m                   deployment-controller  Scaled up replica set metrics-collector-deployment-59f88b7488 to 1 from 0
        Normal  ScalingReplicaSet  132m                   deployment-controller  Scaled up replica set metrics-collector-deployment-6fc8b98f4 to 1
        Normal  ScalingReplicaSet  132m (x2 over 132m)    deployment-controller  Scaled down replica set metrics-collector-deployment-6fc8b98f4 to 0 from 1
        Normal  ScalingReplicaSet  132m (x2 over 135m)    deployment-controller  Scaled down replica set metrics-collector-deployment-59f88b7488 to 0 from 1
        Normal  ScalingReplicaSet  122m                   deployment-controller  Scaled up replica set metrics-collector-deployment-778d66db75 to 1
        Normal  ScalingReplicaSet  122m                   deployment-controller  Scaled down replica set metrics-collector-deployment-778d66db75 to 0 from 1
        Normal  ScalingReplicaSet  122m                   deployment-controller  Scaled up replica set metrics-collector-deployment-778d66db75 to 1 from 0
        Normal  ScalingReplicaSet  121m (x2 over 121m)    deployment-controller  Scaled down replica set metrics-collector-deployment-67bf7d66b6 to 0 from 1
        Normal  ScalingReplicaSet  121m (x2 over 122m)    deployment-controller  Scaled up replica set metrics-collector-deployment-67bf7d66b6 to 1 from 0
        Normal  ScalingReplicaSet  92m (x27 over 132m)    deployment-controller  (combined from similar events): Scaled down replica set metrics-collector-deployment-7cb9666698 to 0 from 1
        Normal  ScalingReplicaSet  72m (x11 over 132m)    deployment-controller  Scaled up replica set metrics-collector-deployment-5975cf4dd5 to 1 from 0
        Normal  ScalingReplicaSet  72m (x12 over 135m)    deployment-controller  Scaled down replica set metrics-collector-deployment-5975cf4dd5 to 0 from 1
        Normal  ScalingReplicaSet  72m                    deployment-controller  Scaled up replica set metrics-collector-deployment-9b7cf969f to 1 from 0
        Normal  ScalingReplicaSet  66m                    deployment-controller  Scaled up replica set metrics-collector-deployment-676d87bd4f to 1
        Normal  ScalingReplicaSet  66m                    deployment-controller  Scaled down replica set metrics-collector-deployment-676d87bd4f to 0 from 1
        Normal  ScalingReplicaSet  59m                    deployment-controller  Scaled up replica set metrics-collector-deployment-845f49d8f7 to 1
        Normal  ScalingReplicaSet  52m                    deployment-controller  Scaled up replica set metrics-collector-deployment-74476d9897 to 1
        Normal  ScalingReplicaSet  52m (x2 over 52m)      deployment-controller  Scaled down replica set metrics-collector-deployment-74476d9897 to 0 from 1
        Normal  ScalingReplicaSet  52m                    deployment-controller  Scaled up replica set metrics-collector-deployment-74476d9897 to 1 from 0
        Normal  ScalingReplicaSet  51m                    deployment-controller  Scaled up replica set metrics-collector-deployment-b75df99cf to 1
        Normal  ScalingReplicaSet  51m                    deployment-controller  Scaled down replica set metrics-collector-deployment-b75df99cf to 0 from 1
        Normal  ScalingReplicaSet  35m                    deployment-controller  Scaled up replica set metrics-collector-deployment-8556767ff8 to 1
        Normal  ScalingReplicaSet  15m                    deployment-controller  Scaled up replica set metrics-collector-deployment-659499dd4f to 1 from 0
        Normal  ScalingReplicaSet  15m (x2 over 15m)      deployment-controller  Scaled up replica set metrics-collector-deployment-5975cf4dd5 to 1 from 0
        Normal  ScalingReplicaSet  15m                    deployment-controller  Scaled down replica set metrics-collector-deployment-659499dd4f to 0 from 1
        Normal  ScalingReplicaSet  15m                    deployment-controller  Scaled down replica set metrics-collector-deployment-8556767ff8 to 0 from 1
        Normal  ScalingReplicaSet  13m                    deployment-controller  Scaled up replica set metrics-collector-deployment-d76fff759 to 1
        Normal  ScalingReplicaSet  13m                    deployment-controller  Scaled down replica set metrics-collector-deployment-d76fff759 to 0 from 1
        Normal  ScalingReplicaSet  13m                    deployment-controller  Scaled up replica set metrics-collector-deployment-cc84469cb to 1
        Normal  ScalingReplicaSet  13m                    deployment-controller  Scaled down replica set metrics-collector-deployment-cc84469cb to 0 from 1
        Normal  ScalingReplicaSet  12m (x3 over 12m)      deployment-controller  (combined from similar events): Scaled up replica set metrics-collector-deployment-68c55d6849 to 1 from 0
        Normal  ScalingReplicaSet  11m (x3 over 35m)      deployment-controller  Scaled down replica set metrics-collector-deployment-5975cf4dd5 to 0 from 1

       

            rh-ee-coquadro Coleen Iona Quadros
            akrzos@redhat.com Alex Krzos
            Xiang Yin Xiang Yin
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: