I1017 20:46:35.357762 1 observer_polling.go:159] Starting file observer W1017 20:46:35.361078 1 cmd.go:227] Using insecure, self-signed certificates I1017 20:46:36.907994 1 leaderelection.go:122] The leader election gives 4 retries and allows for 30s of clock skew. The kube-apiserver downtime tolerance is 78s. Worst non-graceful lease acquisition is 2m43s. Worst graceful lease acquisition is {26s}. I1017 20:46:36.908667 1 observer_polling.go:159] Starting file observer W1017 20:46:36.963291 1 builder.go:230] unable to get owner reference (falling back to namespace): pods is forbidden: User "system:serviceaccount:open-cluster-management-agent:klusterlet-registration-sa" cannot list resource "pods" in API group "" in the namespace "open-cluster-management-agent" I1017 20:46:36.963557 1 builder.go:262] registration-agent version - W1017 20:46:42.302331 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W1017 20:46:42.302358 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W1017 20:46:42.303251 1 builder.go:321] unable to get cluster infrastructure status, using HA cluster values for leader election: infrastructures.config.openshift.io "cluster" is forbidden: User "system:serviceaccount:open-cluster-management-agent:klusterlet-registration-sa" cannot get resource "infrastructures" in API group "config.openshift.io" at the cluster scope I1017 20:46:42.303564 1 leaderelection.go:248] attempting to acquire leader lease open-cluster-management-agent/registration-agent-lock... I1017 20:46:42.304240 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'ClusterInfrastructureStatus' unable to get cluster infrastructure status, using HA cluster values for leader election: infrastructures.config.openshift.io "cluster" is forbidden: User "system:serviceaccount:open-cluster-management-agent:klusterlet-registration-sa" cannot get resource "infrastructures" in API group "config.openshift.io" at the cluster scope I1017 20:46:43.523705 1 leaderelection.go:258] successfully acquired lease open-cluster-management-agent/registration-agent-lock I1017 20:46:43.525032 1 event.go:285] Event(v1.ObjectReference{Kind:"Lease", Namespace:"open-cluster-management-agent", Name:"registration-agent-lock", UID:"83aad9ae-ca34-4c92-85b9-c59521435d53", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"135473547", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' klusterlet-registration-agent-cd79d8c8-z5g4c_f9d19724-22cb-4548-a0a3-44dd40a8e837 became leader I1017 20:46:44.198799 1 secure_serving.go:210] Serving securely on [::]:8443 I1017 20:46:44.198879 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController I1017 20:46:44.198907 1 shared_informer.go:270] Waiting for caches to sync for RequestHeaderAuthRequestController I1017 20:46:44.198985 1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/serving-cert-2635642124/tls.crt::/tmp/serving-cert-2635642124/tls.key" I1017 20:46:44.199121 1 tlsconfig.go:240] "Starting DynamicServingCertificateController" I1017 20:46:44.201350 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I1017 20:46:44.201376 1 shared_informer.go:270] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I1017 20:46:44.201890 1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I1017 20:46:44.201899 1 shared_informer.go:270] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I1017 20:46:44.298929 1 spokeagent.go:138] Cluster name is "build0" and agent name is "9jd7s" I1017 20:46:44.303337 1 base_controller.go:67] Waiting for caches to sync for ManagedClusterCreatingController I1017 20:46:44.303355 1 base_controller.go:73] Caches are synced for ManagedClusterCreatingController I1017 20:46:44.303362 1 base_controller.go:110] Starting #1 worker of ManagedClusterCreatingController controller ... I1017 20:46:44.303431 1 base_controller.go:67] Waiting for caches to sync for HubKubeconfigSecretController I1017 20:46:44.303997 1 shared_informer.go:277] Caches are synced for RequestHeaderAuthRequestController I1017 20:46:44.304643 1 base_controller.go:67] Waiting for caches to sync for AddOnRegistrationController I1017 20:46:44.304660 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'HubClientConfigReady' Client config for hub is ready. I1017 20:46:44.304912 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'FastControllerResync' Controller "ClusterClaimController" resync interval is set to 0s which might lead to client request throttling I1017 20:46:44.304944 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'FastControllerResync' Controller "ManagedClusterAddOnLeaseController" resync interval is set to 30s which might lead to client request throttling I1017 20:46:44.304967 1 base_controller.go:67] Waiting for caches to sync for ClientCertController@cluster:build0 I1017 20:46:44.304981 1 base_controller.go:67] Waiting for caches to sync for ManagedClusterLeaseController I1017 20:46:44.305003 1 base_controller.go:67] Waiting for caches to sync for ManagedClusterStatusController I1017 20:46:44.305028 1 base_controller.go:67] Waiting for caches to sync for ClusterClaimController I1017 20:46:44.305047 1 base_controller.go:67] Waiting for caches to sync for ManagedClusterAddOnLeaseController I1017 20:46:44.305054 1 base_controller.go:73] Caches are synced for ManagedClusterAddOnLeaseController I1017 20:46:44.305061 1 base_controller.go:110] Starting #1 worker of ManagedClusterAddOnLeaseController controller ... I1017 20:46:44.305089 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'FastControllerResync' Controller "ManagedClusterLeaseController" resync interval is set to 0s which might lead to client request throttling I1017 20:46:44.304968 1 base_controller.go:67] Waiting for caches to sync for ManagedClusterJoiningController I1017 20:46:44.316984 1 shared_informer.go:277] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I1017 20:46:44.316986 1 shared_informer.go:277] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I1017 20:46:44.421265 1 base_controller.go:73] Caches are synced for HubKubeconfigSecretController I1017 20:46:44.421308 1 base_controller.go:110] Starting #1 worker of HubKubeconfigSecretController controller ... I1017 20:46:44.504889 1 base_controller.go:73] Caches are synced for AddOnRegistrationController I1017 20:46:44.504922 1 base_controller.go:110] Starting #1 worker of AddOnRegistrationController controller ... I1017 20:46:44.506595 1 base_controller.go:67] Waiting for caches to sync for ClientCertController@addon:cert-policy-controller:signer:kubernetes.io/kube-apiserver-client I1017 20:46:44.506892 1 base_controller.go:67] Waiting for caches to sync for ClientCertController@addon:application-manager:signer:kubernetes.io/kube-apiserver-client I1017 20:46:44.507064 1 base_controller.go:67] Waiting for caches to sync for ClientCertController@addon:cluster-proxy:signer:kubernetes.io/kube-apiserver-client I1017 20:46:44.507212 1 base_controller.go:67] Waiting for caches to sync for ClientCertController@addon:cluster-proxy:signer:open-cluster-management.io/proxy-agent-signer I1017 20:46:44.507361 1 base_controller.go:67] Waiting for caches to sync for ClientCertController@addon:work-manager:signer:kubernetes.io/kube-apiserver-client I1017 20:46:44.507512 1 base_controller.go:67] Waiting for caches to sync for ClientCertController@addon:search-collector:signer:kubernetes.io/kube-apiserver-client I1017 20:46:44.507649 1 base_controller.go:67] Waiting for caches to sync for ClientCertController@addon:config-policy-controller:signer:kubernetes.io/kube-apiserver-client I1017 20:46:44.507796 1 base_controller.go:67] Waiting for caches to sync for ClientCertController@addon:iam-policy-controller:signer:kubernetes.io/kube-apiserver-client I1017 20:46:44.507960 1 base_controller.go:67] Waiting for caches to sync for ClientCertController@addon:governance-policy-framework:signer:kubernetes.io/kube-apiserver-client I1017 20:46:44.517186 1 base_controller.go:73] Caches are synced for ClientCertController@cluster:build0 I1017 20:46:44.517206 1 base_controller.go:110] Starting #1 worker of ClientCertController@cluster:build0 controller ... I1017 20:46:44.524972 1 base_controller.go:73] Caches are synced for ManagedClusterJoiningController I1017 20:46:44.524986 1 base_controller.go:110] Starting #1 worker of ManagedClusterJoiningController controller ... I1017 20:46:44.556238 1 base_controller.go:73] Caches are synced for ManagedClusterLeaseController I1017 20:46:44.556257 1 base_controller.go:110] Starting #1 worker of ManagedClusterLeaseController controller ... I1017 20:46:44.556334 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterLeaseUpdateStarted' Start to update lease "managed-cluster-lease" on cluster "build0" I1017 20:46:44.556567 1 base_controller.go:73] Caches are synced for ManagedClusterStatusController I1017 20:46:44.556575 1 base_controller.go:110] Starting #1 worker of ManagedClusterStatusController controller ... I1017 20:46:44.559252 1 base_controller.go:73] Caches are synced for ClusterClaimController I1017 20:46:44.559263 1 base_controller.go:110] Starting #1 worker of ClusterClaimController controller ... I1017 20:46:44.613167 1 base_controller.go:73] Caches are synced for ClientCertController@addon:governance-policy-framework:signer:kubernetes.io/kube-apiserver-client I1017 20:46:44.613192 1 base_controller.go:110] Starting #1 worker of ClientCertController@addon:governance-policy-framework:signer:kubernetes.io/kube-apiserver-client controller ... I1017 20:46:44.613235 1 base_controller.go:73] Caches are synced for ClientCertController@addon:cert-policy-controller:signer:kubernetes.io/kube-apiserver-client I1017 20:46:44.613241 1 base_controller.go:110] Starting #1 worker of ClientCertController@addon:cert-policy-controller:signer:kubernetes.io/kube-apiserver-client controller ... I1017 20:46:44.613280 1 base_controller.go:73] Caches are synced for ClientCertController@addon:config-policy-controller:signer:kubernetes.io/kube-apiserver-client I1017 20:46:44.613287 1 base_controller.go:110] Starting #1 worker of ClientCertController@addon:config-policy-controller:signer:kubernetes.io/kube-apiserver-client controller ... I1017 20:46:44.613302 1 base_controller.go:73] Caches are synced for ClientCertController@addon:iam-policy-controller:signer:kubernetes.io/kube-apiserver-client I1017 20:46:44.613308 1 base_controller.go:110] Starting #1 worker of ClientCertController@addon:iam-policy-controller:signer:kubernetes.io/kube-apiserver-client controller ... I1017 20:46:44.711312 1 base_controller.go:73] Caches are synced for ClientCertController@addon:search-collector:signer:kubernetes.io/kube-apiserver-client I1017 20:46:44.711350 1 base_controller.go:110] Starting #1 worker of ClientCertController@addon:search-collector:signer:kubernetes.io/kube-apiserver-client controller ... I1017 20:46:44.711448 1 base_controller.go:73] Caches are synced for ClientCertController@addon:application-manager:signer:kubernetes.io/kube-apiserver-client I1017 20:46:44.711454 1 base_controller.go:110] Starting #1 worker of ClientCertController@addon:application-manager:signer:kubernetes.io/kube-apiserver-client controller ... I1017 20:46:44.711478 1 base_controller.go:73] Caches are synced for ClientCertController@addon:cluster-proxy:signer:kubernetes.io/kube-apiserver-client I1017 20:46:44.711485 1 base_controller.go:110] Starting #1 worker of ClientCertController@addon:cluster-proxy:signer:kubernetes.io/kube-apiserver-client controller ... I1017 20:46:44.711510 1 base_controller.go:73] Caches are synced for ClientCertController@addon:cluster-proxy:signer:open-cluster-management.io/proxy-agent-signer I1017 20:46:44.711516 1 base_controller.go:110] Starting #1 worker of ClientCertController@addon:cluster-proxy:signer:open-cluster-management.io/proxy-agent-signer controller ... I1017 20:46:44.711541 1 base_controller.go:73] Caches are synced for ClientCertController@addon:work-manager:signer:kubernetes.io/kube-apiserver-client I1017 20:46:44.711547 1 base_controller.go:110] Starting #1 worker of ClientCertController@addon:work-manager:signer:kubernetes.io/kube-apiserver-client controller ... I1017 20:46:45.428195 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NoValidCertificateFound' No valid client certificate for ClientCertController@addon:config-policy-controller:signer:kubernetes.io/kube-apiserver-client is found. Bootstrap is required I1017 20:46:45.431543 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' A csr "addon-build0-config-policy-controller-phwjp" is created I1017 20:46:45.440828 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NoValidCertificateFound' No valid client certificate for ClientCertController@addon:iam-policy-controller:signer:kubernetes.io/kube-apiserver-client is found. Bootstrap is required I1017 20:46:45.469737 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NoValidCertificateFound' No valid client certificate for ClientCertController@addon:governance-policy-framework:signer:kubernetes.io/kube-apiserver-client is found. Bootstrap is required I1017 20:46:45.469976 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' A csr "addon-build0-iam-policy-controller-mvl4g" is created I1017 20:46:45.493485 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' A csr "addon-build0-governance-policy-framework-s8xqd" is created I1017 20:46:45.713916 1 request.go:690] Waited for 1.002057054s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/api/v1/namespaces/open-cluster-management-agent-addon/secrets/cluster-proxy-hub-kubeconfig I1017 20:46:45.734091 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NoValidCertificateFound' No valid client certificate for ClientCertController@addon:search-collector:signer:kubernetes.io/kube-apiserver-client is found. Bootstrap is required I1017 20:46:45.742933 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' A csr "addon-build0-search-collector-jpl9z" is created I1017 20:46:46.008353 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NoValidCertificateFound' No valid client certificate for ClientCertController@addon:cluster-proxy:signer:kubernetes.io/kube-apiserver-client is found. Bootstrap is required I1017 20:46:46.015311 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' A csr "addon-build0-cluster-proxy-k6j22" is created I1017 20:46:46.018246 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NoValidCertificateFound' No valid client certificate for ClientCertController@addon:application-manager:signer:kubernetes.io/kube-apiserver-client is found. Bootstrap is required I1017 20:46:46.024254 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' A csr "addon-build0-application-manager-dcnmj" is created I1017 20:46:46.393073 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NoValidCertificateFound' No valid client certificate for ClientCertController@addon:cert-policy-controller:signer:kubernetes.io/kube-apiserver-client is found. Bootstrap is required I1017 20:46:46.401043 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' A csr "addon-build0-cert-policy-controller-22fsd" is created I1017 20:46:46.593498 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NoValidCertificateFound' No valid client certificate for ClientCertController@addon:work-manager:signer:kubernetes.io/kube-apiserver-client is found. Bootstrap is required I1017 20:46:46.598469 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' A csr "addon-build0-work-manager-hzhlf" is created I1017 20:46:47.872048 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ClientCertificateCreated' A new client certificate for ClientCertController@addon:config-policy-controller:signer:kubernetes.io/kube-apiserver-client is available I1017 20:46:48.034640 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NoValidCertificateFound' No valid client certificate for ClientCertController@addon:cluster-proxy:signer:open-cluster-management.io/proxy-agent-signer is found. Bootstrap is required I1017 20:46:48.042613 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'CSRCreated' A csr "addon-build0-cluster-proxy-tv9kd" is created I1017 20:46:48.538216 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ClientCertificateCreated' A new client certificate for ClientCertController@addon:governance-policy-framework:signer:kubernetes.io/kube-apiserver-client is available I1017 20:46:48.821132 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ClientCertificateCreated' A new client certificate for ClientCertController@addon:search-collector:signer:kubernetes.io/kube-apiserver-client is available I1017 20:46:49.213007 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ClientCertificateCreated' A new client certificate for ClientCertController@addon:application-manager:signer:kubernetes.io/kube-apiserver-client is available I1017 20:46:50.202873 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ClientCertificateCreated' A new client certificate for ClientCertController@addon:cluster-proxy:signer:kubernetes.io/kube-apiserver-client is available I1017 20:46:50.204764 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ClientCertificateCreated' A new client certificate for ClientCertController@addon:cert-policy-controller:signer:kubernetes.io/kube-apiserver-client is available I1017 20:46:51.119341 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ClientCertificateCreated' A new client certificate for ClientCertController@addon:iam-policy-controller:signer:kubernetes.io/kube-apiserver-client is available I1017 20:46:51.545842 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ClientCertificateCreated' A new client certificate for ClientCertController@addon:work-manager:signer:kubernetes.io/kube-apiserver-client is available I1017 20:46:51.937857 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ClientCertificateCreated' A new client certificate for ClientCertController@addon:cluster-proxy:signer:open-cluster-management.io/proxy-agent-signer is available I1017 21:00:15.887080 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterStatusUpdated' the status of managed cluster "build0" has been updated, available condition is "False", due to "The kube-apiserver is not ok, status code: 500, an error on the server (\"[+]ping ok\\n[+]log ok\\n[-]etcd failed: reason withheld\\n[+]poststarthook/openshift.io-deprecated-api-requests-filter ok\\n[+]poststarthook/openshift.io-startkubeinformers ok\\n[+]poststarthook/openshift.io-openshift-apiserver-reachable ok\\n[+]poststarthook/openshift.io-oauth-apiserver-reachable ok\\n[+]poststarthook/start-kube-apiserver-admission-initializer ok\\n[+]poststarthook/quota.openshift.io-clusterquotamapping ok\\n[+]poststarthook/generic-apiserver-start-informers ok\\n[+]poststarthook/priority-and-fairness-config-consumer ok\\n[+]poststarthook/priority-and-fairness-filter ok\\n[+]poststarthook/start-apiextensions-informers ok\\n[+]poststarthook/start-apiextensions-controllers ok\\n[+]poststarthook/crd-informer-synced ok\\n[+]poststarthook/bootstrap-controller ok\\n[+]poststarthook/rbac/bootstrap-roles ok\\n[+]poststarthook/scheduling/bootstrap-system-priority-classes ok\\n[+]poststarthook/priority-and-fairness-config-producer ok\\n[+]poststarthook/start-cluster-authentication-info-controller ok\\n[+]poststarthook/aggregator-reload-proxy-client-cert ok\\n[+]poststarthook/start-kube-aggregator-informers ok\\n[+]poststarthook/apiservice-registration-controller ok\\n[+]poststarthook/apiservice-status-available-controller ok\\n[+]poststarthook/apiservice-wait-for-first-sync ok\\n[+]poststarthook/kube-apiserver-autoregistration ok\\n[+]autoregister-completion ok\\n[+]poststarthook/apiservice-openapi-controller ok\\n[+]poststarthook/apiservice-openapiv3-controller ok\\nlivez check failed\") has prevented the request from succeeding" I1017 21:00:38.588707 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterStatusUpdated' the status of managed cluster "build0" has been updated, available condition is "True", due to "Managed cluster is available" I1017 21:01:46.275806 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' update managed cluster addon "governance-policy-framework" available condition to "True" with its lease "open-cluster-management-agent-addon"/"governance-policy-framework" status I1017 21:02:18.086361 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' update managed cluster addon "config-policy-controller" available condition to "True" with its lease "open-cluster-management-agent-addon"/"config-policy-controller" status I1017 21:04:44.747072 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' update managed cluster addon "search-collector" available condition to "True" with its lease "open-cluster-management-agent-addon"/"search-collector" status I1017 21:04:46.097958 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' update managed cluster addon "cert-policy-controller" available condition to "True" with its lease "open-cluster-management-agent-addon"/"cert-policy-controller" status I1017 21:04:46.273356 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' update managed cluster addon "application-manager" available condition to "True" with its lease "open-cluster-management-agent-addon"/"application-manager" status I1017 21:04:46.701271 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' update managed cluster addon "cluster-proxy" available condition to "True" with its lease "open-cluster-management-agent-addon"/"cluster-proxy" status E1017 21:08:16.458607 1 base_controller.go:270] "ManagedClusterAddOnLeaseController" controller failed to sync "open-cluster-management-agent-addon/cluster-proxy", err: rpc error: code = Unknown desc = malformed header: missing HTTP content-type I1017 21:16:14.604735 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' update managed cluster addon "work-manager" available condition to "True" with its lease "open-cluster-management-agent-addon"/"work-manager" status I1017 21:20:16.614956 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' update managed cluster addon "iam-policy-controller" available condition to "True" with its lease "open-cluster-management-agent-addon"/"iam-policy-controller" status I1017 22:02:30.662963 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterStatusUpdated' the status of managed cluster "build0" has been updated, available condition is "False", due to "The kube-apiserver is not ok, status code: 500, an error on the server (\"[+]ping ok\\n[+]log ok\\n[-]etcd failed: reason withheld\\n[+]poststarthook/openshift.io-deprecated-api-requests-filter ok\\n[+]poststarthook/openshift.io-startkubeinformers ok\\n[+]poststarthook/openshift.io-openshift-apiserver-reachable ok\\n[+]poststarthook/openshift.io-oauth-apiserver-reachable ok\\n[+]poststarthook/start-kube-apiserver-admission-initializer ok\\n[+]poststarthook/quota.openshift.io-clusterquotamapping ok\\n[+]poststarthook/generic-apiserver-start-informers ok\\n[+]poststarthook/priority-and-fairness-config-consumer ok\\n[+]poststarthook/priority-and-fairness-filter ok\\n[+]poststarthook/start-apiextensions-informers ok\\n[+]poststarthook/start-apiextensions-controllers ok\\n[+]poststarthook/crd-informer-synced ok\\n[+]poststarthook/bootstrap-controller ok\\n[+]poststarthook/rbac/bootstrap-roles ok\\n[+]poststarthook/scheduling/bootstrap-system-priority-classes ok\\n[+]poststarthook/priority-and-fairness-config-producer ok\\n[+]poststarthook/start-cluster-authentication-info-controller ok\\n[+]poststarthook/aggregator-reload-proxy-client-cert ok\\n[+]poststarthook/start-kube-aggregator-informers ok\\n[+]poststarthook/apiservice-registration-controller ok\\n[+]poststarthook/apiservice-status-available-controller ok\\n[+]poststarthook/apiservice-wait-for-first-sync ok\\n[+]poststarthook/kube-apiserver-autoregistration ok\\n[+]autoregister-completion ok\\n[+]poststarthook/apiservice-openapi-controller ok\\n[+]poststarthook/apiservice-openapiv3-controller ok\\nlivez check failed\") has prevented the request from succeeding" I1017 22:02:44.843863 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterStatusUpdated' the status of managed cluster "build0" has been updated, available condition is "True", due to "Managed cluster is available" E1017 23:27:56.609411 1 leaderelection.go:367] Failed to update lock: etcdserver: request timed out E1018 00:02:46.133442 1 leaderelection.go:367] Failed to update lock: etcdserver: request timed out E1018 00:04:21.273852 1 leaderelection.go:367] Failed to update lock: etcdserver: request timed out I1018 02:38:19.883995 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterStatusUpdated' the status of managed cluster "build0" has been updated, available condition is "False", due to "The kube-apiserver is not ok, status code: 500, an error on the server (\"[+]ping ok\\n[+]log ok\\n[-]etcd failed: reason withheld\\n[+]poststarthook/openshift.io-deprecated-api-requests-filter ok\\n[+]poststarthook/openshift.io-startkubeinformers ok\\n[+]poststarthook/openshift.io-openshift-apiserver-reachable ok\\n[+]poststarthook/openshift.io-oauth-apiserver-reachable ok\\n[+]poststarthook/start-kube-apiserver-admission-initializer ok\\n[+]poststarthook/quota.openshift.io-clusterquotamapping ok\\n[+]poststarthook/generic-apiserver-start-informers ok\\n[+]poststarthook/priority-and-fairness-config-consumer ok\\n[+]poststarthook/priority-and-fairness-filter ok\\n[+]poststarthook/start-apiextensions-informers ok\\n[+]poststarthook/start-apiextensions-controllers ok\\n[+]poststarthook/crd-informer-synced ok\\n[+]poststarthook/bootstrap-controller ok\\n[+]poststarthook/rbac/bootstrap-roles ok\\n[+]poststarthook/scheduling/bootstrap-system-priority-classes ok\\n[+]poststarthook/priority-and-fairness-config-producer ok\\n[+]poststarthook/start-cluster-authentication-info-controller ok\\n[+]poststarthook/aggregator-reload-proxy-client-cert ok\\n[+]poststarthook/start-kube-aggregator-informers ok\\n[+]poststarthook/apiservice-registration-controller ok\\n[+]poststarthook/apiservice-status-available-controller ok\\n[+]poststarthook/apiservice-wait-for-first-sync ok\\n[+]poststarthook/kube-apiserver-autoregistration ok\\n[+]autoregister-completion ok\\n[+]poststarthook/apiservice-openapi-controller ok\\n[+]poststarthook/apiservice-openapiv3-controller ok\\nlivez check failed\") has prevented the request from succeeding" I1018 02:38:21.626726 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterStatusUpdated' the status of managed cluster "build0" has been updated, available condition is "True", due to "Managed cluster is available" I1018 03:05:40.687740 1 trace.go:219] Trace[1691795791]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 03:05:29.233) (total time: 11453ms): Trace[1691795791]: ---"Objects listed" error: 11453ms (03:05:40.687) Trace[1691795791]: [11.453942364s] [11.453942364s] END I1018 03:05:40.790242 1 trace.go:219] Trace[232766345]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 03:05:29.010) (total time: 11779ms): Trace[232766345]: ---"Objects listed" error: 11779ms (03:05:40.789) Trace[232766345]: [11.779903136s] [11.779903136s] END I1018 03:05:40.794489 1 trace.go:219] Trace[569577919]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 03:05:30.608) (total time: 10185ms): Trace[569577919]: ---"Objects listed" error: 10185ms (03:05:40.794) Trace[569577919]: [10.185594582s] [10.185594582s] END I1018 03:05:41.631055 1 trace.go:219] Trace[283980271]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 03:05:29.497) (total time: 12133ms): Trace[283980271]: ---"Objects listed" error: 12133ms (03:05:41.630) Trace[283980271]: [12.133577468s] [12.133577468s] END I1018 03:05:41.631065 1 trace.go:219] Trace[1518282771]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 03:05:28.973) (total time: 12657ms): Trace[1518282771]: ---"Objects listed" error: 12656ms (03:05:41.630) Trace[1518282771]: [12.65708644s] [12.65708644s] END I1018 03:05:41.633149 1 trace.go:219] Trace[747204886]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 03:05:29.010) (total time: 12623ms): Trace[747204886]: ---"Objects listed" error: 12622ms (03:05:41.633) Trace[747204886]: [12.623039552s] [12.623039552s] END I1018 03:05:41.643576 1 trace.go:219] Trace[1812106757]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 03:05:29.354) (total time: 12288ms): Trace[1812106757]: ---"Objects listed" error: 12288ms (03:05:41.643) Trace[1812106757]: [12.288603594s] [12.288603594s] END I1018 03:05:41.835877 1 trace.go:219] Trace[1818301633]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 03:05:28.593) (total time: 13242ms): Trace[1818301633]: ---"Objects listed" error: 13242ms (03:05:41.835) Trace[1818301633]: [13.242655456s] [13.242655456s] END I1018 03:05:42.426896 1 trace.go:219] Trace[1187763714]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 03:05:28.505) (total time: 13920ms): Trace[1187763714]: ---"Objects listed" error: 13920ms (03:05:42.426) Trace[1187763714]: [13.920869546s] [13.920869546s] END I1018 03:05:42.514854 1 trace.go:219] Trace[1356681764]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 03:05:29.089) (total time: 13424ms): Trace[1356681764]: ---"Objects listed" error: 13424ms (03:05:42.514) Trace[1356681764]: [13.424837933s] [13.424837933s] END I1018 09:23:25.278458 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterStatusUpdated' the status of managed cluster "build0" has been updated, available condition is "False", due to "The kube-apiserver is not ok, status code: 500, an error on the server (\"[+]ping ok\\n[+]log ok\\n[-]etcd failed: reason withheld\\n[+]poststarthook/openshift.io-deprecated-api-requests-filter ok\\n[+]poststarthook/openshift.io-startkubeinformers ok\\n[+]poststarthook/openshift.io-openshift-apiserver-reachable ok\\n[+]poststarthook/openshift.io-oauth-apiserver-reachable ok\\n[+]poststarthook/start-kube-apiserver-admission-initializer ok\\n[+]poststarthook/quota.openshift.io-clusterquotamapping ok\\n[+]poststarthook/generic-apiserver-start-informers ok\\n[+]poststarthook/priority-and-fairness-config-consumer ok\\n[+]poststarthook/priority-and-fairness-filter ok\\n[+]poststarthook/start-apiextensions-informers ok\\n[+]poststarthook/start-apiextensions-controllers ok\\n[+]poststarthook/crd-informer-synced ok\\n[+]poststarthook/bootstrap-controller ok\\n[+]poststarthook/rbac/bootstrap-roles ok\\n[+]poststarthook/scheduling/bootstrap-system-priority-classes ok\\n[+]poststarthook/priority-and-fairness-config-producer ok\\n[+]poststarthook/start-cluster-authentication-info-controller ok\\n[+]poststarthook/aggregator-reload-proxy-client-cert ok\\n[+]poststarthook/start-kube-aggregator-informers ok\\n[+]poststarthook/apiservice-registration-controller ok\\n[+]poststarthook/apiservice-status-available-controller ok\\n[+]poststarthook/apiservice-wait-for-first-sync ok\\n[+]poststarthook/kube-apiserver-autoregistration ok\\n[+]autoregister-completion ok\\n[+]poststarthook/apiservice-openapi-controller ok\\n[+]poststarthook/apiservice-openapiv3-controller ok\\nlivez check failed\") has prevented the request from succeeding" I1018 09:23:25.560950 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterStatusUpdated' the status of managed cluster "build0" has been updated, available condition is "True", due to "Managed cluster is available" I1018 09:26:09.679208 1 trace.go:219] Trace[746960631]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 09:25:53.702) (total time: 15976ms): Trace[746960631]: ---"Objects listed" error: 15976ms (09:26:09.679) Trace[746960631]: [15.976641897s] [15.976641897s] END I1018 09:26:11.949070 1 trace.go:219] Trace[1355438880]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 09:25:53.747) (total time: 18201ms): Trace[1355438880]: ---"Objects listed" error: 18201ms (09:26:11.948) Trace[1355438880]: [18.201256227s] [18.201256227s] END I1018 09:26:11.980393 1 trace.go:219] Trace[412203935]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 09:25:53.465) (total time: 18514ms): Trace[412203935]: ---"Objects listed" error: 18514ms (09:26:11.980) Trace[412203935]: [18.514535417s] [18.514535417s] END I1018 09:26:14.593578 1 trace.go:219] Trace[1444079499]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 09:25:54.214) (total time: 20379ms): Trace[1444079499]: ---"Objects listed" error: 20379ms (09:26:14.593) Trace[1444079499]: [20.379483736s] [20.379483736s] END I1018 09:26:14.600444 1 trace.go:219] Trace[1459210863]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 09:25:54.240) (total time: 20359ms): Trace[1459210863]: ---"Objects listed" error: 20359ms (09:26:14.600) Trace[1459210863]: [20.359468244s] [20.359468244s] END I1018 09:26:14.603886 1 trace.go:219] Trace[1694262089]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 09:25:54.029) (total time: 20574ms): Trace[1694262089]: ---"Objects listed" error: 20574ms (09:26:14.603) Trace[1694262089]: [20.574509983s] [20.574509983s] END I1018 09:26:14.627793 1 trace.go:219] Trace[1483089669]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 09:25:54.238) (total time: 20388ms): Trace[1483089669]: ---"Objects listed" error: 20388ms (09:26:14.627) Trace[1483089669]: [20.38882249s] [20.38882249s] END I1018 09:26:14.627798 1 trace.go:219] Trace[2125804956]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 09:25:54.086) (total time: 20541ms): Trace[2125804956]: ---"Objects listed" error: 20541ms (09:26:14.627) Trace[2125804956]: [20.541376283s] [20.541376283s] END I1018 09:26:14.629601 1 trace.go:219] Trace[1924967193]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 09:25:53.884) (total time: 20745ms): Trace[1924967193]: ---"Objects listed" error: 20744ms (09:26:14.629) Trace[1924967193]: [20.745007418s] [20.745007418s] END I1018 09:26:14.630824 1 trace.go:219] Trace[932984292]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 09:25:53.992) (total time: 20638ms): Trace[932984292]: ---"Objects listed" error: 20638ms (09:26:14.630) Trace[932984292]: [20.638518035s] [20.638518035s] END I1018 09:30:48.057559 1 trace.go:219] Trace[574638930]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 09:30:17.827) (total time: 30230ms): Trace[574638930]: ---"Objects listed" error: 30230ms (09:30:48.057) Trace[574638930]: [30.23045556s] [30.23045556s] END I1018 09:45:15.319389 1 trace.go:219] Trace[715638944]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 09:45:00.058) (total time: 15261ms): Trace[715638944]: ---"Objects listed" error: 15261ms (09:45:15.319) Trace[715638944]: [15.261181047s] [15.261181047s] END I1018 09:45:15.413365 1 trace.go:219] Trace[1106082410]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 09:44:59.910) (total time: 15503ms): Trace[1106082410]: ---"Objects listed" error: 15502ms (09:45:15.413) Trace[1106082410]: [15.503084216s] [15.503084216s] END I1018 09:45:15.416412 1 trace.go:219] Trace[1771497180]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 09:44:59.539) (total time: 15876ms): Trace[1771497180]: ---"Objects listed" error: 15876ms (09:45:15.416) Trace[1771497180]: [15.876858074s] [15.876858074s] END I1018 09:45:15.556793 1 trace.go:219] Trace[1011535948]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 09:45:00.325) (total time: 15231ms): Trace[1011535948]: ---"Objects listed" error: 15230ms (09:45:15.556) Trace[1011535948]: [15.231055392s] [15.231055392s] END I1018 09:45:15.641738 1 trace.go:219] Trace[642332636]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 09:45:00.233) (total time: 15408ms): Trace[642332636]: ---"Objects listed" error: 15408ms (09:45:15.641) Trace[642332636]: [15.40859349s] [15.40859349s] END I1018 10:09:48.352742 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterStatusUpdated' the status of managed cluster "build0" has been updated, available condition is "False", due to "The kube-apiserver is not ok, status code: 500, an error on the server (\"[+]ping ok\\n[+]log ok\\n[-]etcd failed: reason withheld\\n[+]poststarthook/openshift.io-deprecated-api-requests-filter ok\\n[+]poststarthook/openshift.io-startkubeinformers ok\\n[+]poststarthook/openshift.io-openshift-apiserver-reachable ok\\n[+]poststarthook/openshift.io-oauth-apiserver-reachable ok\\n[+]poststarthook/start-kube-apiserver-admission-initializer ok\\n[+]poststarthook/quota.openshift.io-clusterquotamapping ok\\n[+]poststarthook/generic-apiserver-start-informers ok\\n[+]poststarthook/priority-and-fairness-config-consumer ok\\n[+]poststarthook/priority-and-fairness-filter ok\\n[+]poststarthook/start-apiextensions-informers ok\\n[+]poststarthook/start-apiextensions-controllers ok\\n[+]poststarthook/crd-informer-synced ok\\n[+]poststarthook/bootstrap-controller ok\\n[+]poststarthook/rbac/bootstrap-roles ok\\n[+]poststarthook/scheduling/bootstrap-system-priority-classes ok\\n[+]poststarthook/priority-and-fairness-config-producer ok\\n[+]poststarthook/start-cluster-authentication-info-controller ok\\n[+]poststarthook/aggregator-reload-proxy-client-cert ok\\n[+]poststarthook/start-kube-aggregator-informers ok\\n[+]poststarthook/apiservice-registration-controller ok\\n[+]poststarthook/apiservice-status-available-controller ok\\n[+]poststarthook/apiservice-wait-for-first-sync ok\\n[+]poststarthook/kube-apiserver-autoregistration ok\\n[+]autoregister-completion ok\\n[+]poststarthook/apiservice-openapi-controller ok\\n[+]poststarthook/apiservice-openapiv3-controller ok\\nlivez check failed\") has prevented the request from succeeding" I1018 10:10:31.207929 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterStatusUpdated' the status of managed cluster "build0" has been updated, available condition is "True", due to "Managed cluster is available" I1018 10:12:46.585368 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterStatusUpdated' the status of managed cluster "build0" has been updated, available condition is "False", due to "The kube-apiserver is not ok, status code: 500, an error on the server (\"[+]ping ok\\n[+]log ok\\n[-]etcd failed: reason withheld\\n[+]poststarthook/openshift.io-deprecated-api-requests-filter ok\\n[+]poststarthook/openshift.io-startkubeinformers ok\\n[+]poststarthook/openshift.io-openshift-apiserver-reachable ok\\n[+]poststarthook/openshift.io-oauth-apiserver-reachable ok\\n[+]poststarthook/start-kube-apiserver-admission-initializer ok\\n[+]poststarthook/quota.openshift.io-clusterquotamapping ok\\n[+]poststarthook/generic-apiserver-start-informers ok\\n[+]poststarthook/priority-and-fairness-config-consumer ok\\n[+]poststarthook/priority-and-fairness-filter ok\\n[+]poststarthook/start-apiextensions-informers ok\\n[+]poststarthook/start-apiextensions-controllers ok\\n[+]poststarthook/crd-informer-synced ok\\n[+]poststarthook/bootstrap-controller ok\\n[+]poststarthook/rbac/bootstrap-roles ok\\n[+]poststarthook/scheduling/bootstrap-system-priority-classes ok\\n[+]poststarthook/priority-and-fairness-config-producer ok\\n[+]poststarthook/start-cluster-authentication-info-controller ok\\n[+]poststarthook/aggregator-reload-proxy-client-cert ok\\n[+]poststarthook/start-kube-aggregator-informers ok\\n[+]poststarthook/apiservice-registration-controller ok\\n[+]poststarthook/apiservice-status-available-controller ok\\n[+]poststarthook/apiservice-wait-for-first-sync ok\\n[+]poststarthook/kube-apiserver-autoregistration ok\\n[+]autoregister-completion ok\\n[+]poststarthook/apiservice-openapi-controller ok\\n[+]poststarthook/apiservice-openapiv3-controller ok\\nlivez check failed\") has prevented the request from succeeding" I1018 10:13:23.042683 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterStatusUpdated' the status of managed cluster "build0" has been updated, available condition is "True", due to "Managed cluster is available" I1018 10:13:28.901617 1 trace.go:219] Trace[1837709068]: "Reflector ListAndWatch" name:k8s.io/client-go@v0.26.7/tools/cache/reflector.go:169 (18-Oct-2023 10:13:18.379) (total time: 10522ms): Trace[1837709068]: ---"Objects listed" error: 10522ms (10:13:28.901) Trace[1837709068]: [10.522144388s] [10.522144388s] END I1018 10:13:49.281978 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterStatusUpdated' the status of managed cluster "build0" has been updated, available condition is "False", due to "The kube-apiserver is not ok, status code: 500, an error on the server (\"[+]ping ok\\n[+]log ok\\n[-]etcd failed: reason withheld\\n[+]poststarthook/openshift.io-deprecated-api-requests-filter ok\\n[+]poststarthook/openshift.io-startkubeinformers ok\\n[+]poststarthook/openshift.io-openshift-apiserver-reachable ok\\n[+]poststarthook/openshift.io-oauth-apiserver-reachable ok\\n[+]poststarthook/start-kube-apiserver-admission-initializer ok\\n[+]poststarthook/quota.openshift.io-clusterquotamapping ok\\n[+]poststarthook/generic-apiserver-start-informers ok\\n[+]poststarthook/priority-and-fairness-config-consumer ok\\n[+]poststarthook/priority-and-fairness-filter ok\\n[+]poststarthook/start-apiextensions-informers ok\\n[+]poststarthook/start-apiextensions-controllers ok\\n[+]poststarthook/crd-informer-synced ok\\n[+]poststarthook/bootstrap-controller ok\\n[+]poststarthook/rbac/bootstrap-roles ok\\n[+]poststarthook/scheduling/bootstrap-system-priority-classes ok\\n[+]poststarthook/priority-and-fairness-config-producer ok\\n[+]poststarthook/start-cluster-authentication-info-controller ok\\n[+]poststarthook/aggregator-reload-proxy-client-cert ok\\n[+]poststarthook/start-kube-aggregator-informers ok\\n[+]poststarthook/apiservice-registration-controller ok\\n[+]poststarthook/apiservice-status-available-controller ok\\n[+]poststarthook/apiservice-wait-for-first-sync ok\\n[+]poststarthook/kube-apiserver-autoregistration ok\\n[+]autoregister-completion ok\\n[+]poststarthook/apiservice-openapi-controller ok\\n[+]poststarthook/apiservice-openapiv3-controller ok\\nlivez check failed\") has prevented the request from succeeding" I1018 10:14:01.900464 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterStatusUpdated' the status of managed cluster "build0" has been updated, available condition is "True", due to "Managed cluster is available" E1018 10:14:06.660787 1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"open-cluster-management-agent.178f00c18ffe3370", GenerateName:"", Namespace:"open-cluster-management-agent", SelfLink:"", UID:"", ResourceVersion:"135998264", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, Reason:"ManagedClusterStatusUpdated", Message:"the status of managed cluster \"build0\" has been updated, available condition is \"False\", due to \"The kube-apiserver is not ok, status code: 500, an error on the server (\\\"[+]ping ok\\\\n[+]log ok\\\\n[-]etcd failed: reason withheld\\\\n[+]poststarthook/openshift.io-deprecated-api-requests-filter ok\\\\n[+]poststarthook/openshift.io-startkubeinformers ok\\\\n[+]poststarthook/openshift.io-openshift-apiserver-reachable ok\\\\n[+]poststarthook/openshift.io-oauth-apiserver-reachable ok\\\\n[+]poststarthook/start-kube-apiserver-admission-initializer ok\\\\n[+]poststarthook/quota.openshift.io-clusterquotamapping ok\\\\n[+]poststarthook/generic-apiserver-start-informers ok\\\\n[+]poststarthook/priority-and-fairness-config-consumer ok\\\\n[+]poststarthook/priority-and-fairness-filter ok\\\\n[+]poststarthook/start-apiextensions-informers ok\\\\n[+]poststarthook/start-apiextensions-controllers ok\\\\n[+]poststarthook/crd-informer-synced ok\\\\n[+]poststarthook/bootstrap-controller ok\\\\n[+]poststarthook/rbac/bootstrap-roles ok\\\\n[+]poststarthook/scheduling/bootstrap-system-priority-classes ok\\\\n[+]poststarthook/priority-and-fairness-config-producer ok\\\\n[+]poststarthook/start-cluster-authentication-info-controller ok\\\\n[+]poststarthook/aggregator-reload-proxy-client-cert ok\\\\n[+]poststarthook/start-kube-aggregator-informers ok\\\\n[+]poststarthook/apiservice-registration-controller ok\\\\n[+]poststarthook/apiservice-status-available-controller ok\\\\n[+]poststarthook/apiservice-wait-for-first-sync ok\\\\n[+]poststarthook/kube-apiserver-autoregistration ok\\\\n[+]autoregister-completion ok\\\\n[+]poststarthook/apiservice-openapi-controller ok\\\\n[+]poststarthook/apiservice-openapiv3-controller ok\\\\nlivez check failed\\\") has prevented the request from succeeding\"", Source:v1.EventSource{Component:"registration-agent-managedclusterstatuscontroller", Host:""}, FirstTimestamp:time.Date(2023, time.October, 17, 21, 0, 15, 0, time.Local), LastTimestamp:time.Date(2023, time.October, 18, 10, 13, 49, 279465051, time.Local), Count:7, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'etcdserver: request timed out' (will not retry!) I1018 11:50:53.379135 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' update managed cluster addon "cert-policy-controller" available condition to "False" with its lease "open-cluster-management-agent-addon"/"cert-policy-controller" status I1018 11:53:22.179754 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' update managed cluster addon "application-manager" available condition to "False" with its lease "open-cluster-management-agent-addon"/"application-manager" status I1018 11:56:22.174724 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' update managed cluster addon "iam-policy-controller" available condition to "False" with its lease "open-cluster-management-agent-addon"/"iam-policy-controller" status I1018 12:08:24.063881 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' update managed cluster addon "application-manager" available condition to "True" with its lease "open-cluster-management-agent-addon"/"application-manager" status I1018 12:09:51.345084 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' update managed cluster addon "iam-policy-controller" available condition to "True" with its lease "open-cluster-management-agent-addon"/"iam-policy-controller" status I1018 12:10:23.883215 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterAddOnStatusUpdated' update managed cluster addon "cert-policy-controller" available condition to "True" with its lease "open-cluster-management-agent-addon"/"cert-policy-controller" status E1018 12:49:25.984756 1 leaderelection.go:367] Failed to update lock: etcdserver: request timed out I1018 13:03:18.924904 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterStatusUpdated' the status of managed cluster "build0" has been updated, available condition is "False", due to "The kube-apiserver is not ok, status code: 500, an error on the server (\"[+]ping ok\\n[+]log ok\\n[-]etcd failed: reason withheld\\n[+]poststarthook/openshift.io-deprecated-api-requests-filter ok\\n[+]poststarthook/openshift.io-startkubeinformers ok\\n[+]poststarthook/openshift.io-openshift-apiserver-reachable ok\\n[+]poststarthook/openshift.io-oauth-apiserver-reachable ok\\n[+]poststarthook/start-kube-apiserver-admission-initializer ok\\n[+]poststarthook/quota.openshift.io-clusterquotamapping ok\\n[+]poststarthook/generic-apiserver-start-informers ok\\n[+]poststarthook/priority-and-fairness-config-consumer ok\\n[+]poststarthook/priority-and-fairness-filter ok\\n[+]poststarthook/start-apiextensions-informers ok\\n[+]poststarthook/start-apiextensions-controllers ok\\n[+]poststarthook/crd-informer-synced ok\\n[+]poststarthook/bootstrap-controller ok\\n[+]poststarthook/rbac/bootstrap-roles ok\\n[+]poststarthook/scheduling/bootstrap-system-priority-classes ok\\n[+]poststarthook/priority-and-fairness-config-producer ok\\n[+]poststarthook/start-cluster-authentication-info-controller ok\\n[+]poststarthook/aggregator-reload-proxy-client-cert ok\\n[+]poststarthook/start-kube-aggregator-informers ok\\n[+]poststarthook/apiservice-registration-controller ok\\n[+]poststarthook/apiservice-status-available-controller ok\\n[+]poststarthook/apiservice-wait-for-first-sync ok\\n[+]poststarthook/kube-apiserver-autoregistration ok\\n[+]autoregister-completion ok\\n[+]poststarthook/apiservice-openapi-controller ok\\n[+]poststarthook/apiservice-openapiv3-controller ok\\nlivez check failed\") has prevented the request from succeeding" I1018 13:03:23.302020 1 event.go:285] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"open-cluster-management-agent", Name:"open-cluster-management-agent", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ManagedClusterStatusUpdated' the status of managed cluster "build0" has been updated, available condition is "True", due to "Managed cluster is available"