- Boot eaabd4a200444da280994369aefdc564 -- Jan 29 18:08:47 master-1 systemd[1]: Starting Corosync Cluster Engine... Jan 29 18:08:47 master-1 corosync[1791]: [MAIN ] Corosync Cluster Engine 3.1.9 starting up Jan 29 18:08:47 master-1 corosync[1791]: [MAIN ] Corosync built-in features: dbus systemd xmlconf vqsim nozzle snmp pie relro bindnow Jan 29 18:08:47 master-1 corosync[1791]: [TOTEM ] Initializing transport (Kronosnet). Jan 29 18:08:47 master-1 corosync[1791]: [TOTEM ] totemknet initialized Jan 29 18:08:47 master-1 corosync[1791]: [KNET ] pmtud: MTU manually set to: 0 Jan 29 18:08:47 master-1 corosync[1791]: [KNET ] common: crypto_nss.so has been loaded from /usr/lib64/kronosnet/crypto_nss.so Jan 29 18:08:48 master-1 corosync[1791]: [SERV ] Service engine loaded: corosync configuration map access [0] Jan 29 18:08:48 master-1 corosync[1791]: [QB ] server name: cmap Jan 29 18:08:48 master-1 corosync[1791]: [SERV ] Service engine loaded: corosync configuration service [1] Jan 29 18:08:48 master-1 corosync[1791]: [QB ] server name: cfg Jan 29 18:08:48 master-1 corosync[1791]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2] Jan 29 18:08:48 master-1 corosync[1791]: [QB ] server name: cpg Jan 29 18:08:48 master-1 corosync[1791]: [SERV ] Service engine loaded: corosync profile loading service [4] Jan 29 18:08:48 master-1 corosync[1791]: [QUORUM] Using quorum provider corosync_votequorum Jan 29 18:08:48 master-1 corosync[1791]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2 Jan 29 18:08:48 master-1 corosync[1791]: [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5] Jan 29 18:08:48 master-1 corosync[1791]: [QB ] server name: votequorum Jan 29 18:08:48 master-1 corosync[1791]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3] Jan 29 18:08:48 master-1 corosync[1791]: [QB ] server name: quorum Jan 29 18:08:48 master-1 corosync[1791]: [TOTEM ] Configuring link 0 Jan 29 18:08:48 master-1 corosync[1791]: [TOTEM ] Configured link number 0: local addr: 192.168.111.21, port=5405 Jan 29 18:08:48 master-1 corosync[1791]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1) Jan 29 18:08:48 master-1 corosync[1791]: [KNET ] host: host: 1 has no active links Jan 29 18:08:48 master-1 corosync[1791]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1) Jan 29 18:08:48 master-1 corosync[1791]: [KNET ] host: host: 1 has no active links Jan 29 18:08:48 master-1 corosync[1791]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1) Jan 29 18:08:48 master-1 corosync[1791]: [KNET ] host: host: 1 has no active links Jan 29 18:08:48 master-1 corosync[1791]: [KNET ] link: Resetting MTU for link 0 because host 2 joined Jan 29 18:08:48 master-1 corosync[1791]: [QUORUM] Sync members[1]: 2 Jan 29 18:08:48 master-1 corosync[1791]: [QUORUM] Sync joined[1]: 2 Jan 29 18:08:48 master-1 corosync[1791]: [TOTEM ] A new membership (2.e) was formed. Members joined: 2 Jan 29 18:08:48 master-1 corosync[1791]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2 Jan 29 18:08:48 master-1 corosync[1791]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2 Jan 29 18:08:48 master-1 corosync[1791]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2 Jan 29 18:08:48 master-1 corosync[1791]: [QUORUM] Members[1]: 2 Jan 29 18:08:48 master-1 corosync[1791]: [MAIN ] Completed service synchronization, ready to provide service. Jan 29 18:08:48 master-1 systemd[1]: Started Corosync Cluster Engine. Jan 29 18:08:48 master-1 systemd[1]: Started Pacemaker High Availability Cluster Manager. Jan 29 18:08:48 master-1 pacemakerd[1850]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log Jan 29 18:08:48 master-1 pacemakerd[1850]: notice: Starting Pacemaker 2.1.9-1.2.el9_6 Jan 29 18:08:48 master-1 pacemakerd[1850]: notice: Pacemaker daemon successfully started and accepting connections Jan 29 18:08:48 master-1 pacemaker-attrd[1854]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log Jan 29 18:08:48 master-1 pacemaker-execd[1853]: notice: Starting Pacemaker local executor Jan 29 18:08:48 master-1 pacemaker-execd[1853]: notice: Pacemaker local executor successfully started and accepting connections Jan 29 18:08:48 master-1 pacemaker-execd[1853]: notice: OCF resource agent search path is /usr/lib/ocf/resource.d Jan 29 18:08:48 master-1 pacemaker-fenced[1852]: notice: Additional logging available in /var/log/pacemaker/pacemaker.log Jan 29 18:08:48 master-1 pacemaker-fenced[1852]: notice: Starting Pacemaker fencer Jan 29 18:08:48 master-1 pacemaker-fenced[1852]: notice: Connecting to corosync cluster layer Jan 29 18:08:48 master-1 pacemaker-based[1851]: notice: Starting Pacemaker CIB manager Jan 29 18:08:48 master-1 pacemaker-attrd[1854]: notice: Starting Pacemaker node attribute manager Jan 29 18:08:48 master-1 pacemaker-schedulerd[1855]: notice: Starting Pacemaker scheduler Jan 29 18:08:48 master-1 pacemaker-schedulerd[1855]: notice: Pacemaker scheduler successfully started and accepting connections Jan 29 18:08:48 master-1 pacemaker-controld[1856]: notice: Starting Pacemaker controller Jan 29 18:08:48 master-1 pacemaker-fenced[1852]: notice: Node master-1 state is now member Jan 29 18:08:48 master-1 pacemaker-based[1851]: notice: Connecting to corosync cluster layer Jan 29 18:08:48 master-1 pacemaker-based[1851]: notice: Node master-1 state is now member Jan 29 18:08:48 master-1 pacemaker-based[1851]: notice: Pacemaker CIB manager successfully started and accepting connections Jan 29 18:08:49 master-1 pacemaker-attrd[1854]: notice: Connecting to corosync cluster layer Jan 29 18:08:49 master-1 pacemaker-controld[1856]: notice: Connecting to corosync cluster layer Jan 29 18:08:49 master-1 pacemaker-attrd[1854]: notice: Node master-1 state is now member Jan 29 18:08:49 master-1 pacemaker-fenced[1852]: notice: Pacemaker fencer successfully started and accepting connections Jan 29 18:08:49 master-1 pacemaker-attrd[1854]: notice: Pacemaker node attribute manager successfully started and accepting connections Jan 29 18:08:49 master-1 pacemaker-attrd[1854]: notice: Setting #attrd-protocol[master-1] in instance_attributes: (unset) -> 6 Jan 29 18:08:49 master-1 pacemaker-attrd[1854]: notice: Recorded local node as attribute writer (was unset) Jan 29 18:08:49 master-1 pacemaker-controld[1856]: warning: No quorum Jan 29 18:08:49 master-1 pacemaker-controld[1856]: notice: Node master-1 state is now member Jan 29 18:08:49 master-1 pacemaker-controld[1856]: notice: Pacemaker controller successfully started and accepting connections Jan 29 18:08:49 master-1 pacemaker-fenced[1852]: notice: Added 'master-0_redfish' to device list (1 active device) Jan 29 18:08:49 master-1 pacemaker-fenced[1852]: notice: Added 'master-1_redfish' to device list (2 active devices) Jan 29 18:08:49 master-1 pacemaker-controld[1856]: notice: Fencer successfully connected Jan 29 18:08:49 master-1 pacemaker-controld[1856]: notice: State transition S_STARTING -> S_PENDING Jan 29 18:08:49 master-1 corosync[1791]: [KNET ] rx: host: 1 link: 0 is up Jan 29 18:08:49 master-1 corosync[1791]: [KNET ] link: Resetting MTU for link 0 because host 1 joined Jan 29 18:08:49 master-1 corosync[1791]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1) Jan 29 18:08:49 master-1 corosync[1791]: [KNET ] pmtud: PMTUD link change for host: 1 link: 0 from 469 to 1397 Jan 29 18:08:49 master-1 corosync[1791]: [KNET ] pmtud: Global data MTU changed to: 1397 Jan 29 18:08:50 master-1 corosync[1791]: [QUORUM] Sync members[2]: 1 2 Jan 29 18:08:50 master-1 corosync[1791]: [QUORUM] Sync joined[1]: 1 Jan 29 18:08:50 master-1 corosync[1791]: [TOTEM ] A new membership (1.12) was formed. Members joined: 1 Jan 29 18:08:50 master-1 pacemaker-based[1851]: notice: Node master-0 state is now member Jan 29 18:08:50 master-1 corosync[1791]: [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2 Jan 29 18:08:50 master-1 pacemaker-fenced[1852]: notice: Node master-0 state is now member Jan 29 18:08:50 master-1 corosync[1791]: [QUORUM] This node is within the primary component and will provide service. Jan 29 18:08:50 master-1 corosync[1791]: [QUORUM] Members[2]: 1 2 Jan 29 18:08:50 master-1 corosync[1791]: [MAIN ] Completed service synchronization, ready to provide service. Jan 29 18:08:50 master-1 pacemaker-attrd[1854]: notice: Node master-0 state is now member Jan 29 18:08:50 master-1 pacemaker-attrd[1854]: notice: Detected another attribute writer (master-0), starting new election Jan 29 18:08:50 master-1 pacemaker-attrd[1854]: notice: Setting #attrd-protocol[master-0] in instance_attributes: (unset) -> 6 Jan 29 18:08:50 master-1 pacemaker-attrd[1854]: notice: Setting #feature-set[master-0] in instance_attributes: (unset) -> 3.19.6 Jan 29 18:08:50 master-1 pacemaker-controld[1856]: notice: Quorum acquired Jan 29 18:08:50 master-1 pacemaker-controld[1856]: notice: Node master-0 state is now member Jan 29 18:08:50 master-1 pacemaker-controld[1856]: notice: State transition S_PENDING -> S_NOT_DC Jan 29 18:08:50 master-1 pacemaker-attrd[1854]: notice: Setting #feature-set[master-1] in instance_attributes: (unset) -> 3.19.6 Jan 29 18:08:51 master-1 pacemaker-controld[1856]: notice: Requesting local execution of probe operation for kubelet on master-1 Jan 29 18:08:51 master-1 pacemaker-controld[1856]: notice: Result of probe operation for kubelet on master-1: not running (inactive) Jan 29 18:08:51 master-1 pacemaker-controld[1856]: notice: Requesting local execution of start operation for kubelet on master-1 Jan 29 18:08:52 master-1 pacemaker-controld[1856]: notice: Requesting local execution of probe operation for etcd on master-1 Jan 29 18:08:52 master-1 pacemaker-controld[1856]: notice: Result of probe operation for etcd on master-1: not running Jan 29 18:08:52 master-1 pacemaker-controld[1856]: notice: Requesting local execution of probe operation for master-0_redfish on master-1 Jan 29 18:08:52 master-1 pacemaker-controld[1856]: notice: Result of probe operation for master-0_redfish on master-1: not running Jan 29 18:08:52 master-1 pacemaker-controld[1856]: notice: Requesting local execution of probe operation for master-1_redfish on master-1 Jan 29 18:08:52 master-1 pacemaker-controld[1856]: notice: Result of probe operation for master-1_redfish on master-1: not running Jan 29 18:08:56 master-1 corosync[1791]: [TOTEM ] Token has not been received in 2250 ms Jan 29 18:09:07 master-1 pacemaker-attrd[1854]: notice: Setting force_new_cluster[master-0] in instance_attributes: (unset) -> master-0 Jan 29 18:09:07 master-1 pacemaker-attrd[1854]: notice: Setting last-failure-etcd#monitor_30000[master-0] in instance_attributes: (unset) -> 1769710147 Jan 29 18:09:07 master-1 pacemaker-attrd[1854]: notice: Setting fail-count-etcd#monitor_30000[master-0] in instance_attributes: (unset) -> 1 Jan 29 18:10:36 master-1 pacemaker-controld[1856]: error: Result of start operation for kubelet on master-1: Timed Out after 1m40s (start action for systemd unit kubelet did not complete in time) Jan 29 18:10:36 master-1 pacemaker-attrd[1854]: notice: Setting last-failure-kubelet#start_0[master-1] in instance_attributes: (unset) -> 1769710236 Jan 29 18:10:36 master-1 pacemaker-attrd[1854]: notice: Setting fail-count-kubelet#start_0[master-1] in instance_attributes: (unset) -> INFINITY Jan 29 18:11:08 master-1 pacemaker-controld[1856]: notice: Requesting local execution of stop operation for kubelet on master-1 Jan 29 18:11:09 master-1 pacemaker-controld[1856]: notice: Result of stop operation for kubelet on master-1: ok Jan 29 18:11:14 master-1 pacemaker-attrd[1854]: notice: Setting force_new_cluster[master-0] in instance_attributes: master-0 -> (unset)