-
Bug
-
Resolution: Not a Bug
-
Undefined
-
None
-
4.12.0
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Version:
4.12.0-0.nightly-2022-09-08-114806
multicluster-engine.v2.2.0
Tried to deploy a spoke SNO from hub SNO.
The BMH was in "preparing" status and then switched to "available" and got stuck in this state.
The ironic-agent log on the spoke shows this repeating error:
2022-09-10 14:07:55.529 1 ERROR ironic_python_agent.agent [-] error sending heartbeat to https://10.19.134.13:6385: ironic_python_agent.errors.HeartbeatError: Error heartbeating to agent API: Error 404: Node 48d24898-1911-4f43-82b0-0b15f8484ae7 could not be found. 2022-09-10 14:07:55.529 1 ERROR ironic_python_agent.agent Traceback (most recent call last): 2022-09-10 14:07:55.529 1 ERROR ironic_python_agent.agent File "/usr/lib/python3.6/site-packages/ironic_python_agent/agent.py", line 122, in do_heartbeat 2022-09-10 14:07:55.529 1 ERROR ironic_python_agent.agent generated_cert=self.agent.generated_cert, 2022-09-10 14:07:55.529 1 ERROR ironic_python_agent.agent File "/usr/lib/python3.6/site-packages/ironic_python_agent/ironic_api_client.py", line 175, in heartbeat 2022-09-10 14:07:55.529 1 ERROR ironic_python_agent.agent raise errors.HeartbeatError(error) 2022-09-10 14:07:55.529 1 ERROR ironic_python_agent.agent ironic_python_agent.errors.HeartbeatError: Error heartbeating to agent API: Error 404: Node 48d24898-1911-4f43-82b0-0b15f8484ae7 could not be found. 2022-09-10 14:07:55.529 1 ERROR ironic_python_agent.agent 2022-09-10 14:07:55.529 1 INFO ironic_python_agent.agent [-] sleeping before next heartbeat, interval: 153.94185373943864 2022-09-10 14:08:00.530 1 DEBUG ironic_python_agent.ironic_api_client [-] Heartbeat: announcing callback URL https://10.19.134.5:9999, API version is 1.68 heartbeat /usr/lib/python3.6/site-packages/ironic_python_agent/ironic_api_client.py:162 /usr/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) 2022-09-10 14:08:00.544 1 ERROR ironic_python_agent.agent [-] error sending heartbeat to https://10.19.134.13:6385: ironic_python_agent.errors.HeartbeatError: Error heartbeating to agent API: Error 404: Node 48d24898-1911-4f43-82b0-0b15f8484ae7 could not be found. 2022-09-10 14:08:00.544 1 ERROR ironic_python_agent.agent Traceback (most recent call last): 2022-09-10 14:08:00.544 1 ERROR ironic_python_agent.agent File "/usr/lib/python3.6/site-packages/ironic_python_agent/agent.py", line 122, in do_heartbeat 2022-09-10 14:08:00.544 1 ERROR ironic_python_agent.agent generated_cert=self.agent.generated_cert, 2022-09-10 14:08:00.544 1 ERROR ironic_python_agent.agent File "/usr/lib/python3.6/site-packages/ironic_python_agent/ironic_api_client.py", line 175, in heartbeat 2022-09-10 14:08:00.544 1 ERROR ironic_python_agent.agent raise errors.HeartbeatError(error) 2022-09-10 14:08:00.544 1 ERROR ironic_python_agent.agent ironic_python_agent.errors.HeartbeatError: Error heartbeating to agent API: Error 404: Node 48d24898-1911-4f43-82b0-0b15f8484ae7 could not be found. 2022-09-10 14:08:00.544 1 ERROR ironic_python_agent.agent 2022-09-10 14:08:00.544 1 INFO ironic_python_agent.agent [-] sleeping before next heartbeat, interval: 129.89219707896012 2022-09-10 14:08:05.545 1 DEBUG ironic_python_agent.ironic_api_client [-] Heartbeat: announcing callback URL https://10.19.134.5:9999, API version is 1.68 heartbeat /usr/lib/python3.6/site-packages/ironic_python_agent/ironic_api_client.py:162 /usr/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) 2022-09-10 14:08:05.559 1 ERROR ironic_python_agent.agent [-] error sending heartbeat to https://10.19.134.13:6385: ironic_python_agent.errors.HeartbeatError: Error heartbeating to agent API: Error 404: Node 48d24898-1911-4f43-82b0-0b15f8484ae7 could not be found. 2022-09-10 14:08:05.559 1 ERROR ironic_python_agent.agent Traceback (most recent call last): 2022-09-10 14:08:05.559 1 ERROR ironic_python_agent.agent File "/usr/lib/python3.6/site-packages/ironic_python_agent/agent.py", line 122, in do_heartbeat 2022-09-10 14:08:05.559 1 ERROR ironic_python_agent.agent generated_cert=self.agent.generated_cert, 2022-09-10 14:08:05.559 1 ERROR ironic_python_agent.agent File "/usr/lib/python3.6/site-packages/ironic_python_agent/ironic_api_client.py", line 175, in heartbeat 2022-09-10 14:08:05.559 1 ERROR ironic_python_agent.agent raise errors.HeartbeatError(error) 2022-09-10 14:08:05.559 1 ERROR ironic_python_agent.agent ironic_python_agent.errors.HeartbeatError: Error heartbeating to agent API: Error 404: Node 48d24898-1911-4f43-82b0-0b15f8484ae7 could not be found. 2022-09-10 14:08:05.559 1 ERROR ironic_python_agent.agent 2022-09-10 14:08:05.560 1 INFO ironic_python_agent.agent [-] sleeping before next heartbeat, interval: 132.4694016992394 2022-09-10 14:08:10.560 1 DEBUG ironic_python_agent.ironic_api_client [-] Heartbeat: announcing callback URL https://10.19.134.5:9999, API version is 1.68 heartbeat /usr/lib/python3.6/site-packages/ironic_python_agent/ironic_api_client.py:162 /usr/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning)
Collecting the BMH state from the hub:
oc get bmh master-1-0 -o yaml
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
annotations:
baremetalhost.metal3.io/detached: assisted-service-controller
bmac.agent-install.openshift.io/hostname: master-1-0
bmac.agent-install.openshift.io/role: master
creationTimestamp: "2022-09-10T00:01:18Z"
finalizers:
- baremetalhost.metal3.io
generation: 1
labels:
infraenvs.agent-install.openshift.io: qe2
name: master-1-0
namespace: qe2
resourceVersion: "102761"
uid: ec96109c-a05a-40b9-90ee-1756998d020e
spec:
automatedCleaningMode: disabled
bmc:
address: idrac-virtualmedia://10.19.133.17/redfish/v1/Systems/System.Embedded.1
credentialsName: bmc-secret1
disableCertificateVerification: true
bootMACAddress: 98:03:9b:61:7c:61
online: true
rootDeviceHints:
deviceName: /dev/sda
status:
errorCount: 0
errorMessage: ""
goodCredentials:
credentials:
name: bmc-secret1
namespace: qe2
credentialsVersion: "89066"
hardware:
cpu:
arch: x86_64
clockMegahertz: 3700
count: 64
flags:
- 3dnowprefetch
- abm
- acpi
- adx
- aes
- aperfmperf
- apic
- arat
- arch_capabilities
- arch_perfmon
- art
- avx
- avx2
- avx512bw
- avx512cd
- avx512dq
- avx512f
- avx512vl
- bmi1
- bmi2
- bts
- cat_l3
- cdp_l3
- clflush
- clflushopt
- clwb
- cmov
- constant_tsc
- cpuid
- cpuid_fault
- cqm
- cqm_llc
- cqm_mbm_local
- cqm_mbm_total
- cqm_occup_llc
- cx16
- cx8
- dca
- de
- ds_cpl
- dtes64
- dtherm
- dts
- epb
- ept
- ept_ad
- erms
- est
- f16c
- flexpriority
- flush_l1d
- fma
- fpu
- fsgsbase
- fxsr
- hle
- ht
- ibpb
- ibrs
- ida
- intel_ppin
- intel_pt
- invpcid
- invpcid_single
- lahf_lm
- lm
- mba
- mca
- mce
- md_clear
- mmx
- monitor
- movbe
- mpx
- msr
- mtrr
- nonstop_tsc
- nopl
- nx
- ospke
- pae
- pat
- pbe
- pcid
- pclmulqdq
- pdcm
- pdpe1gb
- pebs
- pge
- pku
- pln
- pni
- popcnt
- pse
- pse36
- pti
- pts
- rdrand
- rdseed
- rdt_a
- rdtscp
- rep_good
- rtm
- sdbg
- sep
- smap
- smep
- smx
- ss
- ssbd
- sse
- sse2
- sse4_1
- sse4_2
- ssse3
- stibp
- syscall
- tm
- tm2
- tpr_shadow
- tsc
- tsc_adjust
- tsc_deadline_timer
- vme
- vmx
- vnmi
- vpid
- x2apic
- xgetbv1
- xsave
- xsavec
- xsaveopt
- xsaves
- xtopology
- xtpr
model: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
firmware:
bios:
date: 12/17/2018
vendor: Dell Inc.
version: 1.6.13
hostname: openshift-worker-0.qe1.kni.lab.eng.bos.redhat.com
nics:
- ip: 10.19.134.5
mac: 98:03:9b:61:7c:61
model: 0x15b3 0x1015
name: eno2
- ip: fe80::9a03:9bff:fe61:7c60%eno1.3100
mac: 98:03:9b:61:7c:60
name: eno1.3100
- ip: fe80::9a03:9bff:fe61:7c60%eno1.2100
mac: 98:03:9b:61:7c:60
name: eno1.2100
- ip: fe80::9a03:9bff:fe61:7c61%eno2.200
mac: 98:03:9b:61:7c:61
name: eno2.200
- mac: 98:03:9b:61:7c:60
model: 0x15b3 0x1015
name: eno1
- ip: fe80::9a03:9bff:fe61:7c60%eno1.1100
mac: 98:03:9b:61:7c:60
name: eno1.1100
ramMebibytes: 196608
storage:
- hctl: "1:2:0:0"
model: PERC H330 Mini
name: /dev/sdb
rotational: true
sizeBytes: 479559942144
type: HDD
vendor: DELL
- model: Dell Express Flash NVMe P4610 1.6TB SFF
name: /dev/nvme0n1
sizeBytes: 1600000000000
type: NVME
- model: Dell Express Flash NVMe P4610 1.6TB SFF
name: /dev/nvme1n1
sizeBytes: 1600000000000
type: NVME
- model: Dell Express Flash NVMe P4610 1.6TB SFF
name: /dev/nvme2n1
sizeBytes: 1600000000000
type: NVME
- model: Dell Express Flash NVMe P4610 1.6TB SFF
name: /dev/nvme3n1
sizeBytes: 1600000000000
type: NVME
systemVendor:
manufacturer: Dell Inc.
productName: PowerEdge R640 (SKU=NotProvided;ModelName=PowerEdge R640)
serialNumber: 176S2W2
hardwareProfile: unknown
lastUpdated: "2022-09-10T00:18:32Z"
operationHistory:
deprovision:
end: null
start: null
inspect:
end: "2022-09-10T00:08:45Z"
start: "2022-09-10T00:01:41Z"
provision:
end: null
start: null
register:
end: "2022-09-10T00:01:41Z"
start: "2022-09-10T00:01:18Z"
operationalStatus: detached
poweredOn: false
provisioning:
ID: 48d24898-1911-4f43-82b0-0b15f8484ae7
bootMode: UEFI
image:
url: ""
raid:
hardwareRAIDVolumes: null
softwareRAIDVolumes: []
rootDeviceHints:
deviceName: /dev/sda
state: available
triedCredentials:
credentials:
name: bmc-secret1
namespace: qe2
credentialsVersion: "89066"
- relates to
-
OCPBUGS-1147 HostFirmwareSettings controller keeps reconciling detached hosts
-
- Closed
-