Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-74415

Introspection fails to download metal3 customization image

    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      Baremetal introspection (run during ocloud cluster preparation) is failing while attempting to download metal3 customization image, as in the logs below.
      The BMH is created as
      
      ```
      Name:         dell-server-randu2
      Namespace:    dell-pool
      Labels:       interfacelabel.clcm.openshift.io/data-interface=ens2f3.3934
                    resources.clcm.openshift.io/resourcePoolId=dell-pool
                    resources.clcm.openshift.io/siteId=local-west
                    resourceselector.clcm.openshift.io/server-id=poweredge-xr11-randu2
                    resourceselector.clcm.openshift.io/server-type=blade
      Annotations:  bmac.agent-install.openshift.io/allow-provisioned-host-management: 
                    resourceinfo.clcm.openshift.io/description: Dell server randu2
                    resourceinfo.clcm.openshift.io/globalAssetId: super1-randu2
                    resourceinfo.clcm.openshift.io/groups: groupA, groupB
                    resourceinfo.clcm.openshift.io/partNumber: GJK05L3
      API Version:  metal3.io/v1alpha1
      Kind:         BareMetalHost
      Metadata:
        Creation Timestamp:  2026-01-16T15:55:09Z
        Finalizers:
          baremetalhost.metal3.io
        Generation:        1
        Resource Version:  1260361470
        UID:               05ecda95-f03d-4c6e-a542-c81d7bf08e18
      Spec:
        Automated Cleaning Mode:  disabled
        Bmc:
          Address:                           idrac-virtualmedia+https://192.168.51.44/redfish/v1/Systems/System.Embedded.1
          Credentials Name:                  bmc-secret-dell-server-randu2
          Disable Certificate Verification:  true
        Boot MAC Address:                    40:A6:B7:82:54:07
        Online:                              false
        Preprovisioning Network Data Name:   network-data-dell-server-randu2
        Root Device Hints:
          Device Name:  /dev/disk/by-path/pci-0000:04:00.0-ata-2
      Status:
        Error Count:    5
        Error Message:  Failed to inspect hardware. Reason: unable to start inspection: Validation of image href http://metal3-image-customization-service.openshift-machine-api.svc.cluster.local/d4181d84-1a8c-416d-8498-63ed09a1a565 failed, reason: HTTPConnectionPool(host='metal3-image-customization-service.openshift-machine-api.svc.cluster.local', port=80): Max retries exceeded with url: /d4181d84-1a8c-416d-8498-63ed09a1a565 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f698eea3fe0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
        Error Type:     inspection error
      ```
      
      The error seems related to failing DNS lookup, as also specified in metal3-ironic logs:
      
      ```
      oc logs metal3-757b75d8b5-b7dd2 -n openshift-machine-api -c metal3-ironic | grep -i error
      2026-01-16 15:34:52.325 1 DEBUG ironic.api.method [None req-a16758d1-b874-41fa-896a-3eea8aeeef10 - - - - - -] Client-side error: Node 6dbaedee-0d8d-44cb-9cf6-eaa2a4b812c0 could not be found. format_exception /usr/lib/python3.12/site-packages/ironic/api/method.py:124
      2026-01-16 15:44:53.369 1 DEBUG ironic.api.method [None req-7cb950bc-f19c-475f-9b30-f380ec8b873d - - - - - -] Client-side error: Node 6dbaedee-0d8d-44cb-9cf6-eaa2a4b812c0 could not be found. format_exception /usr/lib/python3.12/site-packages/ironic/api/method.py:124
      2026-01-16 15:54:54.388 1 DEBUG ironic.api.method [None req-71c80419-5122-4cb5-809c-e729678d8403 - - - - - -] Client-side error: Node 6dbaedee-0d8d-44cb-9cf6-eaa2a4b812c0 could not be found. format_exception /usr/lib/python3.12/site-packages/ironic/api/method.py:124
      2026-01-16 15:55:12.748 1 DEBUG ironic.api.method [None req-a0489774-b93d-4d9f-a2f2-2b7354bcc4a9 - - - - - -] Client-side error: Node dell-pool~dell-server-randu2 could not be found. format_exception /usr/lib/python3.12/site-packages/ironic/api/method.py:124
      2026-01-16 15:55:13.248 1 DEBUG ironic.api.method [None req-d2eddcd9-ec89-418a-9a64-1f1109c7b5a8 - - - - - -] Client-side error: Node dell-server-randu2 could not be found. format_exception /usr/lib/python3.12/site-packages/ironic/api/method.py:124
      2026-01-16 15:55:38.820 1 DEBUG ironic.api.method [None req-8e163852-6519-4d3a-bb0e-39cc86f5dff0 - - - - - -] Client-side error: Node inventory record for node 3 could not be found. format_exception /usr/lib/python3.12/site-packages/ironic/api/method.py:124
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface [None req-858843b9-3ba4-4169-9ec4-2ac123255761 - - - - - -] Unable to start managed inspection for node 2e123f5d-6bfe-4e00-a9d2-1dbdb9e52335: Validation of image href http://metal3-image-customization-service.openshift-machine-api.svc.cluster.local/d4181d84-1a8c-416d-8498-63ed09a1a565 failed, reason: HTTPConnectionPool(host='metal3-image-customization-service.openshift-machine-api.svc.cluster.local', port=80): Max retries exceeded with url: /d4181d84-1a8c-416d-8498-63ed09a1a565 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f69905d80b0>: Failed to establish a new connection: [Errno -2] Name or service not known')): ironic.common.exception.ImageRefValidationFailed: Validation of image href http://metal3-image-customization-service.openshift-machine-api.svc.cluster.local/d4181d84-1a8c-416d-8498-63ed09a1a565 failed, reason: HTTPConnectionPool(host='metal3-image-customization-service.openshift-machine-api.svc.cluster.local', port=80): Max retries exceeded with url: /d4181d84-1a8c-416d-8498-63ed09a1a565 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f69905d80b0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface Traceback (most recent call last):
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/eventlet/support/greendns.py", line 463, in resolve
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     return _proxy.query(name, rdtype, raise_on_no_answer=raises,
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/eventlet/support/greendns.py", line 419, in query
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     return end()
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface            ^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/eventlet/support/greendns.py", line 398, in end
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     raise result[1]
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/eventlet/support/greendns.py", line 379, in step
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     a = fun(*args, **kwargs)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface         ^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/dns/resolver.py", line 1364, in query
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     return self.resolve(
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface            ^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/dns/resolver.py", line 1318, in resolve
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     (nameserver, tcp, backoff) = resolution.next_nameserver()
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/dns/resolver.py", line 763, in next_nameserver
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     raise NoNameservers(request=self.request, errors=self.errors)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface dns.resolver.NoNameservers: All nameservers failed to answer the query metal3-image-customization-service.openshift-machine-api.svc.cluster.local.mgmt.ocloud.lab. IN A: Server Do53:172.30.0.10@53 answered REFUSED
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface 
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface During handling of the above exception, another exception occurred:
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface 
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface Traceback (most recent call last):
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/urllib3/connection.py", line 174, in _new_conn
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     conn = connection.create_connection(
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/urllib3/util/connection.py", line 72, in create_connection
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/eventlet/support/greendns.py", line 558, in getaddrinfo
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     qname, addrs = _getaddrinfo_lookup(host, family, flags)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/eventlet/support/greendns.py", line 529, in _getaddrinfo_lookup
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     raise err
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/eventlet/support/greendns.py", line 518, in _getaddrinfo_lookup
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     answer = resolve(host, qfamily, False, use_network=use_network)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/eventlet/support/greendns.py", line 473, in resolve
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     _raise_new_error(EAI_NODATA_ERROR)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/eventlet/support/greendns.py", line 100, in _raise_new_error
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     raise error_instance.__class__(*error_instance.args)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface socket.gaierror: [Errno -2] Name or service not known
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface 
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface During handling of the above exception, another exception occurred:
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface 
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface Traceback (most recent call last):
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/urllib3/connectionpool.py", line 715, in urlopen
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     httplib_response = self._make_request(
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface                        ^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/urllib3/connectionpool.py", line 416, in _make_request
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     conn.request(method, url, **httplib_request_kw)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/urllib3/connection.py", line 244, in request
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib64/python3.12/http/client.py", line 1327, in request
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     self._send_request(method, url, body, headers, encode_chunked)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib64/python3.12/http/client.py", line 1373, in _send_request
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     self.endheaders(body, encode_chunked=encode_chunked)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib64/python3.12/http/client.py", line 1322, in endheaders
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     self._send_output(message_body, encode_chunked=encode_chunked)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib64/python3.12/http/client.py", line 1081, in _send_output
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     self.send(msg)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib64/python3.12/http/client.py", line 1025, in send
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     self.connect()
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/urllib3/connection.py", line 205, in connect
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     conn = self._new_conn()
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface            ^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/urllib3/connection.py", line 186, in _new_conn
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     raise NewConnectionError(
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f69905d80b0>: Failed to establish a new connection: [Errno -2] Name or service not known
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface 
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface During handling of the above exception, another exception occurred:
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface 
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface Traceback (most recent call last):
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     resp = conn.urlopen(
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface            ^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/urllib3/connectionpool.py", line 799, in urlopen
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     retries = retries.increment(
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface               ^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/urllib3/util/retry.py", line 594, in increment
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     raise MaxRetryError(_pool, url, error or ResponseError(cause))
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='metal3-image-customization-service.openshift-machine-api.svc.cluster.local', port=80): Max retries exceeded with url: /d4181d84-1a8c-416d-8498-63ed09a1a565 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f69905d80b0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface 
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface During handling of the above exception, another exception occurred:
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface 
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface Traceback (most recent call last):
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/ironic/common/image_service.py", line 178, in validate_href
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     response = requests.head(image_href, verify=verify,
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/requests/api.py", line 100, in head
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     return request("head", url, **kwargs)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/requests/api.py", line 59, in request
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     return session.request(method=method, url=url, **kwargs)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     resp = self.send(prep, **send_kwargs)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     r = adapter.send(request, **kwargs)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/requests/adapters.py", line 700, in send
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     raise ConnectionError(e, request=request)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface requests.exceptions.ConnectionError: HTTPConnectionPool(host='metal3-image-customization-service.openshift-machine-api.svc.cluster.local', port=80): Max retries exceeded with url: /d4181d84-1a8c-416d-8498-63ed09a1a565 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f69905d80b0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface 
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface During handling of the above exception, another exception occurred:
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface 
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface Traceback (most recent call last):
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/ironic/drivers/modules/inspector/interface.py", line 234, in inspect_hardware
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     self._start_managed_inspection(task)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/ironic/drivers/modules/inspector/agent.py", line 52, in _start_managed_inspection
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     common.prepare_managed_inspection(task, ep)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/ironic/drivers/modules/inspector/interface.py", line 162, in prepare_managed_inspection
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     task.driver.boot.prepare_ramdisk(task, ramdisk_params=params)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/ironic/drivers/modules/redfish/boot.py", line 823, in prepare_ramdisk
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     iso_ref = image_utils.prepare_deploy_iso(task, ramdisk_params,
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/ironic/drivers/modules/image_utils.py", line 613, in prepare_deploy_iso
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     return prepare_iso_image(inject_files=inject_files)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/ironic/drivers/modules/image_utils.py", line 460, in _prepare_iso_image
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     return prepare_remote_image(task, base_iso,
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/ironic/drivers/modules/image_utils.py", line 394, in prepare_remote_image
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     cache.fetch_image(image_url, tmp_file,
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/ironic/drivers/modules/image_cache.py", line 152, in fetch_image
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     img_info = img_service.show(href)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface                ^^^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/ironic/common/image_service.py", line 273, in show
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface     response = self.validate_href(image_href)
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      2026-01-16 15:55:42.019 1 ERROR ironic.drivers.modules.inspector.interface   File "/usr/lib/python3.12/site-packages/ironic/common/image_service.py", line 219, in validate_href
      ```
      
      Nevertheless metal3-ironic container is able to resolve and reach the expected endpoint, as:
      
      ```
      sh-5.1# curl http://metal3-image-customization-service.openshift-machine-api.svc.cluster.local:80
      <!doctype html>
      <meta name="viewport" content="width=device-width">
      <pre>
      <a href="7847bc6d-36da-48a3-97ba-d7369e61dae5">7847bc6d-36da-48a3-97ba-d7369e61dae5</a>
      </pre>
      
      sh-5.1# dig +short @172.30.0.10 metal3-image-customization-service.openshift-machine-api.svc.cluster.local
      172.30.121.89
      ```
      
      As an additional element, Tracing has been enabled on DNS operator to log queries to the aforementioned endpoint, as in the following trace:
      
      ```
      [INFO] 100.64.0.2:33769 - 50093 "AAAA IN metal3-image-customization-service.openshift-machine-api.svc.cluster.local.openshift-machine-api.svc.cluster.local. udp 132 false 512" NXDOMAIN qr,aa,rd 225 0.000324391s
      [INFO] 100.64.0.2:39826 - 39522 "AAAA IN metal3-image-customization-service.openshift-machine-api.svc.cluster.local.svc.cluster.local. udp 110 false 512" NXDOMAIN qr,aa,rd 203 0.000413708s
      [INFO] 100.64.0.2:57190 - 45179 "AAAA IN metal3-image-customization-service.openshift-machine-api.svc.cluster.local.cluster.local. udp 106 false 512" NXDOMAIN qr,aa,rd 199 0.000271054s
      [INFO] 100.64.0.2:35318 - 49017 "AAAA IN metal3-image-customization-service.openshift-machine-api.svc.cluster.local.mgmt.ocloud.lab. udp 108 false 512" REFUSED qr,rd,ra 108 0.002749578s
      [INFO] 100.64.0.2:50684 - 43565 "A IN metal3-image-customization-service.openshift-machine-api.svc.cluster.local.mgmt.ocloud.lab. udp 108 false 512" REFUSED qr,rd,ra 108 0.001534604s
      ```     

      Version-Release number of selected component (if applicable):

          

      How reproducible:

      Create a BMH resource and wait for introspection to fail    

      Steps to Reproduce:

          1.
          2.
          3.
          

      Actual results:

      Introspection fails    

      Expected results:

      Introspection is completed successfully and cluster deployment is started   

      Additional info:

      A similar discussion has been tracked internally on slack in the past https://redhat-internal.slack.com/archives/CG6252WBY/p1732188647685939    

              eterrell@redhat.com Eduardo Otubo
              rhn-gps-fnebiolo Federico Nebiolo
              None
              None
              Jad Haj Yahya Jad Haj Yahya
              None
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: