-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
None
-
False
-
-
False
-
?
-
nova-rhel9-operator:1.0.14-2
-
rhos-workloads-compute
-
None
-
-
-
Moderate
https://github.com/openstack-k8s-operators/watcher-operator/pull/159 pocs 2 node EDPM job on crc 4.18 with CentOS Stream 10 tcib master containers.
It deploys the controlplane with disabled tls at podlevel and ingress. It also disasbles TLS on edpm node.
While running the watcher-tempest-plugin scenario tests, Almost all scenarios are failing with following error:
raceback (most recent call last): File "/usr/lib/python3.12/site-packages/watcher_tempest_plugin/tests/scenario/test_execute_actuator.py", line 210, in test_execute_scenarios self._fill_actions(actions) File "/usr/lib/python3.12/site-packages/watcher_tempest_plugin/tests/scenario/test_execute_actuator.py", line 145, in _fill_actions parameters = filling_function() ^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/watcher_tempest_plugin/tests/scenario/test_execute_actuator.py", line 107, in _prerequisite_param_for_resize_action created_instances = self._create_one_instance_per_host_with_statistic( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/watcher_tempest_plugin/tests/scenario/base.py", line 288, in _create_one_instance_per_host_with_statistic instance = self.create_server( ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/tempest/scenario/manager.py", line 349, in create_server body, _ = compute.create_test_server( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/tempest/common/compute.py", line 343, in create_test_server with excutils.save_and_reraise_exception(): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/oslo_utils/excutils.py", line 227, in __exit__ self.force_reraise() File "/usr/lib/python3.12/site-packages/oslo_utils/excutils.py", line 200, in force_reraise raise self.value File "/usr/lib/python3.12/site-packages/tempest/common/compute.py", line 326, in create_test_server server = waiters.wait_for_server_status( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/tempest/common/waiters.py", line 80, in wait_for_server_status raise exceptions.BuildErrorException(details, server_id=server_id) tempest.exceptions.BuildErrorException: Server 8684e4d3-bf33-42f9-adb0-3fddacc91e74 failed to build and is in ERROR status Details: Fault: {'code': 500, 'created': '2025-05-15T15:49:13Z', 'message': 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 8684e4d3-bf33-42f9-adb0-3fddacc91e74.'}. Request ID of server operation performed before checking the server status req-7c610b83-25f7-4476-827f-41115c586fde.
While going through nova compute journal logs on EDPM node sosreport, we found following traces for Server 8684e4d3-bf33-42f9-adb0-3fddacc91e74.
req-7c610b83-25f7-4476-827f-41115c586fde 75f302af24364ce5960a8117a7db15ca 52469cf58a7c4263ae00cbe70a48984b - - default default] [instance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] Instance failed to spawn: FileNot> ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] Traceback (most recent call last): ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] File "/usr/lib/python3.12/site-packages/nova/compute/manager.py", line 2901, in _build_resources ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] yield resources ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] File "/usr/lib/python3.12/site-packages/nova/compute/manager.py", line 2648, in _build_and_run_instance ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] self.driver.spawn(context, instance, image_meta, ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] File "/usr/lib/python3.12/site-packages/nova/virt/libvirt/driver.py", line 4864, in spawn ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] self._create_guest_with_network( ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] File "/usr/lib/python3.12/site-packages/nova/virt/libvirt/driver.py", line 8264, in _create_guest_with_network ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] with excutils.save_and_reraise_exception(): ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] File "/usr/lib/python3.12/site-packages/oslo_utils/excutils.py", line 227, in __exit__ ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] self.force_reraise() ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] File "/usr/lib/python3.12/site-packages/oslo_utils/excutils.py", line 200, in force_reraise ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] raise self.value ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] File "/usr/lib/python3.12/site-packages/nova/virt/libvirt/driver.py", line 8242, in _create_guest_with_network ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] guest = self._create_guest( ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] ^^^^^^^^^^^^^^^^^^^ ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] File "/usr/lib/python3.12/site-packages/nova/virt/libvirt/driver.py", line 8177, in _create_guest ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] post_xml_callback() ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] File "/usr/lib/python3.12/site-packages/nova/virt/libvirt/driver.py", line 5491, in _create_configdrive ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] cdb.make_drive(config_disk_local_path) ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] File "/usr/lib/python3.12/site-packages/nova/virt/configdrive.py", line 141, in make_drive ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] self._make_iso9660(path, tmpdir) ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] File "/usr/lib/python3.12/site-packages/nova/virt/configdrive.py", line 84, in _make_iso9660 ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] processutils.execute(CONF.mkisofs_cmd, ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] File "/usr/lib/python3.12/site-packages/oslo_concurrency/processutils.py", line 354, in execute ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] obj = subprocess.Popen(cmd, ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] ^^^^^^^^^^^^^^^^^^^^^ ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] File "/usr/lib/python3.12/site-packages/eventlet/green/subprocess.py", line 56, in __init__ ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds) ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] File "/usr/lib64/python3.12/subprocess.py", line 1026, in __init__ ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] self._execute_child(args, executable, preexec_fn, close_fds, ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] File "/usr/lib64/python3.12/subprocess.py", line 1955, in _execute_child ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] raise child_exception_type(errno_num, err_msg, err_filename) ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74] FileNotFoundError: [Errno 2] No such file or directory: 'genisoimage' ance: 8684e4d3-bf33-42f9-adb0-3fddacc91e74]
If we take a look conf message for nova.
May 15 21:08:14 compute-1 nova_compute[185972]: 2025-05-15 15:38:14.513 2 DEBUG oslo_service.backend.eventlet.service [None req-78b988af-a648-4a9c-ac60-1b98dfd05dcc - - - - - -] state_path = > ESCOC let.service [None req-78b988af-a648-4a9c-ac60-1b98dfd05dcc - - - - - -] mkisofs_cmd = genisoimage log_opt_values /usr/lib/python3.12/site-packages/oslo_config/cfg.py:2817
mkisofs_cmd points to genisoimage. Based on https://github.com/rdo-packages/nova-distgit/commit/702f2fbf0ef344b594d89448e4d06ae3edb42dd6,
mkisofs_cmd = mkisofs
since we donot use `genisoimage` that's why above issue is coming. Logging this jira to track the issue.. We need to use mkisofs command to address this issue.
- depends on
-
OSPRH-16663 Run Master EDPM job on CentOS Stream 10
-
- Closed
-
- links to
