Loading...

XML

Word

Printable

Type: Bug
Resolution: Unresolved
Priority: Normal
Fix Version/s: rhos-17.1.13
Affects Version/s: rhos-17.1.11
Component/s: openstack-tripleo-heat-templates
Labels:
None

Story Points:
0
Epic Link:
[BugEpic]: cinder-backup container is failing with "/etc/multipath.conf does not exist" message
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Docs Approval:
?
Fixed in Build:
openstack-tripleo-heat-templates-14.3.1-17.1.20251118170858.e7c7ce3.el9osttrunk
AssignedTeam:
rhos-storage-cinder
Regression:
None
Release Note Status:
Proposed
Git Pull Request:
https://gitlab.cee.redhat.com/eng/openstack/openstack-tripleo-heat-templates/-/merge_requests/72
Intelligence Requested:
Market:
PX Impact Score:
PX Review Complete:

Severity:
Important

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

To Reproduce Steps to reproduce the behavior:

Update to RHOSO17.1 latest

Expected behavior

cinder-backup works fine without any issues.

Screenshots

Device Info (please complete the following information):

Bug impact

Production environment doesn't work as expected after updating to latest RHOSP17.1.

Known workaround

Modify /var/lib/tripleo-config/container-startup-config/step_4/cinder_backup.json

     # vi /var/lib/tripleo-config/container-startup-config/step_4/cinder_backup.json
       
       <snip>
         "/lib/modules:/lib/modules:ro",
         "/var/lib/cinder:/var/lib/cinder:z",
         "/var/lib/iscsi:/var/lib/iscsi:z",
         "/etc/multipath:/etc/multipath:z",   <===== Add comma
         "/etc/multipath.conf:/etc/multipath.conf:ro"  <===== Add this line
     ]

Recreate cinder-backup container as per https://access.redhat.com/solutions/7047941

Additional context

The logs of cinder-backup is:

2025-11-14 02:07:55.307 17 ERROR os_brick.initiator.linuxscsi [req-897dd41c-1406-4235-b226-f4b7395a15be f2b48f8b42e64ca0a9654d2cdfba8297 1420c4d4966a459c81b136571da0d944 - - -] Failed to get mpath device dm-11 ready for I/O: Unexpected error while running command.
Command: multipath -C dm-11
Exit code: 1
Stdout: ''
Stderr: '403157.750613 | /etc/multipath.conf does not exist, blacklisting all devices.\n403157.750639 | You can run "/sbin/mpathconf --enable" to create\n403157.750643 | /etc/multipath.conf. See man mpathconf(8) for more details\n403157.752330 | sdc: error 2 in pathinfo, discarding path\n403157.752696 | sdd: error 2 in pathinfo, discarding path\n403157.753067 | sde: error 2 in pathinfo, discarding path\n403157.753469 | sdf: error 2 in pathinfo, discarding path\n403157.753478 | 3600a098038314e647524577472386a41: removing empty pathgroup 0\n403157.753484 | dm-11: no usable paths found\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server [req-897dd41c-1406-4235-b226-f4b7395a15be f2b48f8b42e64ca0a9654d2cdfba8297 1420c4d4966a459c81b136571da0d944 - - -] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: multipath -C dm-11
Exit code: 1
Stdout: ''
Stderr: '403157.750613 | /etc/multipath.conf does not exist, blacklisting all devices.\n403157.750639 | You can run "/sbin/mpathconf --enable" to create\n403157.750643 | /etc/multipath.conf. See man mpathconf(8) for more details\n403157.752330 | sdc: error 2 in pathinfo, discarding path\n403157.752696 | sdd: error 2 in pathinfo, discarding path\n403157.753067 | sde: error 2 in pathinfo, discarding path\n403157.753469 | sdf: error 2 in pathinfo, discarding path\n403157.753478 | 3600a098038314e647524577472386a41: removing empty pathgroup 0\n403157.753484 | dm-11: no usable paths found\n'
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/utils.py", line 1035, in wrapper
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     return func(self, *args, **kwargs)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 410, in create_backup
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     volume_utils.update_backup_error(backup, str(err))
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     self.force_reraise()
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     raise self.value
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 399, in create_backup
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     updates = self._run_backup(context, backup, volume)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 467, in _run_backup
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     attach_info = self._attach_device(context,
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 1002, in _attach_device
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     return self._attach_volume(ctxt, backup_device, properties)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 1021, in _attach_volume
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     LOG.warning("Failed to terminate the connection "
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     self.force_reraise()
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     raise self.value
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 1013, in _attach_volume
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     return self._connect_device(conn)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 1055, in _connect_device
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     vol_handle = connector.connect_volume(conn['data'])
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/os_brick/utils.py", line 160, in trace_logging_wrapper
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/os_brick/utils.py", line 288, in change_encrypted
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     res = func(self, connection_properties)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/base.py", line 76, in inner
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/iscsi.py", line 517, in connect_volume
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     self._cleanup_connection(connection_properties, force=True)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     self.force_reraise()
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     raise self.value
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/iscsi.py", line 510, in connect_volume
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     return self._connect_multipath_volume(connection_properties)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/os_brick/utils.py", line 84, in _wrapper
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     return r.call(f, *args, **kwargs)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 411, in call
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     return self.__call__(*args, **kwargs)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     do = self.iter(retry_state=retry_state)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 360, in iter
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     return fut.result()
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 439, in result
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     return self.__get_result()
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     raise self._exception
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     result = fn(*args, **kwargs)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/iscsi.py", line 753, in _connect_multipath_volume
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     self._linuxscsi.wait_for_mpath_device(mpath)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/os_brick/initiator/linuxscsi.py", line 754, in wait_for_mpath_device
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     self._execute('multipath', '-C', mpath,
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/os_brick/executor.py", line 52, in _execute
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     result = self.__execute(*args, **kwargs)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     return execute_root(*cmd, **kwargs)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 253, in _wrap
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     return self.channel.remote_call(name, args, kwargs)
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 226, in remote_call
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server     raise exc_type(*result[2])
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server Command: multipath -C dm-11
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server Exit code: 1
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server Stdout: ''
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server Stderr: '403157.750613 | /etc/multipath.conf does not exist, blacklisting all devices.\n403157.750639 | You can run "/sbin/mpathconf --enable" to create\n403157.750643 | /etc/multipath.conf. See man mpathconf(8) for more details\n403157.752330 | sdc: error 2 in pathinfo, discarding path\n403157.752696 | sdd: error 2 in pathinfo, discarding path\n403157.753067 | sde: error 2 in pathinfo, discarding path\n403157.753469 | sdf: error 2 in pathinfo, discarding path\n403157.753478 | 3600a098038314e647524577472386a41: removing empty pathgroup 0\n403157.753484 | dm-11: no usable paths found\n'
2025-11-14 02:07:56.416 17 ERROR oslo_messaging.rpc.server

The below patch should be merged.
- https://gitlab.cee.redhat.com/eng/openstack/openstack-tripleo-heat-templates/-/merge_requests/72

Assignee:: Alan Bishop

Reporter:: Nozomi Kawamoto

Team:: rhos-storage-cinder

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Created:: 2025/11/14 12:12 AM

Updated:: 2025/11/19 4:50 PM

Details

Description

Attachments

Easy Agile Planning Poker

Activity

People

Dates

PagerDuty

Hide