-
Task
-
Resolution: Done
-
Critical
-
None
-
rhos-16.2.9
-
None
-
2
-
False
-
-
False
-
Not Selected
-
rhos-ops-day1day2-upgrades
-
-
-
RHOS Upgrades 2025 Sprint 19
-
1
Known Issue: Image Tagging Failure During OpenStack Upgrade
-
- Observed Error
```
2025-11-27 09:56:32 | 2025-11-27 09:56:32.158531 | 52540032-4698-d560-1726-00000000024a | FATAL | Tag cluster.common.tag/cinder-backup:pcmklatest to latest cluster.common.tag/openstack-cinder-backup:pcmklatest image | controller-1 | error= {"changed": true, "cmd": "podman tag cluster.common.tag/openstack-cinder-backup:pcmklatest cluster.common.tag/cinder-backup:pcmklatest", "delta": "0:00:00.082326", "end": "2025-11-27 09:56:32.129833", "msg": "non-zero return code", "rc": 125, "start": "2025-11-27 09:56:32.047507", "stderr": "Error: cluster.common.tag/openstack-cinder-backup:pcmklatest: image not known", "stderr_lines": ["Error: cluster.common.tag/openstack-cinder-backup:pcmklatest: image not known"], "stdout": "", "stdout_lines": []}```
- Observed Error
-
- Root Cause
This is because PCS resources cinder-volume and cinder-backup run only on one node and are started by Pacemaker if there is a failover to another node. This means only one node has a running container using the tagged image. When we run an upgrade or update, there is a post-operation that calls cleanup of container images. - Workaround
- Root Cause
1. During upgrade or update run pass following parameter to the run command:
```
--skip-tags podman_purge
```
-
- Recovery
1. Find out on which node we have running cinder-volume and cinder-backup services.
```
pcs status
```
- Recovery
2. On the node that is running cinder-volume, look at the list of images.
```
podman image list
```
3. You will see `cluster.common.tag/openstack-cinder-volume pcmklatest` tag pointing to an image, for example:
`undercloud-0.ctlplane.redhat.local:8787/rhosp-rhel8/openstack-cinder-volume` with example tag `16.2_20251118.1`
4. On remaining nodes, create the `cluster.common.tag/openstack-cinder-volume:pcmklatest` tag.
```
podman tag undercloud-0.ctlplane.redhat.local:8787/rhosp-rhel8/openstack-cinder-volume:16.2_20251118.1 cluster.common.tag/openstack-cinder-volume:pcmklatest
```
-
- Related Jira Issue
OSPRH-22122
- Related Jira Issue
- is caused by
-
OSPRH-22122 Debug the image tagging failure for cinder-backup during OC upgrade
-
- Closed
-