Marking Degraded due to: unexpected on-disk state unexpected on-disk state validating against rendered-master-ee4ff728579addc84ddb1ca63d9e8ac1: expected target osImageURL \" \", have \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:"xxx"

Cluster is in phase of upgrade version 4.11.28 to version 4.12.21 .

  Upgrade is stuck on the MCP phase >

& oc get co | grep machine-config 
machine-config                            4.11.28  True       True         True      9h57m

nodes are stuck with the podman error >>

~~~

message: 'Node is reporting: "Error checking type of
      update image: failed to run command podman (6 tries): [timed out waiting for
      the condition, running podman pull -q --authfile /var/lib/kubelet/config.json   failed:
      Error: invalid reference format\n: exit status 125]"'
    reason: 1 nodes are reporting degraded status on sync
    status: 'True'
    type: NodeDegraded

Tried the below workaround >>  

 We found there is no issue wrt image pull using podman from mirror registry and ICSP seems to be fine.

- We found on of the issue node maste2, rpm-ostreed service was down

- We tried the below command.

# systemctl restart rpm-ostreed && rpm-ostree cleanup -r

- Still the rpmostree was up for sometimes and can go down. We again ran rpm clean up command

# rpm-ostree cleanup -r

- Post that rpm-ostree was up and postman error was sorted.

Post that rpm-ostree was up and postman error was sorted.-

 Later we got a content mismatch error for file "/usr/local/bin/configure-ovs.sh, we manually edited the current config value same as desired config (latest rendered master) value- We found failing with incorrect osImageURLs, we found the expected url missing and we manually added the osImageURL for 4.12.21

its return with below error 

~~

Message: Node is reporting: "unexpected on-disk state validating against rendered-master-ee4ff728579addc84ddb1ca63d9e8ac1: expected target osImageURL \" \", have \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3bf5db684edcaeac5e6385d8540a55283d50d56385d65dd2511dfe8219cd708\" (\"e4b804167aa1a93aac1047649c473a623944543af4668b52740244d8d3b7c677\")"

~~~~

Try with KCS 

https://access.redhat.com/solutions/5598401

still on the same phase.

Already going for BUG.

https://bugzilla.redhat.com/show_bug.cgi?id=2111817

4.12.21

1.
2.
3.

Kindly refer the case- 03556455 , must-gather , SOS report and ournalctl -b -1 -u ostree-finalize-staged.service logs are attched.