While upgrading 3555 SNOs from 4.12.29 to 4.13.9 via 4 CGU objects. Two SNOs in the completed status of the CGUs did not actually complete operator upgrade (vm00357 and vm01725).  This appears to be the case because the two clusters ran into a race condition on when the OLM reads a new catalogsource vs when the ACM policy engine reports that a policy is compliant.

Hub 4.13.10
ACM - 2.9.0-DOWNSTREAM-2023-09-01-02-58-15
TALM - 4.13.0
Deployed SNOs 4.12.29 upgraded to 4.13.9 (with operator upgrades)

Rarely at scale 2 out of 3555 total upgrades, 2 out of 21 total upgrade failures.  This does account for all of the operator upgrade failures however.

1.
2.
3.