Slack thread https://redhat-internal.slack.com/archives/C01CQA76KMX/p1714157947169409

4.15 and 4.14 metal both look significantly worse after GA, and it looks like maybe it's related to MOC.

4.15 last week is worse than GA:
https://sippy.dptools.openshift.org/sippy-ng/component_readiness/main?baseEndTime=2024-02-28%2023%3A59%3A59&baseRelease=4.15&baseStartTime=2024-01-29%2000%3A00%3A00&confidence=95&excludeArches=arm64&excludeArches=heterogeneous&excludeArches=ppc64le&excludeArches=s390x&excludeClouds=openstack&excludeClouds=ibmcloud&excludeClouds=libvirt&excludeClouds=ovirt&excludeClouds=unknown&excludeVariants=hypershift&excludeVariants=osd&excludeVariants=microshift&excludeVariants=techpreview&excludeVariants=single-node&excludeVariants=assisted&excludeVariants=compact&groupBy=cloud&groupBy=arch&groupBy=network&ignoreDisruption=1&ignoreMissing=0&minFail=3&pity=5&sampleEndTime=2024-04-26%2023%3A59%3A59&sampleRelease=4.15&sampleStartTime=2024-04-20%2000%3A00%3A00

4.14 last week is worse than GA:
https://sippy.dptools.openshift.org/sippy-ng/component_readiness/main?baseEndTime=2023-10-31%2023%3A59%3A59&baseRelease=4.14&baseStartTime=2023-10-01%2000%3A00%3A00&confidence=95&excludeArches=arm64&excludeArches=heterogeneous&excludeArches=ppc64le&excludeArches=s390x&excludeClouds=openstack&excludeClouds=ibmcloud&excludeClouds=libvirt&excludeClouds=ovirt&excludeClouds=unknown&excludeVariants=hypershift&excludeVariants=osd&excludeVariants=microshift&excludeVariants=techpreview&excludeVariants=single-node&excludeVariants=assisted&excludeVariants=compact&groupBy=cloud&groupBy=arch&groupBy=network&ignoreDisruption=1&ignoreMissing=0&minFail=3&pity=5&sampleEndTime=2024-04-26%2023%3A59%3A59&sampleRelease=4.14&sampleStartTime=2024-04-20%2000%3A00%3A00

Scott and I dug into some of the failures, for example these jobs:
https://sippy.dptools.openshift.org/sippy-ng/jobs/4.15/runs?filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22failed_test_names%22%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22%5Bsig-cluster-lifecycle%5D%20Cluster%20completes%20upgrade%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22metal-ipi%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22upgrade-minor%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22ovn%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22amd64%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22upgrade%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22ha%22%7D%5D%2C%22linkOperator%22%3A%22and%22%7D&sort=asc&sortField=timestamp

There was a pause between 4/6 and 4/20 where the upgrades were healthy
4/20 they started failing again, and the failures are all on MOC hosts

I also see incidences of this on 4.16, but 4.16 is noisier on metal since there's other regressions