Several of the 4.14 tech preview jobs have been permfailing since January 20th:
https://prow.ci.openshift.org/job-history/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-aws-sdn-techpreview

example failures:
https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-aws-sdn-techpreview/1749740678981619712
https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-azure-sdn-techpreview/1749740680076333056
https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.14-e2e-gcp-sdn-techpreview/1749740673168314368

All the jobs fail in the same way:
fail [github.com/openshift/origin/test/extended/prometheus/prometheus.go:459]: possibly some services didn't register ServiceMonitors to allow metrics collection

It seems that some resource must not be getting installed in techpreview clusters which is causing this test to fail.

Unfortunately these jobs failing prevents us from accepting 4.14 payloads, which makes this extremely important to resolve quickly.

https://amd64.ocp.releases.ci.openshift.org/releasestream/4.14.0-0.nightly/release/4.14.0-0.nightly-2024-01-20-231201 represents the first payload that showed these failures but I did not see anything obviously suspect in the commit changes between it and the last accepted payload.

The failing tech preview jobs block payload acceptance because this payload acceptance job depends on consuming results from the tech preview jobs: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.14-overall-analysis-all/1749818281608351744