[4.11][sig-arch] Check if alerts are firing during or after upgrade success - alert TargetDown fired for openshift-config-operator/metrics

[sig-arch] Check if alerts are firing during or after upgrade success is failing in 4.11 z-stream upgrade-rollback job. The job is testing z stream rollback by installing 4.11.0, updating towards a recent 4.11 nightly, and then, at some random point during that update, rolling back to 4.11.0.

The error is:

{Sep  7 08:49:15.682: Unexpected alerts fired or pending during the upgrade:

alert TargetDown fired for 1470 seconds with labels: {job="metrics", namespace="openshift-config-operator", service="metrics", severity="warning"} Failure Sep  7 08:49:15.682: Unexpected alerts fired or pending during the upgrade:

alert TargetDown fired for 1470 seconds with labels: {job="metrics", namespace="openshift-config-operator", service="metrics", severity="warning"}

github.com/openshift/origin/test/extended/util/disruption.(*chaosMonkeyAdapter).Test(0xc0005ace10, 0xc001457548)
	github.com/openshift/origin/test/extended/util/disruption/disruption.go:192 +0x332
k8s.io/kubernetes/test/e2e/chaosmonkey.(*Chaosmonkey).Do.func1()
	k8s.io/kubernetes@v1.24.0/test/e2e/chaosmonkey/chaosmonkey.go:90 +0x6a
created by k8s.io/kubernetes/test/e2e/chaosmonkey.(*Chaosmonkey).Do
	k8s.io/kubernetes@v1.24.0/test/e2e/chaosmonkey/chaosmonkey.go:87 +0x87}

The failed job is here https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.11-e2e-aws-upgrade-rollback-oldest-supported/1699669129511833600

4.11.49

Flaky

1.
2.
3.

disruption_tests: [sig-arch] Check if alerts are firing during or after upgrade success 

{Sep  7 08:49:15.682: Unexpected alerts fired or pending during the upgrade:

alert TargetDown fired for 1470 seconds with labels: {job="metrics", namespace="openshift-config-operator", service="metrics", severity="warning"} Failure Sep  7 08:49:15.682: Unexpected alerts fired or pending during the upgrade:

alert TargetDown fired for 1470 seconds with labels: {job="metrics", namespace="openshift-config-operator", service="metrics", severity="warning"}

github.com/openshift/origin/test/extended/util/disruption.(*chaosMonkeyAdapter).Test(0xc0005ace10, 0xc001457548)
	github.com/openshift/origin/test/extended/util/disruption/disruption.go:192 +0x332
k8s.io/kubernetes/test/e2e/chaosmonkey.(*Chaosmonkey).Do.func1()
	k8s.io/kubernetes@v1.24.0/test/e2e/chaosmonkey/chaosmonkey.go:90 +0x6a
created by k8s.io/kubernetes/test/e2e/chaosmonkey.(*Chaosmonkey).Do
	k8s.io/kubernetes@v1.24.0/test/e2e/chaosmonkey/chaosmonkey.go:87 +0x87}

The test case passed