-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
None
-
None
job link
log snippet from e2e log:
[It] alert/etcdMembersDown should not be at or above info [Suite:openshift/conformance/parallel] github.com/openshift/origin/test/extended/prometheus/alerts.go:23 Apr 15 08:51:08.297: FAIL: etcdMembersDown was at or above info for at least 30m33s on platformidentification.JobType{Release:"4.11", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=3.49s): pending for 30m3s, firing for 30m33s: Apr 15 08:40:56.000 - 611s E alert/etcdMembersDown ns/openshift-etcd pod/etcd-ip-10-0-179-92.us-west-2.compute.internal ALERTS{alertname="etcdMembersDown", alertstate="firing", job="etcd", namespace="openshift-etcd", pod="etcd-ip-10-0-179-92.us-west-2.compute.internal", prometheus="openshift-monitoring/k8s", service="etcd", severity="critical"} Apr 15 08:40:56.000 - 611s E alert/etcdMembersDown ns/openshift-etcd pod/etcd-ip-10-0-181-161.us-west-2.compute.internal ALERTS{alertname="etcdMembersDown", alertstate="firing", job="etcd", namespace="openshift-etcd", pod="etcd-ip-10-0-181-161.us-west-2.compute.internal", prometheus="openshift-monitoring/k8s", service="etcd", severity="critical"} Apr 15 08:40:56.000 - 611s E alert/etcdMembersDown ns/openshift-etcd pod/etcd-ip-10-0-215-42.us-west-2.compute.internal ALERTS{alertname="etcdMembersDown", alertstate="firing", job="etcd", namespace="openshift-etcd", pod="etcd-ip-10-0-215-42.us-west-2.compute.internal", prometheus="openshift-monitoring/k8s", service="etcd", severity="critical"} Full Stack Trace github.com/openshift/origin/test/extended/prometheus.init.0.func1.1() github.com/openshift/origin/test/extended/prometheus/alerts.go:24 +0x96 github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc0000001a0) github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/leafnodes/runner.go:113 +0xba github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0xc002440ea0) github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/leafnodes/runner.go:64 +0x125 github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run(0x203000) github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/leafnodes/it_node.go:26 +0x7b github.com/onsi/ginkgo/internal/spec.(*Spec).runSample(0xc002a5dd10, 0xc002441268, {0x8c6f6c0, 0xc000426bc0}) github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/spec/spec.go:215 +0x2a9 github.com/onsi/ginkgo/internal/spec.(*Spec).Run(0xc002a5dd10, {0x8c6f6c0, 0xc000426bc0}) github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/spec/spec.go:138 +0xe7 github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec(0xc002e14000, 0xc002a5dd10) github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/specrunner/spec_runner.go:200 +0xe5 github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs(0xc002e14000) github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/specrunner/spec_runner.go:170 +0x1a5 github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc002e14000) github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/specrunner/spec_runner.go:66 +0xc5 github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc000438000, {0x8c6f9e0, 0xc002182500}, {0x0, 0xc00009b800}, {0xc0014dcd60, 0x1, 0x1}, {0x8d84cb8, 0xc000426bc0}, ...) github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/suite/suite.go:62 +0x4b2 github.com/openshift/origin/pkg/test/ginkgo.(*TestOptions).Run(0xc00220b980, {0xc0012b00c0, 0xc6407b0, 0x484f160}) github.com/openshift/origin/pkg/test/ginkgo/cmd_runtest.go:61 +0x3be main.newRunTestCommand.func1.1() github.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:434 +0x32 github.com/openshift/origin/test/extended/util.WithCleanup(0xc00183fc18) github.com/openshift/origin/test/extended/util/test.go:168 +0xad main.newRunTestCommand.func1(0xc00222f680, {0xc0012b00c0, 0x1, 0x1}) github.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:434 +0x349 github.com/spf13/cobra.(*Command).execute(0xc00222f680, {0xc0012b0060, 0x1, 0x1}) github.com/spf13/cobra@v1.2.1/command.go:856 +0x60e github.com/spf13/cobra.(*Command).ExecuteC(0xc00222ec80) github.com/spf13/cobra@v1.2.1/command.go:974 +0x3bc github.com/spf13/cobra.(*Command).Execute(...) github.com/spf13/cobra@v1.2.1/command.go:902 main.main.func1(0xc00049a900) github.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:84 +0x8a main.main() github.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:85 +0x3b6 [AfterEach] [sig-arch][bz-etcd][Late] Alerts github.com/openshift/origin/test/extended/util/client.go:151 [AfterEach] [sig-arch][bz-etcd][Late] Alerts github.com/openshift/origin/test/extended/util/client.go:152 fail [github.com/openshift/origin/test/extended/prometheus/alerts.go:24]: Apr 15 08:51:08.297: etcdMembersDown was at or above info for at least 30m33s on platformidentification.JobType{Release:"4.11", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=3.49s): pending for 30m3s, firing for 30m33s:
link to this job's testgrid for reference.
- clones
-
SDN-2953 failure in test case "[sig-instrumentation][Late] Alerts shouldn't report any unexpected alerts in firing or pending state"
- Closed
- is cloned by
-
SDN-2956 failure in test case "[sig-instrumentation] Prometheus when installed on the cluster should start and expose a secured proxy and unsecured metrics"
- Closed