Uploaded image for project: 'OpenShift SDN'
  1. OpenShift SDN
  2. SDN-2955

failure in test case "[sig-arch][bz-etcd][Late] Alerts alert/etcdMembersDown should not be at or above info"

XMLWordPrintable

    • SDN Sprint 218, SDN Sprint 219
    • 0
    • 0

      job link

      must-gather

      log snippet from e2e log:

      [It] alert/etcdMembersDown should not be at or above info [Suite:openshift/conformance/parallel]
        github.com/openshift/origin/test/extended/prometheus/alerts.go:23
      Apr 15 08:51:08.297: FAIL: etcdMembersDown was at or above info for at least 30m33s on platformidentification.JobType{Release:"4.11", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=3.49s): pending for 30m3s, firing for 30m33s:
      
      Apr 15 08:40:56.000 - 611s  E alert/etcdMembersDown ns/openshift-etcd pod/etcd-ip-10-0-179-92.us-west-2.compute.internal ALERTS{alertname="etcdMembersDown", alertstate="firing", job="etcd", namespace="openshift-etcd", pod="etcd-ip-10-0-179-92.us-west-2.compute.internal", prometheus="openshift-monitoring/k8s", service="etcd", severity="critical"}
      Apr 15 08:40:56.000 - 611s  E alert/etcdMembersDown ns/openshift-etcd pod/etcd-ip-10-0-181-161.us-west-2.compute.internal ALERTS{alertname="etcdMembersDown", alertstate="firing", job="etcd", namespace="openshift-etcd", pod="etcd-ip-10-0-181-161.us-west-2.compute.internal", prometheus="openshift-monitoring/k8s", service="etcd", severity="critical"}
      Apr 15 08:40:56.000 - 611s  E alert/etcdMembersDown ns/openshift-etcd pod/etcd-ip-10-0-215-42.us-west-2.compute.internal ALERTS{alertname="etcdMembersDown", alertstate="firing", job="etcd", namespace="openshift-etcd", pod="etcd-ip-10-0-215-42.us-west-2.compute.internal", prometheus="openshift-monitoring/k8s", service="etcd", severity="critical"}
      
      Full Stack Trace
      github.com/openshift/origin/test/extended/prometheus.init.0.func1.1()
      	github.com/openshift/origin/test/extended/prometheus/alerts.go:24 +0x96
      github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc0000001a0)
      	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/leafnodes/runner.go:113 +0xba
      github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0xc002440ea0)
      	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/leafnodes/runner.go:64 +0x125
      github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run(0x203000)
      	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/leafnodes/it_node.go:26 +0x7b
      github.com/onsi/ginkgo/internal/spec.(*Spec).runSample(0xc002a5dd10, 0xc002441268, {0x8c6f6c0, 0xc000426bc0})
      	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/spec/spec.go:215 +0x2a9
      github.com/onsi/ginkgo/internal/spec.(*Spec).Run(0xc002a5dd10, {0x8c6f6c0, 0xc000426bc0})
      	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/spec/spec.go:138 +0xe7
      github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec(0xc002e14000, 0xc002a5dd10)
      	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/specrunner/spec_runner.go:200 +0xe5
      github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs(0xc002e14000)
      	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/specrunner/spec_runner.go:170 +0x1a5
      github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc002e14000)
      	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/specrunner/spec_runner.go:66 +0xc5
      github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc000438000, {0x8c6f9e0, 0xc002182500}, {0x0, 0xc00009b800}, {0xc0014dcd60, 0x1, 0x1}, {0x8d84cb8, 0xc000426bc0}, ...)
      	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/suite/suite.go:62 +0x4b2
      github.com/openshift/origin/pkg/test/ginkgo.(*TestOptions).Run(0xc00220b980, {0xc0012b00c0, 0xc6407b0, 0x484f160})
      	github.com/openshift/origin/pkg/test/ginkgo/cmd_runtest.go:61 +0x3be
      main.newRunTestCommand.func1.1()
      	github.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:434 +0x32
      github.com/openshift/origin/test/extended/util.WithCleanup(0xc00183fc18)
      	github.com/openshift/origin/test/extended/util/test.go:168 +0xad
      main.newRunTestCommand.func1(0xc00222f680, {0xc0012b00c0, 0x1, 0x1})
      	github.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:434 +0x349
      github.com/spf13/cobra.(*Command).execute(0xc00222f680, {0xc0012b0060, 0x1, 0x1})
      	github.com/spf13/cobra@v1.2.1/command.go:856 +0x60e
      github.com/spf13/cobra.(*Command).ExecuteC(0xc00222ec80)
      	github.com/spf13/cobra@v1.2.1/command.go:974 +0x3bc
      github.com/spf13/cobra.(*Command).Execute(...)
      	github.com/spf13/cobra@v1.2.1/command.go:902
      main.main.func1(0xc00049a900)
      	github.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:84 +0x8a
      main.main()
      	github.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:85 +0x3b6
      [AfterEach] [sig-arch][bz-etcd][Late] Alerts
        github.com/openshift/origin/test/extended/util/client.go:151
      [AfterEach] [sig-arch][bz-etcd][Late] Alerts
        github.com/openshift/origin/test/extended/util/client.go:152
      fail [github.com/openshift/origin/test/extended/prometheus/alerts.go:24]: Apr 15 08:51:08.297: etcdMembersDown was at or above info for at least 30m33s on platformidentification.JobType{Release:"4.11", FromRelease:"", Platform:"aws", Architecture:"amd64", Network:"ovn", Topology:"ha"} (maxAllowed=3.49s): pending for 30m3s, firing for 30m33s:
      

      link to this job's testgrid for reference.

       

              npinaeva@redhat.com Nadia Pinaeva
              jluhrsen Jamo Luhrsen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: