Uploaded image for project: 'OpenShift SDN'
  1. OpenShift SDN
  2. SDN-2953

failure in test case "[sig-instrumentation][Late] Alerts shouldn't report any unexpected alerts in firing or pending state"

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Major
    • None
    • None
    • None
    • 0
    • 0

    Description

      job link
      same failure happened in previous two jobs, here and here

      must-gather

      log snippet from e2e log:

      [It] shouldn't report any unexpected alerts in firing or pending state [Suite:openshift/conformance/parallel]
  github.com/openshift/origin/test/extended/prometheus/prometheus.go:270
Apr 16 10:45:25.127: INFO: Alerts were detected during test run which are allowed:

alert KubePodNotReady fired for 2168 seconds with labels: {namespace="openshift-e2e-loki", pod=[link to this job's testgrid for reference.|https://testgrid.k8s.io/redhat-openshift-ocp-release-4.11-informing#periodic-ci-openshift-release-master-ci-4.11-e2e-aws-ovn]
"loki-promtail-2jqqh", severity="warning"} (allowed: Loki is nice to have, but we can allow it to be down)
alert KubePodNotReady fired for 2168 seconds with labels: {namespace="openshift-e2e-loki", pod="loki-promtail-8r54v", severity="warning"} (allowed: Loki is nice to have, but we can allow it to be down)
alert KubePodNotReady fired for 2168 seconds with labels: {namespace="openshift-e2e-loki", pod="loki-promtail-9q62d", severity="warning"} (allowed: Loki is nice to have, but we can allow it to be down)
alert KubePodNotReady fired for 2168 seconds with labels: {namespace="openshift-e2e-loki", pod="loki-promtail-jwf2q", severity="warning"} (allowed: Loki is nice to have, but we can allow it to be down)
alert KubePodNotReady fired for 2168 seconds with labels: {namespace="openshift-e2e-loki", pod="loki-promtail-pwmgs", severity="warning"} (allowed: Loki is nice to have, but we can allow it to be down)
alert KubePodNotReady fired for 2168 seconds with labels: {namespace="openshift-e2e-loki", pod="loki-promtail-zxffs", severity="warning"} (allowed: Loki is nice to have, but we can allow it to be down)
alert TargetDown fired for 2174 seconds with labels: {job="promtail", namespace="openshift-e2e-loki", service="promtail", severity="warning"} (allowed: Loki is nice to have, but we can allow it to be down)
Apr 16 10:45:25.127: FAIL: Unexpected alerts fired or pending after the test run:

alert CertifiedOperatorsCatalogError fired for 2261 seconds with labels: {container="catalog-operator", endpoint="https-metrics", exported_namespace="openshift-marketplace", instance="10.128.0.20:8443", job="catalog-operator-metrics", name="certified-operators", namespace="openshift-operator-lifecycle-manager", pod="catalog-operator-7445c654c9-qpjt5", service="catalog-operator-metrics", severity="warning"}
alert CommunityOperatorsCatalogError fired for 2241 seconds with labels: {container="catalog-operator", endpoint="https-metrics", exported_namespace="openshift-marketplace", instance="10.128.0.20:8443", job="catalog-operator-metrics", name="community-operators", namespace="openshift-operator-lifecycle-manager", pod="catalog-operator-7445c654c9-qpjt5", service="catalog-operator-metrics", severity="warning"}
alert KubeDaemonSetRolloutStuck fired for 1268 seconds with labels: {container="kube-rbac-proxy-main", daemonset="loki-promtail", endpoint="https-main", job="kube-state-metrics", namespace="openshift-e2e-loki", service="kube-state-metrics", severity="warning"}
alert KubePodNotReady fired for 1958 seconds with labels: {namespace="openshift-marketplace", pod="certified-operators-cbn68", severity="warning"}
alert KubePodNotReady fired for 1958 seconds with labels: {namespace="openshift-marketplace", pod="community-operators-v4kkt", severity="warning"}
alert KubePodNotReady fired for 1958 seconds with labels: {namespace="openshift-marketplace", pod="redhat-marketplace-zglj6", severity="warning"}
alert KubePodNotReady fired for 1958 seconds with labels: {namespace="openshift-marketplace", pod="redhat-operators-ldr7z", severity="warning"}
alert KubePodNotReady fired for 2168 seconds with labels: {namespace="openshift-marketplace", pod="certified-operators-5bftm", severity="warning"}
alert KubePodNotReady fired for 2168 seconds with labels: {namespace="openshift-marketplace", pod="community-operators-tdx7v", severity="warning"}
alert KubePodNotReady fired for 2168 seconds with labels: {namespace="openshift-marketplace", pod="redhat-marketplace-rvd69", severity="warning"}
alert KubePodNotReady fired for 2168 seconds with labels: {namespace="openshift-marketplace", pod="redhat-operators-qwqzk", severity="warning"}
alert RedhatMarketplaceCatalogError fired for 2261 seconds with labels: {container="catalog-operator", endpoint="https-metrics", exported_namespace="openshift-marketplace", instance="10.128.0.20:8443", job="catalog-operator-metrics", name="redhat-marketplace", namespace="openshift-operator-lifecycle-manager", pod="catalog-operator-7445c654c9-qpjt5", service="catalog-operator-metrics", severity="warning"}
alert RedhatOperatorsCatalogError fired for 2251 seconds with labels: {container="catalog-operator", endpoint="https-metrics", exported_namespace="openshift-marketplace", instance="10.128.0.20:8443", job="catalog-operator-metrics", name="redhat-operators", namespace="openshift-operator-lifecycle-manager", pod="catalog-operator-7445c654c9-qpjt5", service="catalog-operator-metrics", severity="warning"}

Full Stack Trace
github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc0000001a0)
	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/leafnodes/runner.go:113 +0xba
github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0xc002ce0ea0)
	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/leafnodes/runner.go:64 +0x125
github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run(0x7f2d81032fff)
	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/leafnodes/it_node.go:26 +0x7b
github.com/onsi/ginkgo/internal/spec.(*Spec).runSample(0xc0029725a0, 0xc002ce1268, {0x8c6fa00, 0xc00046abc0})
	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/spec/spec.go:215 +0x2a9
github.com/onsi/ginkgo/internal/spec.(*Spec).Run(0xc0029725a0, {0x8c6fa00, 0xc00046abc0})
	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/spec/spec.go:138 +0xe7
github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec(0xc002c9c140, 0xc0029725a0)
	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/specrunner/spec_runner.go:200 +0xe5
github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs(0xc002c9c140)
	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/specrunner/spec_runner.go:170 +0x1a5
github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc002c9c140)
	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/specrunner/spec_runner.go:66 +0xc5
github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc00047e000, {0x8c6fd20, 0xc0023bdb30}, {0x0, 0xc000096800}, {0xc0022ef890, 0x1, 0x1}, {0x8d84ff8, 0xc00046abc0}, ...)
	github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/suite/suite.go:62 +0x4b2
github.com/openshift/origin/pkg/test/ginkgo.(*TestOptions).Run(0xc001ec9380, {0xc001014860, 0xc6407f0, 0x484f3a0})
	github.com/openshift/origin/pkg/test/ginkgo/cmd_runtest.go:61 +0x3be
main.newRunTestCommand.func1.1()
	github.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:434 +0x32
github.com/openshift/origin/test/extended/util.WithCleanup(0xc001f3fc18)
	github.com/openshift/origin/test/extended/util/test.go:168 +0xad
main.newRunTestCommand.func1(0xc000d09900, {0xc001014860, 0x1, 0x1})
	github.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:434 +0x349
github.com/spf13/cobra.(*Command).execute(0xc000d09900, {0xc001014830, 0x1, 0x1})
	github.com/spf13/cobra@v1.2.1/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc0020dfb80)
	github.com/spf13/cobra@v1.2.1/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
	github.com/spf13/cobra@v1.2.1/command.go:902
main.main.func1(0xc001bc8500)
	github.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:84 +0x8a
main.main()
	github.com/openshift/origin/cmd/openshift-tests/openshift-tests.go:85 +0x3b6
[AfterEach] [sig-instrumentation][Late] Alerts
  github.com/openshift/origin/test/extended/util/client.go:151
[AfterEach] [sig-instrumentation][Late] Alerts
  github.com/openshift/origin/test/extended/util/client.go:152
fail [github.com/onsi/ginkgo@v4.7.0-origin.0+incompatible/internal/leafnodes/runner.go:113]: Apr 16 10:45:25.127: Unexpected alerts fired or pending after the test run:

alert CertifiedOperatorsCatalogError fired for 2261 seconds with labels: {container="catalog-operator", endpoint="https-metrics", exported_namespace="openshift-marketplace", instance="10.128.0.20:8443", job="catalog-operator-metrics", name="certified-operators", namespace="openshift-operator-lifecycle-manager", pod="catalog-operator-7445c654c9-qpjt5", service="catalog-operator-metrics", severity="warning"}
alert CommunityOperatorsCatalogError fired for 2241 seconds with labels: {container="catalog-operator", endpoint="https-metrics", exported_namespace="openshift-marketplace", instance="10.128.0.20:8443", job="catalog-operator-metrics", name="community-operators", namespace="openshift-operator-lifecycle-manager", pod="catalog-operator-7445c654c9-qpjt5", service="catalog-operator-metrics", severity="warning"}
alert KubeDaemonSetRolloutStuck fired for 1268 seconds with labels: {container="kube-rbac-proxy-main", daemonset="loki-promtail", endpoint="https-main", job="kube-state-metrics", namespace="openshift-e2e-loki", service="kube-state-metrics", severity="warning"}
alert KubePodNotReady fired for 1958 seconds with labels: {namespace="openshift-marketplace", pod="certified-operators-cbn68", severity="warning"}
alert KubePodNotReady fired for 1958 seconds with labels: {namespace="openshift-marketplace", pod="community-operators-v4kkt", severity="warning"}
alert KubePodNotReady fired for 1958 seconds with labels: {namespace="openshift-marketplace", pod="redhat-marketplace-zglj6", severity="warning"}
alert KubePodNotReady fired for 1958 seconds with labels: {namespace="openshift-marketplace", pod="redhat-operators-ldr7z", severity="warning"}
alert KubePodNotReady fired for 2168 seconds with labels: {namespace="openshift-marketplace", pod="certified-operators-5bftm", severity="warning"}
alert KubePodNotReady fired for 2168 seconds with labels: {namespace="openshift-marketplace", pod="community-operators-tdx7v", severity="warning"}
alert KubePodNotReady fired for 2168 seconds with labels: {namespace="openshift-marketplace", pod="redhat-marketplace-rvd69", severity="warning"}
alert KubePodNotReady fired for 2168 seconds with labels: {namespace="openshift-marketplace", pod="redhat-operators-qwqzk", severity="warning"}
alert RedhatMarketplaceCatalogError fired for 2261 seconds with labels: {container="catalog-operator", endpoint="https-metrics", exported_namespace="openshift-marketplace", instance="10.128.0.20:8443", job="catalog-operator-metrics", name="redhat-marketplace", namespace="openshift-operator-lifecycle-manager", pod="catalog-operator-7445c654c9-qpjt5", service="catalog-operator-metrics", severity="warning"}
alert RedhatOperatorsCatalogError fired for 2251 seconds with labels: {container="catalog-operator", endpoint="https-metrics", exported_namespace="openshift-marketplace", instance="10.128.0.20:8443", job="catalog-operator-metrics", name="redhat-operators", namespace="openshift-operator-lifecycle-manager", pod="catalog-operator-7445c654c9-qpjt5", service="catalog-operator-metrics", severity="warning"}

      link to this job's testgrid for reference.

       

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. SDN-2952 failure in test case "[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should honor timeout [Conformance]"

          • Minor - To be worked after urgent, high, and medium priorities are resolved. This term is chosen when the severity of the issue is low, or the complexity or effort to correct it may be higher, relatively speaking. For low priority issues, known workarounds exist or are not needed due to the trivial effort needed to address the issue.
          • Closed
          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. SDN-2955 failure in test case "[sig-arch][bz-etcd][Late] Alerts alert/etcdMembersDown should not be at or above info"

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Bug - A problem that impairs or prevents the functions of the product. SDN-2954 failure in test case "[sig-operator] an end user can use OLM can subscribe to the operator"

          • Normal - To be worked after urgent and high priorities are resolved. This term is chosen when the severity of an issue is relatively close to the level of effort to fix it. The existence of an easily implemented workaround can also lead to this priority level instead of a higher priority.
          • Closed

          Activity

            People

              mmahmoud@redhat.com Mohamed Mahmoud
              jluhrsen Jamo Luhrsen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: