Uploaded image for project: 'OCP Technical Release Team'
  1. OCP Technical Release Team
  2. TRT-735

Investigate MCDPivotError alerts firing

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Done
    • Icon: Normal Normal
    • None
    • None
    • False
    • None
    • False

      This test always flakes now, so this is not failing jobs, but looks like something is happening here. Over last two weeks:

       release   | test_id |                          alert                           |             namespace             | result | alert_count                                                                                            
      ------------+---------+----------------------------------------------------------+-----------------------------------+--------+-------------                                                                                           
       4.13       |       1 | MCDPivotError                                            | openshift-machine-config-operator | reject |          39                     
      

      Looks to have started on Nov 29, we began tracking Nov 21 so it may be new. A few hits per day but some recent examples:

      https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.13-upgrade-from-stable-4.12-e2e-aws-ovn-upgrade/1602196947291082752

      https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.13-upgrade-from-stable-4.12-e2e-aws-sdn-upgrade/1601563440122433536

      https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.13-upgrade-from-stable-4.12-e2e-azure-sdn-upgrade/1600566958846119936

      Seems to cross clouds and ovn/sdn. Looks like it's always 4.12 to 4.13 upgrades though. (aka minor)

      Not super common, but enough to take note and currently our most frequent alert that isn't handled.

      Dig a little for details on the alert, pod logs, and then file a bug (or convert this card) if appropriate.

              kenzhang@redhat.com Ken Zhang
              rhn-engineering-dgoodwin Devan Goodwin
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: