Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-46352

Image registry failing single node installs in 4.19

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • 4.19.0
    • Image Registry
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Critical
    • None
    • None
    • None
    • Proposed
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Three single node aggregated jobs have failed now with 5072 failures, caused by not enough results to aggregate. Roughly half the jobs fail to install because the image registry is complaining about:

      {Operator degraded (ImagePrunerJobFailed): ImagePrunerDegraded: Job has reached the specified backoff limit Operator degraded (ImagePrunerJobFailed): ImagePrunerDegraded: Job has reached the specified backoff limit}
      

      Examples:

      https://prow.ci.openshift.org/view/gs/test-platform-results/logs/aggregated-aws-ovn-single-node-upgrade-4.19-micro-release-openshift-release-analysis-aggregator/1866988278972944384

      https://prow.ci.openshift.org/view/gs/test-platform-results/logs/aggregated-aws-ovn-single-node-upgrade-4.19-micro-release-openshift-release-analysis-aggregator/1867042320000487424

      Or an example of the sub-job with the failure:

      https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.19-e2e-aws-upgrade-ovn-single-node/1866988251177291776

      The closest I was able to get to a root cause was here

      I1212 00:08:31.988786      67 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
      I1212 00:08:31.988805      67 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
      Error from server (Timeout): the server was unable to return a response in the time allotted, but may still be processing the request (get pods)
      

      Marked critical as this is a payload blocker for 4.19 at this point.

              fmissi Flavian Missi
              rhn-engineering-dgoodwin Devan Goodwin
              None
              None
              XiuJuan Wang XiuJuan Wang
              None
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: