Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-18999

Intermittent 504 Gateway Time-out

    XMLWordPrintable

Details

    • No
    • Sprint 242
    • 1
    • Approved
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the default low settings for QPS and Burst caused the image registry to return with a 504 gateway timeout when API server requests were not returned in an appropriate time. To resolve this issue, users had to restart the image registry. With this update, the default settings for QPS and Burst have been increased, and this issue no longer occurs.

      Cause: Default low settings for QPS and Burst caused the registry to return with a 504 gateway timeout when apiserver requests did not return timely.
      Consequence: The registry needed to be manually restarted.
      Fix: Increase QPS and Burst settings.
      Result: Bug doesn’t present anymore.
      Show
      * Previously, the default low settings for QPS and Burst caused the image registry to return with a 504 gateway timeout when API server requests were not returned in an appropriate time. To resolve this issue, users had to restart the image registry. With this update, the default settings for QPS and Burst have been increased, and this issue no longer occurs. Cause: Default low settings for QPS and Burst caused the registry to return with a 504 gateway timeout when apiserver requests did not return timely. Consequence: The registry needed to be manually restarted. Fix: Increase QPS and Burst settings. Result: Bug doesn’t present anymore.
    • Bug Fix
    • Done

    Description

      Description of problem:

      Image pulls fail with http status 504, gateway timeout until image registry pods are restarted.

      Version-Release number of selected component (if applicable):

      4.13.12

      How reproducible:

      Intermittent

      Steps to Reproduce:

      1.
2.
3.

      Actual results:

      Images can't be pulled: 
podman pull registry.ci.openshift.org/ci/applyconfig:latest Trying to pull registry.ci.openshift.org/ci/applyconfig:latest... Getting image source signatures Error: reading signatures: downloading signatures for sha256:83c1b636069c3302f5ba5075ceeca5c4a271767900fee06b919efc3c8fa14984 in registry.ci.openshift.org/ci/applyconfig: received unexpected HTTP status: 504 Gateway Time-out


Image registry pods contain errors:
time="2023-09-01T02:25:39.596485238Z" level=warning msg="error authorizing context: access denied" go.version="go1.19.10 X:strictfipsruntime" http.request.host=registry.ci.openshift.org http.request.id=3e805818-515d-443f-8d9b-04667986611d http.request.method=GET http.request.remoteaddr=18.218.67.82 http.request.uri="/v2/ocp/4-dev-preview/manifests/sha256:caf073ce29232978c331d421c06ca5c2736ce5461962775fdd760b05fb2496a0" http.request.useragent="containers/5.24.1 (github.com/containers/image)" vars.name=ocp/4-dev-preview vars.reference="sha256:caf073ce29232978c331d421c06ca5c2736ce5461962775fdd760b05fb2496a0"

      Expected results:

      Image registry does not return gateway timeouts

      Additional info:

      Must gather(s) attached, additional information in linked OHSS ticket.

       

      Attachments

        Issue Links

          blocks

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-19379 Intermittent 504 Gateway Time-out

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-19379 Intermittent 504 Gateway Time-out

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          links to

          GitHub openshift/image-registry#380: OCPBUGS-18999: increase rest.Config QPS and Burst

          Web Link RHEA-2023:7198 rpm

          Activity

            People

              fmissi Flavian Missi
              cbusse.openshift Claudio Busse
              xiujuan wang xiujuan wang
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: