Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-18999

Intermittent 504 Gateway Time-out

XMLWordPrintable

    • No
    • Sprint 242
    • 1
    • Approved
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the default low settings for QPS and Burst caused the image registry to return with a 504 gateway timeout when API server requests were not returned in an appropriate time. To resolve this issue, users had to restart the image registry. With this update, the default settings for QPS and Burst have been increased, and this issue no longer occurs.

      Cause: Default low settings for QPS and Burst caused the registry to return with a 504 gateway timeout when apiserver requests did not return timely.
      Consequence: The registry needed to be manually restarted.
      Fix: Increase QPS and Burst settings.
      Result: Bug doesn’t present anymore.
      Show
      * Previously, the default low settings for QPS and Burst caused the image registry to return with a 504 gateway timeout when API server requests were not returned in an appropriate time. To resolve this issue, users had to restart the image registry. With this update, the default settings for QPS and Burst have been increased, and this issue no longer occurs. Cause: Default low settings for QPS and Burst caused the registry to return with a 504 gateway timeout when apiserver requests did not return timely. Consequence: The registry needed to be manually restarted. Fix: Increase QPS and Burst settings. Result: Bug doesn’t present anymore.
    • Bug Fix
    • Done

      Description of problem:

      Image pulls fail with http status 504, gateway timeout until image registry pods are restarted.

      Version-Release number of selected component (if applicable):

      4.13.12

      How reproducible:

      Intermittent

      Steps to Reproduce:

      1.
      2.
      3.
      

      Actual results:

      Images can't be pulled: 
      podman pull registry.ci.openshift.org/ci/applyconfig:latest Trying to pull registry.ci.openshift.org/ci/applyconfig:latest... Getting image source signatures Error: reading signatures: downloading signatures for sha256:83c1b636069c3302f5ba5075ceeca5c4a271767900fee06b919efc3c8fa14984 in registry.ci.openshift.org/ci/applyconfig: received unexpected HTTP status: 504 Gateway Time-out
      
      
      Image registry pods contain errors:
      time="2023-09-01T02:25:39.596485238Z" level=warning msg="error authorizing context: access denied" go.version="go1.19.10 X:strictfipsruntime" http.request.host=registry.ci.openshift.org http.request.id=3e805818-515d-443f-8d9b-04667986611d http.request.method=GET http.request.remoteaddr=18.218.67.82 http.request.uri="/v2/ocp/4-dev-preview/manifests/sha256:caf073ce29232978c331d421c06ca5c2736ce5461962775fdd760b05fb2496a0" http.request.useragent="containers/5.24.1 (github.com/containers/image)" vars.name=ocp/4-dev-preview vars.reference="sha256:caf073ce29232978c331d421c06ca5c2736ce5461962775fdd760b05fb2496a0"

      Expected results:

      Image registry does not return gateway timeouts

      Additional info:

      Must gather(s) attached, additional information in linked OHSS ticket.

       

            fmissi Flavian Missi
            cbusse.openshift Claudio Busse
            XiuJuan Wang XiuJuan Wang
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: