Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-10384

Quay v3.16.1 logs repeated [storage.cloud][OTEL] errors with Ceph RGW backend after upgrade

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • False
    • Quay Hosted
      • errors are seen in the quay logs. No functional blockage at the moment.
      • Deploy Red Hat Quay with Ceph RADOS Gateway as the storage backend.
      • Upgrade Quay to v3.16.1.
      • Observe quay logs

      After upgrading Red Hat Quay to v3.16.1, the registry logs show repeated error-level log entries originating from the storage cloud layer with OTEL instrumentation enabled.

      The errors appear during S3 (Ceph RGW) operations and include full AWS-style request headers logged at ERROR level under:

      gunicorn-registry stdout | 2026-01-21 21:13:14,868 [311] [ERROR] [storage.cloud] [OTEL] request xxx}
      gunicorn-registry stdout | 2026-01-21 21:13:14,930 [309] [ERROR] [storage.cloud] [OTEL] request xxx} 

      Despite these errors:

      • Image push operations work as expected
      • Image pull operations work as expected
      • No functional impact has been observed so far

      This behavior was not present before upgrading to v3.16.1, suggesting a regression or logging-level issue introduced in this release, possibly related to OpenTelemetry instrumentation for storage operations.

       

      Additional Note - this is observed with a customer with standalone quay deployment


              marckok Marcus Kok
              rhn-support-snangare Sameer Nangare
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: