• Icon: Task Task
    • Resolution: Done
    • Icon: Major Major
    • omr-v1.2.0
    • None
    • OMR, quay

      Problem: in its current form it seems that all components and workers running inside the quay pod operator with log-level set to debug. This creates massive log traffic, that creates at leats 270MB of log data per day for a registry that's mostly idle. 

      Task: We should reduce the log level down to WARN or higher to keep log traffic reasonable and focus on error or problem scenarios.

      Example of current log data:

      gunicorn-registry stdout | 2022-04-05 10:52:31,465 [225] [DEBUG] [auth.permissions] Identity loaded: <Identity id="dmesser" auth_type="signed_jwt" provides={resource(type='repository', namespace='openshift', name='ansible-automation-platform/platform-operator-bundle', role='write')}>
      gunicorn-registry stdout | 2022-04-05 10:52:31,465 [225] [DEBUG] [auth.permissions] Loaded signed_jwt identity for: dmesser
      gunicorn-registry stdout | 2022-04-05 10:52:31,465 [225] [DEBUG] [auth.registry_jwt_auth] Identity changed to dmesser
      gunicorn-registry stdout | 2022-04-05 10:52:31,465 [225] [DEBUG] [auth.registry_jwt_auth] Auth context set to {'version': 2, 'entity_kind': 'user', 'entity_reference': '17f90c7f-5ffe-46d5-8b15-a63a3b59390c', 'kind': 'user', 'user': 'dmesser', 'com.apostille.roots': {'openshift/ansible-automation-platform/platform-operator-bundle': '$disabled'}, 'com.apostille.root': '$disabled'}
      gunicorn-registry stdout | 2022-04-05 10:52:31,465 [225] [DEBUG] [endpoints.v2] Checking permission <class 'auth.permissions.ModifyRepositoryPermission'> for repo: openshift/ansible-automation-platform/platform-operator-bundle
      gunicorn-registry stdout | 2022-04-05 10:52:31,468 [225] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['openshift', 'ansible-automation-platform/platform-operator-bundle', 3, 1, 0])
      gunicorn-registry stdout | 2022-04-05 10:52:31,480 [225] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size" FROM "manifest" AS "t1" WHERE (("t1"."repository_id" = %s) AND ("t1"."digest" = %s)) LIMIT %s OFFSET %s', [68, 'sha256:6d52549d99baf5395dd91f7444de3763f1ce8f5ca1f97434f470b5d1e58ea6a9', 1, 0])
      gunicorn-registry stdout | 2022-04-05 10:52:31,482 [225] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."repository_id", "t1"."manifest_id", "t1"."lifetime_start_ms", "t1"."lifetime_end_ms", "t1"."hidden", "t1"."reversion", "t1"."tag_kind_id", "t1"."linked_tag_id" FROM "tag" AS "t1" WHERE (("t1"."manifest_id" = %s) AND (("t1"."lifetime_end_ms" IS %s) OR ("t1"."lifetime_end_ms" >= %s))) LIMIT %s OFFSET %s', [1746, None, 1649156251481, 1, 0])
      gunicorn-registry stdout | 2022-04-05 10:52:31,484 [225] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."key", "t1"."value", "t1"."media_type_id", "t1"."source_type_id", "t2"."id", "t2"."name" FROM "label" AS "t1" INNER JOIN "mediatype" AS "t2" ON ("t1"."media_type_id" = "t2"."id") INNER JOIN "labelsourcetype" AS "t3" ON ("t1"."source_type_id" = "t3"."id") INNER JOIN "manifestlabel" AS "t4" ON ("t4"."label_id" = "t1"."id") WHERE (("t4"."manifest_id" = %s) AND ("t1"."key" ILIKE %s ESCAPE \'!\'))', [1746, 'quay%'])
      gunicorn-registry stdout | 2022-04-05 10:52:31,490 [225] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."name" FROM "manifest" AS "t1" INNER JOIN "mediatype" AS "t2" ON ("t1"."media_type_id" = "t2"."id") WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1746, 1, 0])
      gunicorn-registry stdout | 2022-04-05 10:52:31,492 [225] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."repository_id", "t1"."manifest_id", "t1"."lifetime_start_ms", "t1"."lifetime_end_ms", "t1"."hidden", "t1"."reversion", "t1"."tag_kind_id", "t1"."linked_tag_id", "t2"."id", "t2"."repository_id", "t2"."digest", "t2"."media_type_id", "t2"."manifest_bytes", "t2"."config_media_type", "t2"."layers_compressed_size" FROM "tag" AS "t1" INNER JOIN "manifest" AS "t2" ON ("t1"."manifest_id" = "t2"."id") WHERE (((("t1"."repository_id" = %s) AND ("t1"."name" = %s)) AND (("t1"."lifetime_end_ms" IS %s) OR ("t1"."lifetime_end_ms" > %s))) AND ("t1"."hidden" = %s)) LIMIT %s OFFSET %s', [68, 'fa1b457c', None, 1649155951491, False, 1, 0])
      gunicorn-registry stdout | 2022-04-05 10:52:31,494 [225] [DEBUG] [peewee] ('UPDATE "tag" SET "lifetime_end_ms" = %s WHERE (("tag"."id" = %s) AND ("tag"."lifetime_end_ms" IS %s))', [1649155951491, 29533, None])
      gunicorn-registry stdout | 2022-04-05 10:52:31,495 [225] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."tag_id", "t1"."repository_tag_id", "t2"."id", "t2"."name", "t2"."image_id", "t2"."repository_id", "t2"."lifetime_start_ts", "t2"."lifetime_end_ts", "t2"."hidden", "t2"."reversion" FROM "tagtorepositorytag" AS "t1" INNER JOIN "repositorytag" AS "t2" ON ("t1"."repository_tag_id" = "t2"."id") WHERE ("t1"."tag_id" = %s) LIMIT %s OFFSET %s', [29533, 1, 0])
      gunicorn-registry stdout | 2022-04-05 10:52:31,497 [225] [DEBUG] [peewee] ('INSERT INTO "tag" ("name", "repository_id", "manifest_id", "lifetime_start_ms", "lifetime_end_ms", "hidden", "reversion", "tag_kind_id") VALUES (%s, %s, %s, %s, %s, %s, %s, %s) RETURNING "tag"."id"', ['fa1b457c', 68, 1746, 1649155951491, None, False, False, 1]) 

            [PROJQUAY-3504] Decrease log level of quay processes

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory, and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHBA-2022:4986

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:4986

            Dongbo Yan added a comment -

            Verified with 'registry-proxy.engineering.redhat.com/rh-osbs/openshift-mirror-registry-rhel8@sha256:52a0a15de6e22fd640be6dead5413cbeb95821bcf54d9b07ca7de6dfbb836202',
                                          'registry-proxy.engineering.redhat.com/rh-osbs/openshift-mirror-registry-rhel8:v1.2.0-3'

            Do not show debug logs in quay pod

            Dongbo Yan added a comment - Verified with 'registry-proxy.engineering.redhat.com/rh-osbs/openshift-mirror-registry-rhel8@sha256:52a0a15de6e22fd640be6dead5413cbeb95821bcf54d9b07ca7de6dfbb836202',                               'registry-proxy.engineering.redhat.com/rh-osbs/openshift-mirror-registry-rhel8:v1.2.0-3' Do not show debug logs in quay pod

              jonathankingfc Jonathan King
              DanielMesser Daniel Messer
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: