Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-7239

Quay logging exception when caching specific `security_reports`

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Undefined Undefined
    • quay-v3.12.0
    • quay-v3.10.5
    • quay
    • False
    • None
    • False
    • Hide

      Have a heavily loaded container (many packages, java, Python, * components) scanned by Clair, resulting in a large security_report.

      Quay will try to cache the `security_report_sha256:<digest>` key with a way too long value, resulting in a exception which may end up in a additional error message if the logs are forwarded via syslog (as it will complain about a too long message too).

      Show
      Have a heavily loaded container (many packages, java, Python, * components) scanned by Clair, resulting in a large security_report. Quay will try to cache the `security_report_sha256:<digest>` key with a way too long value, resulting in a exception which may end up in a additional error message if the logs are forwarded via syslog (as it will complain about a too long message too).

      Quay version 3.10 (may be also latest one) is raising a exception if the value for the key `security_report_sha:xxxxxxx` exceeds a specific size. While this would only affect caching, the logs will fill up log-space quickly.

      In case the security report exceeds a given size, the following will raise an exception with the key and value included in the message:

      https://github.com/quay/quay/blob/8ecfd47f6a7ed1eb0321c789bc1730cb34ecf9b8/data/cache/impl.py#L234:

      class MemcachedModelCache(DataModelCache):
...
   def retrieve(self, cache_key, loader, should_cache=is_not_none):
   ...
       if client is not None and should_cache(result):
            try:
                logger.debug(
                    "Caching loaded result for key %s with expiration %s: %s",
                    cache_key.key,
                    result,
                    cache_key.expiration,
                )
                expires = (
                    convert_to_timedelta(cache_key.expiration) if cache_key.expiration else None
                )
                client.set(
                    cache_key.key,
                    result,
                    expire=int(expires.total_seconds()) if expires else None,
                )
                logger.debug(
                    "Cached loaded result for key %s with expiration %s: %s",
                    cache_key.key,
                    result,
                    cache_key.expiration,
                )
            except:
                logger.warning(
                    "Got exception when trying to set key %s to %s", cache_key.key, result   <<< This will print key + value which failed to get cached.
                )

      Unfortunately I'm not able to attach the full log-message due to it's size, but the following excerpt can be seen in the logs (as a single line):

      gunicorn-web stdout | 2024-05-21 07:12:07,977 [88643] [WARNING] [data.cache.impl] Got exception when trying to set key security_report_sha256:2bc89bda7a4d62c6940321a24dc890885ba9d252839efb5b36deae573ff2678d to {'manifest_hash': 'sha256:2bc89bda7a4d62c6940321a24dc890885ba9d252839efb5b36deae573ff2678d', 'packages': {'191611': {'id': '191611', 'name': 'org.unbescape:unbescape', 'version': '1.1.6.RELEASE', 'kind': 'binary', 'source': {'id': '1', 'name': '', 'version': ''}},.....
repository_ids': ['76', '77', '356', '29406']}, {'package_db': 'bdb:var/lib/rpm', 'introduced_in': 'sha256:69a6a2a5681367b89b94825aba4b49202b56567c59807908db84e61fad5c9283', 'distribution_id': '', '   <<<< The message ends like this!

      The size of the message may get "ellipsized" to drop non related text in between, if the message would exceed a given length.

            DanielMesser Daniel Messer
            rhn-support-ableisch Andreas Bleischwitz
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: