Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-2743

Clair indexing crashes on scanning OpenJDK image

XMLWordPrintable

    • False
    • False
    • Quay Enterprise

      Image in question is library/openjdk:latest on Docker Hub. When copied to Quay 3.6.0, the scanning appears to crash:

      {"level":"info","component":"internal/indexer/layerscannner/layerScanner.scan","kind":"package","layer":"sha256:67a9c63ed3ba0226d765a092dd6fcc06fe5ece8a2dd0fd11e1add9c0ddee488e","manifest":"sha256:ba0fc847b43cb8756be4497758863d7b51003efcb9b861211a8d5d3afe6c9fa2","scanner":"rpm","state":"ScanLayers","error":"context canceled","time":"2021-10-28T11:06:27Z"}
      {"level":"debug","component":"internal/indexer/layerscannner/layerScanner.scan","kind":"package","layer":"sha256:67a9c63ed3ba0226d765a092dd6fcc06fe5ece8a2dd0fd11e1add9c0ddee488e","manifest":"sha256:ba0fc847b43cb8756be4497758863d7b51003efcb9b861211a8d5d3afe6c9fa2","scanner":"rpm","state":"ScanLayers","time":"2021-10-28T11:06:27Z","message":"scan done"}
      {"level":"info","component":"internal/indexer/controller/Controller.Index","manifest":"sha256:ba0fc847b43cb8756be4497758863d7b51003efcb9b861211a8d5d3afe6c9fa2","state":"ScanLayers","time":"2021-10-28T11:06:27Z","message":"layers scan done"}
      {"level":"error","component":"internal/indexer/controller/Controller.Index","manifest":"sha256:ba0fc847b43cb8756be4497758863d7b51003efcb9b861211a8d5d3afe6c9fa2","state":"ScanLayers","error":"failed to scan all layer contents: jar: unidentified jar: jrt-fs.jar","time":"2021-10-28T11:06:27Z","message":"error during scan"}
      {"level":"info","component":"libindex/Libindex.Index","manifest":"sha256:ba0fc847b43cb8756be4497758863d7b51003efcb9b861211a8d5d3afe6c9fa2","time":"2021-10-28T11:06:27Z","message":"index request done"}
      {"level":"info","component":"httptransport/New","remote_addr":"172.17.0.1:49172","method":"POST","request_uri":"/indexer/api/v1/index_report","status":500,"time":"2021-10-28T11:06:27Z","message":"handled HTTP request"}
      

      In certain cases the following is also reported:

      10:16AM INF  error="signal: killed" component=internal/indexer/layerscannner/layerScanner.scan kind=package layer=sha256:f420596490555ff89c8ec7d6b74b4e46c2b3b98dc775c9d19494d8afeb475852 manifest=sha256:ba0fc847b43cb8756be4497758863d7b51003efcb9b861211a8d5d3afe6c9fa2 scanner=rpm state=ScanLayers
      

      The outcome is the following Quay trace:

      securityworker stdout | 2021-10-28 10:22:56,356 [95] [DEBUG] [urllib3.connectionpool] http://172.24.10.50:8080 "POST /indexer/api/v1/index_report HTTP/1.1" 500 125
      securityworker stdout | 2021-10-28 10:22:56,356 [95] [ERROR] [util.secscan.v4.api] Security scanner endpoint responded with non-200 HTTP status code: 500
      securityworker stdout | NoneType: None
      securityworker stdout | 2021-10-28 10:22:56,356 [95] [ERROR] [data.secscan_model.secscan_v4_model] Failed to perform indexing, security scanner API error
      securityworker stdout | Traceback (most recent call last):
      securityworker stdout |   File "/quay-registry/util/secscan/v4/api.py", line 204, in index
      securityworker stdout |     resp = self._perform(actions["Index"](body))
      securityworker stdout |   File "/quay-registry/util/secscan/v4/api.py", line 296, in _perform
      securityworker stdout |     raise Non200ResponseException(resp)
      securityworker stdout | util.secscan.v4.api.Non200ResponseException
      securityworker stdout | During handling of the above exception, another exception occurred:
      securityworker stdout | Traceback (most recent call last):
      securityworker stdout |   File "/quay-registry/data/secscan_model/secscan_v4_model.py", line 288, in perform_indexing
      securityworker stdout |     (report, state) = self._secscan_api.index(manifest, layers)
      securityworker stdout |   File "/quay-registry/util/secscan/v4/api.py", line 208, in index
      securityworker stdout |     raise APIRequestFailure(ex)
      securityworker stdout | util.secscan.v4.api.APIRequestFailure
      

      The image ends up in the queued state and is always queued and sent again for rescanning to Clair. When full openjdk image is copied over (the Windows variants as well, which are 2.7 GB and 6 GB in size), all subsequent images pushed to Quay end up in queued status for a long time, until the indexer fails with a 500. At that point, they start to get queued, until the openjdk image is hit again which, again, slows down scanning.

      Can you please take a look at this? Thanks!

        1. clair
          33 kB
          Sayali Bhavsar
        2. Error_1.PNG
          145 kB
          Sayali Bhavsar
        3. Error_2.PNG
          113 kB
          Sayali Bhavsar
        4. quay_1.txt
          2.29 MB
          Sayali Bhavsar
        5. quay_3.txt
          16.63 MB
          Sayali Bhavsar

              hdonnay Henry Donnay
              rhn-support-ibazulic Ivan Bazulic
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: