Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-3100

Clair-db grew until running out of diskspace due to "notification_body" table filling up

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • clair-4.4.0
    • clair-4.3.5
    • clair
    • 0

    Description

      Running Quay and Clair in a single vm, using podman

      QUAY_IMAGE=registry.redhat.io/quay/quay-rhel8:v3.6.2
      QUAY_CLAIRV4_IMAGE=registry.redhat.io/quay/clair-rhel8:v3.6.2
      QUAY_DB_IMAGE=registry.redhat.io/rhel8/postgresql-10:1

      Quay + Clair have been updated since 3.5.0

      After some time Clair stopped working due to missing free disk-space: (I just extended with 10G to be able to restart postgresql)

      # df -h |grep db
      /dev/mapper/vg_quay_db-lv_clair_db        160G  149G   12G  94% /opt/clair/db
      /dev/mapper/vg_quay_db-lv_quay_db          10G  1.1G  9.0G  11% /opt/quay/db
       
      clairdb=# \d+
                                        List of relations
       Schema |           Name            |   Type   |   Owner   |    Size    | Description 
      --------+---------------------------+----------+-----------+------------+-------------
       public | dist                      | table    | clairuser | 16 kB      | 
       public | dist_id_seq               | sequence | clairuser | 8192 bytes | 
       public | dist_scanartifact         | table    | clairuser | 128 kB     | 
       public | enrichment                | table    | clairuser | 46 MB      | 
       public | enrichment_id_seq         | sequence | clairuser | 8192 bytes | 
       public | indexreport               | table    | clairuser | 216 MB     | 
       public | key                       | table    | clairuser | 16 kB      | 
       public | latest_vuln               | view     | clairuser | 0 bytes    | 
       public | layer                     | table    | clairuser | 4120 kB    | 
       public | layer_id_seq              | sequence | clairuser | 8192 bytes | 
       public | libindex_migrations       | table    | clairuser | 8192 bytes | 
       public | libvuln_migrations        | table    | clairuser | 8192 bytes | 
       public | manifest                  | table    | clairuser | 3696 kB    | 
       public | manifest_id_seq           | sequence | clairuser | 8192 bytes | 
       public | manifest_index            | table    | clairuser | 1323 MB    | 
       public | manifest_index_id_seq     | sequence | clairuser | 8192 bytes | 
       public | manifest_layer            | table    | clairuser | 5152 kB    | 
       public | notification              | table    | clairuser | 504 kB     | 
       public | notification_body         | table    | clairuser | 112 GB     | 
       public | notifier_migrations       | table    | clairuser | 8192 bytes | 
       public | notifier_update_operation | table    | clairuser | 1072 kB    | 
       public | package                   | table    | clairuser | 5160 kB    | 
       public | package_id_seq            | sequence | clairuser | 8192 bytes | 
       public | package_scanartifact      | table    | clairuser | 735 MB     | 
       public | receipt                   | table    | clairuser | 808 kB     | 
       public | repo                      | table    | clairuser | 56 kB      | 
       public | repo_id_seq               | sequence | clairuser | 8192 bytes | 
       public | repo_scanartifact         | table    | clairuser | 5872 kB    | 
       public | scanned_layer             | table    | clairuser | 20 MB      | 
       public | scanned_manifest          | table    | clairuser | 17 MB      | 
       public | scanner                   | table    | clairuser | 16 kB      | 
       public | scanner_id_seq            | sequence | clairuser | 8192 bytes | 
       public | scannerlist               | table    | clairuser | 8192 bytes | 
       public | scannerlist_id_seq        | sequence | clairuser | 8192 bytes | 
       public | uo_enrich                 | table    | clairuser | 19 MB      | 
       public | uo_vuln                   | table    | clairuser | 533 MB     | 
       public | update_operation          | table    | clairuser | 776 kB     | 
       public | update_operation_id_seq   | sequence | clairuser | 8192 bytes | 
       public | vuln                      | table    | clairuser | 6259 MB    | 
       public | vuln_id_seq               | sequence | clairuser | 8192 bytes | 
      (40 rows)
       

      As my clair instance was also stopped, I don't have logs from it.

      I'll try to start it and collect them - 10G should be filled up quite quickly.

      Attachments

        1. clair-config.yaml
          1 kB
        2. quay-config.yaml
          4 kB
        3. quay-trunc.log
          39.82 MB
        4. Screenshot from 2022-01-28 17-03-44.png
          Screenshot from 2022-01-28 17-03-44.png
          38 kB

        Activity

          People

            hdonnay Henry Donnay
            rhn-support-ableisch Andreas Bleischwitz
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: