Loading...

Details

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: clair-4.5.0, quay-v3.7.11, quay-v3.8.0, quay-v3.10.0, quay-v3.11.0
Affects Version/s: None
Component/s: clair, quay
Labels:
- clair
- needs-info
- triaged

Blocked:
False
Blocked Reason:
Client FS is getting Full
Ready:
False
Product:
Quay Enterprise
Git Pull Request:
https://github.com/quay/claircore/pull/677

RICE Score:
0

SFDC Cases Links:
SFDC Cases Counter:

Description

A client is reporting that Clair's database is growing uncontrollably. They are already hitting 75 GB for their database size. This is a print of their tables:

clairv4=> SELECT nspname || '.' || relname AS "relation",
    pg_size_pretty(pg_relation_size(C.oid)) AS "size"
  FROM pg_class C
  LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
  WHERE nspname NOT IN ('pg_catalog', 'information_schema')
  ORDER BY pg_relation_size(C.oid) DESC
  LIMIT 20;
                             relation                             |  size   
------------------------------------------------------------------+---------
 public.manifest_index_unique                                     | 9071 MB
 public.manifest_index                                            | 6827 MB
 public.manifest_index_manifest_id_package_id_dist_id_repo_id_idx | 6819 MB
 public.vuln                                                      | 6250 MB
 public.uo_vuln                                                   | 3512 MB
 public.uo_vuln_pkey                                              | 2690 MB
 public.manifest_index_pkey                                       | 2340 MB
 pg_toast.pg_toast_826417                                         | 1129 MB
 public.package_scanartifact_pkey                                 | 1112 MB
 public.uo_vuln_vuln_idx                                          | 913 MB
 pg_toast.pg_toast_826828                                         | 883 MB
 public.package_scanartifact                                      | 807 MB
 public.uo_vuln_uo_idx                                            | 525 MB
 public.vuln_hash_kind_hash_key                                   | 316 MB
 public.vuln_lookup_idx                                           | 300 MB
 public.vuln_pkey                                                 | 176 MB
 public.scanned_layer                                             | 82 MB
 public.vuln_updater_idx                                          | 64 MB
 public.scanned_layer_pkey                                        | 63 MB
 pg_toast.pg_toast_826417_index                                   | 51 MB
(20 rows)

clairv4=> select count(*) from manifest;
 count 
-------
 46020
(1 row)

First three entries are already over 20 GB in size. I found this commit which seems to indicate we already had an issue with the manifest_index table growing uncontrollably:

https://github.com/quay/claircore/commit/d7cea80e4b1278c1a095232a4bc9d96c4ebf8a9e

Can you please check and advise what would be the best course of action? Rescanning of all 46 thousand manifests is not the best option.