Uploaded image for project: 'OPCT - OpenShift Provider Compatibility Tool'
  1. OPCT - OpenShift Provider Compatibility Tool
  2. OPCT-339

[OPCT] enhance retrieve cleaner to remove unused `packages.operators.coreos.com_v1_packagemanifests.json` collected by namespace

XMLWordPrintable

    • 3
    • OpenShift SPLAT - Sprint 268
    • Moderate

      User Story:
      As an Engineer I want to enhance the OPCT retrieve cleaner to remove the unused objects `packages.operators.coreos.com_v1_packagemanifests.json` collected by namespace so the final file size can be reduced to 1/3 from the original file (sample 4.19: 170MiB->58MiB), saving CI storage (cloud bills), partner support storage (Partner Connect system), and faster processing time.

      Description:

      Currently Sonobuoy, backed by OPCT, collects all kubernetes objects, cluster and namespace's scope, when the jobs have been finished, putting all together in the final archive, resulting in a file with size around 170 MiB (4.19 sample). The compressor does not apply deduplication, and we don't want to implement it in the Sonobuoy.

      Recently the opct created a hook to patch the archive file when downloading the archive from the cluster to the local disk (client-side) to prevent leaking security credentials eventually exposed to non-secret objects in the cluster (such as pull-secret) - this command can be accessed through 'opct adm cleaner --input <unclear file> --output <cleaned file>' to apply the cleaner in old archives.

      This patch can be enhanced to clean undesired/duplicated files too. Here is the path of cleaner:

       

      Acceptance Criteria:

      • Cleaner implemented in CLI, preventing the final archive to keep duplicated/unused files `packages.operators.coreos.com_v1_packagemanifests.json`
      • Unit tests implemented/updated (see the child cards)
      • Tests the results by comparing the cleaner in a rehearsal CI job
      • PR merged in the OPCT CLI

       

      Other Information:
      Raw files from a opct archive sample:

      $ ls -sha opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/*/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/default/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-hammer-5vvdt/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-hammer-85pdj/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-hammer-nkr4h/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-hammer-vxsgx/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-image-layers-mwp6p-another/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-mallet-9hp6b/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-mallet-krk5x/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-mallet-w5nll/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/kube-node-lease/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/kube-public/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/kube-system/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/opct/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-apiserver-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-apiserver/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-authentication-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-authentication/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-catalogd/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cloud-controller-manager-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cloud-controller-manager/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cloud-credential-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cloud-network-config-controller/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cloud-platform-infra/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-csi-drivers/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-machine-approver/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-node-tuning-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-olm-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-samples-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-storage-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-version/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-config-managed/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-config-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-config/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-console-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-console/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-console-user-settings/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-controller-manager-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-controller-manager/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-dns-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-dns/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-etcd-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-etcd/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-host-network/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-image-registry/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-infra/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-ingress-canary/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-ingress-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-ingress/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-insights/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kni-infra/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-apiserver-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-apiserver/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-controller-manager-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-controller-manager/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-scheduler-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-scheduler/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-storage-version-migrator-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-storage-version-migrator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-machine-api/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-machine-config-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-marketplace/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-monitoring/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-multus/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-network-console/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-network-diagnostics/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-network-node-identity/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-network-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-node/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-nutanix-infra/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-oauth-apiserver/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-openstack-infra/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-operator-controller/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-operator-lifecycle-manager/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-operators/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-ovirt-infra/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-ovn-kubernetes/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-route-controller-manager/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-service-ca-operator/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-service-ca/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-user-workload-monitoring/packages.operators.coreos.com_v1_packagemanifests.json
      9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-vsphere-infra/packages.operators.coreos.com_v1_packagemanifests.json
       

      issue created by splat-bot

              rhn-support-mrbraga Marco Braga
              rhn-support-mrbraga Marco Braga
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: