-
Story
-
Resolution: Done
-
Major
-
None
-
None
User Story:
As an Engineer I want to enhance the OPCT retrieve cleaner to remove the unused objects `packages.operators.coreos.com_v1_packagemanifests.json` collected by namespace so the final file size can be reduced to 1/3 from the original file (sample 4.19: 170MiB->58MiB), saving CI storage (cloud bills), partner support storage (Partner Connect system), and faster processing time.
Description:
Currently Sonobuoy, backed by OPCT, collects all kubernetes objects, cluster and namespace's scope, when the jobs have been finished, putting all together in the final archive, resulting in a file with size around 170 MiB (4.19 sample). The compressor does not apply deduplication, and we don't want to implement it in the Sonobuoy.
Recently the opct created a hook to patch the archive file when downloading the archive from the cluster to the local disk (client-side) to prevent leaking security credentials eventually exposed to non-secret objects in the cluster (such as pull-secret) - this command can be accessed through 'opct adm cleaner --input <unclear file> --output <cleaned file>' to apply the cleaner in old archives.
This patch can be enhanced to clean undesired/duplicated files too. Here is the path of cleaner:
- Acting when retrieving/downloading the results: https://github.com/redhat-openshift-ecosystem/opct/blob/main/pkg/retrieve/retrieve.go#L118-L122
- Cleaner package/rules: https://github.com/redhat-openshift-ecosystem/opct/blob/main/internal/cleaner/cleaner.go
- Adm command to access the cleaner: https://github.com/redhat-openshift-ecosystem/opct/blob/main/pkg/cmd/adm/cleaner.go
Acceptance Criteria:
- Cleaner implemented in CLI, preventing the final archive to keep duplicated/unused files `packages.operators.coreos.com_v1_packagemanifests.json`
- Unit tests implemented/updated (see the child cards)
- Tests the results by comparing the cleaner in a rehearsal CI job
- PR merged in the OPCT CLI
Other Information:
Raw files from a opct archive sample:
$ ls -sha opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/*/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/default/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-hammer-5vvdt/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-hammer-85pdj/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-hammer-nkr4h/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-hammer-vxsgx/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-image-layers-mwp6p-another/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-mallet-9hp6b/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-mallet-krk5x/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-mallet-w5nll/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/kube-node-lease/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/kube-public/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/kube-system/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/opct/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-apiserver-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-apiserver/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-authentication-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-authentication/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-catalogd/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cloud-controller-manager-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cloud-controller-manager/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cloud-credential-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cloud-network-config-controller/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cloud-platform-infra/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-csi-drivers/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-machine-approver/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-node-tuning-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-olm-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-samples-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-storage-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-version/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-config-managed/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-config-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-config/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-console-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-console/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-console-user-settings/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-controller-manager-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-controller-manager/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-dns-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-dns/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-etcd-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-etcd/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-host-network/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-image-registry/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-infra/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-ingress-canary/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-ingress-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-ingress/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-insights/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kni-infra/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-apiserver-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-apiserver/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-controller-manager-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-controller-manager/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-scheduler-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-scheduler/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-storage-version-migrator-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-storage-version-migrator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-machine-api/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-machine-config-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-marketplace/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-monitoring/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-multus/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-network-console/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-network-diagnostics/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-network-node-identity/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-network-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-node/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-nutanix-infra/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-oauth-apiserver/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-openstack-infra/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-operator-controller/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-operator-lifecycle-manager/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-operators/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-ovirt-infra/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-ovn-kubernetes/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-route-controller-manager/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-service-ca-operator/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-service-ca/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-user-workload-monitoring/packages.operators.coreos.com_v1_packagemanifests.json 9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-vsphere-infra/packages.operators.coreos.com_v1_packagemanifests.json
issue created by splat-bot
