Loading...

Type: Story
Resolution: Done
Priority: Major
Fix Version/s: None
Affects Version/s: None
Component/s: CLI
Labels:
- good-for-first-issue
- ready-for-prioritization

Story Points:
3
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Epic Link:
OPCT - Backlog

Original story points:
3
Sprint:
OpenShift SPLAT - Sprint 268
Severity:
Moderate

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

User Story:
As an Engineer I want to enhance the OPCT retrieve cleaner to remove the unused objects `packages.operators.coreos.com_v1_packagemanifests.json` collected by namespace so the final file size can be reduced to 1/3 from the original file (sample 4.19: 170MiB->58MiB), saving CI storage (cloud bills), partner support storage (Partner Connect system), and faster processing time.

Description:

Currently Sonobuoy, backed by OPCT, collects all kubernetes objects, cluster and namespace's scope, when the jobs have been finished, putting all together in the final archive, resulting in a file with size around 170 MiB (4.19 sample). The compressor does not apply deduplication, and we don't want to implement it in the Sonobuoy.

Recently the opct created a hook to patch the archive file when downloading the archive from the cluster to the local disk (client-side) to prevent leaking security credentials eventually exposed to non-secret objects in the cluster (such as pull-secret) - this command can be accessed through 'opct adm cleaner --input <unclear file> --output <cleaned file>' to apply the cleaner in old archives.

This patch can be enhanced to clean undesired/duplicated files too. Here is the path of cleaner:

Acting when retrieving/downloading the results: https://github.com/redhat-openshift-ecosystem/opct/blob/main/pkg/retrieve/retrieve.go#L118-L122
Cleaner package/rules: https://github.com/redhat-openshift-ecosystem/opct/blob/main/internal/cleaner/cleaner.go
Adm command to access the cleaner: https://github.com/redhat-openshift-ecosystem/opct/blob/main/pkg/cmd/adm/cleaner.go

Acceptance Criteria:

Cleaner implemented in CLI, preventing the final archive to keep duplicated/unused files `packages.operators.coreos.com_v1_packagemanifests.json`
Unit tests implemented/updated (see the child cards)
Tests the results by comparing the cleaner in a rehearsal CI job
PR merged in the OPCT CLI

Other Information:
Raw files from a opct archive sample:

$ ls -sha opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/*/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/default/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-hammer-5vvdt/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-hammer-85pdj/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-hammer-nkr4h/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-hammer-vxsgx/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-image-layers-mwp6p-another/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-mallet-9hp6b/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-mallet-krk5x/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/e2e-test-mallet-w5nll/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/kube-node-lease/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/kube-public/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/kube-system/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/opct/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-apiserver-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-apiserver/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-authentication-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-authentication/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-catalogd/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cloud-controller-manager-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cloud-controller-manager/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cloud-credential-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cloud-network-config-controller/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cloud-platform-infra/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-csi-drivers/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-machine-approver/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-node-tuning-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-olm-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-samples-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-storage-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-cluster-version/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-config-managed/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-config-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-config/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-console-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-console/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-console-user-settings/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-controller-manager-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-controller-manager/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-dns-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-dns/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-etcd-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-etcd/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-host-network/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-image-registry/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-infra/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-ingress-canary/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-ingress-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-ingress/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-insights/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kni-infra/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-apiserver-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-apiserver/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-controller-manager-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-controller-manager/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-scheduler-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-scheduler/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-storage-version-migrator-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-kube-storage-version-migrator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-machine-api/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-machine-config-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-marketplace/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-monitoring/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-multus/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-network-console/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-network-diagnostics/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-network-node-identity/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-network-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-node/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-nutanix-infra/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-oauth-apiserver/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-openstack-infra/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-operator-controller/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-operator-lifecycle-manager/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-operators/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-ovirt-infra/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-ovn-kubernetes/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-route-controller-manager/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-service-ca-operator/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-service-ca/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-user-workload-monitoring/packages.operators.coreos.com_v1_packagemanifests.json
9.8M opct_202503062321_eecc44a5-c75d-406b-9588-d21ac4a442a3_raw/resources/ns/openshift-vsphere-infra/packages.operators.coreos.com_v1_packagemanifests.json

issue created by splat-bot

duplicates

OPCT-54 [opct] decrease the archive result file

Closed

is duplicated by

OPCT-54 [opct] decrease the archive result file

Closed

links to

https://github.com/redhat-openshift-ecosystem/opct/pull/161

There are no Sub-Tasks for this issue.

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Sub-Tasks

Activity

People

Dates

PagerDuty

Hide