[OCPBUGS-32547] Hit error when generating pruning plan for delete phase using --generate command with oc-mirror v2

Type: Bug
Resolution: Done-Errata
Priority: Critical
Fix Version/s: 4.16
Affects Version/s: 4.16
Component/s: oc-mirror
Labels:

Severity:
Important
Regression:
None
Sprint:
CLID Sprint 253
sprint_count:
1
Release Blocker:
Rejected
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Release Note Text:
N/A
Release Note Type:
Release Note Not Required
Release Note Status:
In Progress
Customer Impact:

Customer Facing
Target Version:

4.16.0
Target Backport Versions:

4.16

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

PX Impact Score:
PX Priority Data:
PX Review Complete:

Description of problem:

Hit error when create pruning plan generation for delete phase using –generate for mirror2mirror

Version-Release number of selected component (if applicable):

oc-mirror  version WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version.Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.16.0-202404191609.p0.g9ac063b.assembly.stream.el9-9ac063b", GitCommit:"9ac063b0b88466183a50287af277c5ed40a8e238", GitTreeState:"clean", BuildDate:"2024-04-19T22:03:51Z", GoVersion:"go1.21.7 (Red Hat 1.21.7-1.el9) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}

How reproducible:

always

Steps to Reproduce:

1) Use following isc to do mirror2mirror for v2:
cat config.yaml
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
mirror:
  platform:
    channels:
      - name: stable-4.15
  additionalImages: 
    - name: registry.redhat.io/ubi8/ubi:latest
    - name: registry.redhat.io/ubi8/ubi-minimal:latest
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
      packages:
      - name: 3scale-operator
    - catalog: oci:///app1/ibm-catalog
      targetTag: "v14"
      targetCatalog: "zhouy/catalog"
    - catalog: oci:///app1/noo/redhat-operator-index
      packages:
        - name: cluster-kube-descheduler-operator
        - name: advanced-cluster-management

`oc-mirror --config config.yaml --v2 docker://xxx.com:5000/m2m  --workspace file:///app1/0416/clid20/`

2) generate pruning plan for delete phase using --generate
cat config-delete.yaml 
apiVersion: mirror.openshift.io/v1alpha2
kind: DeleteImageSetConfiguration
delete:
  platform:
    channels:
      - name: stable-4.15
  additionalImages: 
    - name: registry.redhat.io/ubi8/ubi:latest
    - name: registry.redhat.io/ubi8/ubi-minimal:latest
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
      packages:
      - name: 3scale-operator
    - catalog: oci:///app1/ibm-catalog
      targetTag: "v14"
      targetCatalog: "zhouy/catalog"
    - catalog: oci:///app1/noo/redhat-operator-index
      packages:
        - name: cluster-kube-descheduler-operator
        - name: advanced-cluster-management


`oc-mirror delete --config config-delete.yaml --workspace file://clid20 --v2 --generate  docker://xxx.com:5000/m2m`

Actual results:

2) Many errors for generate command: 

2024/04/22 10:02:29  [ERROR]  : reading manifest d8e94620237da97e1b65dac4fb616d21d13e2fea08c9385145a02ad3fbd59d88 in localhost:55000/3scale-amp2/3scale-rhel7-operator-metadata: manifest unknown image : map[]
2024/04/22 10:02:29  [ERROR]  : [delete-images] reading manifest d8e94620237da97e1b65dac4fb616d21d13e2fea08c9385145a02ad3fbd59d88 in localhost:55000/3scale-amp2/3scale-rhel7-operator-metadata: manifest unknown
2024/04/22 10:02:29  [ERROR]  : reading manifest 4f72f049436af1a940833c61b075d84ad5910c7bb2df2a8de99618995c067dfe in localhost:55000/3scale-amp2/3scale-rhel7-operator: manifest unknown image : map[]
2024/04/22 10:02:29  [ERROR]  : [delete-images] reading manifest 4f72f049436af1a940833c61b075d84ad5910c7bb2df2a8de99618995c067dfe in localhost:55000/3scale-amp2/3scale-rhel7-operator: manifest unknown
2024/04/22 10:02:29  [ERROR]  : reading manifest 4f72f049436af1a940833c61b075d84ad5910c7bb2df2a8de99618995c067dfe in localhost:55000/3scale-amp2/3scale-rhel7-operator: manifest unknown image : map[]
2024/04/22 10:02:29  [ERROR]  : [delete-images] reading manifest 4f72f049436af1a940833c61b075d84ad5910c7bb2df2a8de99618995c067dfe in localhost:55000/3scale-amp2/3scale-rhel7-operator: manifest unknown
2024/04/22 10:02:29  [ERROR]  : reading manifest 68b310ed3cfd65db893ba015ef1d5442365201c0ced006c1915e90edb99933ea in localhost:55000/3scale-amp2/apicast-gateway-rhel8: manifest unknown image : map[]
2024/04/22 10:02:29  [ERROR]  : [delete-images] reading manifest 68b310ed3cfd65db893ba015ef1d5442365201c0ced006c1915e90edb99933ea in localhost:55000/3scale-amp2/apicast-gateway-rhel8: manifest unknown
2024/04/22 10:02:29  [ERROR]  : reading manifest 2b8525c55cfbd5b5d66b50868ebd8fe6f468b10715653d047cdae25fa28e5983 in localhost:55000/3scale-amp2/backend-rhel8: manifest unknown image : map[]
2024/04/22 10:02:29  [ERROR]  : [delete-images] reading manifest 2b8525c55cfbd5b5d66b50868ebd8fe6f468b10715653d047cdae25fa28e5983 in localhost:55000/3scale-amp2/backend-rhel8: manifest unknown
2024/04/22 10:02:29  [ERROR]  : reading manifest 97e6355fcfadf7fea1f30cc8bbf833c8a518655cef4d63051df29fbcde2c1f00 in localhost:55000/3scale-amp2/memcached-rhel7: manifest unknown image : map[]

Expected results:

2) no error

is cloned by

OCPBUGS-32700 Image delete function for v2 does not delete any images in the registry

Closed

links to

openshift/oc-mirror#848: OCPBUGS-32547: Fix delete generation for mirror to mirror flow

RHEA-2024:0041 OpenShift Container Platform 4.16.z bug fix update

Errata Tool added a comment - 2024/06/27 11:46 AM

Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

For information on the advisory (Critical: OpenShift Container Platform 4.16.0 bug fix and security update), and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2024:0041

Errata Tool added a comment - 2024/06/27 11:46 AM Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Critical: OpenShift Container Platform 4.16.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:0041

Ying Zhou added a comment - 2024/05/10 10:22 AM

with the latest fixed pr, can't see the error logs now :

oc-mirror version 
WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"", Minor:"", GitVersion:"v0.0.0-unknown-6c6308b5", GitCommit:"6c6308b5", GitTreeState:"clean", BuildDate:"2024-05-10T08:38:51Z", GoVersion:"go1.21.3", Compiler:"gc", Platform:"linux/amd64"}


after run the mirror2mirror , we can see the mode file : 
cat info/mode.txt 
mirrorToMirror

oc-mirror delete --config delete-s.yaml  --workspace file://0510 --v2 --generate  docker://localhost:5000/m2m  --dest-tls-verify=false2024/05/10 10:13:40  [WARN]   : ⚠️  --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready.
2024/05/10 10:13:40  [INFO]   : 👋 Hello, welcome to oc-mirror
2024/05/10 10:13:40  [INFO]   : ⚙️  setting up the environment for you...
2024/05/10 10:13:40  [INFO]   : 🔀 workflow mode: mirrorToMirror / delete
2024/05/10 10:13:40  [INFO]   : 🕵️  going to discover the necessary images...
2024/05/10 10:13:40  [INFO]   : 🔍 collecting release images...
2024/05/10 10:13:40  [INFO]   : 🔍 collecting operator images...
2024/05/10 10:13:52  [INFO]   : 🔍 collecting additional images...
2024/05/10 10:13:52  [INFO]   : 📄 Generating delete file...
2024/05/10 10:13:52  [INFO]   : 0510/working-dir/delete file created
2024/05/10 10:15:34  [INFO]   : delete time     : 1m53.839306595s
2024/05/10 10:15:34  [INFO]   : 👋 Goodbye, thank you for using oc-mirror



oc-mirror delete --config config-32547.yaml --delete-yaml-file   0510/working-dir/delete/delete-images.yaml  --v2 docker://localhost:5000/m2m --dest-tls-verify=false --force-cache-delete2024/05/10 10:21:34  [WARN]   : ⚠️  --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready.
2024/05/10 10:21:34  [INFO]   : 👋 Hello, welcome to oc-mirror
2024/05/10 10:21:34  [INFO]   : ⚙️  setting up the environment for you...
2024/05/10 10:21:34  [INFO]   : 🔀 workflow mode: mirrorToMirror / delete
2024/05/10 10:21:34  [INFO]   : 👀 Reading delete file...
2024/05/10 10:21:34  [INFO]   : 🚀 Start deleting the images...
2024/05/10 10:21:34  [INFO]   : === Overall Progress - deleting image 1 / 262 ===
2024/05/10 10:21:34  [INFO]   : === Overall Progress - deleting image 2 / 262 ===
2024/05/10 10:21:34  [INFO]   : === Overall Progress - deleting image 3 / 262 ===
2024/05/10 10:21:34  [INFO]   : === Overall Progress - deleting image 4 / 262 ===
2024/05/10 10:21:34  [INFO]   : === Overall Progress - deleting image 5 / 262 ===
....
2024/05/10 10:21:35  [INFO]   : All images deleted successfully 262 / 262 ✅
2024/05/10 10:21:35  [INFO]   : delete time     : 1.821563778s
2024/05/10 10:21:35  [INFO]   : 📝 Rememeber to execute a garbage collect (or similar) on your remote repository
2024/05/10 10:21:35  [INFO]   : 👋 Goodbye, thank you for using oc-mirror

Ying Zhou added a comment - 2024/05/10 10:22 AM with the latest fixed pr, can't see the error logs now : oc-mirror version WARNING: This version information is deprecated and will be replaced with the output from -- short . Use --output=yaml|json to get the full version. Client Version: version.Info{Major: "", Minor:" ", GitVersion:" v0.0.0-unknown-6c6308b5 ", GitCommit:" 6c6308b5 ", GitTreeState:" clean ", BuildDate:" 2024-05-10T08:38:51Z ", GoVersion:" go1.21.3 ", Compiler :" gc ", Platform:" linux/amd64"} after run the mirror2mirror , we can see the mode file : cat info/mode.txt mirrorToMirror oc-mirror delete --config delete-s.yaml --workspace file: //0510 --v2 --generate docker://localhost:5000/m2m --dest-tls-verify=false2024/05/10 10:13:40 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/05/10 10:13:40 [INFO] : 👋 Hello, welcome to oc-mirror 2024/05/10 10:13:40 [INFO] : ⚙️ setting up the environment for you... 2024/05/10 10:13:40 [INFO] : 🔀 workflow mode: mirrorToMirror / delete 2024/05/10 10:13:40 [INFO] : 🕵️ going to discover the necessary images... 2024/05/10 10:13:40 [INFO] : 🔍 collecting release images... 2024/05/10 10:13:40 [INFO] : 🔍 collecting operator images... 2024/05/10 10:13:52 [INFO] : 🔍 collecting additional images... 2024/05/10 10:13:52 [INFO] : 📄 Generating delete file... 2024/05/10 10:13:52 [INFO] : 0510/working-dir/delete file created 2024/05/10 10:15:34 [INFO] : delete time : 1m53.839306595s 2024/05/10 10:15:34 [INFO] : 👋 Goodbye, thank you for using oc-mirror oc-mirror delete --config config-32547.yaml --delete-yaml-file 0510/working-dir/delete/delete-images.yaml --v2 docker: //localhost:5000/m2m --dest-tls-verify= false --force-cache-delete2024/05/10 10:21:34 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/05/10 10:21:34 [INFO] : 👋 Hello, welcome to oc-mirror 2024/05/10 10:21:34 [INFO] : ⚙️ setting up the environment for you... 2024/05/10 10:21:34 [INFO] : 🔀 workflow mode: mirrorToMirror / delete 2024/05/10 10:21:34 [INFO] : 👀 Reading delete file... 2024/05/10 10:21:34 [INFO] : 🚀 Start deleting the images... 2024/05/10 10:21:34 [INFO] : === Overall Progress - deleting image 1 / 262 === 2024/05/10 10:21:34 [INFO] : === Overall Progress - deleting image 2 / 262 === 2024/05/10 10:21:34 [INFO] : === Overall Progress - deleting image 3 / 262 === 2024/05/10 10:21:34 [INFO] : === Overall Progress - deleting image 4 / 262 === 2024/05/10 10:21:34 [INFO] : === Overall Progress - deleting image 5 / 262 === .... 2024/05/10 10:21:35 [INFO] : All images deleted successfully 262 / 262 ✅ 2024/05/10 10:21:35 [INFO] : delete time : 1.821563778s 2024/05/10 10:21:35 [INFO] : 📝 Rememeber to execute a garbage collect (or similar) on your remote repository 2024/05/10 10:21:35 [INFO] : 👋 Goodbye, thank you for using oc-mirror

Ying Zhou added a comment - 2024/05/09 6:23 AM - edited

with the fixed pr : https://github.com/openshift/oc-mirror/pull/848 , I still could reproduce the issue :

git log
commit a09d76ce1cb660125b5bbc80456ef9fabc18e42e (HEAD -> OCPBUGS-32547, origin/OCPBUGS-32547)
Author: Luigi Mario Zuccarelli <luzuccar@redhat.com>
Date:   Wed May 8 14:27:08 2024 +0200    OCPBUGS-32547: Fix delete generation for mirror to mirror flow


oc-mirror version 
WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"", Minor:"", GitVersion:"v0.0.0-unknown-a09d76ce", GitCommit:"a09d76ce", GitTreeState:"clean", BuildDate:"2024-05-09T04:47:55Z", GoVersion:"go1.21.3", Compiler:"gc", Platform:"linux/amd64"}



oc-mirror delete --config  delete-s.yaml --workspace file://clid20 --v2 --generate  docker://localhost:5000 --dest-tls-verify=false2024/05/09 06:20:44  [WARN]   : ⚠️  --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready.
2024/05/09 06:20:44  [INFO]   : 👋 Hello, welcome to oc-mirror
2024/05/09 06:20:44  [INFO]   : ⚙️  setting up the environment for you...
2024/05/09 06:20:44  [INFO]   : 🔀 workflow mode: diskToMirror / delete
2024/05/09 06:20:44  [ERROR]  : [RunDelete] open clid20/working-dir/hold-release/ocp-release: no such file or directory
2024/05/09 06:20:44  [INFO]   : 🕵️  going to discover the necessary images...
2024/05/09 06:20:44  [INFO]   : 🔍 collecting release images...
2024/05/09 06:20:44  [INFO]   : 🔍 collecting operator images...
2024/05/09 06:20:56  [INFO]   : 🔍 collecting additional images...
2024/05/09 06:20:56  [INFO]   : 📄 Generating delete file...
2024/05/09 06:20:56  [INFO]   : clid20/working-dir/delete file created
2024/05/09 06:20:56  [ERROR]  : reading manifest v14 in localhost:55000/zhouy/catalog: manifest unknown image : map[]
2024/05/09 06:20:56  [ERROR]  : [delete-images] reading manifest v14 in localhost:55000/zhouy/catalog: manifest unknown
2024/05/09 06:20:56  [ERROR]  : reading manifest ce3330c1c90514f8ca58d22415c78dfa1039b42753d8f2dccc653416afb4d202 in localhost:55000/kube-descheduler-operator/descheduler-rhel9: manifest unknown image : map[]
2024/05/09 06:20:56  [ERROR]  : [delete-images] reading manifest ce3330c1c90514f8ca58d22415c78dfa1039b42753d8f2dccc653416afb4d202 in localhost:55000/kube-descheduler-operator/descheduler-rhel9: manifest unknown
2024/05/09 06:20:56  [ERROR]  : reading manifest ebcc836c3052bebfe7747c18b57ccc3a2870b5907e5ead894d87c8b2e33a12d9 in localhost:55000/kube-descheduler-operator/kube-descheduler-operator-bundle: manifest unknown image : map[]
2024/05/09 06:20:56  [ERROR]  : [delete-images] reading manifest ebcc836c3052bebfe7747c18b57ccc3a2870b5907e5ead894d87c8b2e33a12d9 in localhost:55000/kube-descheduler-operator/kube-descheduler-operator-bundle: manifest unknown
2024/05/09 06:20:56  [ERROR]  : reading manifest ea40bc7d39b095dba0649d2052561a75972e3da8fd1b8a44a2cd8081d9326b92 in localhost:55000/kube-descheduler-operator/kube-descheduler-rhel9-operator: manifest unknown image : map[]

....
2024/05/09 06:20:56  [ERROR]  : [delete-images] reading manifest b227736a7d44dd1c332edba9e575608cf38fa76df4dd3ad2bf7dc1ee6d66b06c in localhost:55000/cpopen/ibm-zcon-zosconnect-operator: manifest unknown
2024/05/09 06:20:56  [ERROR]  : reading manifest ac54cb8ff880a935ea3b4b1efc96d35bbf973342c450400d6417d06e59050027 in localhost:55000/openshift4/ose-kube-rbac-proxy: manifest unknown image : map[]
2024/05/09 06:20:56  [ERROR]  : [delete-images] reading manifest ac54cb8ff880a935ea3b4b1efc96d35bbf973342c450400d6417d06e59050027 in localhost:55000/openshift4/ose-kube-rbac-proxy: manifest unknown
2024/05/09 06:20:56  [ERROR]  : reading manifest latest in localhost:55000/ubi8/ubi: manifest unknown image : map[]
2024/05/09 06:20:56  [ERROR]  : [delete-images] reading manifest latest in localhost:55000/ubi8/ubi: manifest unknown
2024/05/09 06:20:56  [ERROR]  : reading manifest latest in localhost:55000/ubi8/ubi-minimal: manifest unknown image : map[]
2024/05/09 06:20:56  [ERROR]  : [delete-images] reading manifest latest in localhost:55000/ubi8/ubi-minimal: manifest unknown
2024/05/09 06:20:56  [INFO]   : delete time     : 11.962449042s
2024/05/09 06:20:56  [INFO]   : 👋 Goodbye, thank you for using oc-mirror
[root@preserve-fedora36 bugv]# echo $?
0

And I didn't see the file "mode.txt" as well as a folder called "info" in the "working-dir" main folder.

cd info/
[root@preserve-fedora36 info]# ll
total 0
[root@preserve-fedora36 info]# tree .
.0 directories, 0 files
[root@preserve-fedora36 info]# pwd
/app1/bugv/clid20/working-dir/info

Ying Zhou added a comment - 2024/05/09 6:23 AM - edited with the fixed pr : https://github.com/openshift/oc-mirror/pull/848 , I still could reproduce the issue : git log commit a09d76ce1cb660125b5bbc80456ef9fabc18e42e (HEAD -> OCPBUGS-32547, origin/OCPBUGS-32547) Author: Luigi Mario Zuccarelli <luzuccar@redhat.com> Date: Wed May 8 14:27:08 2024 +0200 OCPBUGS-32547: Fix delete generation for mirror to mirror flow oc-mirror version WARNING: This version information is deprecated and will be replaced with the output from -- short . Use --output=yaml|json to get the full version. Client Version: version.Info{Major: "", Minor:" ", GitVersion:" v0.0.0-unknown-a09d76ce ", GitCommit:" a09d76ce ", GitTreeState:" clean ", BuildDate:" 2024-05-09T04:47:55Z ", GoVersion:" go1.21.3 ", Compiler :" gc ", Platform:" linux/amd64"} oc-mirror delete --config delete-s.yaml --workspace file: //clid20 --v2 --generate docker://localhost:5000 --dest-tls-verify=false2024/05/09 06:20:44 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/05/09 06:20:44 [INFO] : 👋 Hello, welcome to oc-mirror 2024/05/09 06:20:44 [INFO] : ⚙️ setting up the environment for you... 2024/05/09 06:20:44 [INFO] : 🔀 workflow mode: diskToMirror / delete 2024/05/09 06:20:44 [ERROR] : [RunDelete] open clid20/working-dir/hold-release/ocp-release: no such file or directory 2024/05/09 06:20:44 [INFO] : 🕵️ going to discover the necessary images... 2024/05/09 06:20:44 [INFO] : 🔍 collecting release images... 2024/05/09 06:20:44 [INFO] : 🔍 collecting operator images... 2024/05/09 06:20:56 [INFO] : 🔍 collecting additional images... 2024/05/09 06:20:56 [INFO] : 📄 Generating delete file... 2024/05/09 06:20:56 [INFO] : clid20/working-dir/delete file created 2024/05/09 06:20:56 [ERROR] : reading manifest v14 in localhost:55000/zhouy/catalog: manifest unknown image : map[] 2024/05/09 06:20:56 [ERROR] : [delete-images] reading manifest v14 in localhost:55000/zhouy/catalog: manifest unknown 2024/05/09 06:20:56 [ERROR] : reading manifest ce3330c1c90514f8ca58d22415c78dfa1039b42753d8f2dccc653416afb4d202 in localhost:55000/kube-descheduler- operator /descheduler-rhel9: manifest unknown image : map[] 2024/05/09 06:20:56 [ERROR] : [delete-images] reading manifest ce3330c1c90514f8ca58d22415c78dfa1039b42753d8f2dccc653416afb4d202 in localhost:55000/kube-descheduler- operator /descheduler-rhel9: manifest unknown 2024/05/09 06:20:56 [ERROR] : reading manifest ebcc836c3052bebfe7747c18b57ccc3a2870b5907e5ead894d87c8b2e33a12d9 in localhost:55000/kube-descheduler- operator /kube-descheduler- operator -bundle: manifest unknown image : map[] 2024/05/09 06:20:56 [ERROR] : [delete-images] reading manifest ebcc836c3052bebfe7747c18b57ccc3a2870b5907e5ead894d87c8b2e33a12d9 in localhost:55000/kube-descheduler- operator /kube-descheduler- operator -bundle: manifest unknown 2024/05/09 06:20:56 [ERROR] : reading manifest ea40bc7d39b095dba0649d2052561a75972e3da8fd1b8a44a2cd8081d9326b92 in localhost:55000/kube-descheduler- operator /kube-descheduler-rhel9- operator : manifest unknown image : map[] .... 2024/05/09 06:20:56 [ERROR] : [delete-images] reading manifest b227736a7d44dd1c332edba9e575608cf38fa76df4dd3ad2bf7dc1ee6d66b06c in localhost:55000/cpopen/ibm-zcon-zosconnect- operator : manifest unknown 2024/05/09 06:20:56 [ERROR] : reading manifest ac54cb8ff880a935ea3b4b1efc96d35bbf973342c450400d6417d06e59050027 in localhost:55000/openshift4/ose-kube-rbac-proxy: manifest unknown image : map[] 2024/05/09 06:20:56 [ERROR] : [delete-images] reading manifest ac54cb8ff880a935ea3b4b1efc96d35bbf973342c450400d6417d06e59050027 in localhost:55000/openshift4/ose-kube-rbac-proxy: manifest unknown 2024/05/09 06:20:56 [ERROR] : reading manifest latest in localhost:55000/ubi8/ubi: manifest unknown image : map[] 2024/05/09 06:20:56 [ERROR] : [delete-images] reading manifest latest in localhost:55000/ubi8/ubi: manifest unknown 2024/05/09 06:20:56 [ERROR] : reading manifest latest in localhost:55000/ubi8/ubi-minimal: manifest unknown image : map[] 2024/05/09 06:20:56 [ERROR] : [delete-images] reading manifest latest in localhost:55000/ubi8/ubi-minimal: manifest unknown 2024/05/09 06:20:56 [INFO] : delete time : 11.962449042s 2024/05/09 06:20:56 [INFO] : 👋 Goodbye, thank you for using oc-mirror [root@preserve-fedora36 bugv]# echo $? 0 And I didn't see the file "mode.txt" as well as a folder called "info" in the "working-dir" main folder. cd info/ [root@preserve-fedora36 info]# ll total 0 [root@preserve-fedora36 info]# tree . .0 directories, 0 files [root@preserve-fedora36 info]# pwd /app1/bugv/clid20/working-dir/info

Luigi Mario Zuccarelli added a comment - 2024/05/07 7:00 AM

knarra@redhat.com - good find, I will check the mirrorTomirror flow and see where the problem is

Luigi Mario Zuccarelli added a comment - 2024/05/07 7:00 AM knarra@redhat.com - good find, I will check the mirrorTomirror flow and see where the problem is

Rama Kasturi Narra added a comment - 2024/05/06 5:28 PM

luzuccar@redhat.com so one major difference i could see between yours and ying issue is that you are doing a mirror2disk and then disk2mirror but ying is doing a mirror2mirror and that is the reason you might not have been able to reproduce the issue. I followed yings steps and i could still reproduce the issue. Please find the steps to reproduce below

steps to reproduce:
=================
ImageSetConfigFile:
++++++++++++++++++

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
  platform:
    channels:
      - name: stable-4.15
  additionalImages: 
    - name: registry.redhat.io/ubi8/ubi:latest
    - name: registry.redhat.io/ubi8/ubi-minimal:latest
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
      packages:
      - name: 3scale-operator
    - catalog: oci:///app1/knarra/ibm-catalog
      targetTag: "v14"
      targetCatalog: "zhouy/catalog"

imageSetDeleteConfigFile:
++++++++++++++++++++++

kind: DeleteImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
delete:
  platform:
    channels:
      - name: stable-4.15
  additionalImages: 
    - name: registry.redhat.io/ubi8/ubi:latest
    - name: registry.redhat.io/ubi8/ubi-minimal:latest
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
      packages:
      - name: 3scale-operator
    - catalog: oci:///app1/knarra/ibm-catalog
      targetTag: "v14"
      targetCatalog: "zhouy/catalog"

1. ./oc-mirror --config /tmp/isc.yaml --v2 docker://localhost:5000/m2m --workspace file:///app1/knarra/clid20/ --dest-tls-verify=false
2. ./oc-mirror delete --config /tmp/isc_delete.yaml --workspace file://clid20 --v2 --generate docker://localhost:5000/m2m --dest-tls-verify=false

2024/05/06 16:19:36  [WARN]   : ⚠️  --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready.
2024/05/06 16:19:36  [INFO]   : 👋 Hello, welcome to oc-mirror
2024/05/06 16:19:36  [INFO]   : ⚙️  setting up the environment for you...
2024/05/06 16:19:36  [INFO]   : 🔀 workflow mode: diskToMirror / delete
2024/05/06 16:19:36  [INFO]   : 🕵️  going to discover the necessary images...
2024/05/06 16:19:36  [INFO]   : 🔍 collecting release images...
2024/05/06 16:19:36  [INFO]   : 🔍 collecting operator images...
2024/05/06 16:19:49  [INFO]   : 🔍 collecting additional images...
2024/05/06 16:19:49  [INFO]   : 📄 Generating delete file...
2024/05/06 16:19:49  [INFO]   : clid20/working-dir/delete file created
2024/05/06 16:19:49  [ERROR]  : reading manifest sha256:de31241f782e07d7cc9dfc41436ad3ec3c2d0a30800bf37241ff1a72a349667c in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[]
2024/05/06 16:19:49  [ERROR]  : [delete-images] reading manifest sha256:de31241f782e07d7cc9dfc41436ad3ec3c2d0a30800bf37241ff1a72a349667c in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown
2024/05/06 16:19:49  [ERROR]  : reading manifest sha256:1ef0ac95c372cfcca0e853154672f8c321bf454ddaa56684efce9af1312cb421 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[]
2024/05/06 16:19:49  [ERROR]  : [delete-images] reading manifest sha256:1ef0ac95c372cfcca0e853154672f8c321bf454ddaa56684efce9af1312cb421 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown
2024/05/06 16:19:49  [ERROR]  : reading manifest sha256:66e2b14992a68b1da921da9afd284bfd7ad6eae13d78f9eb2e464eabfe0aabb3 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[]
2024/05/06 16:19:49  [ERROR]  : [delete-images] reading manifest sha256:66e2b14992a68b1da921da9afd284bfd7ad6eae13d78f9eb2e464eabfe0aabb3 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown
2024/05/06 16:19:49  [ERROR]  : reading manifest sha256:86341e1b8d47b0e72e2abba57bb9728ff32dc418e037b0d697519165c334da6c in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[]
2024/05/06 16:19:49  [ERROR]  : [delete-images] reading manifest sha256:86341e1b8d47b0e72e2abba57bb9728ff32dc418e037b0d697519165c334da6c in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown
2024/05/06 16:19:49  [ERROR]  : reading manifest sha256:52b69276b26b735736057dc000fcd0c38adc4ed7f11b2153030c8d240eba1686 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[]
2024/05/06 16:19:49  [ERROR]  : [delete-images] reading manifest sha256:52b69276b26b735736057dc000fcd0c38adc4ed7f11b2153030c8d240eba1686 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown
2024/05/06 16:19:49  [ERROR]  : reading manifest sha256:740104c53885b857e3271dfd999b13b82ee9291960c171a8d649fee25b2d6381 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[]
2024/05/06 16:19:49  [ERROR]  : [delete-images] reading manifest sha256:740104c53885b857e3271dfd999b13b82ee9291960c171a8d649fee25b2d6381 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown
2024/05/06 16:19:49  [ERROR]  : reading manifest sha256:4198145ad020d0394636a0604c35fd49c6439b1c3de365cdf49db9eb8a6ff77c in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[]
2024/05/06 16:19:49  [ERROR]  : [delete-images] reading manifest sha256:4198145ad020d0394636a0604c35fd49c6439b1c3de365cdf49db9eb8a6ff77c in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown
2024/05/06 16:19:49  [ERROR]  : reading manifest sha256:30c80279dc6728c002a1e8253681e3d5fe9eaca7a42d175c88bb736905ac2c5b in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[]
2024/05/06 16:19:49  [ERROR]  : [delete-images] reading manifest sha256:30c80279dc6728c002a1e8253681e3d5fe9eaca7a42d175c88bb736905ac2c5b in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown
2024/05/06 16:19:49  [ERROR]  : reading manifest sha256:b42f85c095a94d0b5eaf15e676229604b67ab4403fd5776e4526e9549fb4d334 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[]
2024/05/06 16:19:49  [ERROR]  : [delete-images] reading manifest sha256:b42f85c095a94d0b5eaf15e676229604b67ab4403fd5776e4526e9549fb4d334 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown
2024/05/06 16:19:49  [ERROR]  : reading manifest sha256:ab4810118ddc7ed3e25fd6535579a6c895c71eefe1dae6f2c8c48086af084f96 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[]
2024/05/06 16:19:49  [ERROR]  : [delete-images] reading manifest sha256:ab4810118ddc7ed3e25fd6535579a6c895c71eefe1dae6f2c8c48086af084f96 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown
2024/05/06 16:19:49  [INFO]   : delete time     : 13.261744947s
2024/05/06 16:19:49  [INFO]   : 👋 Goodbye, thank you for using oc-mirror

Rama Kasturi Narra added a comment - 2024/05/06 5:28 PM luzuccar@redhat.com so one major difference i could see between yours and ying issue is that you are doing a mirror2disk and then disk2mirror but ying is doing a mirror2mirror and that is the reason you might not have been able to reproduce the issue. I followed yings steps and i could still reproduce the issue. Please find the steps to reproduce below steps to reproduce: ================= ImageSetConfigFile: ++++++++++++++++++ kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: platform: channels: - name: stable-4.15 additionalImages: - name: registry.redhat.io/ubi8/ubi:latest - name: registry.redhat.io/ubi8/ubi-minimal:latest operators: - catalog: registry.redhat.io/redhat/redhat- operator -index:v4.14 packages: - name: 3scale- operator - catalog: oci: ///app1/knarra/ibm-catalog targetTag: "v14" targetCatalog: "zhouy/catalog" imageSetDeleteConfigFile: ++++++++++++++++++++++ kind: DeleteImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 delete: platform: channels: - name: stable-4.15 additionalImages: - name: registry.redhat.io/ubi8/ubi:latest - name: registry.redhat.io/ubi8/ubi-minimal:latest operators: - catalog: registry.redhat.io/redhat/redhat- operator -index:v4.14 packages: - name: 3scale- operator - catalog: oci: ///app1/knarra/ibm-catalog targetTag: "v14" targetCatalog: "zhouy/catalog" 1. ./oc-mirror --config /tmp/isc.yaml --v2 docker://localhost:5000/m2m --workspace file:///app1/knarra/clid20/ --dest-tls-verify=false 2. ./oc-mirror delete --config /tmp/isc_delete.yaml --workspace file://clid20 --v2 --generate docker://localhost:5000/m2m --dest-tls-verify=false 2024/05/06 16:19:36 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/05/06 16:19:36 [INFO] : 👋 Hello, welcome to oc-mirror 2024/05/06 16:19:36 [INFO] : ⚙️ setting up the environment for you... 2024/05/06 16:19:36 [INFO] : 🔀 workflow mode: diskToMirror / delete 2024/05/06 16:19:36 [INFO] : 🕵️ going to discover the necessary images... 2024/05/06 16:19:36 [INFO] : 🔍 collecting release images... 2024/05/06 16:19:36 [INFO] : 🔍 collecting operator images... 2024/05/06 16:19:49 [INFO] : 🔍 collecting additional images... 2024/05/06 16:19:49 [INFO] : 📄 Generating delete file... 2024/05/06 16:19:49 [INFO] : clid20/working-dir/delete file created 2024/05/06 16:19:49 [ERROR] : reading manifest sha256:de31241f782e07d7cc9dfc41436ad3ec3c2d0a30800bf37241ff1a72a349667c in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[] 2024/05/06 16:19:49 [ERROR] : [delete-images] reading manifest sha256:de31241f782e07d7cc9dfc41436ad3ec3c2d0a30800bf37241ff1a72a349667c in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown 2024/05/06 16:19:49 [ERROR] : reading manifest sha256:1ef0ac95c372cfcca0e853154672f8c321bf454ddaa56684efce9af1312cb421 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[] 2024/05/06 16:19:49 [ERROR] : [delete-images] reading manifest sha256:1ef0ac95c372cfcca0e853154672f8c321bf454ddaa56684efce9af1312cb421 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown 2024/05/06 16:19:49 [ERROR] : reading manifest sha256:66e2b14992a68b1da921da9afd284bfd7ad6eae13d78f9eb2e464eabfe0aabb3 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[] 2024/05/06 16:19:49 [ERROR] : [delete-images] reading manifest sha256:66e2b14992a68b1da921da9afd284bfd7ad6eae13d78f9eb2e464eabfe0aabb3 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown 2024/05/06 16:19:49 [ERROR] : reading manifest sha256:86341e1b8d47b0e72e2abba57bb9728ff32dc418e037b0d697519165c334da6c in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[] 2024/05/06 16:19:49 [ERROR] : [delete-images] reading manifest sha256:86341e1b8d47b0e72e2abba57bb9728ff32dc418e037b0d697519165c334da6c in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown 2024/05/06 16:19:49 [ERROR] : reading manifest sha256:52b69276b26b735736057dc000fcd0c38adc4ed7f11b2153030c8d240eba1686 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[] 2024/05/06 16:19:49 [ERROR] : [delete-images] reading manifest sha256:52b69276b26b735736057dc000fcd0c38adc4ed7f11b2153030c8d240eba1686 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown 2024/05/06 16:19:49 [ERROR] : reading manifest sha256:740104c53885b857e3271dfd999b13b82ee9291960c171a8d649fee25b2d6381 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[] 2024/05/06 16:19:49 [ERROR] : [delete-images] reading manifest sha256:740104c53885b857e3271dfd999b13b82ee9291960c171a8d649fee25b2d6381 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown 2024/05/06 16:19:49 [ERROR] : reading manifest sha256:4198145ad020d0394636a0604c35fd49c6439b1c3de365cdf49db9eb8a6ff77c in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[] 2024/05/06 16:19:49 [ERROR] : [delete-images] reading manifest sha256:4198145ad020d0394636a0604c35fd49c6439b1c3de365cdf49db9eb8a6ff77c in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown 2024/05/06 16:19:49 [ERROR] : reading manifest sha256:30c80279dc6728c002a1e8253681e3d5fe9eaca7a42d175c88bb736905ac2c5b in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[] 2024/05/06 16:19:49 [ERROR] : [delete-images] reading manifest sha256:30c80279dc6728c002a1e8253681e3d5fe9eaca7a42d175c88bb736905ac2c5b in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown 2024/05/06 16:19:49 [ERROR] : reading manifest sha256:b42f85c095a94d0b5eaf15e676229604b67ab4403fd5776e4526e9549fb4d334 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[] 2024/05/06 16:19:49 [ERROR] : [delete-images] reading manifest sha256:b42f85c095a94d0b5eaf15e676229604b67ab4403fd5776e4526e9549fb4d334 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown 2024/05/06 16:19:49 [ERROR] : reading manifest sha256:ab4810118ddc7ed3e25fd6535579a6c895c71eefe1dae6f2c8c48086af084f96 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown image : map[] 2024/05/06 16:19:49 [ERROR] : [delete-images] reading manifest sha256:ab4810118ddc7ed3e25fd6535579a6c895c71eefe1dae6f2c8c48086af084f96 in localhost:55000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown 2024/05/06 16:19:49 [INFO] : delete time : 13.261744947s 2024/05/06 16:19:49 [INFO] : 👋 Goodbye, thank you for using oc-mirror

Rama Kasturi Narra added a comment - 2024/05/06 1:05 PM

luzuccar@redhat.com Let me try reproducing the issue and i will add info back here.

Rama Kasturi Narra added a comment - 2024/05/06 1:05 PM luzuccar@redhat.com Let me try reproducing the issue and i will add info back here.

Luigi Mario Zuccarelli added a comment - 2024/05/06 7:36 AM

knarra@redhat.com yinzhou@redhat.com I tested this locally using the following ImageSetConfig

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.14
      packages:
      - name: aws-load-balancer-operator
    - catalog: oci:///home/lzuccarelli/go/src/github.com/openshift/oc-mirror/rhopi4-14
      targetTag: "v14"
      targetCatalog: "lmz/catalog"
      packages:
      - name: windows-machine-config-operator
    - catalog: oci:///home/lzuccarelli/go/src/github.com/openshift/oc-mirror/rhopi4-14
      packages:
      - name: cincinnati-operator

I first did a mirorToDisk and then a diskToMirror to the remote registry

I then executed the following command to generate the delete artifacts

bin/oc-mirror delete --config delete-32547.yaml --v2 --workspace file://ocpbugs-32547 docker://localhost:5000/ocpbugs-32547 --dest-tls-verify=false --generate

I then followed this with the following command to delete both cache and remote registry

bin/oc-mirror delete --config delete-32547.yaml --v2 --delete-yaml-file ocpbugs-32547/working-dir/delete/delete-images.yaml docker://localhost:5000/ocpbugs-32547 --dest-tls-verify=false --force-cache-delete

I had no errors.

Could it be that maybe in a previous test you deleted the 3scale operator , and when you re-run with a --generate it can't find the 3scale manifests as they have been deleted ?

Luigi Mario Zuccarelli added a comment - 2024/05/06 7:36 AM knarra@redhat.com yinzhou@redhat.com I tested this locally using the following ImageSetConfig kind: ImageSetConfiguration apiVersion: mirror.openshift.io/v2alpha1 mirror: operators: - catalog: registry.redhat.io/redhat/redhat- operator -index:v4.14 packages: - name: aws-load-balancer- operator - catalog: oci: ///home/lzuccarelli/go/src/github.com/openshift/oc-mirror/rhopi4-14 targetTag: "v14" targetCatalog: "lmz/catalog" packages: - name: windows-machine-config- operator - catalog: oci: ///home/lzuccarelli/go/src/github.com/openshift/oc-mirror/rhopi4-14 packages: - name: cincinnati- operator I first did a mirorToDisk and then a diskToMirror to the remote registry I then executed the following command to generate the delete artifacts bin/oc-mirror delete --config delete-32547.yaml --v2 --workspace file: //ocpbugs-32547 docker://localhost:5000/ocpbugs-32547 --dest-tls-verify= false --generate I then followed this with the following command to delete both cache and remote registry bin/oc-mirror delete --config delete-32547.yaml --v2 --delete-yaml-file ocpbugs-32547/working-dir/delete/delete-images.yaml docker: //localhost:5000/ocpbugs-32547 --dest-tls-verify= false --force-cache-delete I had no errors. Could it be that maybe in a previous test you deleted the 3scale operator , and when you re-run with a --generate it can't find the 3scale manifests as they have been deleted ?

Rama Kasturi Narra added a comment - 2024/04/30 8:38 AM

If oci catalog is not present, this should work fine. Error happens due to having oci catalog.

Rama Kasturi Narra added a comment - 2024/04/30 8:38 AM If oci catalog is not present, this should work fine. Error happens due to having oci catalog.

Assignee:: Luigi Mario Zuccarelli

Reporter:: Ying Zhou

QA Contact:: Ying Zhou

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Created:: 2024/04/22 12:56 PM

Updated:: 2024/06/27 11:46 AM

Resolved:: 2024/06/27 11:46 AM

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

Collapse comment: Errata Tool added a comment - 2024/06/27 11:46 AM

Expand comment: Errata Tool added a comment - 2024/06/27 11:46 AM

Collapse comment: Ying Zhou added a comment - 2024/05/10 10:22 AM

Expand comment: Ying Zhou added a comment - 2024/05/10 10:22 AM

Collapse comment: Ying Zhou added a comment - 2024/05/09 6:23 AM, Edited by Ying Zhou - 2024/05/09 7:11 AM

Expand comment: Ying Zhou added a comment - 2024/05/09 6:23 AM, Edited by Ying Zhou - 2024/05/09 7:11 AM

Collapse comment: Luigi Mario Zuccarelli added a comment - 2024/05/07 7:00 AM

Expand comment: Luigi Mario Zuccarelli added a comment - 2024/05/07 7:00 AM

Collapse comment: Rama Kasturi Narra added a comment - 2024/05/06 5:28 PM

Expand comment: Rama Kasturi Narra added a comment - 2024/05/06 5:28 PM

Collapse comment: Rama Kasturi Narra added a comment - 2024/05/06 1:05 PM

Expand comment: Rama Kasturi Narra added a comment - 2024/05/06 1:05 PM

Collapse comment: Luigi Mario Zuccarelli added a comment - 2024/05/06 7:36 AM

Expand comment: Luigi Mario Zuccarelli added a comment - 2024/05/06 7:36 AM

Collapse comment: Rama Kasturi Narra added a comment - 2024/04/30 8:38 AM

Expand comment: Rama Kasturi Narra added a comment - 2024/04/30 8:38 AM

People

Dates