-
Bug
-
Resolution: Not a Bug
-
Critical
-
None
-
4.14
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
The Operator Pods from the `openshift-marketplace` in ImagePullBackOff State with the below error message: ~~~ DEBU[0002] Error pulling candidate registry.redhat.io/redhat/redhat-operator-index:v4.14: copying system image from manifest list: reading signatures: parsing signature https://registry.redhat.io/containers/sigstore/redhat/redhat-operator-index@sha256=fd81517986ef7a766b6a976fe613a65fca97e802016d331f60a13a4494185a29/signature-1: unrecognized signature format, starting with binary 0x3c Error: copying system image from manifest list: reading signatures: parsing signature https://registry.redhat.io/containers/sigstore/redhat/redhat-operator-index@sha256=fd81517986ef7a766b6a976fe613a65fca97e802016d331f60a13a4494185a29/signature-1: unrecognized signature format, starting with binary 0x3c ~~~
Version-Release number of selected component (if applicable):
How reproducible:
No
Steps to Reproduce:
1.
2.
3.
Actual results:
The pods are not able to pull the images due to that pods stuck in imagepullbackerror error.
Expected results:
The pods should able to pull the images.
Additional info:
[1] Podman pull command from a node:
[root@nonprod3ocpmaster2 ~]# podman pull --log-level=debug --authfile=/var/lib/kubelet/config.json registry.redhat.io/redhat/redhat-operator-index:v4.14
INFO[0000] podman filtering at log level debug
DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level=debug --authfile=/var/lib/kubelet/config.json registry.redhat.io/redhat/redhat-operator-index:v4.14)
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Overriding run root "/run/containers/storage" with "/var/run/containers/storage" from database
DEBU[0000] Overriding tmp dir "/run/libpod" with "/var/run/libpod" from database
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /var/run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /var/run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: skip_mount_home=true
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] NewControl(/var/lib/containers/storage/overlay): nextProjectID = 344652480
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=xfs, projectQuotaSupported=true, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/runc"
INFO[0000] Setting parallel job count to 37
DEBU[0000] Pulling image registry.redhat.io/redhat/redhat-operator-index:v4.14 (policy: always)
DEBU[0000] Looking up image "registry.redhat.io/redhat/redhat-operator-index:v4.14" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Trying "registry.redhat.io/redhat/redhat-operator-index:v4.14" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.skip_mount_home=true]@d08f2918ae67559f72906e95f7b836b555a9bd48877766063bd823f7e630f268"
DEBU[0000] Found image "registry.redhat.io/redhat/redhat-operator-index:v4.14" as "registry.redhat.io/redhat/redhat-operator-index:v4.14" in local containers storage
DEBU[0000] Found image "registry.redhat.io/redhat/redhat-operator-index:v4.14" as "registry.redhat.io/redhat/redhat-operator-index:v4.14" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.skip_mount_home=true]@d08f2918ae67559f72906e95f7b836b555a9bd48877766063bd823f7e630f268)
DEBU[0000] exporting opaque data as blob "sha256:d08f2918ae67559f72906e95f7b836b555a9bd48877766063bd823f7e630f268"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf"
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Attempting to pull candidate registry.redhat.io/redhat/redhat-operator-index:v4.14 for registry.redhat.io/redhat/redhat-operator-index:v4.14
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.skip_mount_home=true]registry.redhat.io/redhat/redhat-operator-index:v4.14"
Trying to pull registry.redhat.io/redhat/redhat-operator-index:v4.14...
DEBU[0000] Copying source image //registry.redhat.io/redhat/redhat-operator-index:v4.14 to destination image [overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.skip_mount_home=true]registry.redhat.io/redhat/redhat-operator-index:v4.14
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Trying to access "registry.redhat.io/redhat/redhat-operator-index:v4.14"
DEBU[0000] Found credentials for registry.redhat.io/redhat/redhat-operator-index in credential helper containers-auth.json in file /var/lib/kubelet/config.json
DEBU[0000] Lookaside configuration: using "docker" namespace registry.redhat.io
DEBU[0000] Using "sigstore" https://registry.redhat.io/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.redhat.io
DEBU[0000] Sigstore attachments: using "docker" namespace registry.redhat.io
DEBU[0000] GET https://registry.redhat.io/v2/
DEBU[0000] Ping https://registry.redhat.io/v2/ status 401
DEBU[0000] GET https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?account=%7Cuhc-pool-c27c7400-bdba-42a2-bb8e-053e15f36488&scope=repository%3Aredhat%2Fredhat-operator-index%3Apull&service=docker-registry
DEBU[0000] GET https://registry.redhat.io/v2/redhat/redhat-operator-index/manifests/v4.14
DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json"
DEBU[0000] Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb
DEBU[0000] Source is a manifest list; copying (only) instance sha256:fd81517986ef7a766b6a976fe613a65fca97e802016d331f60a13a4494185a29 for current system
DEBU[0000] GET https://registry.redhat.io/v2/redhat/redhat-operator-index/manifests/sha256:fd81517986ef7a766b6a976fe613a65fca97e802016d331f60a13a4494185a29
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json"
DEBU[0001] IsRunningImageAllowed for image docker:registry.redhat.io/redhat/redhat-operator-index:v4.14
DEBU[0001] Using default policy section
DEBU[0001] Requirement 0: allowed
DEBU[0001] Overall: allowed
DEBU[0001] Downloading /v2/redhat/redhat-operator-index/blobs/sha256:31d7206826208888b382d073b391c4931b3a6b9ba1de9b1688002ab4a8a1dcf7
DEBU[0001] GET https://registry.redhat.io/v2/redhat/redhat-operator-index/blobs/sha256:31d7206826208888b382d073b391c4931b3a6b9ba1de9b1688002ab4a8a1dcf7
Getting image source signatures
DEBU[0001] GET https://registry.redhat.io/containers/sigstore/redhat/redhat-operator-index@sha256=fd81517986ef7a766b6a976fe613a65fca97e802016d331f60a13a4494185a29/signature-1
WARN[0002] Signature "https://registry.redhat.io/containers/sigstore/redhat/redhat-operator-index@sha256=fd81517986ef7a766b6a976fe613a65fca97e802016d331f60a13a4494185a29/signature-1" has Content-Type "text/html; charset=UTF-8", unexpected for a signature
DEBU[0002] Error pulling candidate registry.redhat.io/redhat/redhat-operator-index:v4.14: copying system image from manifest list: reading signatures: parsing signature https://registry.redhat.io/containers/sigstore/redhat/redhat-operator-index@sha256=fd81517986ef7a766b6a976fe613a65fca97e802016d331f60a13a4494185a29/signature-1: unrecognized signature format, starting with binary 0x3c
Error: copying system image from manifest list: reading signatures: parsing signature https://registry.redhat.io/containers/sigstore/redhat/redhat-operator-index@sha256=fd81517986ef7a766b6a976fe613a65fca97e802016d331f60a13a4494185a29/signature-1: unrecognized signature format, starting with binary 0x3c
DEBU[0002] Shutting down engines
