test-scc-d2dfp                          0/1     CreateContainerError   0             7s
jiazha-mac:~ jiazha$ oc logs test-scc-d2dfp 
Error from server (BadRequest): container "registry-server" in pod "test-scc-d2dfp" is waiting to start: CreateContainerError  Normal   Pulling         3s (x5 over 33s)  kubelet            Pulling image "quay.io/olmqe/etcd-index:v1"
  Warning  Failed          2s (x5 over 32s)  kubelet            Error: reference "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.skip_mount_home=true]quay.io/olmqe/etcd-index@sha256:7fbee03d2619e5c59d70acb59796a7371de6900ff7891ea379de79720132b904" does not resolve to an image ID: identifier is not an image
  Normal   Pulled          2s                kubelet            Successfully pulled image "quay.io/olmqe/etcd-index:v1" in 745ms (745ms including waiting). Image size: 67896952 bytes.

    OCP 4.18

    always

    1. Install OCP 4.18.0.
jiazha-mac:~ jiazha$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.18.0-0.nightly-2024-10-07-200953   True        False         85m     Cluster version is 4.18.0-0.nightly-2024-10-07-200953

     2.Create a CatalogSource, like

jiazha-mac:~ jiazha$ cat cs-withoutCache.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: test-scc
  namespace: openshift-marketplace
spec:
  displayName: Test Operators
  image: quay.io/olmqe/etcd-index:v1
  priority: -100
  publisher: Red Hat QE
  sourceType: grpc
  updateStrategy:
    registryPoll:
      interval: 10m
    

    jiazha-mac:~ jiazha$ oc get pods 
NAME                                    READY   STATUS                 RESTARTS      AGE
certified-operators-fc96c               1/1     Running                0             61m
community-operators-wgl9k               1/1     Running                0             98m
marketplace-operator-5d968c5497-6fhhk   1/1     Running                1 (96m ago)   109m
redhat-marketplace-kcx79                1/1     Running                0             74m
redhat-operators-lbjdx                  1/1     Running                0             98m
test-scc-d2dfp                          0/1     CreateContainerError   0             49m
test-scc-dtlr2                          0/1     CreateContainerError   0             32m

    That catalogsource pods works.

    jiazha-mac:~ jiazha$ oc get pods test-scc-d2dfp -o wide 
NAME             READY   STATUS                 RESTARTS   AGE     IP            NODE                                       NOMINATED NODE   READINESS GATES
test-scc-d2dfp   0/1     CreateContainerError   0          2m22s   10.128.2.23   ip-10-0-2-107.us-east-2.compute.internal   <none>           <none>

jiazha-mac:~ jiazha$ oc debug node/ip-10-0-2-107.us-east-2.compute.internal
Temporary namespace openshift-debug-px68s is created for debugging node...
Starting pod/ip-10-0-2-107us-east-2computeinternal-debug-g9dwg ...
To use host binaries, run `chroot /host`
...
Oct 08 06:31:04 ip-10-0-2-107 kubenswrapper[6608]: E1008 06:31:04.683505    6608 kuberuntime_manager.go:1274] "Unhandled Error" err="container &Container{Name:registry-server,Image:quay.io/olmqe/etcd-index:v1,Command:[],Args:[],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:grpc,HostPort:0,ContainerPort:50051,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{10 -3} {<nil>} 10m DecimalSI},memory: {{52428800 0} {<nil>} 50Mi BinarySI},},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-wv4mj,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,RecursiveReadOnly:nil,},},LivenessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[grpc_health_probe -addr=:50051],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:10,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,},ReadinessProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[grpc_health_probe -addr=:50051],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:5,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,},Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:Always,SecurityContext:&SecurityContext{Capabilities:&Capabilities{Add:[],Drop:[MKNOD],},Privileged:nil,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:*false,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,AppArmorProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:FallbackToLogsOnError,VolumeDevices:[]VolumeDevice{},StartupProbe:&Probe{ProbeHandler:ProbeHandler{Exec:&ExecAction{Command:[grpc_health_probe -addr=:50051],},HTTPGet:nil,TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:0,TimeoutSeconds:5,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:10,TerminationGracePeriodSeconds:nil,},ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod test-scc-d2dfp_openshift-marketplace(e19eb969-d554-45dc-bb88-e073b6d1a7b7): CreateContainerError: reference \"[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.skip_mount_home=true]quay.io/olmqe/etcd-index@sha256:7fbee03d2619e5c59d70acb59796a7371de6900ff7891ea379de79720132b904\" does not resolve to an image ID: identifier is not an image" logger="UnhandledError"
Oct 08 06:31:04 ip-10-0-2-107 kubenswrapper[6608]: E1008 06:31:04.684631    6608 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"registry-server\" with CreateContainerError: \"reference \\\"[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.skip_mount_home=true]quay.io/olmqe/etcd-index@sha256:7fbee03d2619e5c59d70acb59796a7371de6900ff7891ea379de79720132b904\\\" does not resolve to an image ID: identifier is not an image\"" pod="openshift-marketplace/test-scc-d2dfp" podUID="e19eb969-d554-45dc-bb88-e073b6d1a7b7"
Oct 08 06:31:16 ip-10-0-2-107 kubenswrapper[6608]: E1008 06:31:16.681434    6608 log.go:32] "CreateContainer in sandbox from runtime service failed" err="rpc error: code = Unknown desc = reference \"[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.skip_mount_home=true]quay.io/olmqe/etcd-index@sha256:7fbee03d2619e5c59d70acb59796a7371de6900ff7891ea379de79720132b904\" does not resolve to an image ID: identifier is not an image" podSandboxID="98c73dc81c952a49f2e1bd4a2c32b4102868612bd4a5e76b47a105e242f712b9"

[cloud-user@preserve-olm-env2 jian]$ sudo podman pull quay.io/olmqe/etcd-index:v1 
Trying to pull quay.io/olmqe/etcd-index:v1...
Getting image source signatures
Copying blob 5f70bf18a086 skipped: already exists 
Copying blob 5f70bf18a086 skipped: already exists 
Copying blob 900a80043247 done 
Copying blob df20fa9351a1 done 
Copying blob 03260e9c3f62 done 
Copying blob 19d60180458b done 
Copying blob 5f70bf18a086 skipped: already exists 
Copying blob 6c9eb1dc45de done 
Copying blob 5f70bf18a086 skipped: already exists 
Copying blob 5f70bf18a086 skipped: already exists 
Writing manifest to image destination
50530c0bc741800fa7f93b6a2dcb04d46e7353c1883768c816a8b5c81141a270
 

jiazha-mac:~ jiazha$ oc get clusterversion
 NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.17.0-0.nightly-2024-10-07-214907   True        False         54m     Cluster version is 4.17.0-0.nightly-2024-10-07-214907

 jiazha-mac:~ jiazha$ oc create -f cs-withoutCache.yaml 
catalogsource.operators.coreos.com/test-scc created

jiazha-mac:~ jiazha$ oc get pods 
NAME                                    READY   STATUS    RESTARTS      AGE
certified-operators-kkxdl               1/1     Running   0             62m
community-operators-8k8gp               1/1     Running   0             76m
marketplace-operator-54b75c6b98-9tffp   1/1     Running   1 (83m ago)   92m
qe-app-registry-dwxr4                   1/1     Running   0             64m
redhat-marketplace-9tr2d                1/1     Running   0             76m
redhat-operators-dk4lx                  1/1     Running   0             76m
test-scc-nxbfc                          1/1     Running   0             11m