We were aiming to write a monitor e2e test that starts a pod during setup of test and monitors the pod throughout test lifecycle.

With https://github.com/openshift/origin/pull/28676, we discovered that we are unable to pull CI images.

Logs say that the image pull secret is invalid, secrets are not found and it fails.

This is odd because we should be able this pod.

We have tried openshift/tools but we get an authenticate error for this pod.


	],Args:[],WorkingDir:,Ports:[]ContainerPort{},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{10 -3} {<nil>} 10m DecimalSI},memory: {{52428800 0} {<nil>} 50Mi BinarySI},},Claims:[]ResourceClaim{},},VolumeMounts:[]VolumeMount{VolumeMount{Name:kube-api-access-7v6sn,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:nil,ReadinessProbe:nil,Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:nil,Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,ResizePolicy:[]ContainerResizePolicy{},RestartPolicy:nil,} start failed in pod label-0_e2e-selinux-label-pod-x9pl4(491d4d98-ad90-4267-8a34-ea1eda675c79): ErrImagePull: reading manifest latest in image-registry.openshift-image-registry.svc:5000/openshift/tools: authentication required2024-04-05 11:00:23E0405 15:00:23.135639    2502 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"label-0\" with ErrImagePull: \"reading manifest latest in image-registry.openshift-image-registry.svc:5000/openshift/tools: authentication required\"" pod="e2e-selinux-label-pod-x9pl4/label-0" podUID="491d4d98-ad90-4267-8a34-ea1eda675c79"2024-04-05 11:00:36{"metadata":{"name":"label-0.17c36a470275d90d","namespace":"e2e-selinux-label-pod-x9pl4","uid":"d8be3cf7-5314-46e9-9829-f25251acd64d","resourceVersion":"34438","creationTimestamp":"2024-04-05T14:58:56Z"},"reason":"BackOff","message":"Back-off pulling image \"image-registry.openshift-image-registry.svc:5000/openshift/tools:latest\"","source":{"component":"kubelet","host":"ci-op-zgwzmn83-fcf39-g5d89-master-0"},"firstTimestamp":"2024-04-05T14:58:56Z","lastTimestamp":"2024-04-05T15:00:36Z","count":6,"type":"Normal","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"ci-op-zgwzmn83-fcf39-g5d89-master-0","involvedObject":{"kind":"Pod","namespace":"e2e-selinux-label-pod-x9pl4","name":"label-0","uid":"491d4d98-ad90-4267-8a34-ea1eda675c79","apiVersion":"v1","resourceVersion":"32193","fieldPath":"spec.containers{label-0}","annotations":{"k8s.ovn.org/pod-networks":"{\"default\":{\"ip_addresses\":[\"10.130.0.69/23\"],\"mac_address\":\"0a:58:0a:82:00:45\",\"gateway_ips\":[\"10.130.0.1\"],\"routes\":[{\"dest\":\"10.128.0.0/14\",\"nextHop\":\"10.130.0.1\"},{\"dest\":\"172.30.0.0/16\",\"nextHop\":\"10.130.0.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.130.0.1\"}],\"ip_address\":\"10.130.0.69/23\",\"gateway_ip\":\"10.130.0.1\"}}","k8s.v1.cni.cncf.io/network-status":"[{\n    \"name\": \"ovn-kubernetes\",\n    \"interface\": \"eth0\",\n    \"ips\": [\n        \"10.130.0.69\"\n    ],\n    \"mac\": \"0a:58:0a:82:00:45\",\n    \"default\": true,\n    \"dns\": {}\n}]"}}}2024-04-05 11:00:36{"metadata":{"name":"label-0.17c36a47027648bb","namespace":"e2e-selinux-label-pod-x9pl4","uid":"cfd1d6ce-7d78-42dd-861f-1d3d7482cd8e","resourceVersion":"34440","creationTimestamp":"2024-04-05T14:58:56Z"},"reason":"Failed","message":"Error: ImagePullBackOff","source":{"component":"kubelet","host":"ci-op-zgwzmn83-fcf39-g5d89-master-0"},"firstTimestamp":"2024-04-05T14:58:56Z","lastTimestamp":"2024-04-05T15:00:36Z","count":6,"type":"Warning","eventTime":null,"reportingComponent":"kubelet","reportingInstance":"ci-op-zgwzmn83-fcf39-g5d89-master-0","involvedObject":{"kind":"Pod","namespace":"e2e-selinux-label-pod-x9pl4","name":"label-0","uid":"491d4d98-ad90-4267-8a34-ea1eda675c79","apiVersion":"v1","resourceVersion":"32193","fieldPath":"spec.containers{label-0}","annotations":{"k8s.ovn.org/pod-networks":"{\"default\":{\"ip_addresses\":[\"10.130.0.69/23\"],\"mac_address\":\"0a:58:0a:82:00:45\",\"gateway_ips\":[\"10.130.0.1\"],\"routes\":[{\"dest\":\"10.128.0.0/14\",\"nextHop\":\"10.130.0.1\"},{\"dest\":\"172.30.0.0/16\",\"nextHop\":\"10.130.0.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.130.0.1\"}],\"ip_address\":\"10.130.0.69/23\",\"gateway_ip\":\"10.130.0.1\"}}","k8s.v1.cni.cncf.io/network-status":"[{\n    \"name\": \"ovn-kubernetes\",\n    \"interface\": \"eth0\",\n    \"ips\": [\n        \"10.130.0.69\"\n    ],\n    \"mac\": \"0a:58:0a:82:00:45\",\n    \"default\": true,\n    \"dns\": {}\n}]"}}}2024-04-05 11:00:36E0

    4.16 nightly

    Always reproduced With https://github.com/openshift/origin/pull/28676.

On all tests.