Time | Namespace | Component | RelatedObject | Reason | Message
--- | --- | --- | --- | --- | ---
15:08:00 | rhods-notebooks | default-scheduler | jupyter-nb-ldap-2dadmin1-0 | Scheduled | Successfully assigned rhods-notebooks/jupyter-nb-ldap-2dadmin1-0 to ip-10-0-192-209.us-east-2.compute.internal by ip-10-0-227-16
15:08:02 | rhods-notebooks | attachdetach-controller | jupyter-nb-ldap-2dadmin1-0 | SuccessfulAttachVolume | AttachVolume.Attach succeeded for volume "pvc-949d2f45-312e-4666-bad7-63724b0a3d5b"
15:08:04 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Pulling | Pulling image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1"
15:08:04 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | Pulled | Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Successfully pulled image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1" in 188.576655ms
15:08:04 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | AddedInterface | Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Add eth0 [10.131.0.73/23] from ovn-kubernetes
15:08:04 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Pulled | Successfully pulled image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1" in 188.576655ms
15:08:04 | rhods-notebooks | multus | jupyter-nb-ldap-2dadmin1-0 | AddedInterface | Add eth0 [10.131.0.73/23] from ovn-kubernetes
15:08:05 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Pulled | Successfully pulled image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" in 760.787478ms
15:08:05 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Created | Created container jupyter-nb-ldap-2dadmin1
15:08:05 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Started | Started container jupyter-nb-ldap-2dadmin1
15:08:05 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Pulling | Pulling image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46"
15:08:06 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Started | Started container oauth-proxy
15:08:06 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Created | Created container oauth-proxy
15:11:33 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Killing | Stopping container oauth-proxy
15:11:33 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Killing | Stopping container jupyter-nb-ldap-2dadmin1
16:15:21 (x4) | rhods-notebooks | statefulset-controller | jupyter-nb-ldap-2dadmin1 | SuccessfulCreate | create Pod jupyter-nb-ldap-2dadmin1-0 in StatefulSet jupyter-nb-ldap-2dadmin1 successful
16:15:21 (x4) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | SuccessfulCreate | Reissued from statefulset/jupyter-nb-ldap-2dadmin1: create Pod jupyter-nb-ldap-2dadmin1-0 in StatefulSet jupyter-nb-ldap-2dadmin1 successful
16:15:21 | rhods-notebooks | default-scheduler | jupyter-nb-ldap-2dadmin1-0 | Scheduled | Successfully assigned rhods-notebooks/jupyter-nb-ldap-2dadmin1-0 to ip-10-0-192-209.us-east-2.compute.internal by ip-10-0-227-16
16:15:21 (x3) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | Scheduled | Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Successfully assigned rhods-notebooks/jupyter-nb-ldap-2dadmin1-0 to ip-10-0-192-209.us-east-2.compute.internal by ip-10-0-227-16
16:15:24 | rhods-notebooks | attachdetach-controller | jupyter-nb-ldap-2dadmin1-0 | SuccessfulAttachVolume | AttachVolume.Attach succeeded for volume "pvc-949d2f45-312e-4666-bad7-63724b0a3d5b"
16:15:24 (x3) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | SuccessfulAttachVolume | Reissued from pod/jupyter-nb-ldap-2dadmin1-0: AttachVolume.Attach succeeded for volume "pvc-949d2f45-312e-4666-bad7-63724b0a3d5b"
16:15:26 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Pulling | Pulling image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1"
16:15:26 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Pulled | Successfully pulled image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1" in 103.702264ms
16:15:26 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Created | Created container jupyter-nb-ldap-2dadmin1
16:15:26 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Pulling | Pulling image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46"
16:15:26 (x3) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | Pulling | Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Pulling image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1"
16:15:26 (x3) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | Created | Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Created container jupyter-nb-ldap-2dadmin1
16:15:26 (x3) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | Started | Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Started container jupyter-nb-ldap-2dadmin1
16:15:26 (x3) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | Pulling | Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Pulling image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46"
16:15:26 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Started | Started container jupyter-nb-ldap-2dadmin1
16:15:26 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | AddedInterface | Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Add eth0 [10.131.0.80/23] from ovn-kubernetes
16:15:26 | rhods-notebooks | multus | jupyter-nb-ldap-2dadmin1-0 | AddedInterface | Add eth0 [10.131.0.80/23] from ovn-kubernetes
16:15:26 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | Pulled | Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Successfully pulled image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1" in 103.702264ms
16:15:27 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Pulled | Successfully pulled image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" in 377.356654ms
16:15:27 (x3) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | Started | Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Started container oauth-proxy
16:15:27 (x3) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | Created | Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Created container oauth-proxy
16:15:27 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | Pulled | Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Successfully pulled image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" in 377.356654ms
16:15:27 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Started | Started container oauth-proxy
16:15:27 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Created | Created container oauth-proxy
16:30:58 | redhat-ods-applications | job-controller | anaconda-ce-periodic-validator-job-custom-run | SuccessfulCreate | Created pod: anaconda-ce-periodic-validator-job-custom-run-rxwf9
16:30:58 | redhat-ods-applications | default-scheduler | anaconda-ce-periodic-validator-job-custom-run-rxwf9 | Scheduled | Successfully assigned redhat-ods-applications/anaconda-ce-periodic-validator-job-custom-run-rxwf9 to ip-10-0-192-137.us-east-2.compute.internal by ip-10-0-227-16
16:31:00 | redhat-ods-applications | multus | anaconda-ce-periodic-validator-job-custom-run-rxwf9 | AddedInterface | Add eth0 [10.128.2.27/23] from ovn-kubernetes
16:32:32 (x4) | redhat-ods-applications | kubelet | anaconda-ce-periodic-validator-job-custom-run-rxwf9 | Pulling | Pulling image "quay.io/openshift/origin-deployer@sha256:257914781b9037805681530e40dc173850ca1e39ea11d3c641df6d28a334d908"
16:32:33 (x4) | redhat-ods-applications | kubelet | anaconda-ce-periodic-validator-job-custom-run-rxwf9 | Failed | Failed to pull image "quay.io/openshift/origin-deployer@sha256:257914781b9037805681530e40dc173850ca1e39ea11d3c641df6d28a334d908": rpc error: code = Unknown desc = reading manifest sha256:257914781b9037805681530e40dc173850ca1e39ea11d3c641df6d28a334d908 in quay.io/openshift/origin-deployer: manifest unknown: manifest unknown
16:32:33 (x4) | redhat-ods-applications | kubelet | anaconda-ce-periodic-validator-job-custom-run-rxwf9 | Failed | Error: ErrImagePull
16:32:43 (x6) | redhat-ods-applications | kubelet | anaconda-ce-periodic-validator-job-custom-run-rxwf9 | Failed | Error: ImagePullBackOff
16:56:09 (x110) | redhat-ods-applications | kubelet | anaconda-ce-periodic-validator-job-custom-run-rxwf9 | BackOff | Back-off pulling image "quay.io/openshift/origin-deployer@sha256:257914781b9037805681530e40dc173850ca1e39ea11d3c641df6d28a334d908"
16:56:12 (x1714) | redhat-ods-applications | kubelet | anaconda-ce-periodic-validator-job-custom-run-cmclv | BackOff | Back-off pulling image "quay.io/openshift/origin-deployer@sha256:257914781b9037805681530e40dc173850ca1e39ea11d3c641df6d28a334d908"
16:56:28 (x1369) | redhat-ods-applications | kubelet | anaconda-ce-periodic-validator-job-custom-run-m4rlw | BackOff | Back-off pulling image "quay.io/openshift/origin-deployer@sha256:257914781b9037805681530e40dc173850ca1e39ea11d3c641df6d28a334d908"
16:59:28 | redhat-ods-applications | default-scheduler | anaconda-ce-periodic-validator-job-custom-run-vnjdh | Scheduled | Successfully assigned redhat-ods-applications/anaconda-ce-periodic-validator-job-custom-run-vnjdh to ip-10-0-192-137.us-east-2.compute.internal by ip-10-0-227-16
16:59:28 | redhat-ods-applications | job-controller | anaconda-ce-periodic-validator-job-custom-run | SuccessfulCreate | Created pod: anaconda-ce-periodic-validator-job-custom-run-vnjdh
16:59:30 | redhat-ods-applications | multus | anaconda-ce-periodic-validator-job-custom-run-vnjdh | AddedInterface | Add eth0 [10.128.2.42/23] from ovn-kubernetes
16:59:46 (x2) | redhat-ods-applications | kubelet | anaconda-ce-periodic-validator-job-custom-run-vnjdh | Pulling | Pulling image "quay.io/openshift/origin-deployer@sha256:257914781b9037805681530e40dc173850ca1e39ea11d3c641df6d28a334d908"
16:59:47 (x2) | redhat-ods-applications | kubelet | anaconda-ce-periodic-validator-job-custom-run-vnjdh | Failed | Failed to pull image "quay.io/openshift/origin-deployer@sha256:257914781b9037805681530e40dc173850ca1e39ea11d3c641df6d28a334d908": rpc error: code = Unknown desc = reading manifest sha256:257914781b9037805681530e40dc173850ca1e39ea11d3c641df6d28a334d908 in quay.io/openshift/origin-deployer: manifest unknown: manifest unknown
16:59:47 (x2) | redhat-ods-applications | kubelet | anaconda-ce-periodic-validator-job-custom-run-vnjdh | Failed | Error: ErrImagePull
16:59:58 (x2) | redhat-ods-applications | kubelet | anaconda-ce-periodic-validator-job-custom-run-vnjdh | Failed | Error: ImagePullBackOff
16:59:58 (x2) | redhat-ods-applications | kubelet | anaconda-ce-periodic-validator-job-custom-run-vnjdh | BackOff | Back-off pulling image "quay.io/openshift/origin-deployer@sha256:257914781b9037805681530e40dc173850ca1e39ea11d3c641df6d28a334d908"
17:00:24 | redhat-ods-applications | job-controller | anaconda-ce-periodic-validator-job-custom-run | SuccessfulCreate | Created pod: anaconda-ce-periodic-validator-job-custom-run-n8m8t
17:00:24 | redhat-ods-applications | default-scheduler | anaconda-ce-periodic-validator-job-custom-run-n8m8t | Scheduled | Successfully assigned redhat-ods-applications/anaconda-ce-periodic-validator-job-custom-run-n8m8t to ip-10-0-192-137.us-east-2.compute.internal by ip-10-0-227-16
17:00:26 | redhat-ods-applications | kubelet | anaconda-ce-periodic-validator-job-custom-run-n8m8t | Pulling | Pulling image "quay.io/openshift/origin-deployer@sha256:baa1a8672911cd3dd71a83f91f534ba865b74027f7c1f53196100c8779d930b7"
17:00:26 | redhat-ods-applications | multus | anaconda-ce-periodic-validator-job-custom-run-n8m8t | AddedInterface | Add eth0 [10.128.2.44/23] from ovn-kubernetes
17:00:29 | redhat-ods-applications | kubelet | anaconda-ce-periodic-validator-job-custom-run-n8m8t | Pulled | Successfully pulled image "quay.io/openshift/origin-deployer@sha256:baa1a8672911cd3dd71a83f91f534ba865b74027f7c1f53196100c8779d930b7" in 2.54152794s
17:00:29 | redhat-ods-applications | kubelet | anaconda-ce-periodic-validator-job-custom-run-n8m8t | Created | Created container anaconda-ce-validator
17:00:29 | redhat-ods-applications | kubelet | anaconda-ce-periodic-validator-job-custom-run-n8m8t | Started | Started container anaconda-ce-validator
17:00:44 | redhat-ods-applications | default-scheduler | s2i-minimal-notebook-anaconda-1-build | Scheduled | Successfully assigned redhat-ods-applications/s2i-minimal-notebook-anaconda-1-build to ip-10-0-192-137.us-east-2.compute.internal by ip-10-0-227-16
17:00:46 | redhat-ods-applications | kubelet | s2i-minimal-notebook-anaconda-1-build | Started | Started container git-clone
17:00:46 | redhat-ods-applications | kubelet | s2i-minimal-notebook-anaconda-1-build | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cbbf6bff06a9897b3188b6df9705879bd99ba17ea7de8090fc68f2260451c04" already present on machine
17:00:46 | redhat-ods-applications | multus | s2i-minimal-notebook-anaconda-1-build | AddedInterface | Add eth0 [10.128.2.45/23] from ovn-kubernetes
17:00:46 | redhat-ods-applications | kubelet | s2i-minimal-notebook-anaconda-1-build | Created | Created container git-clone
17:00:47 | redhat-ods-applications | kubelet | s2i-minimal-notebook-anaconda-1-build | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cbbf6bff06a9897b3188b6df9705879bd99ba17ea7de8090fc68f2260451c04" already present on machine
17:00:47 | redhat-ods-applications | build-controller | s2i-minimal-notebook-anaconda-1 | BuildStarted | Build redhat-ods-applications/s2i-minimal-notebook-anaconda-1 is now running
17:00:48 | redhat-ods-applications | kubelet | s2i-minimal-notebook-anaconda-1-build | Created | Created container manage-dockerfile
17:00:48 | redhat-ods-applications | kubelet | s2i-minimal-notebook-anaconda-1-build | Started | Started container manage-dockerfile
17:00:48 | redhat-ods-applications | kubelet | s2i-minimal-notebook-anaconda-1-build | Pulled | Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cbbf6bff06a9897b3188b6df9705879bd99ba17ea7de8090fc68f2260451c04" already present on machine
17:00:49 | redhat-ods-applications | kubelet | s2i-minimal-notebook-anaconda-1-build | Created | Created container docker-build
17:00:49 | redhat-ods-applications | kubelet | s2i-minimal-notebook-anaconda-1-build | Started | Started container docker-build
17:00:57 | redhat-ods-applications | job-controller | anaconda-ce-periodic-validator-job-custom-run | Completed | Job completed
17:02:12 | rhods-notebooks | persistentvolume-controller | jupyterhub-nb-ldap-2dadmin20-pvc | WaitForFirstConsumer | waiting for first consumer to be created before binding
17:02:14 | rhods-notebooks | ebs.csi.aws.com_ip-10-0-157-105_9c3ab3db-c26f-4e65-bafa-db7628ecfa2b | jupyterhub-nb-ldap-2dadmin20-pvc | Provisioning | External provisioner is provisioning volume for claim "rhods-notebooks/jupyterhub-nb-ldap-2dadmin20-pvc"
17:02:14 (x2) | rhods-notebooks | persistentvolume-controller | jupyterhub-nb-ldap-2dadmin20-pvc | ExternalProvisioning | waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
17:02:14 | rhods-notebooks | statefulset-controller | jupyter-nb-ldap-2dadmin20 | SuccessfulCreate | create Pod jupyter-nb-ldap-2dadmin20-0 in StatefulSet jupyter-nb-ldap-2dadmin20 successful
17:02:14 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | SuccessfulCreate | Reissued from statefulset/jupyter-nb-ldap-2dadmin20: create Pod jupyter-nb-ldap-2dadmin20-0 in StatefulSet jupyter-nb-ldap-2dadmin20 successful
17:02:17 | rhods-notebooks | ebs.csi.aws.com_ip-10-0-157-105_9c3ab3db-c26f-4e65-bafa-db7628ecfa2b | jupyterhub-nb-ldap-2dadmin20-pvc | ProvisioningSucceeded | Successfully provisioned volume pvc-f52de66b-892b-4a98-ba27-cfc8218d1204
17:02:18 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Scheduled | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Successfully assigned rhods-notebooks/jupyter-nb-ldap-2dadmin20-0 to ip-10-0-192-137.us-east-2.compute.internal by ip-10-0-227-16
17:02:18 | rhods-notebooks | default-scheduler | jupyter-nb-ldap-2dadmin20-0 | Scheduled | Successfully assigned rhods-notebooks/jupyter-nb-ldap-2dadmin20-0 to ip-10-0-192-137.us-east-2.compute.internal by ip-10-0-227-16
17:02:20 | rhods-notebooks | attachdetach-controller | jupyter-nb-ldap-2dadmin20-0 | SuccessfulAttachVolume | AttachVolume.Attach succeeded for volume "pvc-f52de66b-892b-4a98-ba27-cfc8218d1204"
17:02:20 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | SuccessfulAttachVolume | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: AttachVolume.Attach succeeded for volume "pvc-f52de66b-892b-4a98-ba27-cfc8218d1204"
17:02:27 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | AddedInterface | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Add eth0 [10.128.2.46/23] from ovn-kubernetes
17:02:27 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Pulling | Pulling image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1"
17:02:27 | rhods-notebooks | multus | jupyter-nb-ldap-2dadmin20-0 | AddedInterface | Add eth0 [10.128.2.46/23] from ovn-kubernetes
17:02:27 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Pulled | Successfully pulled image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1" in 126.199728ms
17:02:27 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Pulled | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Successfully pulled image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1" in 126.199728ms
17:02:27 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Pulling | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Pulling image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1"
17:02:28 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Pulling | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Pulling image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46"
17:02:28 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Pulled | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Successfully pulled image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" in 408.524352ms
17:02:28 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Started | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Started container jupyter-nb-ldap-2dadmin20
17:02:28 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Created | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Created container jupyter-nb-ldap-2dadmin20
17:02:28 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Pulled | Successfully pulled image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" in 408.524352ms
17:02:28 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Started | Started container jupyter-nb-ldap-2dadmin20
17:02:28 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Created | Created container jupyter-nb-ldap-2dadmin20
17:02:28 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Pulling | Pulling image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46"
17:02:29 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Started | Started container oauth-proxy
17:02:29 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Created | Created container oauth-proxy
17:02:29 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Created | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Created container oauth-proxy
17:02:29 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Started | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Started container oauth-proxy
17:02:41 (x4) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | SuccessfulDelete | Reissued from statefulset/jupyter-nb-ldap-2dadmin1: delete Pod jupyter-nb-ldap-2dadmin1-0 in StatefulSet jupyter-nb-ldap-2dadmin1 successful
17:02:41 (x4) | rhods-notebooks | statefulset-controller | jupyter-nb-ldap-2dadmin1 | SuccessfulDelete | delete Pod jupyter-nb-ldap-2dadmin1-0 in StatefulSet jupyter-nb-ldap-2dadmin1 successful
17:02:42 (x3) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | Killing | Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Stopping container oauth-proxy
17:02:42 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Killing | Stopping container oauth-proxy
17:02:42 (x3) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | Killing | Reissued from pod/jupyter-nb-ldap-2dadmin1-0: Stopping container jupyter-nb-ldap-2dadmin1
17:02:44 (x2) | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin1-0 | Killing | Stopping container jupyter-nb-ldap-2dadmin1
17:17:59 | redhat-ods-applications | build-controller | s2i-minimal-notebook-anaconda-1 | BuildCompleted | Build redhat-ods-applications/s2i-minimal-notebook-anaconda-1 completed successfully
17:44:29 (x41) | redhat-ods-applications | kfdef-controller | rhods-model-mesh | KfDefCreationSuccessful | KfDef instance rhods-model-mesh created and deployed successfully
17:53:58 (x56) | redhat-ods-monitoring | kfdef-controller | modelmesh-monitoring | KfDefCreationSuccessful | KfDef instance modelmesh-monitoring created and deployed successfully
17:55:33 (x53) | redhat-ods-monitoring | kfdef-controller | monitoring | KfDefCreationSuccessful | KfDef instance monitoring created and deployed successfully
17:57:34 (x42) | redhat-ods-applications | kfdef-controller | rhods-anaconda | KfDefCreationSuccessful | KfDef instance rhods-anaconda created and deployed successfully
17:59:47 (x598) | rhods-notebooks | kfdef-controller | rhods-osd-config | KfDefCreationSuccessful | KfDef instance rhods-osd-config created and deployed successfully
18:00:46 (x96) | rhods-notebooks | kfdef-controller | rhods-notebooks | KfDefCreationSuccessful | KfDef instance rhods-notebooks created and deployed successfully
18:00:57 (x599) | redhat-ods-applications | kfdef-controller | rhods-dashboard | KfDefCreationSuccessful | KfDef instance rhods-dashboard created and deployed successfully
18:01:11 (x599) | redhat-ods-applications | kfdef-controller | rhods-nbc | KfDefCreationSuccessful | KfDef instance rhods-nbc created and deployed successfully
18:01:20 (x599) | redhat-ods-applications | kfdef-controller | rhods-osd-config | KfDefCreationSuccessful | KfDef instance rhods-osd-config created and deployed successfully
04:18:17 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_prometheus-b6bd4f9b-rwn9d_redhat-ods-monitoring_aaa60268-ed3d-477d-8391-581f34010a52_0(716bfb0dbbb1066b5d237c2f6b82f1dfe1a0ff896d5cfb374221c7412ea49880): error adding pod redhat-ods-monitoring_prometheus-b6bd4f9b-rwn9d to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-monitoring/prometheus-b6bd4f9b-rwn9d/aaa60268-ed3d-477d-8391-581f34010a52]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:17 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_jupyter-nb-ldap-2dadmin20-0_rhods-notebooks_b8ec74a9-ec7c-46a4-8b4a-60840b29b818_0(1c6c9daec7bfec2136fcd86890bbd991fe3af6453a51375ee96b38fdc70d4d8e): error adding pod rhods-notebooks_jupyter-nb-ldap-2dadmin20-0 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [rhods-notebooks/jupyter-nb-ldap-2dadmin20-0/b8ec74a9-ec7c-46a4-8b4a-60840b29b818]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:17 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-xg585 | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_rhods-dashboard-85bdcbdf6d-xg585_redhat-ods-applications_66123cb1-c372-484b-b0f6-0a0cddb4a8f0_0(69216883285dad407cc4958f3b4fb02a67458009dd9ccf2fde41976fdc5903dd): error adding pod redhat-ods-applications_rhods-dashboard-85bdcbdf6d-xg585 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-applications/rhods-dashboard-85bdcbdf6d-xg585/66123cb1-c372-484b-b0f6-0a0cddb4a8f0]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:17 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-gbrsp | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_modelmesh-controller-6cb56cd5f6-gbrsp_redhat-ods-applications_486ddf28-d399-4d63-b35d-7b7b2887de9f_0(d595b57014e97f735ea021815b85d9afee9861f1a01bca3bbae5e5e1ea646e0f): error adding pod redhat-ods-applications_modelmesh-controller-6cb56cd5f6-gbrsp to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-applications/modelmesh-controller-6cb56cd5f6-gbrsp/486ddf28-d399-4d63-b35d-7b7b2887de9f]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:17 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-r8zwc | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_rhods-dashboard-85bdcbdf6d-r8zwc_redhat-ods-applications_5f3d54f0-4875-4196-acee-47d5b66eac2c_0(7993f47225be45c38f8ce5b9f963e9338d3c9e6020afe0a5765d5925a52eee53): error adding pod redhat-ods-applications_rhods-dashboard-85bdcbdf6d-r8zwc to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-applications/rhods-dashboard-85bdcbdf6d-r8zwc/5f3d54f0-4875-4196-acee-47d5b66eac2c]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:17 | redhat-ods-applications | kubelet | notebook-controller-deployment-5cc87dcf7b-frgfh | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_notebook-controller-deployment-5cc87dcf7b-frgfh_redhat-ods-applications_1d6b9a7b-cbad-47f0-a727-2785fd26e337_0(870f7d518d4e0a2aed6fdd795cea0b61ecfbf2be51e5b7217b490ce39023372a): error adding pod redhat-ods-applications_notebook-controller-deployment-5cc87dcf7b-frgfh to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-applications/notebook-controller-deployment-5cc87dcf7b-frgfh/1d6b9a7b-cbad-47f0-a727-2785fd26e337]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:18 | redhat-ods-monitoring | kubelet | rhods-prometheus-operator-5c5b978c57-86wbz | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_rhods-prometheus-operator-5c5b978c57-86wbz_redhat-ods-monitoring_52699804-7998-4203-b0dc-001ad63865b5_0(25db14b8a002d28d476ab3a30b2d8a3c9ef79439a670bd65347f7e1bf2a81ce6): error adding pod redhat-ods-monitoring_rhods-prometheus-operator-5c5b978c57-86wbz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-monitoring/rhods-prometheus-operator-5c5b978c57-86wbz/52699804-7998-4203-b0dc-001ad63865b5]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:18 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-wbnf5 | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_modelmesh-controller-6cb56cd5f6-wbnf5_redhat-ods-applications_c36d5841-4dc5-4148-aee2-2497c419503b_0(c32ce367508d15381c17b01f04be2042a660dec818cf24066fb75bb31dc78c0b): error adding pod redhat-ods-applications_modelmesh-controller-6cb56cd5f6-wbnf5 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-applications/modelmesh-controller-6cb56cd5f6-wbnf5/c36d5841-4dc5-4148-aee2-2497c419503b]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:18 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-bv7gd | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_odh-model-controller-577b785c96-bv7gd_redhat-ods-applications_12320aa0-ddfa-4786-a596-fda57eb22eb4_0(cc62e1430c64e9be3d08164504119391f73a6e36e7d8270a4088174dc567480e): error adding pod redhat-ods-applications_odh-model-controller-577b785c96-bv7gd to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-applications/odh-model-controller-577b785c96-bv7gd/12320aa0-ddfa-4786-a596-fda57eb22eb4]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:18 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-wtcrp | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_odh-model-controller-577b785c96-wtcrp_redhat-ods-applications_8be57a7d-b3a2-4a76-98e0-b490eb567516_0(7d4daabe53264477c197b4a130995e4cbd344e3a24defa6d3280e2faa7071210): error adding pod redhat-ods-applications_odh-model-controller-577b785c96-wtcrp to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-applications/odh-model-controller-577b785c96-wtcrp/8be57a7d-b3a2-4a76-98e0-b490eb567516]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:18 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-0 | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_prometheus-rhods-model-monitoring-0_redhat-ods-monitoring_5419c70c-7219-47fe-af1d-56559b35d893_0(435abb319004914a0fae21f92d08a73b74f2f9d26de51ef3fd400dfce64ecdf5): error adding pod redhat-ods-monitoring_prometheus-rhods-model-monitoring-0 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-monitoring/prometheus-rhods-model-monitoring-0/5419c70c-7219-47fe-af1d-56559b35d893]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:18 | redhat-ods-operator | kubelet | addon-managed-odh-catalog-f49nx | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_addon-managed-odh-catalog-f49nx_redhat-ods-operator_58b828ae-d7ee-476e-9766-f83d63de0352_0(ed44104b8fca0a5d789d4dd7e56441cbc45d85c0fe12ff9632c8bce2aabaa4e2): error adding pod redhat-ods-operator_addon-managed-odh-catalog-f49nx to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-operator/addon-managed-odh-catalog-f49nx/58b828ae-d7ee-476e-9766-f83d63de0352]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:18 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-2 | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_prometheus-rhods-model-monitoring-2_redhat-ods-monitoring_1e5b892f-73a6-4c29-920c-9d31bfd54c5d_0(7a7bbf964df521db770f021b5751742262ffb8efb7c69021a6b9c2b6a9f9279d): error adding pod redhat-ods-monitoring_prometheus-rhods-model-monitoring-2 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-monitoring/prometheus-rhods-model-monitoring-2/1e5b892f-73a6-4c29-920c-9d31bfd54c5d]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:18 | redhat-ods-operator | kubelet | rhods-operator-58bcdb9cfd-hpvmp | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_rhods-operator-58bcdb9cfd-hpvmp_redhat-ods-operator_66ff505a-2483-48da-ac31-804a2d312ce9_0(4786f0d28a2c623974d15e80510c25fd2df26a3f89bf35a8d56f74ae70ee7252): error adding pod redhat-ods-operator_rhods-operator-58bcdb9cfd-hpvmp to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-operator/rhods-operator-58bcdb9cfd-hpvmp/66ff505a-2483-48da-ac31-804a2d312ce9]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:22 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-p7455 | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_odh-model-controller-577b785c96-p7455_redhat-ods-applications_7412e008-ebff-49a0-ae79-4b1cb67bef45_0(acee9d67aa756d3a8ed66946c4e4938e15793f1916981de1bdaae0c4e8d31b52): error adding pod redhat-ods-applications_odh-model-controller-577b785c96-p7455 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-applications/odh-model-controller-577b785c96-p7455/7412e008-ebff-49a0-ae79-4b1cb67bef45]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:22 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-kjpg9 | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_rhods-dashboard-85bdcbdf6d-kjpg9_redhat-ods-applications_dbdeb41e-78b4-472a-94e9-a7184ee6eace_0(f4fe1d39e88e625ba2d68207be2b39e82aad398bdd2225b359ccda1754c12121): error adding pod redhat-ods-applications_rhods-dashboard-85bdcbdf6d-kjpg9 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-applications/rhods-dashboard-85bdcbdf6d-kjpg9/dbdeb41e-78b4-472a-94e9-a7184ee6eace]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:23 | redhat-ods-monitoring | kubelet | blackbox-exporter-645f4478b7-nz8rz | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_blackbox-exporter-645f4478b7-nz8rz_redhat-ods-monitoring_a43bd484-1007-4662-9f6e-6c2e9aad7891_0(8aabc81fc793e6ebc5f67480b79d8af1e59f9841b629812a38d1efbe59967cd1): error adding pod redhat-ods-monitoring_blackbox-exporter-645f4478b7-nz8rz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-monitoring/blackbox-exporter-645f4478b7-nz8rz/a43bd484-1007-4662-9f6e-6c2e9aad7891]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:23 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-1 | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_prometheus-rhods-model-monitoring-1_redhat-ods-monitoring_4ea6feda-f543-4fdc-b7d4-d78eedc27cc1_0(6570fab456e95591ed0d752ef1a2cf543dc8b57691d4f4ff61f11e83a5b5f0bc): error adding pod redhat-ods-monitoring_prometheus-rhods-model-monitoring-1 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-monitoring/prometheus-rhods-model-monitoring-1/4ea6feda-f543-4fdc-b7d4-d78eedc27cc1]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:23 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-dfvgt | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_rhods-dashboard-85bdcbdf6d-dfvgt_redhat-ods-applications_02e2986b-5a04-408b-bb7f-a0513fd7d61d_0(d24b23130b1eca53351a4dd1de989d7d3fc1454fb9510bf849ac1ec9f0f9dff6): error adding pod redhat-ods-applications_rhods-dashboard-85bdcbdf6d-dfvgt to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-applications/rhods-dashboard-85bdcbdf6d-dfvgt/02e2986b-5a04-408b-bb7f-a0513fd7d61d]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:23 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin10-0 | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_jupyter-nb-ldap-2dadmin10-0_rhods-notebooks_e081b2b4-3bc3-45f7-b37e-abd98c1aa48a_0(b5f1f07664d5d63b326fe24c2d9cc6fb3a58e838fc281967b5d67cb760cb00b2): error adding pod rhods-notebooks_jupyter-nb-ldap-2dadmin10-0 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [rhods-notebooks/jupyter-nb-ldap-2dadmin10-0/e081b2b4-3bc3-45f7-b37e-abd98c1aa48a]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:23 | redhat-ods-applications | kubelet | etcd-5fbdcff5d8-kxhpz | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_etcd-5fbdcff5d8-kxhpz_redhat-ods-applications_71465682-27b4-4285-9453-7469d554abc3_0(c4ebce205ddcf26eeaccf3d7fa7d74344075556fb833af9674c599e5643f352b): error adding pod redhat-ods-applications_etcd-5fbdcff5d8-kxhpz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-applications/etcd-5fbdcff5d8-kxhpz/71465682-27b4-4285-9453-7469d554abc3]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:23 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-djnj5 | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_modelmesh-controller-6cb56cd5f6-djnj5_redhat-ods-applications_89879cc0-b1c4-4223-a420-192915e0d4b9_0(246c8f7d2b467f4b4022999a5c21fb65a042372fbc9eb2134777d275780fe4c1): error adding pod redhat-ods-applications_modelmesh-controller-6cb56cd5f6-djnj5 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-applications/modelmesh-controller-6cb56cd5f6-djnj5/89879cc0-b1c4-4223-a420-192915e0d4b9]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:23 | redhat-ods-applications | kubelet | odh-notebook-controller-manager-7d744469-l6k69 | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_odh-notebook-controller-manager-7d744469-l6k69_redhat-ods-applications_042a7929-86f0-44a7-893a-ce676d38b1ac_0(71b1ba331f046264796f8289444798533b81a5acea8081722ee9a51fd7ab1105): error adding pod redhat-ods-applications_odh-notebook-controller-manager-7d744469-l6k69 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-applications/odh-notebook-controller-manager-7d744469-l6k69/042a7929-86f0-44a7-893a-ce676d38b1ac]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:23 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-b4lms | FailedCreatePodSandBox | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_rhods-dashboard-85bdcbdf6d-b4lms_redhat-ods-applications_c46a0365-0942-4eb0-8115-68cdc0e101dc_0(3458ba8e8e1b134e62803c5eeee9698eb58a160289e5a4a2e14bc3cb2e62c54a): error adding pod redhat-ods-applications_rhods-dashboard-85bdcbdf6d-b4lms to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [redhat-ods-applications/rhods-dashboard-85bdcbdf6d-b4lms/c46a0365-0942-4eb0-8115-68cdc0e101dc]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:31 | redhat-ods-operator | multus | rhods-operator-58bcdb9cfd-hpvmp | AddedInterface | Add eth0 [10.128.2.28/23] from ovn-kubernetes
04:18:31 | redhat-ods-applications | multus | rhods-dashboard-85bdcbdf6d-xg585 | AddedInterface | Add eth0 [10.128.2.31/23] from ovn-kubernetes
04:18:31 | redhat-ods-monitoring | multus | prometheus-b6bd4f9b-rwn9d | AddedInterface | Add eth0 [10.128.2.29/23] from ovn-kubernetes
04:18:31 | redhat-ods-applications | multus | notebook-controller-deployment-5cc87dcf7b-frgfh | AddedInterface | Add eth0 [10.128.2.33/23] from ovn-kubernetes
04:18:31 | redhat-ods-applications | kubelet | notebook-controller-deployment-5cc87dcf7b-frgfh | Pulling | Pulling image "registry.redhat.io/rhods/odh-kf-notebook-controller-rhel8@sha256:9bd9c2041095683c58b4a2d7494b9dbfead979e22b46aed169c6cc914fc6d472"
04:18:31 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Pulled | Container image "registry.access.redhat.com/ubi8/ubi-minimal@sha256:31ccb79b1b2c2d6eff1bee0db23d5b8ab598eafd6238417d9813f1346f717c11" already present on machine
04:18:31 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Created | Created container wait-for-deployment
04:18:31 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Started | Started container wait-for-deployment
04:18:31 | rhods-notebooks | multus | jupyter-nb-ldap-2dadmin20-0 | AddedInterface | Add eth0 [10.128.2.46/23] from ovn-kubernetes
04:18:32 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-xg585 | Pulling | Pulling image "registry.redhat.io/rhods/odh-dashboard-rhel8@sha256:ab9d65f0604c5d4f9610a1669c6fcbc9262fb4513c8e28611bba356cdeeeef4b"
04:18:33 | redhat-ods-applications | kubelet | notebook-controller-deployment-5cc87dcf7b-frgfh | Pulled | Successfully pulled image "registry.redhat.io/rhods/odh-kf-notebook-controller-rhel8@sha256:9bd9c2041095683c58b4a2d7494b9dbfead979e22b46aed169c6cc914fc6d472" in 2.044555132s
04:18:33 | redhat-ods-applications | multus | modelmesh-controller-6cb56cd5f6-gbrsp | AddedInterface | Add eth0 [10.128.2.36/23] from ovn-kubernetes
04:18:33 | redhat-ods-operator | multus | addon-managed-odh-catalog-f49nx | AddedInterface | Add eth0 [10.128.2.25/23] from ovn-kubernetes
04:18:33 | redhat-ods-applications | kubelet | notebook-controller-deployment-5cc87dcf7b-frgfh | Started | Started container manager
04:18:33 | redhat-ods-applications | kubelet | notebook-controller-deployment-5cc87dcf7b-frgfh | Created | Created container manager
04:18:34 | redhat-ods-applications | multus | odh-model-controller-577b785c96-bv7gd | AddedInterface | Add eth0 [10.128.2.39/23] from ovn-kubernetes
04:18:34 | redhat-ods-operator | kubelet | rhods-operator-58bcdb9cfd-hpvmp | Pulled | Successfully pulled image "registry.redhat.io/rhods/odh-deployer-rhel8@sha256:acdec11cb738709b95fda839c04799355a8fe8ba45533ae82c8f8dccd4944437" in 2.915172269s
04:18:34 | redhat-ods-applications | multus | rhods-dashboard-85bdcbdf6d-r8zwc | AddedInterface | Add eth0 [10.128.2.30/23] from ovn-kubernetes
04:18:34 | redhat-ods-applications | multus | modelmesh-controller-6cb56cd5f6-wbnf5 | AddedInterface | Add eth0 [10.128.2.37/23] from ovn-kubernetes
04:18:34 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-bv7gd | Pulling | Pulling image "registry.redhat.io/rhods/odh-model-controller-rhel8@sha256:7bb4b593dd199fcc5dd86130872a23d96b2df14a204ba0772ead6eb9ad100f97"
04:18:34 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-2 | Pulled | Container image "registry.redhat.io/openshift4/ose-prometheus-config-reloader:v4.11.0-202211072116.p0.gcb3afa2.assembly.stream" already present on machine
04:18:34 | redhat-ods-monitoring | multus | prometheus-rhods-model-monitoring-0 | AddedInterface | Add eth0 [10.128.2.34/23] from ovn-kubernetes
04:18:34 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-gbrsp | Pulled | Container image "quay.io/opendatahub/modelmesh-controller:v0.9.3-auth" already present on machine
04:18:34 | redhat-ods-monitoring | multus | prometheus-rhods-model-monitoring-2 | AddedInterface | Add eth0 [10.128.2.35/23] from ovn-kubernetes
04:18:35 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-0 | Pulled | Container image "registry.redhat.io/openshift4/ose-prometheus-config-reloader:v4.11.0-202211072116.p0.gcb3afa2.assembly.stream" already present on machine
04:18:35 | redhat-ods-operator | kubelet | addon-managed-odh-catalog-f49nx | Pulled | Successfully pulled image "brew.registry.redhat.io/rh-osbs/iib:389297" in 376.824143ms
04:18:35 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-xg585 | Pulled | Successfully pulled image "registry.redhat.io/rhods/odh-dashboard-rhel8@sha256:ab9d65f0604c5d4f9610a1669c6fcbc9262fb4513c8e28611bba356cdeeeef4b" in 3.420739097s
04:18:35 | redhat-ods-operator | kubelet | addon-managed-odh-catalog-f49nx | Pulling | Pulling image "brew.registry.redhat.io/rh-osbs/iib:389297"
04:18:35 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-r8zwc | Pulling | Pulling image "registry.redhat.io/rhods/odh-dashboard-rhel8@sha256:ab9d65f0604c5d4f9610a1669c6fcbc9262fb4513c8e28611bba356cdeeeef4b"
04:18:35 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-wbnf5 | Pulled | Container image "quay.io/opendatahub/modelmesh-controller:v0.9.3-auth" already present on machine
04:18:35 | redhat-ods-monitoring | multus | rhods-prometheus-operator-5c5b978c57-86wbz | AddedInterface | Add eth0 [10.128.2.32/23] from ovn-kubernetes
04:18:36 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-wbnf5 | Created | Created container manager
04:18:36 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-gbrsp | Started | Started container manager
04:18:36 | redhat-ods-operator | kubelet | addon-managed-odh-catalog-f49nx | Created | Created container registry-server
04:18:36 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-2 | Created | Created container init-config-reloader
04:18:36 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-0 | Created | Created container init-config-reloader
04:18:36 | redhat-ods-applications | multus | odh-model-controller-577b785c96-wtcrp | AddedInterface | Add eth0 [10.128.2.38/23] from ovn-kubernetes
04:18:36 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-gbrsp | Created | Created container manager
04:18:36 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-wbnf5 | Started | Started container manager
04:18:36 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-r8zwc | Pulled | Successfully pulled image "registry.redhat.io/rhods/odh-dashboard-rhel8@sha256:ab9d65f0604c5d4f9610a1669c6fcbc9262fb4513c8e28611bba356cdeeeef4b" in 378.742444ms
04:18:36 | redhat-ods-applications | multus | rhods-dashboard-85bdcbdf6d-b4lms | AddedInterface | Add eth0 [10.131.0.11/23] from ovn-kubernetes
04:18:37 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-0 | Started | Started container init-config-reloader
04:18:37 | redhat-ods-applications | multus | etcd-5fbdcff5d8-kxhpz | AddedInterface | Add eth0 [10.131.0.14/23] from ovn-kubernetes
04:18:37 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-2 | Started | Started container init-config-reloader
04:18:37 | redhat-ods-operator | kubelet | addon-managed-odh-catalog-f49nx | Started | Started container registry-server
04:18:38 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-0 | Pulled | Container image "registry.redhat.io/openshift4/ose-prometheus:v4.11.0-202211072116.p0.ge751c61.assembly.stream" already present on machine
04:18:38 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-r8zwc | Created | Created container rhods-dashboard
04:18:38 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-xg585 | Created | Created container rhods-dashboard
04:18:38 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-bv7gd | Pulled | Successfully pulled image "registry.redhat.io/rhods/odh-model-controller-rhel8@sha256:7bb4b593dd199fcc5dd86130872a23d96b2df14a204ba0772ead6eb9ad100f97" in 3.068500075s
04:18:38 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-2 | Pulled | Container image "registry.redhat.io/openshift4/ose-prometheus:v4.11.0-202211072116.p0.ge751c61.assembly.stream" already present on machine
04:18:39 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-r8zwc | Started | Started container rhods-dashboard
04:18:39 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-r8zwc | Pulled | Container image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" already present on machine
04:18:39 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-xg585 | Started | Started container rhods-dashboard
04:18:39 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-xg585 | Pulled | Container image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" already present on machine
04:18:40 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-1 | Pulled | Container image "registry.redhat.io/openshift4/ose-prometheus-config-reloader:v4.11.0-202211072116.p0.gcb3afa2.assembly.stream" already present on machine
04:18:40 | redhat-ods-applications | kubelet | etcd-5fbdcff5d8-kxhpz | Pulled | Container image "quay.io/modh/etcd-container:3.2.32-rhel7.9-20221017084818" already present on machine
04:18:40 | redhat-ods-monitoring | kubelet | rhods-prometheus-operator-5c5b978c57-86wbz | Pulled | Container image "registry.redhat.io/openshift4/ose-prometheus-operator:v4.11.0-202211072116.p0.gcb3afa2.assembly.stream" already present on machine
04:18:40 | redhat-ods-monitoring | multus | prometheus-rhods-model-monitoring-1 | AddedInterface | Add eth0 [10.131.0.13/23] from ovn-kubernetes
04:18:40 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-b4lms | Pulling | Pulling image "registry.redhat.io/rhods/odh-dashboard-rhel8@sha256:ab9d65f0604c5d4f9610a1669c6fcbc9262fb4513c8e28611bba356cdeeeef4b"
04:18:40 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-wtcrp | Pulling | Pulling image "registry.redhat.io/rhods/odh-model-controller-rhel8@sha256:7bb4b593dd199fcc5dd86130872a23d96b2df14a204ba0772ead6eb9ad100f97"
04:18:40 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-b4lms | Pulled | Successfully pulled image "registry.redhat.io/rhods/odh-dashboard-rhel8@sha256:ab9d65f0604c5d4f9610a1669c6fcbc9262fb4513c8e28611bba356cdeeeef4b" in 465.675738ms
04:18:41 | redhat-ods-applications | multus | odh-notebook-controller-manager-7d744469-l6k69 | AddedInterface | Add eth0 [10.131.0.12/23] from ovn-kubernetes
04:18:41 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-wtcrp | Pulled | Successfully pulled image "registry.redhat.io/rhods/odh-model-controller-rhel8@sha256:7bb4b593dd199fcc5dd86130872a23d96b2df14a204ba0772ead6eb9ad100f97" in 386.160135ms
04:18:41 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-1 | Started | Started container init-config-reloader
04:18:41 | redhat-ods-applications | kubelet | notebook-controller-deployment-5cc87dcf7b-frgfh | Unhealthy | Readiness probe failed: Get "http://10.128.2.33:8081/readyz": dial tcp 10.128.2.33:8081: connect: connection refused
04:18:41 | redhat-ods-applications | kubelet | etcd-5fbdcff5d8-kxhpz | Created | Created container etcd
04:18:41 | redhat-ods-monitoring | multus | blackbox-exporter-645f4478b7-nz8rz | AddedInterface | Add eth0 [10.131.0.5/23] from ovn-kubernetes
04:18:41 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-bv7gd | Created | Created container manager
04:18:41 | rhods-notebooks | multus | jupyter-nb-ldap-2dadmin10-0 | AddedInterface | Add eth0 [10.131.0.6/23] from ovn-kubernetes
04:18:41 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-b4lms | Created | Created container rhods-dashboard
04:18:41 | redhat-ods-applications | kubelet | etcd-5fbdcff5d8-kxhpz | Started | Started container etcd
04:18:41 | redhat-ods-applications | kubelet | notebook-controller-deployment-5cc87dcf7b-frgfh | Unhealthy | Liveness probe failed: Get "http://10.128.2.33:8081/healthz": dial tcp 10.128.2.33:8081: connect: connection refused
04:18:41 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-1 | Created | Created container init-config-reloader
04:18:42 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-bv7gd | Started | Started container manager
04:18:42 | redhat-ods-applications | multus | odh-model-controller-577b785c96-p7455 | AddedInterface | Add eth0 [10.131.0.16/23] from ovn-kubernetes
04:18:42 | redhat-ods-applications | multus | rhods-dashboard-85bdcbdf6d-dfvgt | AddedInterface | Add eth0 [10.131.0.10/23] from ovn-kubernetes
04:18:42 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-b4lms | Started | Started container rhods-dashboard
04:18:42 | redhat-ods-applications | multus | modelmesh-controller-6cb56cd5f6-djnj5 | AddedInterface | Add eth0 [10.131.0.15/23] from ovn-kubernetes
04:18:42 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-b4lms | Pulled | Container image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" already present on machine
04:18:42 | redhat-ods-applications | multus | rhods-dashboard-85bdcbdf6d-kjpg9 | AddedInterface | Add eth0 [10.131.0.7/23] from ovn-kubernetes
04:18:43 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-1 | Created | Created container prometheus
04:18:43 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-1 | Started | Started container prometheus
04:18:43 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-1 | Pulled | Container image "registry.redhat.io/openshift4/ose-prometheus:v4.11.0-202211072116.p0.ge751c61.assembly.stream" already present on machine
04:18:43 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-1 | Pulled | Container image "registry.redhat.io/openshift4/ose-prometheus-config-reloader:v4.11.0-202211072116.p0.gcb3afa2.assembly.stream" already present on machine
04:18:43 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-0 | Created | Created container prometheus
04:18:43 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-0 | Started | Started container prometheus
04:18:43 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-2 | Created | Created container prometheus
04:18:43 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-2 | Pulled | Container image "registry.redhat.io/openshift4/ose-prometheus-config-reloader:v4.11.0-202211072116.p0.gcb3afa2.assembly.stream" already present on machine
04:18:43 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-0 | Pulled | Container image "registry.redhat.io/openshift4/ose-prometheus-config-reloader:v4.11.0-202211072116.p0.gcb3afa2.assembly.stream" already present on machine
04:18:43 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-2 | Started | Started container prometheus
04:18:44 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-xg585 | Created | Created container oauth-proxy
04:18:44 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-xg585 | Started | Started container oauth-proxy
04:18:44 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-wtcrp | Created | Created container manager
04:18:44 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-r8zwc | Created | Created container oauth-proxy
04:18:44 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-wtcrp | Started | Started container manager
04:18:44 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-b4lms | Started | Started container oauth-proxy
04:18:44 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-b4lms | Created | Created container oauth-proxy
04:18:44 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-r8zwc | Started | Started container oauth-proxy
04:18:45 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-1 | Created | Created container config-reloader
04:18:45 | redhat-ods-monitoring | kubelet | blackbox-exporter-645f4478b7-nz8rz | Pulled | Container image "registry.access.redhat.com/ubi8/ubi-minimal@sha256:31ccb79b1b2c2d6eff1bee0db23d5b8ab598eafd6238417d9813f1346f717c11" already present on machine
04:18:45 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-kjpg9 | Pulling | Pulling image "registry.redhat.io/rhods/odh-dashboard-rhel8@sha256:ab9d65f0604c5d4f9610a1669c6fcbc9262fb4513c8e28611bba356cdeeeef4b"
04:18:45 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-1 | Started | Started container config-reloader
04:18:45 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-1 | Pulling | Pulling image "registry.redhat.io/openshift4/ose-oauth-proxy:v4.8"
04:18:45 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-0 | Pulling | Pulling image "registry.redhat.io/openshift4/ose-oauth-proxy:v4.8"
04:18:45 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-0 | Started | Started container config-reloader
04:18:45 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-0 | Created | Created container config-reloader
04:18:45 | redhat-ods-monitoring | kubelet | rhods-prometheus-operator-5c5b978c57-86wbz | Started | Started container prometheus-operator
04:18:45 | redhat-ods-monitoring | kubelet | rhods-prometheus-operator-5c5b978c57-86wbz | Created | Created container prometheus-operator
04:18:45 | redhat-ods-applications | kubelet | odh-notebook-controller-manager-7d744469-l6k69 | Pulling | Pulling image "registry.redhat.io/rhods/odh-notebook-controller-rhel8@sha256:0eef244e3668fae16db521d2e4a0efe15da17a3d6680bab6aca97a03ed01bcc8"
04:18:46 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-2 | Pulling | Pulling image "registry.redhat.io/openshift4/ose-oauth-proxy:v4.8"
04:18:46 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-p7455 | Pulling | Pulling image "registry.redhat.io/rhods/odh-model-controller-rhel8@sha256:7bb4b593dd199fcc5dd86130872a23d96b2df14a204ba0772ead6eb9ad100f97"
04:18:46 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin10-0 | Pulling | Pulling image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1"
04:18:46 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-2 | Pulled | Successfully pulled image "registry.redhat.io/openshift4/ose-oauth-proxy:v4.8" in 682.34278ms
04:18:46 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-djnj5 | Pulled | Container image "quay.io/opendatahub/modelmesh-controller:v0.9.3-auth" already present on machine
04:18:46 | redhat-ods-monitoring | kubelet | blackbox-exporter-645f4478b7-nz8rz | Started | Started container wait-for-deployment
04:18:46 | redhat-ods-monitoring | kubelet | blackbox-exporter-645f4478b7-nz8rz | Created | Created container wait-for-deployment
04:18:46 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-kjpg9 | Pulled | Successfully pulled image "registry.redhat.io/rhods/odh-dashboard-rhel8@sha256:ab9d65f0604c5d4f9610a1669c6fcbc9262fb4513c8e28611bba356cdeeeef4b" in 413.531012ms
04:18:46 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-2 | Started | Started container config-reloader
04:18:46 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-2 | Created | Created container config-reloader
04:18:46 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-0 | Created | Created container oauth-proxy
04:18:46 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-dfvgt | Pulled | Successfully pulled image "registry.redhat.io/rhods/odh-dashboard-rhel8@sha256:ab9d65f0604c5d4f9610a1669c6fcbc9262fb4513c8e28611bba356cdeeeef4b" in 432.162337ms
04:18:46 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-0 | Started | Started container oauth-proxy
04:18:46 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-1 | Pulled | Successfully pulled image "registry.redhat.io/openshift4/ose-oauth-proxy:v4.8" in 768.777439ms
04:18:46 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-dfvgt | Pulling | Pulling image "registry.redhat.io/rhods/odh-dashboard-rhel8@sha256:ab9d65f0604c5d4f9610a1669c6fcbc9262fb4513c8e28611bba356cdeeeef4b"
04:18:46 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-p7455 | Pulled | Successfully pulled image "registry.redhat.io/rhods/odh-model-controller-rhel8@sha256:7bb4b593dd199fcc5dd86130872a23d96b2df14a204ba0772ead6eb9ad100f97" in 415.127621ms
04:18:46 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-0 | Pulled | Successfully pulled image "registry.redhat.io/openshift4/ose-oauth-proxy:v4.8" in 820.07731ms
04:18:47 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Pulled | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Successfully pulled image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1" in 126.199728ms
04:18:47 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | SuccessfulCreate | Reissued from statefulset/jupyter-nb-ldap-2dadmin20: create Pod jupyter-nb-ldap-2dadmin20-0 in StatefulSet jupyter-nb-ldap-2dadmin20 successful
04:18:47 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | FailedCreatePodSandBox | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_jupyter-nb-ldap-2dadmin20-0_rhods-notebooks_b8ec74a9-ec7c-46a4-8b4a-60840b29b818_0(1c6c9daec7bfec2136fcd86890bbd991fe3af6453a51375ee96b38fdc70d4d8e): error adding pod rhods-notebooks_jupyter-nb-ldap-2dadmin20-0 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [rhods-notebooks/jupyter-nb-ldap-2dadmin20-0/b8ec74a9-ec7c-46a4-8b4a-60840b29b818]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:47 (x2) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | AddedInterface | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Add eth0 [10.128.2.46/23] from ovn-kubernetes
04:18:47 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Pulled | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Successfully pulled image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" in 408.524352ms
04:18:47 (x2) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Pulling | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Pulling image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1"
04:18:47 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin10 | AddedInterface | Reissued from pod/jupyter-nb-ldap-2dadmin10-0: Add eth0 [10.131.0.6/23] from ovn-kubernetes
04:18:47 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Started | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Started container jupyter-nb-ldap-2dadmin20
04:18:47 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | SuccessfulDelete | Reissued from statefulset/jupyter-nb-ldap-2dadmin1: delete Pod jupyter-nb-ldap-2dadmin1-0 in StatefulSet jupyter-nb-ldap-2dadmin1 successful
04:18:47 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Created | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Created container jupyter-nb-ldap-2dadmin20
04:18:47 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | SuccessfulAttachVolume | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: AttachVolume.Attach succeeded for volume "pvc-f52de66b-892b-4a98-ba27-cfc8218d1204"
04:18:47 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-2 | Created | Created container oauth-proxy
04:18:47 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin10 | FailedCreatePodSandBox | Reissued from pod/jupyter-nb-ldap-2dadmin10-0: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_jupyter-nb-ldap-2dadmin10-0_rhods-notebooks_e081b2b4-3bc3-45f7-b37e-abd98c1aa48a_0(b5f1f07664d5d63b326fe24c2d9cc6fb3a58e838fc281967b5d67cb760cb00b2): error adding pod rhods-notebooks_jupyter-nb-ldap-2dadmin10-0 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [rhods-notebooks/jupyter-nb-ldap-2dadmin10-0/e081b2b4-3bc3-45f7-b37e-abd98c1aa48a]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
04:18:47 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-2 | Started | Started container oauth-proxy
04:18:47 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin1 | SuccessfulCreate | Reissued from statefulset/jupyter-nb-ldap-2dadmin1: create Pod jupyter-nb-ldap-2dadmin1-0 in StatefulSet jupyter-nb-ldap-2dadmin1 successful
04:18:47 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Scheduled | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Successfully assigned rhods-notebooks/jupyter-nb-ldap-2dadmin20-0 to ip-10-0-192-137.us-east-2.compute.internal by ip-10-0-227-16
04:18:48 | redhat-ods-applications | kubelet | odh-notebook-controller-manager-7d744469-l6k69 | Pulled | Successfully pulled image "registry.redhat.io/rhods/odh-notebook-controller-rhel8@sha256:0eef244e3668fae16db521d2e4a0efe15da17a3d6680bab6aca97a03ed01bcc8" in 2.905759136s
04:18:50 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-kjpg9 | Pulled | Container image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" already present on machine
04:18:50 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-p7455 | Created | Created container manager
04:18:50 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-djnj5 | Created | Created container manager
04:18:50 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-1 | Created | Created container oauth-proxy
04:18:50 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-kjpg9 | Created | Created container rhods-dashboard
04:18:50 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-djnj5 | Started | Started container manager
04:18:50 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-dfvgt | Created | Created container rhods-dashboard
04:18:50 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-kjpg9 | Started | Started container rhods-dashboard
04:18:50 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-bv7gd | Unhealthy | Readiness probe failed: Get "http://10.128.2.39:8081/readyz": dial tcp 10.128.2.39:8081: connect: connection refused
04:18:51 (x2) | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-wbnf5 | Unhealthy | Readiness probe failed: Get "http://10.128.2.37:8081/readyz": dial tcp 10.128.2.37:8081: connect: connection refused
04:18:51 (x2) | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-gbrsp | Unhealthy | Readiness probe failed: Get "http://10.128.2.36:8081/readyz": dial tcp 10.128.2.36:8081: connect: connection refused
04:18:51 | redhat-ods-applications | kubelet | odh-notebook-controller-manager-7d744469-l6k69 | Created | Created container manager
04:18:51 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-gbrsp | Unhealthy | Liveness probe failed: Get "http://10.128.2.36:8081/healthz": dial tcp 10.128.2.36:8081: connect: connection refused
04:18:51 (x2) | redhat-ods-operator | kubelet | rhods-operator-58bcdb9cfd-hpvmp | Started | Started container rhods-deployer
04:18:51 | redhat-ods-operator | kubelet | rhods-operator-58bcdb9cfd-hpvmp | Pulled | Successfully pulled image "registry.redhat.io/rhods/odh-deployer-rhel8@sha256:acdec11cb738709b95fda839c04799355a8fe8ba45533ae82c8f8dccd4944437" in 447.081312ms
04:18:51 (x2) | redhat-ods-operator | kubelet | rhods-operator-58bcdb9cfd-hpvmp | Pulling | Pulling image "registry.redhat.io/rhods/odh-deployer-rhel8@sha256:acdec11cb738709b95fda839c04799355a8fe8ba45533ae82c8f8dccd4944437"
04:18:51 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-wtcrp | Unhealthy | Readiness probe failed: Get "http://10.128.2.38:8081/readyz": dial tcp 10.128.2.38:8081: connect: connection refused
04:18:51 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-wbnf5 | Unhealthy | Liveness probe failed: Get "http://10.128.2.37:8081/healthz": dial tcp 10.128.2.37:8081: connect: connection refused
04:18:51 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-dfvgt | Started | Started container rhods-dashboard
04:18:51 (x2) | redhat-ods-operator | kubelet | rhods-operator-58bcdb9cfd-hpvmp | Created | Created container rhods-deployer
04:18:51 | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-p7455 | Started | Started container manager
04:18:52 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-dfvgt | Pulled | Container image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" already present on machine
04:18:52 | redhat-ods-monitoring | kubelet | prometheus-rhods-model-monitoring-1 | Started | Started container oauth-proxy
04:18:53 | redhat-ods-applications | kubelet | odh-notebook-controller-manager-7d744469-l6k69 | Started | Started container manager
04:18:53 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin10-0 | Pulled | Successfully pulled image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1" in 7.84148762s
04:18:54 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin10 | Pulling | Reissued from pod/jupyter-nb-ldap-2dadmin10-0: Pulling image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1"
04:18:54 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin10-0 | Created | Created container jupyter-nb-ldap-2dadmin10
04:18:54 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-kjpg9 | Created | Created container oauth-proxy
04:18:54 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-kjpg9 | Started | Started container oauth-proxy
04:18:55 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin10-0 | Pulled | Successfully pulled image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" in 829.692707ms
04:18:55 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-dfvgt | Started | Started container oauth-proxy
04:18:55 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin10-0 | Started | Started container jupyter-nb-ldap-2dadmin10
04:18:55 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-dfvgt | Created | Created container oauth-proxy
04:18:55 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin10-0 | Pulling | Pulling image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46"
04:18:56 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin10-0 | Created | Created container oauth-proxy
04:18:56 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin10-0 | Started | Started container oauth-proxy
04:18:58 | redhat-ods-applications | kubelet | odh-notebook-controller-manager-7d744469-l6k69 | Unhealthy | Readiness probe failed: Get "http://10.131.0.12:8081/readyz": dial tcp 10.131.0.12:8081: connect: connection refused
04:18:58 | redhat-ods-operator | operator-lifecycle-manager | rhods-operator.1.20.0-11 | ComponentUnhealthy | installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability.
04:18:58 (x2) | redhat-ods-applications | kubelet | odh-model-controller-577b785c96-p7455 | Unhealthy | Readiness probe failed: Get "http://10.131.0.16:8081/readyz": dial tcp 10.131.0.16:8081: connect: connection refused
04:19:01 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Failed | Failed to pull image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1": rpc error: code = Unknown desc = pinging container registry image-registry.openshift-image-registry.svc:5000: Get "https://image-registry.openshift-image-registry.svc:5000/v2/": dial tcp 172.30.202.150:5000: i/o timeout
04:19:01 | redhat-ods-operator | operator-lifecycle-manager | rhods-operator.1.20.0-11 | NeedsReinstall | installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability.
04:19:01 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Pulling | Pulling image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46"
04:19:01 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Failed | Error: ErrImagePull
04:19:02 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-gbrsp | Unhealthy | Liveness probe failed: Get "http://10.128.2.36:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
04:19:02 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-wbnf5 | Unhealthy | Liveness probe failed: Get "http://10.128.2.37:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
04:19:02 (x2) | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-gbrsp | Unhealthy | Readiness probe failed: Get "http://10.128.2.36:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
04:19:02 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Pulled | Successfully pulled image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" in 692.245755ms
04:19:02 (x2) | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-wbnf5 | Unhealthy | Readiness probe failed: Get "http://10.128.2.37:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
04:19:02 | redhat-ods-operator | operator-lifecycle-manager | rhods-operator.1.20.0-11 | AllRequirementsMet | all requirements found, attempting install
04:19:03 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Started | Started container oauth-proxy
04:19:03 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Created | Created container oauth-proxy
04:19:03 | redhat-ods-operator | operator-lifecycle-manager | rhods-operator.1.20.0-11 | InstallSucceeded | waiting for install components to report healthy
04:19:04 | redhat-ods-operator | operator-lifecycle-manager | rhods-operator.1.20.0-11 | InstallWaiting | installing: waiting for deployment rhods-operator to become ready: deployment "rhods-operator" not available: Deployment does not have minimum availability.
04:19:06 | redhat-ods-applications | odh-model-controller-577b785c96-bv7gd_0528cea3-0179-4664-84c2-07a00ef10f71 | odh-model-controller | LeaderElection | odh-model-controller-577b785c96-bv7gd_0528cea3-0179-4664-84c2-07a00ef10f71 became leader
04:19:07 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin10 | Pulled | Reissued from pod/jupyter-nb-ldap-2dadmin10-0: Successfully pulled image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1" in 7.84148762s
04:19:08 (x2) | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-djnj5 | Unhealthy | Readiness probe failed: Get "http://10.131.0.15:8081/readyz": dial tcp 10.131.0.15:8081: connect: connection refused
04:19:08 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Failed | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Failed to pull image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1": rpc error: code = Unknown desc = pinging container registry image-registry.openshift-image-registry.svc:5000: Get "https://image-registry.openshift-image-registry.svc:5000/v2/": dial tcp 172.30.202.150:5000: i/o timeout
04:19:08 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-djnj5 | Unhealthy | Liveness probe failed: Get "http://10.131.0.15:8081/healthz": dial tcp 10.131.0.15:8081: connect: connection refused
04:19:09 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin10 | Created | Reissued from pod/jupyter-nb-ldap-2dadmin10-0: Created container jupyter-nb-ldap-2dadmin10
04:19:09 (x4) | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | BackOff | Back-off pulling image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1"
04:19:09 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Failed | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Error: ErrImagePull
04:19:09 (x4) | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Failed | Error: ImagePullBackOff
04:19:09 (x2) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Pulling | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Pulling image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46"
04:19:10 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-xg585 | Unhealthy | Liveness probe failed: dial tcp 10.128.2.31:8080: connect: connection refused
04:19:10 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Pulled | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Successfully pulled image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" in 692.245755ms
04:19:10 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-xg585 | Unhealthy | Readiness probe failed: Get "http://10.128.2.31:8080/api/health": dial tcp 10.128.2.31:8080: connect: connection refused
04:19:10 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin10 | Pulling | Reissued from pod/jupyter-nb-ldap-2dadmin10-0: Pulling image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46"
04:19:10 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin10 | Started | Reissued from pod/jupyter-nb-ldap-2dadmin10-0: Started container jupyter-nb-ldap-2dadmin10
04:19:11 (x2) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Created | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Created container oauth-proxy
04:19:11 (x2) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Started | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Started container oauth-proxy
04:19:12 (x2) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | BackOff | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Back-off pulling image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1"
04:19:12 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin10 | Pulled | Reissued from pod/jupyter-nb-ldap-2dadmin10-0: Successfully pulled image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" in 829.692707ms
04:19:12 (x2) | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin20 | Failed | Reissued from pod/jupyter-nb-ldap-2dadmin20-0: Error: ImagePullBackOff
04:19:12 (x4) | redhat-ods-operator | kubelet | addon-managed-odh-catalog-f49nx | Unhealthy | Startup probe failed: timeout: failed to connect service ":50051" within 1s
04:19:13 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin10 | Started | Reissued from pod/jupyter-nb-ldap-2dadmin10-0: Started container oauth-proxy
04:19:13 | rhods-notebooks | notebook-controller | jupyter-nb-ldap-2dadmin10 | Created | Reissued from pod/jupyter-nb-ldap-2dadmin10-0: Created container oauth-proxy
04:19:14 | redhat-ods-applications | kubelet | modelmesh-controller-6cb56cd5f6-djnj5 | Unhealthy | Readiness probe failed: Get "http://10.131.0.15:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
04:19:15 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Pulling | Pulling image "registry.redhat.io/openshift4/ose-prometheus@sha256:62c89e82fdf9639eace286d11c91e6d1c3fdd3437e081ed9dce6f5ea783c660f"
04:19:15 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Started | Started container oauth-proxy
04:19:15 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Created | Created container oauth-proxy
04:19:15 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Pulled | Container image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" already present on machine
04:19:17 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Created | Created container prometheus
04:19:17 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Started | Started container prometheus
04:19:17 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Pulled | Successfully pulled image "registry.redhat.io/openshift4/ose-prometheus@sha256:62c89e82fdf9639eace286d11c91e6d1c3fdd3437e081ed9dce6f5ea783c660f" in 1.411411448s
04:19:17 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Pulling | Pulling image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46"
04:19:18 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-b4lms | Unhealthy | Readiness probe failed: Get "http://10.131.0.11:8080/api/health": dial tcp 10.131.0.11:8080: connect: connection refused
04:19:18 | redhat-ods-applications | kubelet | rhods-dashboard-85bdcbdf6d-b4lms | Unhealthy | Liveness probe failed: dial tcp 10.131.0.11:8080: connect: connection refused
04:19:18 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Created | Created container alertmanager-proxy
04:19:18 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Pulled | Successfully pulled image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" in 1.071365496s
04:19:19 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Pulled | Successfully pulled image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1" in 79.150777ms
04:19:19 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Started | Started container alertmanager-proxy
04:19:19 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Pulling | Pulling image "registry.redhat.io/openshift4/ose-prometheus-alertmanager@sha256:b180f86ebeccbab28b05f75a570ead59fc1462a77dde648b2b1d1ebe9e33cbdb"
04:19:19 (x2) | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Pulling | Pulling image "image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/s2i-minimal-notebook:py3.8-v1"
04:19:20 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Started | Started container alertmanager
04:19:20 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Started | Started container jupyter-nb-ldap-2dadmin20
04:19:20 | redhat-ods-applications | modelmesh-controller-6cb56cd5f6-gbrsp_aa00c57c-94b1-4a51-ba27-6d7fee18b907 | modelmesh-controller-leader-lock | LeaderElection | modelmesh-controller-6cb56cd5f6-gbrsp_aa00c57c-94b1-4a51-ba27-6d7fee18b907 became leader
04:19:20 | redhat-ods-applications | modelmesh-controller-6cb56cd5f6-gbrsp_aa00c57c-94b1-4a51-ba27-6d7fee18b907 | modelmesh-controller-leader-lock | LeaderElection | modelmesh-controller-6cb56cd5f6-gbrsp_aa00c57c-94b1-4a51-ba27-6d7fee18b907 became leader
04:19:20 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Pulled | Successfully pulled image "registry.redhat.io/openshift4/ose-prometheus-alertmanager@sha256:b180f86ebeccbab28b05f75a570ead59fc1462a77dde648b2b1d1ebe9e33cbdb" in 1.092696827s
04:19:20 | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Created | Created container alertmanager
04:19:20 | rhods-notebooks | kubelet | jupyter-nb-ldap-2dadmin20-0 | Created | Created container jupyter-nb-ldap-2dadmin20
04:19:23 (x2) | redhat-ods-monitoring | kubelet | prometheus-b6bd4f9b-rwn9d | Unhealthy | Readiness probe failed: HTTP probe failed with statuscode: 503
04:19:38 | redhat-ods-monitoring | kubelet | blackbox-exporter-645f4478b7-nz8rz | Pulled | Container image "quay.io/integreatly/prometheus-blackbox-exporter@sha256:35b9d2c1002201723b7f7a9f54e9406b2ec4b5b0f73d114f47c70e15956103b5" already present on machine
04:19:38 | redhat-ods-monitoring | kubelet | blackbox-exporter-645f4478b7-nz8rz | Pulled | Container image "registry.redhat.io/openshift4/ose-oauth-proxy@sha256:4bef31eb993feb6f1096b51b4876c65a6fb1f4401fee97fa4f4542b6b7c9bc46" already present on machine
04:19:38 | redhat-ods-monitoring | kubelet | blackbox-exporter-645f4478b7-nz8rz | Created | Created container oauth-proxy
04:19:38 | redhat-ods-monitoring | kubelet | blackbox-exporter-645f4478b7-nz8rz | Started | Started container oauth-proxy
04:19:40 | redhat-ods-monitoring | kubelet | blackbox-exporter-645f4478b7-nz8rz | Created | Created container blackbox-exporter
04:19:40 | redhat-ods-monitoring | kubelet | blackbox-exporter-645f4478b7-nz8rz | Started | Started container blackbox-exporter
04:21:12 | redhat-ods-operator | kubelet | rhods-operator-58bcdb9cfd-hpvmp | Pulling | Pulling image "registry.redhat.io/rhods/odh-operator-rhel8@sha256:55b24f0ec2f047aafc1927ba9fb915bedb2017897cfb663964980af9cf045910"
04:21:15 | redhat-ods-operator | kubelet | rhods-operator-58bcdb9cfd-hpvmp | Started | Started container rhods-operator
04:21:15 | redhat-ods-operator | kubelet | rhods-operator-58bcdb9cfd-hpvmp | Pulled | Successfully pulled image "registry.redhat.io/rhods/odh-operator-rhel8@sha256:55b24f0ec2f047aafc1927ba9fb915bedb2017897cfb663964980af9cf045910" in 3.01788194s
04:21:15 | redhat-ods-operator | kubelet | rhods-operator-58bcdb9cfd-hpvmp | Created | Created container rhods-operator
04:21:51 | redhat-ods-operator | operator-lifecycle-manager | rhods-operator.1.20.0-11 | InstallSucceeded | install strategy completed with no errors
04:49:38 (x17) | rhods-notebooks | kfdef-controller | rhods-notebooks | KfDefCreationSuccessful | KfDef instance rhods-notebooks created and deployed successfully
04:50:02 (x17) | redhat-ods-monitoring | kfdef-controller | monitoring | KfDefCreationSuccessful | KfDef instance monitoring created and deployed successfully
04:50:27 (x17) | redhat-ods-monitoring | kfdef-controller | modelmesh-monitoring | KfDefCreationSuccessful | KfDef instance modelmesh-monitoring created and deployed successfully
04:50:54 (x23) | redhat-ods-applications | kfdef-controller | rhods-dashboard | KfDefCreationSuccessful | KfDef instance rhods-dashboard created and deployed successfully
04:51:01 (x17) | redhat-ods-applications | kfdef-controller | rhods-anaconda | KfDefCreationSuccessful | KfDef instance rhods-anaconda created and deployed successfully
04:51:19 (x23) | redhat-ods-applications | kfdef-controller | rhods-nbc | KfDefCreationSuccessful | KfDef instance rhods-nbc created and deployed successfully
04:51:36 (x17) | redhat-ods-applications | kfdef-controller | rhods-model-mesh | KfDefCreationSuccessful | KfDef instance rhods-model-mesh created and deployed successfully
04:51:45 (x23) | redhat-ods-applications | kfdef-controller | rhods-osd-config | KfDefCreationSuccessful | KfDef instance rhods-osd-config created and deployed successfully
04:51:52 (x24) | rhods-notebooks | kfdef-controller | rhods-osd-config | KfDefCreationSuccessful | KfDef instance rhods-osd-config created and deployed successfully