subscription.operators.coreos.com/quay-operator patched [ec2-user@ip-10-0-12-54 ~]$ oc get pod NAME READY STATUS RESTARTS AGE quay-operator.v3.8.10-f97d5b966-fdtcj 1/1 Running 0 5m33s quayregistry-clair-app-7559d977fc-dgmnk 1/1 Running 0 4m48s quayregistry-clair-app-7559d977fc-p7g58 1/1 Running 0 4m33s quayregistry-clair-postgres-64d8c4b85-x2c94 1/1 Running 1 (4m18s ago) 4m48s quayregistry-quay-app-6845dc6559-fmn2t 1/1 Running 0 3m54s quayregistry-quay-app-6845dc6559-mt9zd 1/1 Running 0 3m51s quayregistry-quay-app-upgrade-26hjt 0/1 Completed 0 4m49s quayregistry-quay-config-editor-55ccdf9b6-5s7jt 1/1 Running 0 4m48s quayregistry-quay-database-7b9cbdb798-d2gpr 1/1 Running 0 4m48s quayregistry-quay-mirror-5dfbc74b4c-fm8vc 1/1 Running 0 4m36s quayregistry-quay-mirror-5dfbc74b4c-tpbnt 1/1 Running 0 4m48s quayregistry-quay-redis-788c7f598c-jn6l8 1/1 Running 0 4m48s [ec2-user@ip-10-0-12-54 ~]$ oc patch Subscription -n quay-enterprise-14080 quay-operator --type json -p '[{"op": "add", "path": "/spec/channel", "value": "stable-3.9"}]' [ec2-user@ip-10-0-12-54 ~]$ oc get pod NAME READY STATUS RESTARTS AGE quay-operator.v3.9.0-645db47fd4-b8hnr 1/1 Running 0 38s quayregistry-clair-postgres-64d8c4b85-x2c94 1/1 Terminating 1 (4m58s ago) 5m28s quayregistry-clair-postgres-upgrade-v24lz 0/1 Init:0/1 0 20s quayregistry-quay-app-6845dc6559-fmn2t 1/1 Terminating 0 4m34s quayregistry-quay-app-upgrade-26hjt 0/1 Completed 0 5m29s quayregistry-quay-config-editor-55ccdf9b6-5s7jt 1/1 Terminating 0 5m28s quayregistry-quay-config-editor-5c64bf849b-84496 1/1 Running 0 20s quayregistry-quay-mirror-5dfbc74b4c-fm8vc 1/1 Terminating 0 5m16s quayregistry-quay-mirror-5dfbc74b4c-tpbnt 1/1 Terminating 0 5m28s quayregistry-quay-postgres-upgrade-rxtbm 0/1 Init:0/1 0 22s quayregistry-quay-redis-55c448565d-kk4tk 1/1 Running 0 20s [ec2-user@ip-10-0-12-54 ~]$ oc get pod NAME READY STATUS RESTARTS AGE quay-operator.v3.9.0-645db47fd4-b8hnr 1/1 Running 0 3m32s quayregistry-clair-postgres-upgrade-v24lz 0/1 Init:0/1 0 3m14s quayregistry-quay-app-upgrade-26hjt 0/1 Completed 0 8m23s quayregistry-quay-config-editor-5c64bf849b-84496 1/1 Running 0 3m14s quayregistry-quay-mirror-bb76fdb6c-b6jmv 0/1 Init:0/1 1 (33s ago) 2m44s quayregistry-quay-mirror-bb76fdb6c-d2m2q 0/1 Init:0/1 1 (31s ago) 2m44s quayregistry-quay-postgres-upgrade-rxtbm 0/1 Completed 0 3m16s quayregistry-quay-redis-55c448565d-kk4tk 1/1 Running 0 3m14s [ec2-user@ip-10-0-12-54 ~]$ oc describe pod/quayregistry-clair-postgres-upgrade-v24lz Name: quayregistry-clair-postgres-upgrade-v24lz Namespace: quay-enterprise-14080 Priority: 0 Service Account: quayregistry-clair-postgres Node: ip-10-0-222-246.us-east-2.compute.internal/10.0.222.246 Start Time: Thu, 20 Jul 2023 05:45:16 +0000 Labels: controller-uid=0e2d4c2b-bb43-4209-8da3-59018211e992 job-name=quayregistry-clair-postgres-upgrade quay-operator/quayregistry=quayregistry Annotations: k8s.v1.cni.cncf.io/network-status: [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.130.2.10" ], "default": true, "dns": {} }] openshift.io/scc: restricted-v2 quay-buildmanager-hostname: quay-operator-service-endpoint: http://quay-operator.quay-enterprise-14080.svc.cluster.local:7071 quay-registry-hostname: quayregistry-quay-quay-enterprise-14080.apps.quaytest-14080.qe.devcluster.openshift.com seccomp.security.alpha.kubernetes.io/pod: runtime/default Status: Pending IP: 10.130.2.10 IPs: IP: 10.130.2.10 Controlled By: Job/quayregistry-clair-postgres-upgrade Init Containers: postgres-old: Container ID: cri-o://0d8cad185c2d0cda95d429e1320857697e23c48c13aaa8d32b3a8254af5b2173 Image: registry.redhat.io/rhscl/postgresql-12-rhel7@sha256:68b969143f4c638098a13c2db8693b60339122ef004f28875ebc4a4afc23035d Image ID: registry.redhat.io/rhscl/postgresql-12-rhel7@sha256:68b969143f4c638098a13c2db8693b60339122ef004f28875ebc4a4afc23035d Port: 5432/TCP Host Port: 0/TCP Command: /bin/bash -c cp -r /var/lib/pgsql/data /var/lib/pgsql/backup/clair && run-postgresql --help State: Running Started: Thu, 20 Jul 2023 05:48:28 +0000 Ready: False Restart Count: 0 Requests: cpu: 500m memory: 2Gi Environment: POSTGRESQL_UPGRADE: copy POSTGRESQL_USER: postgres POSTGRESQL_DATABASE: postgres POSTGRESQL_PASSWORD: postgres POSTGRESQL_ADMIN_PASSWORD: postgres POSTGRESQL_MAX_CONNECTIONS: 1000 Mounts: /usr/share/pgsql/postgresql.conf.sample from clair-postgres-conf-sample (rw,path="postgresql.conf.sample") /var/lib/pgsql/backup from migration-data (rw) /var/lib/pgsql/data from postgres-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tp7nm (ro) Containers: postgres-new: Container ID: Image: registry.redhat.io/rhscl/postgresql-13-rhel7@sha256:05ccee5096af0393d7a8f47e85fc321d771538829f0bcebcfa90d333d89b7993 Image ID: Port: 5432/TCP Host Port: 0/TCP Command: run-postgresql Args: --version State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Requests: cpu: 500m memory: 2Gi Environment: POSTGRESQL_UPGRADE: copy POSTGRESQL_USER: postgres POSTGRESQL_DATABASE: postgres POSTGRESQL_PASSWORD: postgres POSTGRESQL_ADMIN_PASSWORD: postgres POSTGRESQL_MAX_CONNECTIONS: 1000 Mounts: /usr/share/pgsql/postgresql.conf.sample from clair-postgres-conf-sample (rw,path="postgresql.conf.sample") /var/lib/pgsql/backup from migration-data (rw) /var/lib/pgsql/data from postgres-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tp7nm (ro) Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: clair-postgres-conf-sample: Type: ConfigMap (a volume populated by a ConfigMap) Name: quayregistry-clair-postgres-conf-sample Optional: false postgres-data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: quayregistry-clair-postgres ReadOnly: false migration-data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: quayregistry-clair-postgres-migration ReadOnly: false kube-api-access-tp7nm: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 3m25s default-scheduler Successfully assigned quay-enterprise-14080/quayregistry-clair-postgres-upgrade-v24lz to ip-10-0-222-246.us-east-2.compute.internal Warning FailedAttachVolume 3m25s attachdetach-controller Multi-Attach error for volume "pvc-38b6a080-2102-4305-8f19-9e77bd1ad75d" Volume is already used by pod(s) quayregistry-clair-postgres-64d8c4b85-x2c94 Normal SuccessfulAttachVolume 3m22s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-5900ad31-5654-4d94-bc29-c8a521282942" Warning FailedMount 82s kubelet Unable to attach or mount volumes: unmounted volumes=[postgres-data], unattached volumes=[migration-data kube-api-access-tp7nm clair-postgres-conf-sample postgres-data]: timed out waiting for the condition Normal SuccessfulAttachVolume 15s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-38b6a080-2102-4305-8f19-9e77bd1ad75d" Normal AddedInterface 13s multus Add eth0 [10.130.2.10/23] from openshift-sdn Normal Pulled 13s kubelet Container image "registry.redhat.io/rhscl/postgresql-12-rhel7@sha256:68b969143f4c638098a13c2db8693b60339122ef004f28875ebc4a4afc23035d" already present on machine Normal Created 13s kubelet Created container postgres-old Normal Started 13s kubelet Started container postgres-old [ec2-user@ip-10-0-12-54 ~]$ oc get pod NAME READY STATUS RESTARTS AGE quay-operator.v3.9.0-645db47fd4-b8hnr 1/1 Running 0 15m quayregistry-clair-postgres-upgrade-v24lz 0/1 Init:CrashLoopBackOff 6 (61s ago) 15m quayregistry-quay-app-upgrade-26hjt 0/1 Completed 0 20m quayregistry-quay-config-editor-5c64bf849b-84496 1/1 Running 0 15m quayregistry-quay-mirror-bb76fdb6c-b6jmv 0/1 Init:0/1 5 (2m34s ago) 14m quayregistry-quay-mirror-bb76fdb6c-d2m2q 0/1 Init:0/1 5 (2m29s ago) 14m quayregistry-quay-postgres-upgrade-rxtbm 0/1 Completed 0 15m quayregistry-quay-redis-55c448565d-kk4tk 1/1 Running 0 15m [ec2-user@ip-10-0-12-54 ~]$ oc describe pod/quayregistry-clair-postgres-upgrade-v24lz Name: quayregistry-clair-postgres-upgrade-v24lz Namespace: quay-enterprise-14080 Priority: 0 Service Account: quayregistry-clair-postgres Node: ip-10-0-222-246.us-east-2.compute.internal/10.0.222.246 Start Time: Thu, 20 Jul 2023 05:45:16 +0000 Labels: controller-uid=0e2d4c2b-bb43-4209-8da3-59018211e992 job-name=quayregistry-clair-postgres-upgrade quay-operator/quayregistry=quayregistry Annotations: k8s.v1.cni.cncf.io/network-status: [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.130.2.10" ], "default": true, "dns": {} }] openshift.io/scc: restricted-v2 quay-buildmanager-hostname: quay-operator-service-endpoint: http://quay-operator.quay-enterprise-14080.svc.cluster.local:7071 quay-registry-hostname: quayregistry-quay-quay-enterprise-14080.apps.quaytest-14080.qe.devcluster.openshift.com seccomp.security.alpha.kubernetes.io/pod: runtime/default Status: Pending IP: 10.130.2.10 IPs: IP: 10.130.2.10 Controlled By: Job/quayregistry-clair-postgres-upgrade Init Containers: postgres-old: Container ID: cri-o://67e8d85bd6e656b3ac60afc9bc4fc854d4524f99e1b93d71da732f0bafe227cc Image: registry.redhat.io/rhscl/postgresql-12-rhel7@sha256:68b969143f4c638098a13c2db8693b60339122ef004f28875ebc4a4afc23035d Image ID: registry.redhat.io/rhscl/postgresql-12-rhel7@sha256:68b969143f4c638098a13c2db8693b60339122ef004f28875ebc4a4afc23035d Port: 5432/TCP Host Port: 0/TCP Command: /bin/bash -c cp -r /var/lib/pgsql/data /var/lib/pgsql/backup/clair && run-postgresql --help State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Thu, 20 Jul 2023 05:58:45 +0000 Finished: Thu, 20 Jul 2023 05:59:33 +0000 Ready: False Restart Count: 6 Requests: cpu: 500m memory: 2Gi Environment: POSTGRESQL_UPGRADE: copy POSTGRESQL_USER: postgres POSTGRESQL_DATABASE: postgres POSTGRESQL_PASSWORD: postgres POSTGRESQL_ADMIN_PASSWORD: postgres POSTGRESQL_MAX_CONNECTIONS: 1000 Mounts: /usr/share/pgsql/postgresql.conf.sample from clair-postgres-conf-sample (rw,path="postgresql.conf.sample") /var/lib/pgsql/backup from migration-data (rw) /var/lib/pgsql/data from postgres-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tp7nm (ro) Containers: postgres-new: Container ID: Image: registry.redhat.io/rhscl/postgresql-13-rhel7@sha256:05ccee5096af0393d7a8f47e85fc321d771538829f0bcebcfa90d333d89b7993 Image ID: Port: 5432/TCP Host Port: 0/TCP Command: run-postgresql Args: --version State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Requests: cpu: 500m memory: 2Gi Environment: POSTGRESQL_UPGRADE: copy POSTGRESQL_USER: postgres POSTGRESQL_DATABASE: postgres POSTGRESQL_PASSWORD: postgres POSTGRESQL_ADMIN_PASSWORD: postgres POSTGRESQL_MAX_CONNECTIONS: 1000 Mounts: /usr/share/pgsql/postgresql.conf.sample from clair-postgres-conf-sample (rw,path="postgresql.conf.sample") /var/lib/pgsql/backup from migration-data (rw) /var/lib/pgsql/data from postgres-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tp7nm (ro) Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: clair-postgres-conf-sample: Type: ConfigMap (a volume populated by a ConfigMap) Name: quayregistry-clair-postgres-conf-sample Optional: false postgres-data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: quayregistry-clair-postgres ReadOnly: false migration-data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: quayregistry-clair-postgres-migration ReadOnly: false kube-api-access-tp7nm: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 15m default-scheduler Successfully assigned quay-enterprise-14080/quayregistry-clair-postgres-upgrade-v24lz to ip-10-0-222-246.us-east-2.compute.internal Warning FailedAttachVolume 15m attachdetach-controller Multi-Attach error for volume "pvc-38b6a080-2102-4305-8f19-9e77bd1ad75d" Volume is already used by pod(s) quayregistry-clair-postgres-64d8c4b85-x2c94 Normal SuccessfulAttachVolume 15m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-5900ad31-5654-4d94-bc29-c8a521282942" Warning FailedMount 13m kubelet Unable to attach or mount volumes: unmounted volumes=[postgres-data], unattached volumes=[migration-data kube-api-access-tp7nm clair-postgres-conf-sample postgres-data]: timed out waiting for the condition Normal SuccessfulAttachVolume 12m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-38b6a080-2102-4305-8f19-9e77bd1ad75d" Normal AddedInterface 12m multus Add eth0 [10.130.2.10/23] from openshift-sdn Normal Pulled 7m41s (x5 over 12m) kubelet Container image "registry.redhat.io/rhscl/postgresql-12-rhel7@sha256:68b969143f4c638098a13c2db8693b60339122ef004f28875ebc4a4afc23035d" already present on machine Normal Created 7m41s (x5 over 12m) kubelet Created container postgres-old Normal Started 7m41s (x5 over 12m) kubelet Started container postgres-old Warning BackOff 3m11s (x20 over 11m) kubelet Back-off restarting failed container postgres-old in pod quayregistry-clair-postgres-upgrade-v24lz_quay-enterprise-14080(61aa21b2-fe72-4ed9-a47b-aadedc638eac) [ec2-user@ip-10-0-12-54 ~]$ oc get pod NAME READY STATUS RESTARTS AGE quay-operator.v3.9.0-645db47fd4-b8hnr 1/1 Running 0 21m quayregistry-clair-postgres-upgrade-v24lz 0/1 Init:CrashLoopBackOff 7 (32s ago) 20m quayregistry-quay-app-upgrade-26hjt 0/1 Completed 0 25m quayregistry-quay-config-editor-5c64bf849b-84496 1/1 Running 0 20m quayregistry-quay-mirror-bb76fdb6c-b6jmv 0/1 Init:0/1 6 (4m25s ago) 20m quayregistry-quay-mirror-bb76fdb6c-d2m2q 0/1 Init:0/1 6 (4m20s ago) 20m quayregistry-quay-postgres-upgrade-rxtbm 0/1 Completed 0 20m quayregistry-quay-redis-55c448565d-kk4tk 1/1 Running 0 20m [ec2-user@ip-10-0-12-54 ~]$ oc logs quayregistry-clair-postgres-upgrade-v24lz Defaulted container "postgres-new" out of: postgres-new, postgres-old (init) Error from server (BadRequest): container "postgres-new" in pod "quayregistry-clair-postgres-upgrade-v24lz" is waiting to start: PodInitializing [ec2-user@ip-10-0-12-54 ~]$ oc logs quayregistry-clair-postgres-upgrade-v24lz -c postgres-old ========== $PGDATA upgrade: 10 -> 12 ========== ===> Starting old postgresql once again for a clean shutdown... pg_ctl: another server might be running; trying to start server anyway waiting for server to start....2023-07-20 06:05:26.707 GMT [33] FATAL: lock file "postmaster.pid" already exists 2023-07-20 06:05:26.707 GMT [33] HINT: Is another postmaster (PID 1) running in data directory "/var/lib/pgsql/data/userdata"? stopped waiting pg_ctl: could not start server Examine the log output.