-
Bug
-
Resolution: Done
-
Major
-
fuse-7.7-GA
Copying the comment from ENTESB-12534:
Also the pg_upgrade seems to break when there is an integration that uses the sampleDb database:
oc logs -f syndesis-db-2-fdztf -c postgresql psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... /usr/share/container-scripts/postgresql/common.sh: line 121: [: 9.6: integer expression expected ========== $PGDATA upgrade: 9.5 -> 9.6 ========== ===> Initialize new data directory; we will migrate to that. The files belonging to this database system will be owned by user "postgres". This user must also own the server process. The database cluster will be initialized with locale "en_US.utf8". The default database encoding has accordingly been set to "UTF8". The default text search configuration will be set to "english". Data page checksums are disabled. creating directory /var/lib/pgsql/data/userdata-new ... ok creating subdirectories ... ok selecting default max_connections ... 100 selecting default shared_buffers ... 128MB selecting dynamic shared memory implementation ... posix creating configuration files ... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... ok running bootstrap script ... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... ok performing post-bootstrap initialization ... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... ok syncing data to disk ... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... WARNING: enabling "trust" authentication for local connections You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. ok Success. You can now start the database server using: /opt/rh/rh-postgresql96/root/usr/bin/pg_ctl -D /var/lib/pgsql/data/userdata-new -l logfile start ===> Starting the pg_upgrade process. psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... *failure* Consult the last few lines of "pg_upgrade_server.log" for the probable cause of the failure. There seems to be a postmaster servicing the old cluster. Please shutdown that postmaster and try again. Failure, exiting ----------------------------------------------------------------- pg_upgrade run on Tue Feb 25 09:04:00 2020 ----------------------------------------------------------------- command: "/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/userdata" -o "-p 50432 -c autovacuum=off -c autovacuum_freeze_max_age=2000000000 -c listen_addresses='' -c unix_socket_permissions=0700" start >> "pg_upgrade_server.log" 2>&1 pg_ctl: another server might be running; trying to start server anyway waiting for server to start....2020-02-25 09:04:00 UTC F0001 5e54e300.61 FATAL: lock file "postmaster.pid" already exists 2020-02-25 09:04:00 UTC F0001 5e54e300.61 HINT: Is another postmaster (PID 1) running in data directory "/var/lib/pgsql/data/userdata"? stopped waiting pg_ctl: could not start server Examine the log output. ----------------------------------------------------------------- pg_upgrade run on Tue Feb 25 09:04:31 2020 ----------------------------------------------------------------- command: "/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/userdata" -o "-p 50432 -c autovacuum=off -c autovacuum_freeze_max_age=2000000000 -c listen_addresses='' -c unix_socket_permissions=0700" start >> "pg_upgrade_server.log" 2>&1 pg_ctl: another server might be running; trying to start server anyway waiting for server to start....2020-02-25 09:04:31 UTC F0001 5e54e31f.64 FATAL: lock file "postmaster.pid" already exists 2020-02-25 09:04:31 UTC F0001 5e54e31f.64 HINT: Is another postmaster (PID 1) running in data directory "/var/lib/pgsql/data/userdata"? stopped waiting pg_ctl: could not start server Examine the log output. psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server... psql: could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432? Waiting for Postgres server...
oc describe pod syndesis-db-2-fdztf
Name: syndesis-db-2-fdztf
Namespace: avano-syndesis
Priority: 0
PriorityClassName: <none>
Node: node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com/10.0.145.81
Start Time: Tue, 25 Feb 2020 10:03:32 +0100
Labels: app=syndesis
deployment=syndesis-db-2
deploymentconfig=syndesis-db
syndesis.io/app=syndesis
syndesis.io/component=syndesis-db
Annotations: openshift.io/deployment-config.latest-version: 2
openshift.io/deployment-config.name: syndesis-db
openshift.io/deployment.name: syndesis-db-2
openshift.io/scc: anyuid
Status: Running
IP: 10.129.0.95
Controlled By: ReplicationController/syndesis-db-2
Containers:
postgresql:
Container ID: docker://b8b2f343982f2d4d79bc665052fc7d429645458f43b27137bb70350f27feb962
Image: centos/postgresql-96-centos7
Image ID: docker-pullable://docker.io/centos/postgresql-96-centos7@sha256:617c75aab798753ee5e0323d60069757b17a2255abd9de44ca342e4a8bb68b15
Port: 5432/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 25 Feb 2020 10:04:57 +0100
Finished: Tue, 25 Feb 2020 10:05:20 +0100
Ready: False
Restart Count: 2
Limits:
memory: 255Mi
Requests:
memory: 255Mi
Liveness: tcp-socket :5432 delay=60s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [/bin/sh -i -c psql -h 127.0.0.1 -U $POSTGRESQL_USER -q -d $POSTGRESQL_DATABASE -c 'SELECT 1'] delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
POSTGRESQL_USER: syndesis
POSTGRESQL_PASSWORD: 2jwaYDIXovJjSsNc
POSTGRESQL_DATABASE: syndesis
POSTGRESQL_SAMPLEDB_PASSWORD: dtDCIZwIEvGzdtsN
POSTGRESQL_UPGRADE: copy
Mounts:
/opt/app-root/src/postgresql-cfg/ from syndesis-db-conf (rw)
/var/lib/pgsql/data from syndesis-db-data (rw)
/var/lib/pgsql/sampledb from syndesis-sampledb-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from syndesis-default-token-b2svk (ro)
syndesis-db-metrics:
Container ID: docker://251e985e5e3b6935f478c8df495a717553c2dad276cdc5298affb77fb94da972
Image: docker.io/wrouesnel/postgres_exporter:v0.4.7
Image ID: docker-pullable://docker.io/wrouesnel/postgres_exporter@sha256:dd8051322ceb8995d3d7f116041a2116815e01e88232a90f635ebde8dcc4d3f4
Port: 9187/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 25 Feb 2020 10:04:05 +0100
Ready: True
Restart Count: 0
Limits:
memory: 256Mi
Requests:
memory: 20Mi
Liveness: tcp-socket :9187 delay=60s timeout=1s period=10s #success=1 #failure=5
Readiness: tcp-socket :9187 delay=30s timeout=1s period=10s #success=1 #failure=5
Environment:
DATA_SOURCE_NAME: postgresql://syndesis:2jwaYDIXovJjSsNc@localhost:5432/syndesis?sslmode=disable
PG_EXPORTER_EXTEND_QUERY_PATH: /etc/postgres/exporter/queries.yaml
Mounts:
/etc/postgres/exporter from syndesis-db-metrics-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from syndesis-default-token-b2svk (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
syndesis-db-metrics-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: syndesis-db-metrics-config
Optional: false
syndesis-db-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: syndesis-db
ReadOnly: false
syndesis-sampledb-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: syndesis-sampledb-config
Optional: false
syndesis-db-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: syndesis-db-conf
Optional: false
syndesis-default-token-b2svk:
Type: Secret (a volume populated by a Secret)
SecretName: syndesis-default-token-b2svk
Optional: false
QoS Class: Burstable
Node-Selectors: region=primary
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 115s default-scheduler Successfully assigned avano-syndesis/syndesis-db-2-fdztf to node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com
Normal Pulled 83s kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com Container image "docker.io/wrouesnel/postgres_exporter:v0.4.7" already present on machine
Normal Created 82s kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com Created container
Normal Started 82s kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com Started container
Normal Pulled 31s (x3 over 112s) kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com Container image "centos/postgresql-96-centos7" already present on machine
Normal Started 30s (x3 over 109s) kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com Started container
Normal Created 30s (x3 over 110s) kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com Created container
Warning FailedPostStartHook 7s (x3 over 83s) kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com Exec lifecycle hook ([/bin/sh -c /var/lib/pgsql/sampledb/postStart.sh]) for Container "postgresql" in Pod "syndesis-db-2-fdztf_avano-syndesis(b2c4510e-57ad-11ea-a75b-fa163e748b89)" failed - error: command '/bin/sh -c /var/lib/pgsql/sampledb/postStart.sh' exited with 137: , message: ""
Normal Killing 7s (x3 over 83s) kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com Killing container with id docker://postgresql:FailedPostStartHook
Warning BackOff 5s (x5 over 51s) kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com Back-off restarting failed container
oc get pods
NAME READY STATUS RESTARTS AGE
db-backup-1582621158-fhlb6 0/2 Completed 0 8m
i-zzzz-1-build 0/1 Completed 0 11m
i-zzzz-1-zqzv5 1/1 Running 0 11m
syndesis-db-2-deploy 1/1 Running 0 4m
syndesis-db-2-fdztf 1/2 CrashLoopBackOff 3 3m
syndesis-meta-1-sbxcz 1/1 Running 0 4m
syndesis-oauthproxy-2-bzv4n 1/1 Running 0 4m
syndesis-operator-3-kgcjg 1/1 Running 0 8m
syndesis-prometheus-1-z2fng 1/1 Running 0 4m
syndesis-server-1-bxtck 0/1 CrashLoopBackOff 4 4m
syndesis-server-1-deploy 1/1 Running 0 4m
syndesis-ui-1-nvjnq 1/1 Running 0 4m
upgrade-db-migration-r2flk 0/1 Completed 0 7m
- relates to
-
-
- Done
-