Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-13074

Upgrade db fails with sampledb integration active

    XMLWordPrintable

Details

    • Bug
    • Status: Done
    • Major
    • Resolution: Done
    • fuse-7.7-GA
    • fuse-7.7-GA
    • Fuse Online

    Description

      Copying the comment from ENTESB-12534:

      Also the pg_upgrade seems to break when there is an integration that uses the sampleDb database:

       oc logs -f syndesis-db-2-fdztf -c postgresql
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      /usr/share/container-scripts/postgresql/common.sh: line 121: [: 9.6: integer expression expected
      
      ==========  $PGDATA upgrade: 9.5 -> 9.6  ==========
      
      
      ===>  Initialize new data directory; we will migrate to that.
      
      The files belonging to this database system will be owned by user "postgres".
      This user must also own the server process.
      
      The database cluster will be initialized with locale "en_US.utf8".
      The default database encoding has accordingly been set to "UTF8".
      The default text search configuration will be set to "english".
      
      Data page checksums are disabled.
      
      creating directory /var/lib/pgsql/data/userdata-new ... ok
      creating subdirectories ... ok
      selecting default max_connections ... 100
      selecting default shared_buffers ... 128MB
      selecting dynamic shared memory implementation ... posix
      creating configuration files ... psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      ok
      running bootstrap script ... psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      ok
      performing post-bootstrap initialization ... psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      ok
      syncing data to disk ... psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      
      WARNING: enabling "trust" authentication for local connections
      You can change this by editing pg_hba.conf or using the option -A, or
      --auth-local and --auth-host, the next time you run initdb.
      ok
      
      Success. You can now start the database server using:
      
          /opt/rh/rh-postgresql96/root/usr/bin/pg_ctl -D /var/lib/pgsql/data/userdata-new -l logfile start
      
      
      ===>  Starting the pg_upgrade process.
      
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      
      *failure*
      Consult the last few lines of "pg_upgrade_server.log" for
      the probable cause of the failure.
      
      There seems to be a postmaster servicing the old cluster.
      Please shutdown that postmaster and try again.
      Failure, exiting
      
      -----------------------------------------------------------------
        pg_upgrade run on Tue Feb 25 09:04:00 2020
      -----------------------------------------------------------------
      
      command: "/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/userdata" -o "-p 50432 -c autovacuum=off -c autovacuum_freeze_max_age=2000000000  -c listen_addresses='' -c unix_socket_permissions=0700" start >> "pg_upgrade_server.log" 2>&1
      pg_ctl: another server might be running; trying to start server anyway
      waiting for server to start....2020-02-25 09:04:00 UTC   F0001 5e54e300.61 FATAL:  lock file "postmaster.pid" already exists
      2020-02-25 09:04:00 UTC   F0001 5e54e300.61 HINT:  Is another postmaster (PID 1) running in data directory "/var/lib/pgsql/data/userdata"?
       stopped waiting
      pg_ctl: could not start server
      Examine the log output.
      
      
      
      -----------------------------------------------------------------
        pg_upgrade run on Tue Feb 25 09:04:31 2020
      -----------------------------------------------------------------
      
      command: "/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/pgsql/data/userdata" -o "-p 50432 -c autovacuum=off -c autovacuum_freeze_max_age=2000000000  -c listen_addresses='' -c unix_socket_permissions=0700" start >> "pg_upgrade_server.log" 2>&1
      pg_ctl: another server might be running; trying to start server anyway
      waiting for server to start....2020-02-25 09:04:31 UTC   F0001 5e54e31f.64 FATAL:  lock file "postmaster.pid" already exists
      2020-02-25 09:04:31 UTC   F0001 5e54e31f.64 HINT:  Is another postmaster (PID 1) running in data directory "/var/lib/pgsql/data/userdata"?
       stopped waiting
      pg_ctl: could not start server
      Examine the log output.
      
      
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      psql: could not connect to server: Connection refused
      	Is the server running on host "127.0.0.1" and accepting
      	TCP/IP connections on port 5432?
      Waiting for Postgres server...
      
       oc describe pod syndesis-db-2-fdztf                          
      Name:               syndesis-db-2-fdztf
      Namespace:          avano-syndesis
      Priority:           0
      PriorityClassName:  <none>
      Node:               node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com/10.0.145.81
      Start Time:         Tue, 25 Feb 2020 10:03:32 +0100
      Labels:             app=syndesis
                          deployment=syndesis-db-2
                          deploymentconfig=syndesis-db
                          syndesis.io/app=syndesis
                          syndesis.io/component=syndesis-db
      Annotations:        openshift.io/deployment-config.latest-version: 2
                          openshift.io/deployment-config.name: syndesis-db
                          openshift.io/deployment.name: syndesis-db-2
                          openshift.io/scc: anyuid
      Status:             Running
      IP:                 10.129.0.95
      Controlled By:      ReplicationController/syndesis-db-2
      Containers:
        postgresql:
          Container ID:   docker://b8b2f343982f2d4d79bc665052fc7d429645458f43b27137bb70350f27feb962
          Image:          centos/postgresql-96-centos7
          Image ID:       docker-pullable://docker.io/centos/postgresql-96-centos7@sha256:617c75aab798753ee5e0323d60069757b17a2255abd9de44ca342e4a8bb68b15
          Port:           5432/TCP
          Host Port:      0/TCP
          State:          Waiting
            Reason:       CrashLoopBackOff
          Last State:     Terminated
            Reason:       Error
            Exit Code:    1
            Started:      Tue, 25 Feb 2020 10:04:57 +0100
            Finished:     Tue, 25 Feb 2020 10:05:20 +0100
          Ready:          False
          Restart Count:  2
          Limits:
            memory:  255Mi
          Requests:
            memory:   255Mi
          Liveness:   tcp-socket :5432 delay=60s timeout=1s period=10s #success=1 #failure=3
          Readiness:  exec [/bin/sh -i -c psql -h 127.0.0.1 -U $POSTGRESQL_USER -q -d $POSTGRESQL_DATABASE -c 'SELECT 1'] delay=5s timeout=1s period=10s #success=1 #failure=3
          Environment:
            POSTGRESQL_USER:               syndesis
            POSTGRESQL_PASSWORD:           2jwaYDIXovJjSsNc
            POSTGRESQL_DATABASE:           syndesis
            POSTGRESQL_SAMPLEDB_PASSWORD:  dtDCIZwIEvGzdtsN
            POSTGRESQL_UPGRADE:            copy
          Mounts:
            /opt/app-root/src/postgresql-cfg/ from syndesis-db-conf (rw)
            /var/lib/pgsql/data from syndesis-db-data (rw)
            /var/lib/pgsql/sampledb from syndesis-sampledb-config (rw)
            /var/run/secrets/kubernetes.io/serviceaccount from syndesis-default-token-b2svk (ro)
        syndesis-db-metrics:
          Container ID:   docker://251e985e5e3b6935f478c8df495a717553c2dad276cdc5298affb77fb94da972
          Image:          docker.io/wrouesnel/postgres_exporter:v0.4.7
          Image ID:       docker-pullable://docker.io/wrouesnel/postgres_exporter@sha256:dd8051322ceb8995d3d7f116041a2116815e01e88232a90f635ebde8dcc4d3f4
          Port:           9187/TCP
          Host Port:      0/TCP
          State:          Running
            Started:      Tue, 25 Feb 2020 10:04:05 +0100
          Ready:          True
          Restart Count:  0
          Limits:
            memory:  256Mi
          Requests:
            memory:   20Mi
          Liveness:   tcp-socket :9187 delay=60s timeout=1s period=10s #success=1 #failure=5
          Readiness:  tcp-socket :9187 delay=30s timeout=1s period=10s #success=1 #failure=5
          Environment:
            DATA_SOURCE_NAME:               postgresql://syndesis:2jwaYDIXovJjSsNc@localhost:5432/syndesis?sslmode=disable
            PG_EXPORTER_EXTEND_QUERY_PATH:  /etc/postgres/exporter/queries.yaml
          Mounts:
            /etc/postgres/exporter from syndesis-db-metrics-config (rw)
            /var/run/secrets/kubernetes.io/serviceaccount from syndesis-default-token-b2svk (ro)
      Conditions:
        Type              Status
        Initialized       True 
        Ready             False 
        ContainersReady   False 
        PodScheduled      True 
      Volumes:
        syndesis-db-metrics-config:
          Type:      ConfigMap (a volume populated by a ConfigMap)
          Name:      syndesis-db-metrics-config
          Optional:  false
        syndesis-db-data:
          Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
          ClaimName:  syndesis-db
          ReadOnly:   false
        syndesis-sampledb-config:
          Type:      ConfigMap (a volume populated by a ConfigMap)
          Name:      syndesis-sampledb-config
          Optional:  false
        syndesis-db-conf:
          Type:      ConfigMap (a volume populated by a ConfigMap)
          Name:      syndesis-db-conf
          Optional:  false
        syndesis-default-token-b2svk:
          Type:        Secret (a volume populated by a Secret)
          SecretName:  syndesis-default-token-b2svk
          Optional:    false
      QoS Class:       Burstable
      Node-Selectors:  region=primary
      Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
      Events:
        Type     Reason               Age                 From                                                               Message
        ----     ------               ----                ----                                                               -------
        Normal   Scheduled            115s                default-scheduler                                                  Successfully assigned avano-syndesis/syndesis-db-2-fdztf to node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com
        Normal   Pulled               83s                 kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com  Container image "docker.io/wrouesnel/postgres_exporter:v0.4.7" already present on machine
        Normal   Created              82s                 kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com  Created container
        Normal   Started              82s                 kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com  Started container
        Normal   Pulled               31s (x3 over 112s)  kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com  Container image "centos/postgresql-96-centos7" already present on machine
        Normal   Started              30s (x3 over 109s)  kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com  Started container
        Normal   Created              30s (x3 over 110s)  kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com  Created container
        Warning  FailedPostStartHook  7s (x3 over 83s)    kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com  Exec lifecycle hook ([/bin/sh -c /var/lib/pgsql/sampledb/postStart.sh]) for Container "postgresql" in Pod "syndesis-db-2-fdztf_avano-syndesis(b2c4510e-57ad-11ea-a75b-fa163e748b89)" failed - error: command '/bin/sh -c /var/lib/pgsql/sampledb/postStart.sh' exited with 137: , message: ""
        Normal   Killing              7s (x3 over 83s)    kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com  Killing container with id docker://postgresql:FailedPostStartHook
        Warning  BackOff              5s (x5 over 51s)    kubelet, node2.dv-syndesis-v311-6.dos.fuse-qe.eng.rdu2.redhat.com  Back-off restarting failed container
      
       oc get pods
      NAME                          READY   STATUS             RESTARTS   AGE
      db-backup-1582621158-fhlb6    0/2     Completed          0          8m
      i-zzzz-1-build                0/1     Completed          0          11m
      i-zzzz-1-zqzv5                1/1     Running            0          11m
      syndesis-db-2-deploy          1/1     Running            0          4m
      syndesis-db-2-fdztf           1/2     CrashLoopBackOff   3          3m
      syndesis-meta-1-sbxcz         1/1     Running            0          4m
      syndesis-oauthproxy-2-bzv4n   1/1     Running            0          4m
      syndesis-operator-3-kgcjg     1/1     Running            0          8m
      syndesis-prometheus-1-z2fng   1/1     Running            0          4m
      syndesis-server-1-bxtck       0/1     CrashLoopBackOff   4          4m
      syndesis-server-1-deploy      1/1     Running            0          4m
      syndesis-ui-1-nvjnq           1/1     Running            0          4m
      upgrade-db-migration-r2flk    0/1     Completed          0          7m
      
      

      Attachments

        Issue Links

          Activity

            People

              parichar@redhat.com Paul Richardson
              avano@redhat.com Andrej Vano
              Andrej Vano Andrej Vano
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: