-
Bug
-
Resolution: Done
-
Blocker
-
fuse-7.8-GA
-
None
After update Fuse Online from 7.7 to 7.8 via fuse-online-install script, the syndesis-db is in the CrashLoopBackOff state.
Pod description:
Normal Pulled 49s (x3 over 73s) kubelet, crc-zqfk6-master-0 Container image "registry.redhat.io/rhscl/postgresql-10-rhel7:latest" already present on machine Warning FailedPostStartHook 48s kubelet, crc-zqfk6-master-0 Exec lifecycle hook ([/bin/sh -c /var/lib/pgsql/sampledb/postStart.sh]) for Container "postgresql" in Pod "syndesis-db-2-98gb5_mkralik(4118a418-cdfa-4130-a93a-5772d6757727)" failed - error: rpc error: code = Unknown desc = command error: time="2020-09-30T13:24:45Z" level=error msg="exec failed: cannot exec a container that has stopped" exec failed: cannot exec a container that has stopped , stdout: , stderr: , exit code -1, message: "" Normal Killing 48s (x3 over 72s) kubelet, crc-zqfk6-master-0 FailedPostStartHook
Postgresql log:
Incompatible data directory. This container image provides PostgreSQL '10', but data directory is of version '9.6'. This image supports automatic data directory upgrade from '9.6', please _carefully_ consult image documentation about how to use the '$POSTGRESQL_UPGRADE' startup option.
Operator log:
[mkralik@localhost fuse-online-install]$ oc logs -f syndesis-operator-3-pj5ft {"level":"info","ts":1601472168.3732114,"logger":"cmd","msg":"Go Version: go1.13.4"} {"level":"info","ts":1601472168.373305,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1601472168.3733287,"logger":"cmd","msg":"Version of operator-sdk: v0.17.0+git"} {"level":"info","ts":1601472168.3733435,"logger":"cmd","msg":"Syndesis Operator Version: 1.8"} {"level":"info","ts":1601472168.3733509,"logger":"cmd","msg":"Syndesis Operator Image: registry.redhat.io/fuse7/fuse-online-operator"} {"level":"info","ts":1601472168.3733664,"logger":"cmd","msg":"Syndesis Build Time: 2020-09-24T11:28:41+0000"} I0930 13:22:49.431352 1 request.go:621] Throttling request took 1.035934631s, request: GET:https://172.25.0.1:443/apis/networking.k8s.io/v1?timeout=32s {"level":"info","ts":1601472173.189353,"logger":"leader","msg":"Trying to become the leader."} {"level":"info","ts":1601472175.5547626,"logger":"leader","msg":"No pre-existing lock was found."} {"level":"info","ts":1601472175.560202,"logger":"leader","msg":"Became the leader."} {"level":"info","ts":1601472177.9266758,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"} {"level":"info","ts":1601472177.934258,"logger":"cmd","msg":"registering resource schemes."} {"level":"info","ts":1601472177.960696,"logger":"versions","msg":"updating syndesis api","version from":"v1alpha1","version to":"v1beta1","from name":"","from version":"","to name":"app","to version":"1.7"} I0930 13:22:59.477339 1 request.go:621] Throttling request took 1.444773693s, request: GET:https://172.25.0.1:443/apis/storage.k8s.io/v1beta1?timeout=32s {"level":"info","ts":1601472182.7889469,"logger":"metrics","msg":"Metrics Service object updated","Service.Name":"syndesis-operator-metrics","Service.Namespace":"mkralik"} {"level":"info","ts":1601472185.1537335,"logger":"cmd","msg":"Could not create ServiceMonitor object","error":"servicemonitors.monitoring.coreos.com \"syndesis-operator-metrics\" already exists"} {"level":"info","ts":1601472185.1540997,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"syndesis-controller","source":"kind source: /, Kind="} {"level":"info","ts":1601472185.156269,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1601472185.254737,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"syndesis-controller"} {"level":"info","ts":1601472185.2548916,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"syndesis-controller","worker count":1} {"level":"info","ts":1601472188.2699525,"logger":"controller","msg":"syndesis resource changed in the meantime, requeue and rerun in 5 seconds","name":"app"} {"level":"info","ts":1601472188.2701402,"logger":"action","msg":"Upgrading syndesis resource ","type":"upgrade","name":"app","current version":"1.7","target version":"1.8"} {"level":"info","ts":1601472188.2701688,"logger":"action","msg":"running step","type":"upgrade","step":"Scale down"} {"level":"info","ts":1601472188.2702014,"logger":"action","msg":"scale DeploymentConfig","type":"upgrade","direction":"down","deployments":["syndesis-meta","syndesis-server"]} {"level":"info","ts":1601472188.370737,"logger":"action","msg":"scaling DeploymentConfigs","type":"upgrade","name":"syndesis-meta","desired replicas":0,"replicas":1} {"level":"info","ts":1601472188.395437,"logger":"action","msg":"scaling DeploymentConfigs","type":"upgrade","name":"syndesis-server","desired replicas":0,"replicas":1} {"level":"info","ts":1601472188.4446785,"logger":"action","msg":"waiting for DeploymentConfig to scale","type":"upgrade","direction":"down","deployments":["syndesis-meta","syndesis-server"]} {"level":"info","ts":1601472198.450857,"logger":"action","msg":"deploymentConfig successfully scaled","type":"upgrade","name":"syndesis-meta","desired replicas":0,"available replicas":0} {"level":"info","ts":1601472198.4509351,"logger":"action","msg":"deploymentConfig successfully scaled","type":"upgrade","name":"syndesis-server","desired replicas":0,"available replicas":0} {"level":"info","ts":1601472198.450944,"logger":"action","msg":"running step","type":"upgrade","step":"Backup"} {"level":"info","ts":1601472198.4509482,"logger":"backup","msg":"starting backup for syndesis","action":"backup"} {"level":"info","ts":1601472198.5696769,"logger":"backup","msg":"Initiating database backup ...","action":"backup"} I0930 13:23:19.621942 1 request.go:621] Throttling request took 1.043259258s, request: GET:https://172.25.0.1:443/apis/autoscaling/v1?timeout=32s {"level":"info","ts":1601472211.1728082,"logger":"backup","msg":"Backup compiler container terminated so extracting data from logger container","action":"backup"} {"level":"info","ts":1601472211.3054497,"logger":"backup","msg":"Backup extraction to complete","action":"backup","Running time":"10s"} {"level":"info","ts":1601472211.3323293,"logger":"backup","msg":"backup for syndesis done","action":"backup"} {"level":"info","ts":1601472211.332504,"logger":"action","msg":"running step","type":"upgrade","step":"Database migration"} I0930 13:23:32.383585 1 request.go:621] Throttling request took 1.041208854s, request: GET:https://172.25.0.1:443/apis/certificates.k8s.io/v1beta1?timeout=32s {"level":"info","ts":1601472223.7674992,"logger":"action","msg":"database migration job successfully finished","type":"upgrade","active jobs":0,"succeeded jobs":1,"running time":"10s"} I0930 13:23:44.818794 1 request.go:621] Throttling request took 1.031352249s, request: GET:https://172.25.0.1:443/apis/packages.operators.coreos.com/v1?timeout=32s {"level":"info","ts":1601472226.1702237,"logger":"util","msg":"Connecting to PostgreSQL server running at postgresql://syndesis-db:5432/syndesis?sslmode=disable"} {"level":"info","ts":1601472226.1759822,"logger":"action","msg":"Determined versions of PostgreSQL database","type":"upgrade","target":9.6,"current":9.6} {"level":"info","ts":1601472226.1760743,"logger":"action","msg":"running step","type":"upgrade","step":"Install"} {"level":"info","ts":1601472226.1761036,"logger":"action","msg":"running step","type":"upgrade","step":"Cleanup"} {"level":"info","ts":1601472226.183551,"logger":"action","msg":"force deleted DeploymentConfig","type":"upgrade","name":"syndesis-meta"} {"level":"info","ts":1601472226.191016,"logger":"action","msg":"force deleted DeploymentConfig","type":"upgrade","name":"syndesis-server"} {"level":"info","ts":1601472226.2282119,"logger":"action","msg":"force deleted DeploymentConfig","type":"upgrade","name":"syndesis-ui"} {"level":"info","ts":1601472226.2549593,"logger":"action","msg":"force deleted DeploymentConfig","type":"upgrade","name":"syndesis-prometheus"} {"level":"info","ts":1601472226.3612761,"logger":"action","msg":"Syndesis resource upgraded","type":"upgrade","name":"app","target version":"1.8"} {"level":"info","ts":1601472229.3788736,"logger":"controller","msg":"syndesis resource changed in the meantime, requeue and rerun in 5 seconds","name":"app"} {"level":"info","ts":1601472229.3792052,"logger":"action","msg":"failure while running post upgrade run","type":"upgrade","name":"app","target version":"1.8"} {"level":"info","ts":1601472229.379308,"logger":"action","msg":"attempting again to run post upgrade","type":"upgrade","name":"app"} {"level":"info","ts":1601472229.3793964,"logger":"action","msg":"installing Syndesis resource for the first time after upgrading","type":"install","name":"app"} {"level":"info","ts":1601472232.0854385,"logger":"action","msg":"resource updated","type":"install","kind":"ImageStream","name":"syndesis-s2i","namespace":"mkralik"} {"level":"info","ts":1601472232.0969605,"logger":"action","msg":"resource updated","type":"install","kind":"Secret","name":"syndesis-server-secret","namespace":"mkralik"} {"level":"info","ts":1601472232.1109123,"logger":"action","msg":"resource updated","type":"install","kind":"Secret","name":"syndesis-global-config","namespace":"mkralik"} {"level":"info","ts":1601472232.1400337,"logger":"action","msg":"resource updated","type":"install","kind":"ServiceAccount","name":"syndesis-default","namespace":"mkralik"} {"level":"info","ts":1601472232.158678,"logger":"action","msg":"resource updated","type":"install","kind":"ServiceAccount","name":"syndesis-server","namespace":"mkralik"} {"level":"info","ts":1601472232.1736894,"logger":"action","msg":"resource updated","type":"install","kind":"ServiceAccount","name":"syndesis-integration","namespace":"mkralik"} {"level":"info","ts":1601472232.190851,"logger":"action","msg":"resource updated","type":"install","kind":"ServiceAccount","name":"syndesis-prometheus","namespace":"mkralik"} {"level":"info","ts":1601472232.2171793,"logger":"action","msg":"resource updated","type":"install","kind":"ConfigMap","name":"syndesis-server-config","namespace":"mkralik"} {"level":"info","ts":1601472232.2395794,"logger":"action","msg":"resource updated","type":"install","kind":"Service","name":"syndesis-ui","namespace":"mkralik"} {"level":"info","ts":1601472232.2998533,"logger":"action","msg":"resource created","type":"install","kind":"DeploymentConfig","name":"syndesis-ui","namespace":"mkralik"} {"level":"info","ts":1601472232.312291,"logger":"action","msg":"resource updated","type":"install","kind":"ConfigMap","name":"syndesis-ui-config","namespace":"mkralik"} {"level":"info","ts":1601472232.3622074,"logger":"action","msg":"resource updated","type":"install","kind":"ImageStream","name":"jboss-amq-63","namespace":"mkralik"} {"level":"info","ts":1601472232.377311,"logger":"action","msg":"resource updated","type":"install","kind":"Service","name":"broker-amq-tcp","namespace":"mkralik"} {"level":"info","ts":1601472232.4163666,"logger":"action","msg":"resource updated","type":"install","kind":"DeploymentConfig","name":"broker-amq","namespace":"mkralik"} {"level":"info","ts":1601472232.4313598,"logger":"action","msg":"resource updated","type":"install","kind":"Service","name":"syndesis-meta","namespace":"mkralik"} {"level":"info","ts":1601472232.4941256,"logger":"action","msg":"resource updated","type":"install","kind":"PersistentVolumeClaim","name":"syndesis-meta","namespace":"mkralik"} {"level":"info","ts":1601472232.586162,"logger":"action","msg":"resource created","type":"install","kind":"DeploymentConfig","name":"syndesis-meta","namespace":"mkralik"} {"level":"info","ts":1601472232.6282291,"logger":"action","msg":"resource updated","type":"install","kind":"ConfigMap","name":"syndesis-meta-config","namespace":"mkralik"} {"level":"info","ts":1601472232.6711116,"logger":"action","msg":"resource updated","type":"install","kind":"Service","name":"syndesis-oauthproxy","namespace":"mkralik"} {"level":"info","ts":1601472232.7615163,"logger":"action","msg":"resource updated","type":"install","kind":"DeploymentConfig","name":"syndesis-oauthproxy","namespace":"mkralik"} {"level":"info","ts":1601472232.781472,"logger":"action","msg":"resource updated","type":"install","kind":"Service","name":"syndesis-server","namespace":"mkralik"} {"level":"info","ts":1601472232.8898404,"logger":"action","msg":"resource created","type":"install","kind":"DeploymentConfig","name":"syndesis-server","namespace":"mkralik"} {"level":"info","ts":1601472232.913532,"logger":"action","msg":"resource updated","type":"install","kind":"Role","name":"syndesis-editor","namespace":"mkralik"} {"level":"info","ts":1601472232.934851,"logger":"action","msg":"resource updated","type":"install","kind":"RoleBinding","name":"syndesis-server-editor","namespace":"mkralik"} {"level":"info","ts":1601472232.9537718,"logger":"action","msg":"resource updated","type":"install","kind":"Role","name":"syndesis-viewer","namespace":"mkralik"} {"level":"info","ts":1601472233.024392,"logger":"action","msg":"resource updated","type":"install","kind":"RoleBinding","name":"syndesis-prometheus-viewer","namespace":"mkralik"} {"level":"info","ts":1601472233.5285022,"logger":"action","msg":"resource created","type":"install","kind":"Role","name":"syndesis-server","namespace":"mkralik"} {"level":"info","ts":1601472233.7621543,"logger":"action","msg":"resource created","type":"install","kind":"RoleBinding","name":"syndesis-server","namespace":"mkralik"} {"level":"info","ts":1601472233.8316095,"logger":"action","msg":"resource updated","type":"install","kind":"ConfigMap","name":"syndesis-prometheus-config","namespace":"mkralik"} {"level":"info","ts":1601472233.967227,"logger":"action","msg":"resource updated","type":"install","kind":"Service","name":"syndesis-prometheus","namespace":"mkralik"} {"level":"info","ts":1601472234.03511,"logger":"action","msg":"resource updated","type":"install","kind":"PersistentVolumeClaim","name":"syndesis-prometheus","namespace":"mkralik"} {"level":"info","ts":1601472234.083608,"logger":"action","msg":"resource created","type":"install","kind":"DeploymentConfig","name":"syndesis-prometheus","namespace":"mkralik"} {"level":"info","ts":1601472234.1136742,"logger":"action","msg":"resource updated","type":"install","kind":"ConfigMap","name":"syndesis-db-metrics-config","namespace":"mkralik"} {"level":"info","ts":1601472234.1781406,"logger":"action","msg":"resource updated","type":"install","kind":"ConfigMap","name":"syndesis-sampledb-config","namespace":"mkralik"} {"level":"info","ts":1601472234.1912189,"logger":"action","msg":"resource updated","type":"install","kind":"ConfigMap","name":"syndesis-db-conf","namespace":"mkralik"} {"level":"info","ts":1601472234.2631075,"logger":"action","msg":"resource updated","type":"install","kind":"Service","name":"syndesis-db","namespace":"mkralik"} {"level":"info","ts":1601472234.3080082,"logger":"action","msg":"resource updated","type":"install","kind":"PersistentVolumeClaim","name":"syndesis-db","namespace":"mkralik"} {"level":"info","ts":1601472234.5037932,"logger":"action","msg":"resource updated","type":"install","kind":"DeploymentConfig","name":"syndesis-db","namespace":"mkralik"} E0930 13:23:55.164575 1 reflector.go:382] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:224: Failed to watch *v1.PackageManifest: unknown (get packagemanifests.packages.operators.coreos.com) E0930 13:23:55.630389 1 reflector.go:382] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:224: Failed to watch *v1.PackageManifest: unknown (get packagemanifests.packages.operators.coreos.com) {"level":"info","ts":1601472242.0929532,"logger":"action","msg":"resource updated","type":"install","kind":"Jaeger","name":"syndesis-jaeger","namespace":"mkralik"} {"level":"info","ts":1601472242.4212048,"logger":"action","msg":"resource updated","type":"install","kind":"Service","name":"noauth-syndesis-jaeger-query","namespace":"mkralik"} I0930 13:24:03.596480 1 request.go:621] Throttling request took 1.049728491s, request: GET:https://172.25.0.1:443/apis/autoscaling.openshift.io/v1?timeout=32s E0930 13:24:03.987894 1 reflector.go:382] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:224: Failed to watch *v1.PackageManifest: unknown (get packagemanifests.packages.operators.coreos.com) {"level":"info","ts":1601472245.2441225,"logger":"action","msg":"resource deleted","type":"install","kind":"ConfigMap","name":"syndesis-prometheus-agent-config","namespace":"mkralik"} {"level":"info","ts":1601472246.0705957,"logger":"action","msg":"Syndesis resource installed after upgrading","type":"install","name":"app"} {"level":"info","ts":1601472246.0706773,"logger":"controller","msg":"syndesis resource changed in the meantime, requeue and rerun in 5 seconds","name":"app"} {"level":"info","ts":1601472246.0707622,"logger":"action","msg":"syndesis resource post upgrade ran successfully","type":"upgrade","name":"app","previous version":"1.7","target version":"1.8"} {"level":"info","ts":1601472249.2457685,"logger":"controller","msg":"syndesis resource changed in the meantime, requeue and rerun in 5 seconds","name":"app"} E0930 13:24:12.240874 1 reflector.go:382] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:224: Failed to watch *v1.PackageManifest: unknown (get packagemanifests.packages.operators.coreos.com) I0930 13:24:16.430822 1 request.go:621] Throttling request took 1.03277178s, request: GET:https://172.25.0.1:443/apis/operator.openshift.io/v1?timeout=32s {"level":"info","ts":1601472265.8218637,"logger":"action","msg":"resource updated","type":"install","kind":"Jaeger","name":"syndesis-jaeger","namespace":"mkralik"} E0930 13:24:26.848170 1 reflector.go:382] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:224: Failed to watch *v1.PackageManifest: unknown (get packagemanifests.packages.operators.coreos.com) I0930 13:24:27.242238 1 request.go:621] Throttling request took 1.076103506s, request: GET:https://172.25.0.1:443/apis/autoscaling.openshift.io/v1?timeout=32s I0930 13:24:37.321862 1 request.go:621] Throttling request took 1.754741063s, request: GET:https://172.25.0.1:443/apis/camel.apache.org/v1alpha1?timeout=32s E0930 13:24:48.840334 1 reflector.go:382] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:224: Failed to watch *v1.PackageManifest: unknown (get packagemanifests.packages.operators.coreos.com) I0930 13:24:58.641076 1 request.go:621] Throttling request took 1.03300136s, request: GET:https://172.25.0.1:443/apis/operators.coreos.com/v1alpha2?timeout=32s I0930 13:25:20.489039 1 request.go:621] Throttling request took 1.047924368s, request: GET:https://172.25.0.1:443/apis/operator.openshift.io/v1alpha1?timeout=32s E0930 13:25:22.946567 1 reflector.go:382] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:224: Failed to watch *v1.PackageManifest: unknown (get packagemanifests.packages.operators.coreos.com) I0930 13:25:42.111055 1 request.go:621] Throttling request took 1.034648637s, request: GET:https://172.25.0.1:443/apis/config.openshift.io/v1?timeout=32s I0930 13:26:03.978891 1 request.go:621] Throttling request took 1.040909975s, request: GET:https://172.25.0.1:443/apis/flowcontrol.apiserver.k8s.io/v1alpha1?timeout=32s E0930 13:26:15.232677 1 reflector.go:382] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:224: Failed to watch *v1.PackageManifest: unknown (get packagemanifests.packages.operators.coreos.com) I0930 13:26:25.696115 1 request.go:621] Throttling request took 1.03417976s, request: GET:https://172.25.0.1:443/apis/storage.k8s.io/v1beta1?timeout=32s I0930 13:26:47.218105 1 request.go:621] Throttling request took 1.039792332s, request: GET:https://172.25.0.1:443/apis/apiextensions.k8s.io/v1beta1?timeout=32s ...
- relates to
-
ENTESB-15421 Syndesis fails when updating via operatorhub
- Done