-
Bug
-
Resolution: Cannot Reproduce
-
Critical
-
None
-
None
-
%
-
Cannot reproduce, please reopen when run into the issue again
Sometimes it happens to me that some of the deployments do not come back up after the upgrade, it happened to me with various pods (server, meta, ui, prometheus)
The state after the upgrade - missing server and ui pods:
syndesis-db-2-fpwqp 2/2 Running 0 11m
syndesis-meta-1-8sskj 1/1 Running 0 11m
syndesis-oauthproxy-2-hzcm7 1/1 Running 0 11m
syndesis-operator-2-wbx6t 1/1 Running 0 14m
syndesis-prometheus-1-hfbtw 1/1 Running 0 11m
todo-1-build 0/1 Completed 0 18m
todo-1-wvwg2 1/1 Running 0 17m
upgrade-db-migration-bn6wd 0/1 Completed 0 14m
oc get dc NAME REVISION DESIRED CURRENT TRIGGERED BY syndesis-db 2 1 1 config syndesis-meta 1 1 1 config syndesis-oauthproxy 2 1 1 config syndesis-operator 2 1 1 config,image(syndesis-operator:1.9.1-20200203) syndesis-prometheus 1 1 1 config syndesis-server 1 1 0 config syndesis-ui 1 1 0 config
related part of the operator log:
"level":"info","ts":1580981281.3294973,"logger":"action","msg":"Syndesis resource installed after upgrading","type":"install","name":"app"} {"level":"info","ts":1580981281.3340726,"logger":"action","msg":"failure while running post upgrade run","type":"upgrade","name":"app","target version":"1.9.1-20200203"} {"level":"info","ts":1580981281.3341372,"logger":"action","msg":"attempting again to run post upgrade","type":"upgrade","name":"app"} {"level":"info","ts":1580981281.334163,"logger":"action","msg":"installing Syndesis resource for the first time after upgrading","type":"install","name":"app"} {"level":"info","ts":1580981281.340141,"logger":"action","msg":"force deleted DeploymentConfig","type":"install","name":"syndesis-meta","app":"app"} {"level":"info","ts":1580981281.3462307,"logger":"action","msg":"force deleted DeploymentConfig","type":"install","name":"syndesis-server","app":"app"} {"level":"info","ts":1580981281.3534343,"logger":"action","msg":"force deleted DeploymentConfig","type":"install","name":"syndesis-ui","app":"app"} {"level":"info","ts":1580981281.366618,"logger":"action","msg":"force deleted DeploymentConfig","type":"install","name":"syndesis-prometheus","app":"app"} {"level":"info","ts":1580981286.4191034,"logger":"action","msg":"resource created","type":"install","kind":"DeploymentConfig","name":"syndesis-ui","namespace":"avano-syndesis"} {"level":"info","ts":1580981286.4474368,"logger":"action","msg":"resource created","type":"install","kind":"DeploymentConfig","name":"syndesis-meta","namespace":"avano-syndesis"} {"level":"info","ts":1580981286.6045678,"logger":"action","msg":"resource created","type":"install","kind":"DeploymentConfig","name":"syndesis-server","namespace":"avano-syndesis"} {"level":"info","ts":1580981286.6597192,"logger":"action","msg":"resource created","type":"install","kind":"DeploymentConfig","name":"syndesis-prometheus","namespace":"avano-syndesis"} {"level":"error","ts":1580981291.4210515,"logger":"controller","msg":"Error reconciling","action":"*action.installAction","phase":"PostUpgradeRun","error":"Operation cannot be fulfilled on syndesises.syndesis.io \"app\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/syndesisio/syndesis/install/operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/syndesisio/syndesis/install/operator/pkg/controller/syndesis.(*ReconcileSyndesis).Reconcile\n\tsrc/github.com/syndesisio/syndesis/install/operator/pkg/controller/syndesis/syndesis_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsrc/github.com/syndesisio/syndesis/install/operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsrc/github.com/syndesisio/syndesis/install/operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tsrc/github.com/syndesisio/syndesis/install/operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tsrc/github.com/syndesisio/syndesis/install/operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tsrc/github.com/syndesisio/syndesis/install/operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\tsrc/github.com/syndesisio/syndesis/install/operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} {"level":"info","ts":1580981291.4213526,"logger":"action","msg":"syndesis resource post upgrade ran successfully","type":"upgrade","name":"app","previous version":"1.8.13","target version":"1.9.1-20200203"} {"level":"info","ts":1580981294.4262102,"logger":"controller","msg":"syndesis resource changed in the meantime, requeue and rerun in 5 seconds","name":"app"}
I can then rollout them manually and everything works:
⏽ ~ :: oc rollout latest dc/syndesis-server deploymentconfig.apps.openshift.io/syndesis-server rolled out ⏽ ~ :: oc rollout latest dc/syndesis-ui deploymentconfig.apps.openshift.io/syndesis-ui rolled out
NAME READY STATUS RESTARTS AGE
syndesis-db-2-fpwqp 2/2 Running 0 15m
syndesis-meta-1-8sskj 1/1 Running 0 15m
syndesis-oauthproxy-2-hzcm7 1/1 Running 0 15m
syndesis-operator-2-wbx6t 1/1 Running 0 17m
syndesis-prometheus-1-hfbtw 1/1 Running 0 15m
syndesis-server-2-rgf8h 1/1 Running 0 1m
syndesis-ui-2-sb8vc 1/1 Running 0 1m
todo-1-build 0/1 Completed 0 22m
todo-1-wvwg2 1/1 Running 0 21m
upgrade-db-migration-bn6wd 0/1 Completed 0 17m