Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-12923

Sometimes some of the pods do not come up after upgrade

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Critical Critical
    • fuse-7.7-GA
    • None
    • Fuse Online
    • None
    • % %
    • Cannot reproduce, please reopen when run into the issue again

      Sometimes it happens to me that some of the deployments do not come back up after the upgrade, it happened to me with various pods (server, meta, ui, prometheus)

      The state after the upgrade - missing server and ui pods:

      syndesis-db-2-fpwqp           2/2     Running     0          11m
      syndesis-meta-1-8sskj         1/1     Running     0          11m
      syndesis-oauthproxy-2-hzcm7   1/1     Running     0          11m
      syndesis-operator-2-wbx6t     1/1     Running     0          14m
      syndesis-prometheus-1-hfbtw   1/1     Running     0          11m
      todo-1-build                  0/1     Completed   0          18m
      todo-1-wvwg2                  1/1     Running     0          17m
      upgrade-db-migration-bn6wd    0/1     Completed   0          14m
      
      oc get dc                           
      NAME                  REVISION   DESIRED   CURRENT   TRIGGERED BY
      syndesis-db           2          1         1         config
      syndesis-meta         1          1         1         config
      syndesis-oauthproxy   2          1         1         config
      syndesis-operator     2          1         1         config,image(syndesis-operator:1.9.1-20200203)
      syndesis-prometheus   1          1         1         config
      syndesis-server       1          1         0         config
      syndesis-ui           1          1         0         config
      

      related part of the operator log:

      "level":"info","ts":1580981281.3294973,"logger":"action","msg":"Syndesis resource installed after upgrading","type":"install","name":"app"}
      {"level":"info","ts":1580981281.3340726,"logger":"action","msg":"failure while running post upgrade run","type":"upgrade","name":"app","target version":"1.9.1-20200203"}
      {"level":"info","ts":1580981281.3341372,"logger":"action","msg":"attempting again to run post upgrade","type":"upgrade","name":"app"}
      {"level":"info","ts":1580981281.334163,"logger":"action","msg":"installing Syndesis resource for the first time after upgrading","type":"install","name":"app"}
      {"level":"info","ts":1580981281.340141,"logger":"action","msg":"force deleted DeploymentConfig","type":"install","name":"syndesis-meta","app":"app"}
      {"level":"info","ts":1580981281.3462307,"logger":"action","msg":"force deleted DeploymentConfig","type":"install","name":"syndesis-server","app":"app"}
      {"level":"info","ts":1580981281.3534343,"logger":"action","msg":"force deleted DeploymentConfig","type":"install","name":"syndesis-ui","app":"app"}
      {"level":"info","ts":1580981281.366618,"logger":"action","msg":"force deleted DeploymentConfig","type":"install","name":"syndesis-prometheus","app":"app"}
      {"level":"info","ts":1580981286.4191034,"logger":"action","msg":"resource created","type":"install","kind":"DeploymentConfig","name":"syndesis-ui","namespace":"avano-syndesis"}
      {"level":"info","ts":1580981286.4474368,"logger":"action","msg":"resource created","type":"install","kind":"DeploymentConfig","name":"syndesis-meta","namespace":"avano-syndesis"}
      {"level":"info","ts":1580981286.6045678,"logger":"action","msg":"resource created","type":"install","kind":"DeploymentConfig","name":"syndesis-server","namespace":"avano-syndesis"}
      {"level":"info","ts":1580981286.6597192,"logger":"action","msg":"resource created","type":"install","kind":"DeploymentConfig","name":"syndesis-prometheus","namespace":"avano-syndesis"}
      {"level":"error","ts":1580981291.4210515,"logger":"controller","msg":"Error reconciling","action":"*action.installAction","phase":"PostUpgradeRun","error":"Operation cannot be fulfilled on syndesises.syndesis.io \"app\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsrc/github.com/syndesisio/syndesis/install/operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/syndesisio/syndesis/install/operator/pkg/controller/syndesis.(*ReconcileSyndesis).Reconcile\n\tsrc/github.com/syndesisio/syndesis/install/operator/pkg/controller/syndesis/syndesis_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsrc/github.com/syndesisio/syndesis/install/operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsrc/github.com/syndesisio/syndesis/install/operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tsrc/github.com/syndesisio/syndesis/install/operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tsrc/github.com/syndesisio/syndesis/install/operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tsrc/github.com/syndesisio/syndesis/install/operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\tsrc/github.com/syndesisio/syndesis/install/operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
      {"level":"info","ts":1580981291.4213526,"logger":"action","msg":"syndesis resource post upgrade ran successfully","type":"upgrade","name":"app","previous version":"1.8.13","target version":"1.9.1-20200203"}
      {"level":"info","ts":1580981294.4262102,"logger":"controller","msg":"syndesis resource changed in the meantime, requeue and rerun in 5 seconds","name":"app"}
      

      I can then rollout them manually and everything works:

      ⏽ ~ :: oc rollout latest dc/syndesis-server    
      deploymentconfig.apps.openshift.io/syndesis-server rolled out
      ⏽ ~ :: oc rollout latest dc/syndesis-ui    
      deploymentconfig.apps.openshift.io/syndesis-ui rolled out
      
      NAME                          READY   STATUS      RESTARTS   AGE
      syndesis-db-2-fpwqp           2/2     Running     0          15m
      syndesis-meta-1-8sskj         1/1     Running     0          15m
      syndesis-oauthproxy-2-hzcm7   1/1     Running     0          15m
      syndesis-operator-2-wbx6t     1/1     Running     0          17m
      syndesis-prometheus-1-hfbtw   1/1     Running     0          15m
      syndesis-server-2-rgf8h       1/1     Running     0          1m
      syndesis-ui-2-sb8vc           1/1     Running     0          1m
      todo-1-build                  0/1     Completed   0          22m
      todo-1-wvwg2                  1/1     Running     0          21m                                                                                                                                                   
      upgrade-db-migration-bn6wd    0/1     Completed   0          17m
      
      

            parichar@redhat.com Paul Richardson
            avano@redhat.com Andrej Vano
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: