Uploaded image for project: 'Red Hat Decision Manager'
  1. Red Hat Decision Manager
  2. RHDM-531

OptaWeb employee rostering image using Mysql as external db has TimeoutException during startup

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Partially Completed
    • Affects Version/s: 7.0.0.GA
    • Fix Version/s: None
    • Component/s: Deployment, OptaPlanner
    • Labels:
      None
    • Environment:

      OCP 3.9
      RHDM 7.0 OptaShift employee rostering image
      MySql 5.5

    • Steps to Reproduce:
      Hide

      oc new-app \
      -e MYSQL_USER=mysqluser \
      -e MYSQL_PASSWORD=password \
      -e MYSQL_DATABASE=root \
      mysql:5.5 \
      --name=external-db

      oc process -n calvin-test -f rhdm70-optashift-externaldb.yaml -o yaml \
      -p IMAGE_STREAM_NAMESPACE=calvin-test -p IMAGE_STREAM_TAG=1.0 \
      -p DBE_USERNAME=mysqluser -p DBE_PASSWORD=password -p DBE_DATABASE=root -p DBE_DRIVER=mysql -p DBE_SERVICE_HOST=external-db -p DBE_SERVICE_PORT=3306 \
      -p OPTASHIFT_ROSTERING_PERSISTENCE_DATASOURCE=java:jboss/datasources/dbe -p OPTASHIFT_ROSTERING_PERSISTENCE_DIALECT=org.hibernate.dialect.MySQL5Dialect \
      > myapp_rhdm70-optashift-externaldb_mysql.yaml
      oc create -f myapp_rhdm70-optashift-externaldb_mysql.yaml

      Show
      oc new-app \ -e MYSQL_USER=mysqluser \ -e MYSQL_PASSWORD=password \ -e MYSQL_DATABASE=root \ mysql:5.5 \ --name=external-db oc process -n calvin-test -f rhdm70-optashift-externaldb.yaml -o yaml \ -p IMAGE_STREAM_NAMESPACE=calvin-test -p IMAGE_STREAM_TAG=1.0 \ -p DBE_USERNAME=mysqluser -p DBE_PASSWORD=password -p DBE_DATABASE=root -p DBE_DRIVER=mysql -p DBE_SERVICE_HOST=external-db -p DBE_SERVICE_PORT=3306 \ -p OPTASHIFT_ROSTERING_PERSISTENCE_DATASOURCE=java:jboss/datasources/dbe -p OPTASHIFT_ROSTERING_PERSISTENCE_DIALECT=org.hibernate.dialect.MySQL5Dialect \ > myapp_rhdm70-optashift-externaldb_mysql.yaml oc create -f myapp_rhdm70-optashift-externaldb_mysql.yaml
    • Workaround:
      Workaround Exists
    • Workaround Description:
      Hide

      There is a workaround I found: -Djboss.as.management.blocking.timeout=600

      After add this argument for JVM, I sometime still saw exception in the pod, then the solution is wait for a few more minutes (total time might be around 10 minutes), then just restart the pod.

      2nd time startup, there was no more error in the pod's log.

      I felt somehow the ORM hibernate layer is doing something really slow.

      Show
      There is a workaround I found: -Djboss.as.management.blocking.timeout=600 After add this argument for JVM, I sometime still saw exception in the pod, then the solution is wait for a few more minutes (total time might be around 10 minutes), then just restart the pod. 2nd time startup, there was no more error in the pod's log. I felt somehow the ORM hibernate layer is doing something really slow.

      Description

      When use MySql as external DB for Optashift in OCP 3.9, during 1st startup, the pod would run into timeout exception around 5 minutes.

      And when switch to Postgresql as external DB and keep everything else the same, there is no problem.

      There is a workaround I found: -Djboss.as.management.blocking.timeout=600
      even with this, the pod sometimes has to be restarted again.

      Error log:

      
      17:34:28,252 INFO  [org.hibernate.hql.internal.QueryTranslatorFactoryInitiator] (ServerService Thread Pool -- 69) HHH000397: Using ASTQueryTranslatorFactory
      17:34:28,346 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 71) WFLYCLINF0002: Started client-mappings cache from ejb container
      17:39:14,147 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[
          ("core-service" => "management"),
          ("management-interface" => "http-interface")
      ]'
      17:39:19,152 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler@63dda7c6 for operation add-deployer-chains at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
      	at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:522)
      	at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1485)
      	at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1439)
      	at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1412)
      	at org.jboss.as.controller.AbstractOperationContext$Step.access$400(AbstractOperationContext.java:1286)
      	at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:859)
      	at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:709)
      	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:450)
      	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1402)
      	at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:516)
      	at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:468)
      	at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:430)
      	at org.jboss.as.server.ServerService.boot(ServerService.java:437)
      	at org.jboss.as.server.ServerService.boot(ServerService.java:396)
      	at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:370)
      	at java.lang.Thread.run(Thread.java:748)
      
      17:39:19,153 ERROR [org.jboss.as.controller.client] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler@63dda7c6 for operation add-deployer-chains at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException
      17:39:24,154 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.domain.management.security.SecurityRealmAddHandler$ServiceInstallStepHandler@63e6563f for operation add at address [
          ("core-service" => "management"),
          ("security-realm" => "ApplicationRealm")
      ] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
      	at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:522)
      	at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1485)
      	at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1439)
      	at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1422)
      	at org.jboss.as.controller.AbstractOperationContext$Step.access$400(AbstractOperationContext.java:1286)
      	at org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:859)
      	at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:709)
      	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:450)
      	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1402)
      	at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:516)
      	at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:468)
      	at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:430)
      	at org.jboss.as.server.ServerService.boot(ServerService.java:437)
      	at org.jboss.as.server.ServerService.boot(ServerService.java:396)
      	at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:370)
      	at java.lang.Thread.run(Thread.java:748)
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  cchianel Christopher Chianelli
                  Reporter:
                  czhu calvin zhu
                  Tester:
                  Karel Suta
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  7 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: