Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-25306

[GSS](8.0.z) Sync model operations fail when a HC with stopped managed servers is registered back in the domain

XMLWordPrintable

    • False
    • None
    • False
    • Workaround Exists
    • Hide

      To get the HC registered we should start the managed server:

      • Connect it to the HC management interface and start the servers on that HC.

      This workaround only makes sense if we want the managed servers started, if those servers cannot be started for any reason, we have to remove them from the HC

      If the root case of the synchronization is an application re-deployment, avoid the re-deployment and replace it with an undeploy followed by a deploy operation.

      Show
      To get the HC registered we should start the managed server: Connect it to the HC management interface and start the servers on that HC. This workaround only makes sense if we want the managed servers started, if those servers cannot be started for any reason, we have to remove them from the HC If the root case of the synchronization is an application re-deployment, avoid the re-deployment and replace it with an undeploy followed by a deploy operation.
    • Hide
      • Start a domain with a DC and a remote HC. The remote HC has several managed servers. The DC has no managed servers. All the managed servers are started and running.
      • Stop a managed server on the remote HC: [domain@localhost:9990 /] /host=remote/server=server-one-001:stop
      • Deploy and application from the DC and then force its re-deployment with --force. This will trigger a resynchronization after restarting the DC.
      • Restart the DC without stopping the HC: [domain@localhost:9990 /] /host=primary:shutdown(restart=true)
      Show
      Start a domain with a DC and a remote HC. The remote HC has several managed servers. The DC has no managed servers. All the managed servers are started and running. Stop a managed server on the remote HC:  [domain@localhost:9990 /]  /host=remote/server=server-one-001:stop Deploy and application from the DC and then force its re-deployment with --force. This will trigger a resynchronization after restarting the DC. Restart the DC without stopping the HC: [domain@localhost:9990 /]  /host=primary:shutdown(restart=true)

      When a DC is restarted any running HC will try to automatically register itself into the domain. In the registration process, the HC pulls down the DC configuration, compares it with the current copy stored in the HC, and if there are DC configuration updates, these updates are propagated as management operations to the HC managed servers.

      When there are pending updates, these updates are also applied to any stopped managed server.

      On running managed servers, a ProxyController is in charge of deriving the required managed operations to the running servers. This ProxyController returns a handler for anything. However, for stopped managed servers, we use StoppedServerResource which doesn't register a handler for ServerProcessStateHandler.REQUIRE_RESTART_OPERATION or ServerProcessStateHandler.REQUIRE_RELOAD_OPERATION. 

      This situation leads to the following error trace logged on the HC side:

      [Host Controller] 11:01:02,820 ERROR [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 11) WFLYCTL0013: Operation ("apply-remote-domain-model") failed - address: ([]): java.lang.IllegalArgumentException: Parameter 'step' may not be null
[Host Controller]     at org.wildfly.common@1.6.0.Final//org.wildfly.common.Assert.checkNotNullParamChecked(Assert.java:71)
[Host Controller]     at org.wildfly.common@1.6.0.Final//org.wildfly.common.Assert.checkNotNullParam(Assert.java:49)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.AbstractOperationContext.addStep(AbstractOperationContext.java:314)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.AbstractOperationContext.addStep(AbstractOperationContext.java:299)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.AbstractOperationContext.addStep(AbstractOperationContext.java:266)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.domain.controller.operations.SyncServerStateOperationHandler$1.execute(SyncServerStateOperationHandler.java:121)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:1040)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:772)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:466)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1435)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:448)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:356)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.AbstractControllerService.internalExecute(AbstractControllerService.java:466)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.DomainModelControllerService.access$100(DomainModelControllerService.java:203)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.DomainModelControllerService$InternalExecutor$1.lambda$apply$0(DomainModelControllerService.java:1458)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.access.InVmAccess.runInVm(InVmAccess.java:63)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.DomainModelControllerService$InternalExecutor$1.apply(DomainModelControllerService.java:1458)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.DomainModelControllerService$InternalExecutor$1.apply(DomainModelControllerService.java:1455)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.SecurityActions$Execution$1.execute(SecurityActions.java:77)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.SecurityActions.privilegedExecution(SecurityActions.java:47)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.DomainModelControllerService$InternalExecutor.execute(DomainModelControllerService.java:1461)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.RemoteDomainConnectionService.applyRemoteDomainModel(RemoteDomainConnectionService.java:595)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.RemoteDomainConnectionService$3.applyDomainModel(RemoteDomainConnectionService.java:538)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.RemoteDomainConnection.applyDomainModel(RemoteDomainConnection.java:315)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.RemoteDomainConnection$RegisterSubsystemsRequest$1.execute(RemoteDomainConnection.java:458)
[Host Controller]     at org.jboss.as.protocol@21.0.0.Beta2-SNAPSHOT//org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$1.doExecute(ManagementRequestContextImpl.java:70)
[Host Controller]     at org.jboss.as.protocol@21.0.0.Beta2-SNAPSHOT//org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$AsyncTaskRunner.run(ManagementRequestContextImpl.java:160)
[Host Controller]     at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[Host Controller]     at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
[Host Controller]     at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
[Host Controller]     at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
[Host Controller]     at java.base/java.lang.Thread.run(Thread.java:834)
[Host Controller]     at org.jboss.threads@2.4.0.Final//org.jboss.threads.JBossThread.run(JBossThread.java:513)
[Host Controller]
[Host Controller] 11:01:02,822 ERROR [org.jboss.as.host.controller] (Host Controller Service Threads - 11) WFLYHC0143: Failed to apply domain-wide configuration from the Domain Controller. Operation outcome: failed. Failure description "WFLYCTL0158: Operation handler failed: java.lang.IllegalArgumentException: Parameter 'step' may not be null"
[Host Controller] 11:01:02,824 WARN  [org.jboss.as.host.controller] (Host Controller Service Threads - 9) WFLYHC0146: Could not discover the domain controller using discovery option StaticDiscovery{protocol=remote+http,host=localhost,port=9990}. Error was: 1-$-
[Host Controller] 11:01:02,837 WARN  [org.jboss.as.host.controller] (Host Controller Service Threads - 9) WFLYHC0147: No domain controller discovery options remain.

      Which makes the sync process fail. We can see on the DC side the following error:

      [Host Controller] 11:01:02,828 INFO  [org.jboss.as.protocol] (management I/O-1) WFLYPRT0057:  cancelled task by interrupting thread Thread[Host Controller Service Threads - 29,5,Host Controller Service Threads]

            yborgess1@redhat.com Yeray Borges Santana
            pberan@redhat.com Petr Beran
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: