Uploaded image for project: 'WildFly Core'
  1. WildFly Core
  2. WFCORE-6375

Sync model operations fail when an HC with stopped managed servers registers back in the domain

    XMLWordPrintable

Details

    • Hide
      • Start a domain with a DC and a remote HC. The remote HC has several managed servers. The DC has no managed servers. All the managed servers are started and running.
      • Stop a managed server on the remote HC: [domain@localhost:9990 /] /host=remote/server=server-one-001:stop
      • Deploy and application from the DC and then force its re-deployment with --force. This will trigger a resynchronization after stopping the
      • Restart the DC without stopping the HC: [domain@localhost:9990 /] /host=primary:shutdown(restart=true)
      Show
      Start a domain with a DC and a remote HC. The remote HC has several managed servers. The DC has no managed servers. All the managed servers are started and running. Stop a managed server on the remote HC:  [domain@localhost:9990 /]  /host=remote/server=server-one-001:stop Deploy and application from the DC and then force its re-deployment with --force. This will trigger a resynchronization after stopping the Restart the DC without stopping the HC: [domain@localhost:9990 /]  /host=primary:shutdown(restart=true)
    • Workaround Exists
    • Hide

      To get the HC registered we should start the managed server:

      • Connect it to the HC management interface and start the servers on that HC.

      This workaround only makes sense if we want the managed servers started, if those servers cannot be started for any reason, we have to remove them from the HC

      Fi the root case of the syncronization is an application re-deployment, avoid an re-deployment and replace it by a undeploy followef by a deploy

      Show
      To get the HC registered we should start the managed server: Connect it to the HC management interface and start the servers on that HC. This workaround only makes sense if we want the managed servers started, if those servers cannot be started for any reason, we have to remove them from the HC Fi the root case of the syncronization is an application re-deployment, avoid an re-deployment and replace it by a undeploy followef by a deploy

    Description

      When a DC is restarted any running HC will try to automatically register itself into the domain. In the registration process, the HC pulls down the DC configuration, compares it with the current copy stored in the HC, and if there are DC configuration updates, these updates are propagated as management operations to the HC managed servers.

      When there are pending updates, these updates are also applied to any stopped managed server.

      On running managed servers, a ProxyController is in charge of deriving the required managed operations to the running servers. This ProxyController returns a handler for anything. However, for stopped managed servers, we use StoppedServerResource which doesn't register a handler for ServerProcessStateHandler.REQUIRE_RESTART_OPERATION or ServerProcessStateHandler.REQUIRE_RELOAD_OPERATION. 

      This situation leads to the following error trace logged on the HC side:

      [Host Controller] 11:01:02,820 ERROR [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 11) WFLYCTL0013: Operation ("apply-remote-domain-model") failed - address: ([]): java.lang.IllegalArgumentException: Parameter 'step' may not be null
[Host Controller]     at org.wildfly.common@1.6.0.Final//org.wildfly.common.Assert.checkNotNullParamChecked(Assert.java:71)
[Host Controller]     at org.wildfly.common@1.6.0.Final//org.wildfly.common.Assert.checkNotNullParam(Assert.java:49)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.AbstractOperationContext.addStep(AbstractOperationContext.java:314)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.AbstractOperationContext.addStep(AbstractOperationContext.java:299)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.AbstractOperationContext.addStep(AbstractOperationContext.java:266)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.domain.controller.operations.SyncServerStateOperationHandler$1.execute(SyncServerStateOperationHandler.java:121)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:1040)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:772)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:466)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1435)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:448)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:356)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.AbstractControllerService.internalExecute(AbstractControllerService.java:466)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.DomainModelControllerService.access$100(DomainModelControllerService.java:203)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.DomainModelControllerService$InternalExecutor$1.lambda$apply$0(DomainModelControllerService.java:1458)
[Host Controller]     at org.jboss.as.controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.controller.access.InVmAccess.runInVm(InVmAccess.java:63)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.DomainModelControllerService$InternalExecutor$1.apply(DomainModelControllerService.java:1458)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.DomainModelControllerService$InternalExecutor$1.apply(DomainModelControllerService.java:1455)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.SecurityActions$Execution$1.execute(SecurityActions.java:77)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.SecurityActions.privilegedExecution(SecurityActions.java:47)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.DomainModelControllerService$InternalExecutor.execute(DomainModelControllerService.java:1461)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.RemoteDomainConnectionService.applyRemoteDomainModel(RemoteDomainConnectionService.java:595)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.RemoteDomainConnectionService$3.applyDomainModel(RemoteDomainConnectionService.java:538)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.RemoteDomainConnection.applyDomainModel(RemoteDomainConnection.java:315)
[Host Controller]     at org.jboss.as.host-controller@21.0.0.Beta2-SNAPSHOT//org.jboss.as.host.controller.RemoteDomainConnection$RegisterSubsystemsRequest$1.execute(RemoteDomainConnection.java:458)
[Host Controller]     at org.jboss.as.protocol@21.0.0.Beta2-SNAPSHOT//org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$1.doExecute(ManagementRequestContextImpl.java:70)
[Host Controller]     at org.jboss.as.protocol@21.0.0.Beta2-SNAPSHOT//org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$AsyncTaskRunner.run(ManagementRequestContextImpl.java:160)
[Host Controller]     at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[Host Controller]     at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
[Host Controller]     at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
[Host Controller]     at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
[Host Controller]     at java.base/java.lang.Thread.run(Thread.java:834)
[Host Controller]     at org.jboss.threads@2.4.0.Final//org.jboss.threads.JBossThread.run(JBossThread.java:513)
[Host Controller]
[Host Controller] 11:01:02,822 ERROR [org.jboss.as.host.controller] (Host Controller Service Threads - 11) WFLYHC0143: Failed to apply domain-wide configuration from the Domain Controller. Operation outcome: failed. Failure description "WFLYCTL0158: Operation handler failed: java.lang.IllegalArgumentException: Parameter 'step' may not be null"
[Host Controller] 11:01:02,824 WARN  [org.jboss.as.host.controller] (Host Controller Service Threads - 9) WFLYHC0146: Could not discover the domain controller using discovery option StaticDiscovery{protocol=remote+http,host=localhost,port=9990}. Error was: 1-$-
[Host Controller] 11:01:02,837 WARN  [org.jboss.as.host.controller] (Host Controller Service Threads - 9) WFLYHC0147: No domain controller discovery options remain.

      Which makes the sync process fail. We can see on the DC side the following error:

      [Host Controller] 11:01:02,828 INFO  [org.jboss.as.protocol] (management I/O-1) WFLYPRT0057:  cancelled task by interrupting thread Thread[Host Controller Service Threads - 29,5,Host Controller Service Threads]

      Attachments

        Issue Links

          is incorporated by

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-24931 [GSS](7.4.z) Sync model operations fail when a HC with stopped managed servers is registered back in the domain

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Verified
          is related to

          Bug - A problem that impairs or prevents the functions of the product. WFCORE-6434 Managed servers could ignore restart/reload required operations when HC reconnects to the domain

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved

          Activity

            People

              yborgess1@redhat.com Yeray Borges Santana
              yborgess1@redhat.com Yeray Borges Santana
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: