Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-15547

[GSS] (7.1.z) HC cannot connect to DC after lost connect with error "WFLYCTL0332: Permission denied\"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 7.1.6.CR1, 7.1.6.GA
    • None
    • Security
    • None
    • Hide

      Attached to the issue there is domain and slave configurations that can be used to reproduce. Steps are described here:

      1. Create a management user which will be used for DC / HC authentication
        ./bin/add-user.sh -u admin -p admin -g management -ds
      2. Edit host-slave.xml and :
        • Replace the existing secret the for ManagementRealm security with the one generated for the user admin
        • Add the attribute username="admin" in the domain-controller/remote endpoint
      3. Start the DC: bin/domain.sh --host-config=host-master.xml
      4. Start the HC: bin/domain.sh --host-config=host-slave.xml -Djboss.domain.master.address=127.0.0.1 -Djboss.management.native.port=19999 -Djboss.domain.base.dir=slave
      5. Enable RBAC for the user 'admin': 
        /core-service=management/access=authorization:write-attribute(name=provider,value=rbac)
/core-service=management/access=authorization/role-mapping=SuperUser/include=ManagementRealm:add(name=admin,type=USER)
      6. Restart HC
      7. This step is not required if you are using a different machine for DC and HC. In a single, it allow us to to force the uses of EXTENAL authentication mchanism instead of JBOSS-LOCAL-AUTH. Configure SSL for the Management interface.
      8. Restart DC and HC
      9. Force a disconnection of the HC stopping the process 
         ps -fea | grep 'host-slave' | grep 'Host Controller' | awk '{print $2}' | xargs kill -STOP
      10. After some seconds this error is displayed in th DC log: 
        [Host Controller] 13:04:53,840 WARN  [org.jboss.as.domain.controller] (management task-6) WFLYHC0030: Connection to remote host "slave" closed unexpectedly
      11. Change the domain model, for example modifying the jvm configuration used in a server group 
        /server-group=main-server-group/jvm=default:write-attribute(name=heap-size, value=500m)
      12. Send the continue signal to the HC process 
        ps -fea | grep 'host-slave' | grep 'Host Controller' | awk '{print $2}' | xargs kill -CONT

      These messages are shown in the DC:

      [Host Controller] 13:07:17,931 INFO  [org.jboss.as.protocol] (management I/O-2) WFLYPRT0057:  cancelled task by interrupting thread Thread[Host Controller Service Threads - 13,5,Host Controller Service Threads]

      These messages are shown in the HC:

      13:21:05,009 ERROR [org.jboss.as.host.controller] (Host Controller Service Threads - 9) WFLYHC0143: Failed to apply domain-wide configuration from master host controller. Operation outcome: failed. Failure description "WFLYCTL0313: Unauthorized to execute operation 'server-set-restart-required' for resource '[]' -- \"WFLYCTL0332: Permission denied\""
13:21:05,012 WARN  [org.jboss.as.host.controller] (Host Controller Service Threads - 3) WFLYHC0146: Could not discover master using discovery option StaticDiscovery{protocol=remote,host=127.0.0.1,port=9999}. Error was: 1-$-
13:21:05,012 WARN  [org.jboss.as.host.controller] (Host Controller Service Threads - 3) WFLYHC0147: No domain controller discovery options remain.
13:21:06,015 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 3) WFLYHC0150: Trying to reconnect to master host controller.

      These messages in the server-one:

      13:21:04,829 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 67) WFLYCTL0013: Operation ("server-set-reload-required") failed - address: ([]) - failure description: "WFLYCTL0313: Unauthorized to execute operation 'server-set-reload-required' for resource '[]' -- \"WFLYCTL0332: Permission denied\""
13:21:05,006 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 67) WFLYCTL0013: Operation ("server-set-restart-required") failed - address: ([]) - failure description: "WFLYCTL0313: Unauthorized to execute operation 'server-set-restart-required' for resource '[]' -- \"WFLYCTL0332: Permission denied\""
      Show
      Attached to the issue there is domain and slave configurations that can be used to reproduce. Steps are described here: Create a management user which will be used for DC / HC authentication ./bin/add-user.sh -u admin -p admin -g management -ds Edit host-slave.xml and : Replace the existing secret the for ManagementRealm security with the one generated for the user admin Add the attribute username="admin" in the domain-controller/remote endpoint Start the DC: bin/domain.sh --host-config=host-master.xml Start the HC: bin/domain.sh --host-config=host-slave.xml -Djboss.domain.master.address=127.0.0.1 -Djboss.management.native.port=19999 -Djboss.domain.base.dir=slave Enable RBAC for the user 'admin': /core-service=management/access=authorization:write-attribute(name=provider,value=rbac) /core-service=management/access=authorization/role-mapping=SuperUser/include=ManagementRealm:add(name=admin,type=USER) Restart HC This step is not required if you are using a different machine for DC and HC. In a single, it allow us to to force the uses of EXTENAL authentication mchanism instead of JBOSS-LOCAL-AUTH. Configure SSL for the Management interface. Restart DC and HC Force a disconnection of the HC stopping the process ps -fea | grep 'host-slave' | grep 'Host Controller' | awk '{print $2}' | xargs kill -STOP After some seconds this error is displayed in th DC log: [Host Controller] 13:04:53,840 WARN [org.jboss.as.domain.controller] (management task-6) WFLYHC0030: Connection to remote host "slave" closed unexpectedly Change the domain model, for example modifying the jvm configuration used in a server group /server-group=main-server-group/jvm=default:write-attribute(name=heap-size, value=500m) Send the continue signal to the HC process ps -fea | grep 'host-slave' | grep 'Host Controller' | awk '{print $2}' | xargs kill -CONT These messages are shown in the DC: [Host Controller] 13:07:17,931 INFO [org.jboss.as.protocol] (management I/O-2) WFLYPRT0057: cancelled task by interrupting thread Thread[Host Controller Service Threads - 13,5,Host Controller Service Threads] These messages are shown in the HC: 13:21:05,009 ERROR [org.jboss.as.host.controller] (Host Controller Service Threads - 9) WFLYHC0143: Failed to apply domain-wide configuration from master host controller. Operation outcome: failed. Failure description "WFLYCTL0313: Unauthorized to execute operation 'server-set-restart-required' for resource '[]' -- \"WFLYCTL0332: Permission denied\"" 13:21:05,012 WARN [org.jboss.as.host.controller] (Host Controller Service Threads - 3) WFLYHC0146: Could not discover master using discovery option StaticDiscovery{protocol=remote,host=127.0.0.1,port=9999}. Error was: 1-$- 13:21:05,012 WARN [org.jboss.as.host.controller] (Host Controller Service Threads - 3) WFLYHC0147: No domain controller discovery options remain. 13:21:06,015 INFO [org.jboss.as.host.controller] (Host Controller Service Threads - 3) WFLYHC0150: Trying to reconnect to master host controller. These messages in the server-one: 13:21:04,829 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 67) WFLYCTL0013: Operation ("server-set-reload-required") failed - address: ([]) - failure description: "WFLYCTL0313: Unauthorized to execute operation 'server-set-reload-required' for resource '[]' -- \"WFLYCTL0332: Permission denied\"" 13:21:05,006 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 67) WFLYCTL0013: Operation ("server-set-restart-required") failed - address: ([]) - failure description: "WFLYCTL0313: Unauthorized to execute operation 'server-set-restart-required' for resource '[]' -- \"WFLYCTL0332: Permission denied\""

      Customer has domain mode, they have the following enabled

      • RBAC
      • Management realm with ssl and ldap

      When HC is disconnected from the DC due to bad GC performance, it then cannot connect to the DC with the following errors

      2018-08-15 04:30:19,035 WARN [org.jboss.as.host.controller] (management task-3) WFLYHC0015: Connection to remote host-controller closed. Trying to reconnect.
      2018-08-15 04:30:19,036 INFO [org.jboss.as.host.controller] (Host Controller Service Threads - 149) WFLYHC0150: Trying to reconnect to master host controller.
      2018-08-15 04:30:21,006 ERROR [org.jboss.as.host.controller] (Host Controller Service Threads - 151) WFLYHC0143: Failed to apply domain-wide configuration from master host controller. Operation outcome: failed. Failure description "WFLYCTL0313: Unauthorized to execute operation 'server-set-reload-required' for resource '[]' – \"WFLYCTL0332: Permission denied\""
      due to this , We are not able to restart any JVMs in this domain. The only way we could recover was to restart all DC/HC & JVMs. I have collected the logs and config files for DC/HC/JVM and I am uploading it to the case. Please review and let us know what is the root cause of this issue and what can be done to prevent it.

      There is a management operation is requires reload in the log.

              chaowan@redhat.com Chao Wang
              rhn-support-leiyu Lei Yu (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: