Uploaded image for project: 'WildFly Core'
  1. WildFly Core
  2. WFCORE-4152

HC cannot connect to DC after lost connect with error "WFLYCTL0332: Permission denied\"

XMLWordPrintable

    • Hide

      What we need to reproduce the issue is a domain mode environment with a master and slave where the slave is using an RBAC user for its authentication. We have to have a reconnection with a model out of sync, that can be achieved setting the DC in admin-only mode, executing a management operation that affects to the HC or the Servers, bringing back the DC. The HC is unable to connect in that scenario.

      1. Create a management user which will be used for DC / HC authentication
        ./bin/add-user.sh -u admin -p admin -ds
      2. Edit host-slave.xml and :
        • Replace the existing secret the for ManagementRealm security with the one generated for the user admin
        • Add the attribute username="admin" in the domain-controller/remote endpoint
      3. Start the DC: bin/domain.sh --host-config=host-master.xml
      4. Start the HC: bin/domain.sh --host-config=host-slave.xml -Djboss.domain.master.address=127.0.0.1 -Djboss.management.native.port=19999 -Djboss.domain.base.dir=slave
      5. Enable RBAC for the user 'admin':
        /core-service=management/access=authorization:write-attribute(name=provider,value=rbac)
        /core-service=management/access=authorization/role-mapping=SuperUser/include=ManagementRealm:add(name=admin,type=USER)
        
      6. Remove the local authentication:
        /host=master/core-service=management/security-realm=ManagementRealm/authentication=local:remove
        /host=slave/core-service=management/security-realm=ManagementRealm/authentication=local:remove
        
      7. Restart HC and DC
      8. Reload the DC in admin-only mode
        reload --host=master --admin-only
        
      9. Change the domain model, for example modifying the jvm configuration used in a server group
        /server-group=main-server-group/jvm=default:write-attribute(name=heap-size, value=500m)
        
      10. Reload the DC
        reload --host=master --admin-only
        

      These messages are shown in the DC:

      [Host Controller] 13:07:17,931 INFO  [org.jboss.as.protocol] (management I/O-2) WFLYPRT0057:  cancelled task by interrupting thread Thread[Host Controller Service Threads - 13,5,Host Controller Service Threads]
      

      These messages are shown in the HC:

      13:21:05,009 ERROR [org.jboss.as.host.controller] (Host Controller Service Threads - 9) WFLYHC0143: Failed to apply domain-wide configuration from master host controller. Operation outcome: failed. Failure description "WFLYCTL0313: Unauthorized to execute operation 'server-set-restart-required' for resource '[]' -- \"WFLYCTL0332: Permission denied\""
      13:21:05,012 WARN  [org.jboss.as.host.controller] (Host Controller Service Threads - 3) WFLYHC0146: Could not discover master using discovery option StaticDiscovery{protocol=remote,host=127.0.0.1,port=9999}. Error was: 1-$-
      13:21:05,012 WARN  [org.jboss.as.host.controller] (Host Controller Service Threads - 3) WFLYHC0147: No domain controller discovery options remain.
      13:21:06,015 INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 3) WFLYHC0150: Trying to reconnect to master host controller.
      

      These messages in the server-one:

      13:21:04,829 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 67) WFLYCTL0013: Operation ("server-set-reload-required") failed - address: ([]) - failure description: "WFLYCTL0313: Unauthorized to execute operation 'server-set-reload-required' for resource '[]' -- \"WFLYCTL0332: Permission denied\""
      13:21:05,006 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 67) WFLYCTL0013: Operation ("server-set-restart-required") failed - address: ([]) - failure description: "WFLYCTL0313: Unauthorized to execute operation 'server-set-restart-required' for resource '[]' -- \"WFLYCTL0332: Permission denied\""
      
      Show
      What we need to reproduce the issue is a domain mode environment with a master and slave where the slave is using an RBAC user for its authentication. We have to have a reconnection with a model out of sync, that can be achieved setting the DC in admin-only mode, executing a management operation that affects to the HC or the Servers, bringing back the DC. The HC is unable to connect in that scenario. Create a management user which will be used for DC / HC authentication ./bin/add-user.sh -u admin -p admin -ds Edit host-slave.xml and : Replace the existing secret the for ManagementRealm security with the one generated for the user admin Add the attribute username="admin" in the domain-controller/remote endpoint Start the DC: bin/domain.sh --host-config=host-master.xml Start the HC: bin/domain.sh --host-config=host-slave.xml -Djboss.domain.master.address=127.0.0.1 -Djboss.management.native.port=19999 -Djboss.domain.base.dir=slave Enable RBAC for the user 'admin': /core-service=management/access=authorization:write-attribute(name=provider,value=rbac) /core-service=management/access=authorization/role-mapping=SuperUser/include=ManagementRealm:add(name=admin,type=USER) Remove the local authentication: /host=master/core-service=management/security-realm=ManagementRealm/authentication=local:remove /host=slave/core-service=management/security-realm=ManagementRealm/authentication=local:remove Restart HC and DC Reload the DC in admin-only mode reload --host=master --admin-only Change the domain model, for example modifying the jvm configuration used in a server group /server-group=main-server-group/jvm=default:write-attribute(name=heap-size, value=500m) Reload the DC reload --host=master --admin-only These messages are shown in the DC: [Host Controller] 13:07:17,931 INFO [org.jboss.as.protocol] (management I/O-2) WFLYPRT0057: cancelled task by interrupting thread Thread[Host Controller Service Threads - 13,5,Host Controller Service Threads] These messages are shown in the HC: 13:21:05,009 ERROR [org.jboss.as.host.controller] (Host Controller Service Threads - 9) WFLYHC0143: Failed to apply domain-wide configuration from master host controller. Operation outcome: failed. Failure description "WFLYCTL0313: Unauthorized to execute operation 'server-set-restart-required' for resource '[]' -- \"WFLYCTL0332: Permission denied\"" 13:21:05,012 WARN [org.jboss.as.host.controller] (Host Controller Service Threads - 3) WFLYHC0146: Could not discover master using discovery option StaticDiscovery{protocol=remote,host=127.0.0.1,port=9999}. Error was: 1-$- 13:21:05,012 WARN [org.jboss.as.host.controller] (Host Controller Service Threads - 3) WFLYHC0147: No domain controller discovery options remain. 13:21:06,015 INFO [org.jboss.as.host.controller] (Host Controller Service Threads - 3) WFLYHC0150: Trying to reconnect to master host controller. These messages in the server-one: 13:21:04,829 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 67) WFLYCTL0013: Operation ("server-set-reload-required") failed - address: ([]) - failure description: "WFLYCTL0313: Unauthorized to execute operation 'server-set-reload-required' for resource '[]' -- \"WFLYCTL0332: Permission denied\"" 13:21:05,006 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 67) WFLYCTL0013: Operation ("server-set-restart-required") failed - address: ([]) - failure description: "WFLYCTL0313: Unauthorized to execute operation 'server-set-restart-required' for resource '[]' -- \"WFLYCTL0332: Permission denied\""

      Customer has domain mode, they have the following enabled

      • RBAC
      • Management realm with ssl and ldap

      When HC is disconnected from the DC due to bad GC performance, it then cannot connect to the DC with the following errors

      2018-08-15 04:30:19,035 WARN [org.jboss.as.host.controller] (management task-3) WFLYHC0015: Connection to remote host-controller closed. Trying to reconnect.
      2018-08-15 04:30:19,036 INFO [org.jboss.as.host.controller] (Host Controller Service Threads - 149) WFLYHC0150: Trying to reconnect to master host controller.
      2018-08-15 04:30:21,006 ERROR [org.jboss.as.host.controller] (Host Controller Service Threads - 151) WFLYHC0143: Failed to apply domain-wide configuration from master host controller. Operation outcome: failed. Failure description "WFLYCTL0313: Unauthorized to execute operation 'server-set-reload-required' for resource '[]' – \"WFLYCTL0332: Permission denied\""
      due to this , We are not able to restart any JVMs in this domain. The only way we could recover was to restart all DC/HC & JVMs. I have collected the logs and config files for DC/HC/JVM and I am uploading it to the case. Please review and let us know what is the root cause of this issue and what can be done to prevent it.

      There is a management operation is requires reload in the log.

              yborgess1@redhat.com Yeray Borges Santana
              yborgess1@redhat.com Yeray Borges Santana
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: