Uploaded image for project: 'Application Server 3 4 5 and 6'
  1. Application Server 3 4 5 and 6
  2. JBAS-73

Singleton and/or deployer is too brittle

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Obsolete
    • Icon: Major Major
    • No Release
    • JBossAS-3.2.6 Final
    • Clustering
    • None
    • High
    • Workaround Exists
    • Hide

      For users who experience this kind of problem, a workaround to help prevent it is to space the starting of servers in your cluster by 10 seconds or so. For example, if you have a script that starts several servers, add some kind of pause or sleep to the script between each start. The delay helps ensure that the first server becomes the coordinator, and later servers join its group, rather than more than multiple servers becoming the coordinator of a "one-server group" and then the groups have to merge.

      Show
      For users who experience this kind of problem, a workaround to help prevent it is to space the starting of servers in your cluster by 10 seconds or so. For example, if you have a script that starts several servers, add some kind of pause or sleep to the script between each start. The delay helps ensure that the first server becomes the coordinator, and later servers join its group, rather than more than multiple servers becoming the coordinator of a "one-server group" and then the groups have to merge.

      Trying to bootstrap two JBoss servers, but the cluster join failed
      because the message took too long (long garbage collection/paging on my computer?)

      Server 2:

      GMS: address is htimes2:32925 (additional data: 14 bytes)
      -------------------------------------------------------
      2004-12-21 15:14:30,909 DEBUG [org.jboss.ha.framework.server.ClusterPartition] Starting channel
      2004-12-21 15:14:30,910 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] get nodeName
      2004-12-21 15:14:30,990 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] Get current members
      2004-12-21 15:14:30,991 INFO [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] Number of cluster members: 1
      2004-12-21 15:14:30,992 INFO [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] Other members: 0
      2004-12-21 15:14:30,992 INFO [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] Fetching state (will wait for 60000 milliseconds):
      2004-12-21 15:14:30,999 INFO [org.jboss.ha.framework.interfaces.HAPartition.lifecycle.DefaultPartition] New cluster view for partition DefaultPartition (id: 0, delta:
      0) : [127.0.0.1:1199]
      2004-12-21 15:14:31,008 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] membership changed from 1 to 1
      2004-12-21 15:14:31,009 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] Begin notifyListeners, viewID: 0
      2004-12-21 15:14:31,009 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] I am (null) received membershipChanged event:
      2004-12-21 15:14:31,010 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] Dead members: 0 ([])
      2004-12-21 15:14:31,010 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] New Members : 0 ([])
      2004-12-21 15:14:31,010 INFO [org.jboss.ha.framework.server.DistributedReplicantManagerImpl.DefaultPartition] All Members : 1 ([127.0.0.1:1199])
      2004-12-21 15:14:31,011 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] End notifyListeners, viewID: 0
      2004-12-21 15:14:31,058 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] State could not be retrieved, (must be first member of group)
      2004-12-21 15:14:31,137 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] setState called
      2004-12-21 15:14:31,138 DEBUG [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] state is null
      2004-12-21 15:14:31,139 DEBUG [org.jboss.ha.framework.server.ClusterPartition] Started ClusterPartition: DefaultPartition

      The message was rejected by the other server (too late?) but the cluster
      eventually joined together

      Server 1:

      15:13:05,825 INFO [Server] JBoss (MX MicroKernel) [3.2.7RC2 (build: CVSTag=Branch_3_2 date=200411160220)] Started in 4m:43s:249ms
      15:15:39,184 WARN [NAKACK] [htimes2:32920 (additional data: 14 bytes)] discarded message from non-member htimes2:32925 (additional data: 14 bytes)
      15:15:41,080 WARN [NAKACK] [htimes2:32920 (additional data: 14 bytes)] discarded message from non-member htimes2:32925 (additional data: 14 bytes)
      15:15:55,106 INFO [DefaultPartition] New cluster view for partition DefaultPartition (id: 1, delta: 1) : [127.0.0.1:1099, 127.0.0.1:1199]
      15:15:55,107 INFO [DefaultPartition] Merging partitions...
      15:15:55,107 INFO [DefaultPartition] Dead members: 0

      This meant that server2 tried to bootstrap the deploy-hasingleton
      but part way through doing that (when it eventually joined the cluster)
      it was told to undeploy it.

      Since the original deploy hadn't finished, this caused havoc
      in particular trying to use a classloader that no longer existed:
      2004-12-21 15:16:28,614 ERROR [org.jboss.mq.sm.jdbc.JDBCStateManager] Starting failed JDBCStateManager
      org.jboss.mq.SpyJMSException: Error creating connection to the database.; - nested throwable: (java.lang.NullPointerException)
      at org.jboss.mq.sm.jdbc.JDBCStateManager$JDBCSession.<init>(JDBCStateManager.java:542)
      at org.jboss.mq.sm.jdbc.JDBCStateManager.initDB(JDBCStateManager.java:432)
      at org.jboss.mq.sm.jdbc.JDBCStateManager.startService(JDBCStateManager.java:399)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:271)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:221)
      at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
      at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:884)
      at $Proxy18.start(Unknown Source)
      at org.jboss.system.ServiceController.start(ServiceController.java:414)
      at org.jboss.system.ServiceController.start(ServiceController.java:436)
      at org.jboss.system.ServiceController.start(ServiceController.java:436)
      at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
      at $Proxy4.start(Unknown Source)
      at org.jboss.deployment.SARDeployer.start(SARDeployer.java:251)
      at org.jboss.deployment.MainDeployer.start(MainDeployer.java:829)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:641)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:604)
      at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
      at $Proxy8.deploy(Unknown Source)
      at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:304)
      at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:478)
      at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:201)
      at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:274)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:271)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:221)
      at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
      at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:884)
      at $Proxy0.start(Unknown Source)
      at org.jboss.system.ServiceController.start(ServiceController.java:414)
      at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
      at $Proxy4.start(Unknown Source)
      at org.jboss.deployment.SARDeployer.start(SARDeployer.java:251)
      at org.jboss.deployment.MainDeployer.start(MainDeployer.java:829)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:641)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:604)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:588)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:198)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
      at $Proxy5.deploy(Unknown Source)
      at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:407)
      at org.jboss.system.server.ServerImpl.start(ServerImpl.java:311)
      at org.jboss.Main.boot(Main.java:147)
      at org.jboss.Main$1.run(Main.java:422)
      at java.lang.Thread.run(Thread.java:534)
      Caused by: java.lang.NullPointerException
      at org.jboss.mx.loading.LoadMgr3.beginLoadTask(LoadMgr3.java:143)
      at org.jboss.mx.loading.UnifiedClassLoader3.loadClassImpl(UnifiedClassLoader3.java:186)
      at org.jboss.mx.loading.UnifiedClassLoader3.loadClass(UnifiedClassLoader3.java:136)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:235)
      at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:302)
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:219)
      at javax.security.auth.login.LoginContext.invoke(LoginContext.java:637)
      at javax.security.auth.login.LoginContext.access$000(LoginContext.java:129)
      at javax.security.auth.login.LoginContext$4.run(LoginContext.java:610)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.login.LoginContext.invokeModule(LoginContext.java:607)
      at javax.security.auth.login.LoginContext.login(LoginContext.java:534)
      at org.jboss.security.plugins.JaasSecurityManager.defaultLogin(JaasSecurityManager.java:480)
      at org.jboss.security.plugins.JaasSecurityManager.authenticate(JaasSecurityManager.java:420)
      at org.jboss.security.plugins.JaasSecurityManager.isValid(JaasSecurityManager.java:237)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2.getSubject(BaseConnectionManager2.java:665)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)
      at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:888)
      at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102)
      at org.jboss.mq.sm.jdbc.JDBCStateManager$JDBCSession.<init>(JDBCStateManager.java:518)
      ... 95 more

      I would guess a similar race condition exists on all the HASingleton processing
      and with all attempts to multi-threadedly deploy/undeploy the same url?

        1.
        		 Pluggable policy for handling HASingleton failover issues Sub-task Closed Major Brian Stansberry

            bstansbe@redhat.com Brian Stansberry
            adrian.brock Adrian Brock (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: