Uploaded image for project: 'JBoss A-MQ'
  1. JBoss A-MQ
  2. ENTMQ-1388

Browsing Queues in Hawtio after Broker Restart Causes Exception and Queue Browsing Becomes Unavailable

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • JBoss A-MQ 6.3
    • JBoss A-MQ 6.1, 6.1.1
    • hawtio (console)
    • None
    • Hide
      • Configure and start up JBoss AMQ 6.1.0
      • Send a few messages to queues (I sent 20 messages to 36 queues)
      • Open a browser to Hawtio, login, and select the ActiveMQ tab.
      • Expand the queues and select a queue for viewing
        • Messages are present and may be viewed
      • Log out of Hawtio using the upper right-hand corner drop-down menu
      • Restart the broker
      • Log back into Hawtio
        • Queues appear to be empty and a stacktrace is present in the logs / command-line console

      Note: With R3 applied, the error changes slightly:

      • The first time the broker is rebooted and Hawtio login occurs, the following trace was present in the cmd-line console, but the queues were still browseable:
      ...
      javax.management.openmbean.OpenDataException: Argument's element itemNames[25]="JMSCorrelationID" duplicates a previous item names.
      javax.management.openmbean.OpenDataException: Argument's element itemNames[25]="JMSCorrelationID" duplicates a previous item names.
      	at javax.management.openmbean.CompositeType.<init>(CompositeType.java:153)
      	at org.apache.activemq.broker.jmx.OpenTypeSupport$AbstractOpenTypeFactory.createCompositeType(OpenTypeSupport.java:82)
      	at org.apache.activemq.broker.jmx.OpenTypeSupport$AbstractOpenTypeFactory.getCompositeType(OpenTypeSupport.java:70)
      	at org.apache.activemq.broker.jmx.OpenTypeSupport.convert(OpenTypeSupport.java:543)
      	at org.apache.activemq.broker.jmx.DestinationView.browse(DestinationView.java:206)
      	at org.apache.activemq.broker.jmx.DestinationView.browse(DestinationView.java:186)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
      	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
      	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
      	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
      	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
      	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
      	at javax.management.StandardMBean.invoke(StandardMBean.java:405)
      	at org.apache.activemq.broker.jmx.AnnotatedMBean.invoke(AnnotatedMBean.java:216)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
      	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
      	at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:98)
      	at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:40)
      	at org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:89)
      	at org.jolokia.backend.MBeanServerExecutorLocal.handleRequest(MBeanServerExecutorLocal.java:109)
      	at org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:102)
      	at org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:98)
      	at org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:410)
      	at org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:158)
      	at org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:197)
      	at org.jolokia.http.HttpRequestHandler.handlePostRequest(HttpRequestHandler.java:137)
      	at org.jolokia.http.AgentServlet$2.handleRequest(AgentServlet.java:409)
      	at org.jolokia.http.AgentServlet.handle(AgentServlet.java:288)
      	at org.jolokia.http.AgentServlet.doPost(AgentServlet.java:256)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
      	at io.hawt.web.AuthenticationFilter$2.run(AuthenticationFilter.java:145)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at io.hawt.web.AuthenticationFilter.executeAs(AuthenticationFilter.java:142)
      	at io.hawt.web.AuthenticationFilter.doFilter(AuthenticationFilter.java:110)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at io.hawt.web.CORSFilter.doFilter(CORSFilter.java:42)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at io.hawt.web.CacheHeadersFilter.doFilter(CacheHeadersFilter.java:37)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at io.hawt.web.SessionExpiryFilter.process(SessionExpiryFilter.java:142)
      	at io.hawt.web.SessionExpiryFilter.doFilter(SessionExpiryFilter.java:46)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at org.ops4j.pax.web.service.internal.WelcomeFilesFilter.doFilter(WelcomeFilesFilter.java:185)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
      	at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
      	at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:240)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:77)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      	at org.eclipse.jetty.server.Server.handle(Server.java:370)
      	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
      	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
      	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
      	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
      	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
      	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
      	at java.lang.Thread.run(Thread.java:745)
      ...
      
      • Log out of Hawtio again using the drop-down menu
      • Restart the broker a second time
      • Log back into Hawtio
      • The queues are now unbrowsable with the original error from 6.1.0 GA
      Show
      Configure and start up JBoss AMQ 6.1.0 Send a few messages to queues (I sent 20 messages to 36 queues) Open a browser to Hawtio, login, and select the ActiveMQ tab. Expand the queues and select a queue for viewing Messages are present and may be viewed Log out of Hawtio using the upper right-hand corner drop-down menu Restart the broker Log back into Hawtio Queues appear to be empty and a stacktrace is present in the logs / command-line console Note: With R3 applied, the error changes slightly: The first time the broker is rebooted and Hawtio login occurs, the following trace was present in the cmd-line console, but the queues were still browseable: ... javax.management.openmbean.OpenDataException: Argument's element itemNames[25]= "JMSCorrelationID" duplicates a previous item names. javax.management.openmbean.OpenDataException: Argument's element itemNames[25]= "JMSCorrelationID" duplicates a previous item names. at javax.management.openmbean.CompositeType.<init>(CompositeType.java:153) at org.apache.activemq.broker.jmx.OpenTypeSupport$AbstractOpenTypeFactory.createCompositeType(OpenTypeSupport.java:82) at org.apache.activemq.broker.jmx.OpenTypeSupport$AbstractOpenTypeFactory.getCompositeType(OpenTypeSupport.java:70) at org.apache.activemq.broker.jmx.OpenTypeSupport.convert(OpenTypeSupport.java:543) at org.apache.activemq.broker.jmx.DestinationView.browse(DestinationView.java:206) at org.apache.activemq.broker.jmx.DestinationView.browse(DestinationView.java:186) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252) at javax.management.StandardMBean.invoke(StandardMBean.java:405) at org.apache.activemq.broker.jmx.AnnotatedMBean.invoke(AnnotatedMBean.java:216) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801) at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:98) at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:40) at org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:89) at org.jolokia.backend.MBeanServerExecutorLocal.handleRequest(MBeanServerExecutorLocal.java:109) at org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:102) at org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:98) at org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:410) at org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:158) at org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:197) at org.jolokia.http.HttpRequestHandler.handlePostRequest(HttpRequestHandler.java:137) at org.jolokia.http.AgentServlet$2.handleRequest(AgentServlet.java:409) at org.jolokia.http.AgentServlet.handle(AgentServlet.java:288) at org.jolokia.http.AgentServlet.doPost(AgentServlet.java:256) at javax.servlet.http.HttpServlet.service(HttpServlet.java:595) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496) at io.hawt.web.AuthenticationFilter$2.run(AuthenticationFilter.java:145) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at io.hawt.web.AuthenticationFilter.executeAs(AuthenticationFilter.java:142) at io.hawt.web.AuthenticationFilter.doFilter(AuthenticationFilter.java:110) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at io.hawt.web.CORSFilter.doFilter(CORSFilter.java:42) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at io.hawt.web.CacheHeadersFilter.doFilter(CacheHeadersFilter.java:37) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at io.hawt.web.SessionExpiryFilter.process(SessionExpiryFilter.java:142) at io.hawt.web.SessionExpiryFilter.doFilter(SessionExpiryFilter.java:46) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.ops4j.pax.web.service.internal.WelcomeFilesFilter.doFilter(WelcomeFilesFilter.java:185) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:240) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:77) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang. Thread .run( Thread .java:745) ... Log out of Hawtio again using the drop-down menu Restart the broker a second time Log back into Hawtio The queues are now unbrowsable with the original error from 6.1.0 GA

      When queues are re-accessed through Hawtio after a broker restart, the queues appear empty in Hawtio and a stacktrace appears in the log and in the command-line console:

      ...
      java.lang.IllegalArgumentException: Argument's element itemNames[22] cannot be null.
      	at javax.management.openmbean.CompositeType.checkForNullElement(CompositeType.java:167)
      	at javax.management.openmbean.CompositeType.<init>(CompositeType.java:130)
      	at org.apache.activemq.broker.jmx.OpenTypeSupport$AbstractOpenTypeFactory.createCompositeType(OpenTypeSupport.java:82)
      	at org.apache.activemq.broker.jmx.OpenTypeSupport$AbstractOpenTypeFactory.getCompositeType(OpenTypeSupport.java:70)
      	at org.apache.activemq.broker.jmx.OpenTypeSupport.convert(OpenTypeSupport.java:543)
      	at org.apache.activemq.broker.jmx.DestinationView.browse(DestinationView.java:206)
      	at org.apache.activemq.broker.jmx.DestinationView.browse(DestinationView.java:186)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
      java.lang.IllegalArgumentException: Argument's element itemNames[22] cannot be null.
      	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
      	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
      	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
      	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
      	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
      	at javax.management.StandardMBean.invoke(StandardMBean.java:405)
      	at org.apache.activemq.broker.jmx.AnnotatedMBean.invoke(AnnotatedMBean.java:198)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
      	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
      	at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:98)
      	at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:40)
      	at org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:89)
      	at org.jolokia.backend.MBeanServerExecutorLocal.handleRequest(MBeanServerExecutorLocal.java:109)
      	at org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:102)
      	at org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:98)
      	at org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:409)
      	at org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:158)
      	at org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:197)
      	at org.jolokia.http.HttpRequestHandler.handlePostRequest(HttpRequestHandler.java:137)
      	at org.jolokia.http.AgentServlet$2.handleRequest(AgentServlet.java:399)
      	at org.jolokia.http.AgentServlet.handle(AgentServlet.java:287)
      	at org.jolokia.http.AgentServlet.doPost(AgentServlet.java:256)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
      	at io.hawt.web.AuthenticationFilter$2.run(AuthenticationFilter.java:143)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at io.hawt.web.AuthenticationFilter.executeAs(AuthenticationFilter.java:140)
      	at io.hawt.web.AuthenticationFilter.doFilter(AuthenticationFilter.java:108)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at io.hawt.web.CORSFilter.doFilter(CORSFilter.java:42)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at io.hawt.web.CacheHeadersFilter.doFilter(CacheHeadersFilter.java:37)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at org.ops4j.pax.web.service.internal.WelcomeFilesFilter.doFilter(WelcomeFilesFilter.java:185)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
      	at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
      	at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:219)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:77)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      	at org.eclipse.jetty.server.Server.handle(Server.java:370)
      	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
      	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
      	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
      	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
      	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
      	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
      	at java.lang.Thread.run(Thread.java:745)
      java.lang.IllegalArgumentException: Argument's element itemNames[22] cannot be null.
      	at javax.management.openmbean.CompositeType.checkForNullElement(CompositeType.java:167)
      	at javax.management.openmbean.CompositeType.<init>(CompositeType.java:130)
      	at org.apache.activemq.broker.jmx.OpenTypeSupport$AbstractOpenTypeFactory.createCompositeType(OpenTypeSupport.java:82)
      	at org.apache.activemq.broker.jmx.OpenTypeSupport$AbstractOpenTypeFactory.getCompositeType(OpenTypeSupport.java:70)
      	at org.apache.activemq.broker.jmx.OpenTypeSupport.convert(OpenTypeSupport.java:543)
      	at org.apache.activemq.broker.jmx.DestinationView.browse(DestinationView.java:206)
      	at org.apache.activemq.broker.jmx.DestinationView.browse(DestinationView.java:186)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
      	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
      	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
      	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
      	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
      	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
      	at javax.management.StandardMBean.invoke(StandardMBean.java:405)
      	at org.apache.activemq.broker.jmx.AnnotatedMBean.invoke(AnnotatedMBean.java:198)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
      	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
      	at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:98)
      	at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:40)
      	at org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:89)
      	at org.jolokia.backend.MBeanServerExecutorLocal.handleRequest(MBeanServerExecutorLocal.java:109)
      	at org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:102)
      	at org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:98)
      	at org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:409)
      	at org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:158)
      	at org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:197)
      	at org.jolokia.http.HttpRequestHandler.handlePostRequest(HttpRequestHandler.java:137)
      	at org.jolokia.http.AgentServlet$2.handleRequest(AgentServlet.java:399)
      	at org.jolokia.http.AgentServlet.handle(AgentServlet.java:287)
      	at org.jolokia.http.AgentServlet.doPost(AgentServlet.java:256)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
      	at io.hawt.web.AuthenticationFilter$2.run(AuthenticationFilter.java:143)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at io.hawt.web.AuthenticationFilter.executeAs(AuthenticationFilter.java:140)
      	at io.hawt.web.AuthenticationFilter.doFilter(AuthenticationFilter.java:108)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at io.hawt.web.CORSFilter.doFilter(CORSFilter.java:42)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at io.hawt.web.CacheHeadersFilter.doFilter(CacheHeadersFilter.java:37)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at org.ops4j.pax.web.service.internal.WelcomeFilesFilter.doFilter(WelcomeFilesFilter.java:185)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
      	at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
      	at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:219)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:77)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      	at org.eclipse.jetty.server.Server.handle(Server.java:370)
      	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
      	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
      	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
      	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
      	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
      	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:667)
      	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
      	at java.lang.Thread.run(Thread.java:745)
      java.lang.IllegalArgumentException: Argument's element itemNames[22] cannot be null.
      java.lang.IllegalArgumentException: Argument's element itemNames[22] cannot be null.
      	at javax.management.openmbean.CompositeType.checkForNullElement(CompositeType.java:167)
      	at javax.management.openmbean.CompositeType.<init>(CompositeType.java:130)
      	at org.apache.activemq.broker.jmx.OpenTypeSupport$AbstractOpenTypeFactory.createCompositeType(OpenTypeSupport.java:82)
      	at org.apache.activemq.broker.jmx.OpenTypeSupport$AbstractOpenTypeFactory.getCompositeType(OpenTypeSupport.java:70)
      	at org.apache.activemq.broker.jmx.OpenTypeSupport.convert(OpenTypeSupport.java:543)
      	at org.apache.activemq.broker.jmx.DestinationView.browse(DestinationView.java:206)
      	at org.apache.activemq.broker.jmx.DestinationView.browse(DestinationView.java:186)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
      	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
      	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
      	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
      	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
      	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
      	at javax.management.StandardMBean.invoke(StandardMBean.java:405)
      	at org.apache.activemq.broker.jmx.AnnotatedMBean.invoke(AnnotatedMBean.java:198)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
      	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
      	at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:98)
      	at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:40)
      	at org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:89)
      	at org.jolokia.backend.MBeanServerExecutorLocal.handleRequest(MBeanServerExecutorLocal.java:109)
      	at org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:102)
      	at org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:98)
      	at org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:409)
      	at org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:158)
      	at org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:197)
      	at org.jolokia.http.HttpRequestHandler.handlePostRequest(HttpRequestHandler.java:137)
      	at org.jolokia.http.AgentServlet$2.handleRequest(AgentServlet.java:399)
      	at org.jolokia.http.AgentServlet.handle(AgentServlet.java:287)
      	at org.jolokia.http.AgentServlet.doPost(AgentServlet.java:256)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:595)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
      	at io.hawt.web.AuthenticationFilter$2.run(AuthenticationFilter.java:143)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at io.hawt.web.AuthenticationFilter.executeAs(AuthenticationFilter.java:140)
      	at io.hawt.web.AuthenticationFilter.doFilter(AuthenticationFilter.java:108)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at io.hawt.web.CORSFilter.doFilter(CORSFilter.java:42)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at io.hawt.web.CacheHeadersFilter.doFilter(CacheHeadersFilter.java:37)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at org.ops4j.pax.web.service.internal.WelcomeFilesFilter.doFilter(WelcomeFilesFilter.java:185)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
      	at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
      	at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:219)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
      	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:77)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
      java.lang.IllegalArgumentException: Argument's element itemNames[22] cannot be null.	at org.eclipse.jetty.server.Server.handle(Server.java:370)
      

      Querying the queues via the command-line console reveals that the queues still retain messages:

      JBossA-MQ:karaf@root> activemq:query -QQueue=Host1.QueueB.5.5
      AverageMessageSize = 1090.55
      UseCache = true
      MemoryPercentUsage = 2
      Name = Host1.QueueB.5.5
      BlockedProducerWarningInterval = 30000
      MaxMessageSize = 1091
      MemoryUsagePortion = 0.0
      CacheEnabled = true
      destinationType = Queue
      BlockedSends = 0
      EnqueueCount = 20
      Options = 
      TotalBlockedTime = 0
      MinMessageSize = 1090
      Subscriptions = []
      PrioritizedMessages = false
      DLQ = false
      CursorPercentUsage = 2
      MessageGroupType = cached
      CursorMemoryUsage = 21811
      AlwaysRetroactive = false
      AverageEnqueueTime = 0.0
      DispatchCount = 0
      AverageBlockedTime = 0.0
      destinationName = Host1.QueueB.5.5
      MemoryLimit = 1048576
      MaxProducersToAudit = 64
      ProducerFlowControl = true
      QueueSize = 20
      ConsumerCount = 0
      MaxEnqueueTime = 0
      MessageGroups = {}
      CursorFull = false
      MaxAuditDepth = 2048
      MemoryUsageByteCount = 21811
      InFlightCount = 0
      ExpiredCount = 0
      brokerName = amq
      DequeueCount = 0
      MaxPageSize = 200
      MinEnqueueTime = 0
      ProducerCount = 0
      type = Broker
      

        1. amq.log
          593 kB
          Duane Hawkins
        2. threads.txt
          44 kB
          Duane Hawkins

              dejanbosanac Dejan Bosanac
              rhn-support-dhawkins Duane Hawkins
              Dominik Lenosi Dominik Lenosi (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: