-
Bug
-
Resolution: Done
-
Major
-
1.0.0.Alpha7
-
None
-
None
Trying to shutdown Hawkular (Mac OS X, Standalone Cassandra) using Ctrl-C does not shuts the server down, something seems to be hanging. Complete stopping log is attached.
Some relevant parts:
2015-11-30 12:09:31,222 WARN [org.hawkular.agent.monitor.service.MonitorService] (MSC service thread 1-6) HAWKMONITOR010055: Agent encountered errors during shutdown.: java.lang.IllegalStateException: [org.hawkular.agent.monitor.protocol.dmr.DMREndpointService] must be in state [RUNNING] rather than [RUNNING] to perfrom [removeInventoryListener()] at org.hawkular.agent.monitor.service.ServiceStatus.assertInitialOrStopped(ServiceStatus.java:42) at org.hawkular.agent.monitor.protocol.EndpointService.removeInventoryListener(EndpointService.java:328) at org.hawkular.agent.monitor.protocol.ProtocolService.removeInventoryListener(ProtocolService.java:91) at org.hawkular.agent.monitor.protocol.ProtocolServices.removeInventoryListener(ProtocolServices.java:221) at org.hawkular.agent.monitor.service.MonitorService.stopMonitorService(MonitorService.java:522) at org.hawkular.agent.monitor.service.MonitorService.stop(MonitorService.java:399) at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056) at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) (...) 2015-11-30 12:09:31,286 ERROR [org.apache.activemq.ra.ActiveMQEndpointWorker] (ActiveMQ Connection Executor: vm://org.hawkular.bus.broker.bucephalus#2) Connection to broker failed: peer (vm://org.hawkular.bus.broker.bucephalus#3) stopped.: javax.jms.JMSException: peer (vm://org.hawkular.bus.broker.bucephalus#3) stopped. at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54) at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1998) at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017) at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126) at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101) at org.apache.activemq.transport.vm.VMTransport.stop(VMTransport.java:206) at org.apache.activemq.transport.TransportFilter.stop(TransportFilter.java:65) at org.apache.activemq.transport.TransportFilter.stop(TransportFilter.java:65) at org.apache.activemq.transport.ResponseCorrelator.stop(ResponseCorrelator.java:132) at org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1102) at org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.activemq.transport.TransportDisposedIOException: peer (vm://org.hawkular.bus.broker.bucephalus#3) stopped. ... 9 more (...) 2015-11-30 12:09:42,006 ERROR [org.hawkular.agent.monitor.storage.MetricsOnlyStorageAdapter] (OkHttp http://127.0.0.1:8080/hawkular/metrics/metrics/data) HAWKMONITOR010010: Failed to store metric data: {"counters":[],"availabilities":[],"gauges":[{"data":[{"value":3.3524988E8,"timestamp":1448885381998}],"id":"MI~R~[bucephalus~Local~~]~MT~WildFly Memory Metrics~Heap Used"}]}: java.io.IOException: status-code=[404], reason=[Not Found], url=[http://127.0.0.1:8080/hawkular/metrics/metrics/data] at org.hawkular.agent.monitor.storage.MetricsOnlyStorageAdapter$1.onResponse(MetricsOnlyStorageAdapter.java:118) at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:170) at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2015-11-30 12:09:42,011 ERROR [org.hawkular.agent.monitor.protocol.EndpointService] (Hawkular-Monitor-Scheduler-Metrics-2) HAWKMONITOR010049: Could not access resources of endpoint [DMREndpointService[Endpoint[Local]:[null]]]: org.hawkular.agent.monitor.protocol.ProtocolException: Unsuccessful fetching DMR attribute [TimedOut] at org.hawkular.agent.monitor.protocol.dmr.DMRDriver.fetchAttribute(DMRDriver.java:151) at org.hawkular.agent.monitor.protocol.EndpointService.measureMetrics(EndpointService.java:299) at org.hawkular.agent.monitor.scheduler.IntervalBasedScheduler$MetricsJob.run(IntervalBasedScheduler.java:94) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at org.jboss.threads.JBossThread.run(JBossThread.java:320) Caused by: org.hawkular.dmr.api.OperationFailureException: Could not perform operation [read-attribute]: WFLYCTL0030: No resource definition is registered for address [ ("subsystem" => "datasources"), ("data-source" => "HawkularDS"), ("statistics" => "pool") ] at org.hawkular.dmr.api.OperationBuilder$OperationResult.assertSuccess(OperationBuilder.java:422) at org.hawkular.agent.monitor.protocol.dmr.DMRDriver.fetchAttribute(DMRDriver.java:148) ... 10 more 2015-11-30 12:09:42,035 INFO [com.squareup.okhttp.OkHttpClient] (OkHttp http://127.0.0.1:8080/hawkular/metrics/metrics/data) Callback failure for call to http://127.0.0.1:8080/...: java.io.IOException: status-code=[404], reason=[Not Found], url=[http://127.0.0.1:8080/hawkular/metrics/metrics/data] at org.hawkular.agent.monitor.storage.MetricsOnlyStorageAdapter$1.onResponse(MetricsOnlyStorageAdapter.java:118) at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:170) at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) (...)
It seems the agent fails to stop and continues to try collecting and storing metrics. Not sure if it's the cause of the problem being reported.