-
Bug
-
Resolution: Done
-
Major
-
1.0.0.Alpha7
-
None
-
None
-
Hawkular (master) on Mac OS X 10.11.1 (Standalone Cassandra)
Trying to shutdown Hawkular (Mac OS X, Standalone Cassandra) using Ctrl-C does not shuts the server down, something seems to be hanging. Complete stopping log is attached.
Some relevant parts:
2015-11-30 12:09:31,222 WARN [org.hawkular.agent.monitor.service.MonitorService] (MSC service thread 1-6) HAWKMONITOR010055: Agent encountered errors during shutdown.: java.lang.IllegalStateException: [org.hawkular.agent.monitor.protocol.dmr.DMREndpointService] must be in state [RUNNING] rather than [RUNNING] to perfrom [removeInventoryListener()]
at org.hawkular.agent.monitor.service.ServiceStatus.assertInitialOrStopped(ServiceStatus.java:42)
at org.hawkular.agent.monitor.protocol.EndpointService.removeInventoryListener(EndpointService.java:328)
at org.hawkular.agent.monitor.protocol.ProtocolService.removeInventoryListener(ProtocolService.java:91)
at org.hawkular.agent.monitor.protocol.ProtocolServices.removeInventoryListener(ProtocolServices.java:221)
at org.hawkular.agent.monitor.service.MonitorService.stopMonitorService(MonitorService.java:522)
at org.hawkular.agent.monitor.service.MonitorService.stop(MonitorService.java:399)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
(...)
2015-11-30 12:09:31,286 ERROR [org.apache.activemq.ra.ActiveMQEndpointWorker] (ActiveMQ Connection Executor: vm://org.hawkular.bus.broker.bucephalus#2) Connection to broker failed: peer (vm://org.hawkular.bus.broker.bucephalus#3) stopped.: javax.jms.JMSException: peer (vm://org.hawkular.bus.broker.bucephalus#3) stopped.
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1998)
at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at org.apache.activemq.transport.vm.VMTransport.stop(VMTransport.java:206)
at org.apache.activemq.transport.TransportFilter.stop(TransportFilter.java:65)
at org.apache.activemq.transport.TransportFilter.stop(TransportFilter.java:65)
at org.apache.activemq.transport.ResponseCorrelator.stop(ResponseCorrelator.java:132)
at org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1102)
at org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.activemq.transport.TransportDisposedIOException: peer (vm://org.hawkular.bus.broker.bucephalus#3) stopped.
... 9 more
(...)
2015-11-30 12:09:42,006 ERROR [org.hawkular.agent.monitor.storage.MetricsOnlyStorageAdapter] (OkHttp http://127.0.0.1:8080/hawkular/metrics/metrics/data) HAWKMONITOR010010: Failed to store metric data: {"counters":[],"availabilities":[],"gauges":[{"data":[{"value":3.3524988E8,"timestamp":1448885381998}],"id":"MI~R~[bucephalus~Local~~]~MT~WildFly Memory Metrics~Heap Used"}]}: java.io.IOException: status-code=[404], reason=[Not Found], url=[http://127.0.0.1:8080/hawkular/metrics/metrics/data]
at org.hawkular.agent.monitor.storage.MetricsOnlyStorageAdapter$1.onResponse(MetricsOnlyStorageAdapter.java:118)
at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:170)
at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2015-11-30 12:09:42,011 ERROR [org.hawkular.agent.monitor.protocol.EndpointService] (Hawkular-Monitor-Scheduler-Metrics-2) HAWKMONITOR010049: Could not access resources of endpoint [DMREndpointService[Endpoint[Local]:[null]]]: org.hawkular.agent.monitor.protocol.ProtocolException: Unsuccessful fetching DMR attribute [TimedOut]
at org.hawkular.agent.monitor.protocol.dmr.DMRDriver.fetchAttribute(DMRDriver.java:151)
at org.hawkular.agent.monitor.protocol.EndpointService.measureMetrics(EndpointService.java:299)
at org.hawkular.agent.monitor.scheduler.IntervalBasedScheduler$MetricsJob.run(IntervalBasedScheduler.java:94)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: org.hawkular.dmr.api.OperationFailureException: Could not perform operation [read-attribute]: WFLYCTL0030: No resource definition is registered for address [
("subsystem" => "datasources"),
("data-source" => "HawkularDS"),
("statistics" => "pool")
]
at org.hawkular.dmr.api.OperationBuilder$OperationResult.assertSuccess(OperationBuilder.java:422)
at org.hawkular.agent.monitor.protocol.dmr.DMRDriver.fetchAttribute(DMRDriver.java:148)
... 10 more
2015-11-30 12:09:42,035 INFO [com.squareup.okhttp.OkHttpClient] (OkHttp http://127.0.0.1:8080/hawkular/metrics/metrics/data) Callback failure for call to http://127.0.0.1:8080/...: java.io.IOException: status-code=[404], reason=[Not Found], url=[http://127.0.0.1:8080/hawkular/metrics/metrics/data]
at org.hawkular.agent.monitor.storage.MetricsOnlyStorageAdapter$1.onResponse(MetricsOnlyStorageAdapter.java:118)
at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:170)
at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
(...)
It seems the agent fails to stop and continues to try collecting and storing metrics. Not sure if it's the cause of the problem being reported.