Uploaded image for project: 'Hawkular'
  1. Hawkular
  2. HAWKULAR-848

Hawkular doesn't shutdown with Ctrl-C

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 1.0.0.Alpha8
    • 1.0.0.Alpha7
    • None
    • None

      Trying to shutdown Hawkular (Mac OS X, Standalone Cassandra) using Ctrl-C does not shuts the server down, something seems to be hanging. Complete stopping log is attached.

      Some relevant parts:

      2015-11-30 12:09:31,222 WARN  [org.hawkular.agent.monitor.service.MonitorService] (MSC service thread 1-6) HAWKMONITOR010055: Agent encountered errors during shutdown.: java.lang.IllegalStateException: [org.hawkular.agent.monitor.protocol.dmr.DMREndpointService] must be in state [RUNNING] rather than [RUNNING] to perfrom [removeInventoryListener()]
      	at org.hawkular.agent.monitor.service.ServiceStatus.assertInitialOrStopped(ServiceStatus.java:42)
      	at org.hawkular.agent.monitor.protocol.EndpointService.removeInventoryListener(EndpointService.java:328)
      	at org.hawkular.agent.monitor.protocol.ProtocolService.removeInventoryListener(ProtocolService.java:91)
      	at org.hawkular.agent.monitor.protocol.ProtocolServices.removeInventoryListener(ProtocolServices.java:221)
      	at org.hawkular.agent.monitor.service.MonitorService.stopMonitorService(MonitorService.java:522)
      	at org.hawkular.agent.monitor.service.MonitorService.stop(MonitorService.java:399)
      	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
      	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      (...)
      
      2015-11-30 12:09:31,286 ERROR [org.apache.activemq.ra.ActiveMQEndpointWorker] (ActiveMQ Connection Executor: vm://org.hawkular.bus.broker.bucephalus#2) Connection to broker failed: peer (vm://org.hawkular.bus.broker.bucephalus#3) stopped.: javax.jms.JMSException: peer (vm://org.hawkular.bus.broker.bucephalus#3) stopped.
      	at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:54)
      	at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1998)
      	at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017)
      	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
      	at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
      	at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
      	at org.apache.activemq.transport.vm.VMTransport.stop(VMTransport.java:206)
      	at org.apache.activemq.transport.TransportFilter.stop(TransportFilter.java:65)
      	at org.apache.activemq.transport.TransportFilter.stop(TransportFilter.java:65)
      	at org.apache.activemq.transport.ResponseCorrelator.stop(ResponseCorrelator.java:132)
      	at org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1102)
      	at org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:1068)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.activemq.transport.TransportDisposedIOException: peer (vm://org.hawkular.bus.broker.bucephalus#3) stopped.
      	... 9 more
      
      (...)
      
      2015-11-30 12:09:42,006 ERROR [org.hawkular.agent.monitor.storage.MetricsOnlyStorageAdapter] (OkHttp http://127.0.0.1:8080/hawkular/metrics/metrics/data) HAWKMONITOR010010: Failed to store metric data: {"counters":[],"availabilities":[],"gauges":[{"data":[{"value":3.3524988E8,"timestamp":1448885381998}],"id":"MI~R~[bucephalus~Local~~]~MT~WildFly Memory Metrics~Heap Used"}]}: java.io.IOException: status-code=[404], reason=[Not Found], url=[http://127.0.0.1:8080/hawkular/metrics/metrics/data]
      	at org.hawkular.agent.monitor.storage.MetricsOnlyStorageAdapter$1.onResponse(MetricsOnlyStorageAdapter.java:118)
      	at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:170)
      	at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      2015-11-30 12:09:42,011 ERROR [org.hawkular.agent.monitor.protocol.EndpointService] (Hawkular-Monitor-Scheduler-Metrics-2) HAWKMONITOR010049: Could not access resources of endpoint [DMREndpointService[Endpoint[Local]:[null]]]: org.hawkular.agent.monitor.protocol.ProtocolException: Unsuccessful fetching DMR attribute [TimedOut]
      	at org.hawkular.agent.monitor.protocol.dmr.DMRDriver.fetchAttribute(DMRDriver.java:151)
      	at org.hawkular.agent.monitor.protocol.EndpointService.measureMetrics(EndpointService.java:299)
      	at org.hawkular.agent.monitor.scheduler.IntervalBasedScheduler$MetricsJob.run(IntervalBasedScheduler.java:94)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      	at org.jboss.threads.JBossThread.run(JBossThread.java:320)
      Caused by: org.hawkular.dmr.api.OperationFailureException: Could not perform operation [read-attribute]: WFLYCTL0030: No resource definition is registered for address [
          ("subsystem" => "datasources"),
          ("data-source" => "HawkularDS"),
          ("statistics" => "pool")
      ]
      	at org.hawkular.dmr.api.OperationBuilder$OperationResult.assertSuccess(OperationBuilder.java:422)
      	at org.hawkular.agent.monitor.protocol.dmr.DMRDriver.fetchAttribute(DMRDriver.java:148)
      	... 10 more
      
      2015-11-30 12:09:42,035 INFO  [com.squareup.okhttp.OkHttpClient] (OkHttp http://127.0.0.1:8080/hawkular/metrics/metrics/data) Callback failure for call to http://127.0.0.1:8080/...: java.io.IOException: status-code=[404], reason=[Not Found], url=[http://127.0.0.1:8080/hawkular/metrics/metrics/data]
      	at org.hawkular.agent.monitor.storage.MetricsOnlyStorageAdapter$1.onResponse(MetricsOnlyStorageAdapter.java:118)
      	at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:170)
      	at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      (...)
      

      It seems the agent fails to stop and continues to try collecting and storing metrics. Not sure if it's the cause of the problem being reported.

        1. hawkular-848-server.log
          1.74 MB
          Alexandre Mendonça

            jmazzitelli John Mazzitelli
            alexandrem_jira Alexandre Mendonça (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: