Uploaded image for project: 'Hawkular'
  1. Hawkular
  2. HAWKULAR-1103

inventory initialization should retry when C*node not ready

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • None
    • None
    • Inventory
    • None

    Description

      Hawkular Services and Cassandra are deployed in the same OpenShift pod. Metrics and Alerts are running properly as indicated by the status html page. Inventory on the other hand seems to try to connect to C*node before it's ready and does not retry.

      20:03:00,369 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 68) MSC000001: Failed to start service jboss.deployment.unit."hawkular-inventory-dist.war".component.InventoryJNDIPublisher.START: org.jboss.msc.service.StartException in service jboss.deployment.unit."hawkular-inventory-dist.war".component.InventoryJNDIPublisher.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
      	at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:57)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      	at org.jboss.threads.JBossThread.run(JBossThread.java:320)
      Caused by: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
      	at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:163)
      	at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134)
      	at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88)
      	at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:124)
      	at org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:138)
      	at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54)
      	... 6 more
      Caused by: javax.ejb.EJBException: java.lang.IllegalStateException: Could not initialize inventory. Last error message was: Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInNoTx(CMTTxInterceptor.java:213)
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:265)
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.notSupported(CMTTxInterceptor.java:313)
      	at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:62)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      	at org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      	at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
      	at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
      	at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161)
      	... 11 more
      Caused by: java.lang.IllegalStateException: Could not initialize inventory. Last error message was: Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager
      	at org.hawkular.inventory.cdi.OfficialInventoryProducer.initInventory(OfficialInventoryProducer.java:119)
      	at org.hawkular.inventory.cdi.OfficialInventoryProducer.getInventory(OfficialInventoryProducer.java:64)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88)
      	at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78)
      	at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:99)
      	at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161)
      	at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:181)
      	at org.jboss.weld.context.AbstractContext.get(AbstractContext.java:96)
      	at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:101)
      	at org.jboss.weld.bean.ContextualInstanceStrategy$ApplicationScopedContextualInstanceStrategy.get(ContextualInstanceStrategy.java:141)
      	at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
      	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:742)
      	at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:842)
      	at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92)
      	at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:378)
      	at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:389)
      	at org.jboss.weld.injection.producer.DefaultInjector$1.proceed(DefaultInjector.java:71)
      	at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48)
      	at org.jboss.weld.injection.producer.DefaultInjector.inject(DefaultInjector.java:73)
      	at org.jboss.weld.injection.producer.ejb.SessionBeanInjectionTarget.inject(SessionBeanInjectionTarget.java:140)
      	at org.jboss.as.weld.injection.WeldInjectionContext.inject(WeldInjectionContext.java:39)
      	at org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:51)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      	at org.jboss.as.ee.component.AroundConstructInterceptorFactory$1.processInvocation(AroundConstructInterceptorFactory.java:28)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      	at org.jboss.as.weld.injection.WeldInterceptorInjectionInterceptor.processInvocation(WeldInterceptorInjectionInterceptor.java:56)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      	at org.jboss.as.weld.ejb.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:100)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
      	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:263)
      	... 27 more
      Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager
      	at com.thinkaurelius.titan.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:55)
      	at com.thinkaurelius.titan.diskstorage.Backend.getImplementationClass(Backend.java:421)
      	at com.thinkaurelius.titan.diskstorage.Backend.getStorageManager(Backend.java:361)
      	at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1275)
      	at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:93)
      	at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:73)
      	at org.hawkular.inventory.impl.tinkerpop.provider.TitanProvider.instantiateGraph(TitanProvider.java:108)
      	at org.hawkular.inventory.impl.tinkerpop.provider.TitanProvider.instantiateGraph(TitanProvider.java:53)
      	at org.hawkular.inventory.impl.tinkerpop.TinkerpopInventory.ensureIndices(TinkerpopInventory.java:72)
      	at org.hawkular.inventory.impl.tinkerpop.TinkerpopInventory.loadGraph(TinkerpopInventory.java:66)
      	at org.hawkular.inventory.impl.tinkerpop.TinkerpopInventory.doInitialize(TinkerpopInventory.java:57)
      	at org.hawkular.inventory.base.BaseInventory.initialize(BaseInventory.java:137)
      	at org.hawkular.inventory.cdi.OfficialInventoryProducer.initInventory(OfficialInventoryProducer.java:105)
      	... 62 more
      Caused by: java.lang.reflect.InvocationTargetException
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      	at com.thinkaurelius.titan.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:44)
      	... 74 more
      Caused by: com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Temporary failure in storage backend
      	at com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager.getCassandraPartitioner(CassandraThriftStoreManager.java:218)
      	at com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager.<init>(CassandraThriftStoreManager.java:196)
      	... 79 more
      Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
      	at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
      	at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
      	at com.thinkaurelius.titan.diskstorage.cassandra.thrift.thriftpool.CTConnectionFactory.makeRawConnection(CTConnectionFactory.java:88)
      	at com.thinkaurelius.titan.diskstorage.cassandra.thrift.thriftpool.CTConnectionFactory.makeObject(CTConnectionFactory.java:52)
      	at com.thinkaurelius.titan.diskstorage.cassandra.thrift.thriftpool.CTConnectionFactory.makeObject(CTConnectionFactory.java:21)
      	at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
      	at com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager.getCassandraPartitioner(CassandraThriftStoreManager.java:215)
      	... 80 more
      Caused by: java.net.ConnectException: Connection refused
      	at java.net.PlainSocketImpl.socketConnect(Native Method)
      	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
      	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
      	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
      	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
      	at java.net.Socket.connect(Socket.java:589)
      	at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
      	... 86 more
      
      20:03:00,428 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("deployment" => "hawkular-inventory-dist.war")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.unit.\"hawkular-inventory-dist.war\".component.InventoryJNDIPublisher.START" => "org.jboss.msc.service.StartException in service jboss.deployment.unit.\"hawkular-inventory-dist.war\".component.InventoryJNDIPublisher.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
          Caused by: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
          Caused by: javax.ejb.EJBException: java.lang.IllegalStateException: Could not initialize inventory. Last error message was: Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager
          Caused by: java.lang.IllegalStateException: Could not initialize inventory. Last error message was: Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager
      Caused by: java.lang.IllegalArgumentException: Could not instantiate implementation: com.thinkaurelius.titan.diskstorage.cassandra.thrift.CassandraThriftStoreManager
      

      Attachments

        Activity

          People

            rhn-engineering-lkrejci Lukas Krejci
            vietn_jira viet nguyen (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: