Uploaded image for project: 'FlightPath'
  1. FlightPath
  2. FLPATH-1037

MTA workflow hangs in Running state

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False

      Description of the problem: MTA workflow hangs for long period, never moving to Completed state. 

      How reproducible: 100%

      Steps to reproduce:

      1. Deploy orchestrator with 0.1.26 Helm chart

      2. Deploy workflows using 0.2.6 Helm chart

      3. Deploy MTA workflow, navigate to workflow instance.

      Actual results:  MTA workflow pod is running normally but workflow never finishes execution.

      Expected results: MTA will finish deployment within a few minutes.

      Additional information:
      From the MTA workflow pod logs:

      2024-02-22 12:31:22,343 ERROR [org.kie.kog.eve.pro.ReactiveMessagingEventPublisher] (executor-thread-1) Error while publishing message org.eclipse.microprofile.reactive.messaging.Message$8@41e5bf7c: java.lang.IllegalStateException: SRMSG00034: Insufficient downstream requests to emit item	at io.smallrye.reactive.messaging.providers.extension.ThrowingEmitter.emit(ThrowingEmitter.java:60)	at io.smallrye.reactive.messaging.providers.extension.AbstractEmitter.emit(AbstractEmitter.java:162)	at io.smallrye.reactive.messaging.providers.extension.MutinyEmitterImpl.lambda$sendMessage$4(MutinyEmitterImpl.java:72)	at io.smallrye.context.impl.wrappers.SlowContextualConsumer.accept(SlowContextualConsumer.java:21)	at io.smallrye.mutiny.operators.uni.builders.UniCreateWithEmitter.subscribe(UniCreateWithEmitter.java:22)	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)	at io.smallrye.mutiny.operators.uni.UniEmitOn.subscribe(UniEmitOn.java:22)	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)	at io.smallrye.mutiny.groups.UniSubscribe.withSubscriber(UniSubscribe.java:51)	at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:110)	at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:88)	at io.smallrye.reactive.messaging.providers.extension.MutinyEmitterImpl.sendMessageAndForget(MutinyEmitterImpl.java:50)	at org.kie.kogito.events.process.ReactiveMessagingEventPublisher$ReactiveMessageEmitter.accept(ReactiveMessagingEventPublisher.java:164)	at org.kie.kogito.events.process.ReactiveMessagingEventPublisher$ReactiveMessageEmitter.accept(ReactiveMessagingEventPublisher.java:161)	at org.kie.kogito.events.process.ReactiveMessagingEventPublisher.publishToTopic(ReactiveMessagingEventPublisher.java:135)	at org.kie.kogito.events.process.ReactiveMessagingEventPublisher.publish(ReactiveMessagingEventPublisher.java:98)	at org.kie.kogito.events.process.ReactiveMessagingEventPublisher.publish(ReactiveMessagingEventPublisher.java:120)	at org.kie.kogito.event.impl.BaseEventManager.lambda$publish$0(BaseEventManager.java:58)	at java.base/java.lang.Iterable.forEach(Iterable.java:75)	at org.kie.kogito.event.impl.BaseEventManager.publish(BaseEventManager.java:58)	at org.kie.kogito.services.uow.CollectingUnitOfWork.end(CollectingUnitOfWork.java:69)	at org.kie.kogito.services.uow.ManagedUnitOfWork.end(ManagedUnitOfWork.java:54)	at org.kie.kogito.services.uow.UnitOfWorkExecutor.executeInUnitOfWork(UnitOfWorkExecutor.java:41)	at org.kie.kogito.services.jobs.impl.TriggerJobCommand.execute(TriggerJobCommand.java:59)	at org.kie.kogito.jobs.quarkus.common.CallbackJobsServiceResource.triggerTimer(CallbackJobsServiceResource.java:88)	at jdk.internal.reflect.GeneratedMethodAccessor139.invoke(Unknown Source)	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)	at java.base/java.lang.reflect.Method.invoke(Method.java:568)	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:154)	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:118)	at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:560)	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:452)	at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:413)	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:321)	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:415)	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:378)	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:356)	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:70)	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:429)	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:240)	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:154)	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:321)	at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:157)	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:229)	at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:82)	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:147)	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:93)	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:576)	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)	at java.base/java.lang.Thread.run(Thread.java:840)
      2024-02-22 12:31:22,343 WARN  [io.sma.rea.mes.provider] (vert.x-eventloop-thread-3) SRMSG00234: Failed to emit a Message to the channel: java.lang.IllegalStateException: SRMSG00034: Insufficient downstream requests to emit item	at io.smallrye.reactive.messaging.providers.extension.ThrowingEmitter.emit(ThrowingEmitter.java:60)	at io.smallrye.reactive.messaging.providers.extension.AbstractEmitter.emit(AbstractEmitter.java:162)	at io.smallrye.reactive.messaging.providers.extension.MutinyEmitterImpl.lambda$sendMessage$4(MutinyEmitterImpl.java:72)	at io.smallrye.context.impl.wrappers.SlowContextualConsumer.accept(SlowContextualConsumer.java:21)	at io.smallrye.mutiny.operators.uni.builders.UniCreateWithEmitter.subscribe(UniCreateWithEmitter.java:22)	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)	at io.smallrye.mutiny.operators.uni.UniEmitOn.subscribe(UniEmitOn.java:22)	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)	at io.smallrye.mutiny.groups.UniSubscribe.withSubscriber(UniSubscribe.java:51)	at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:110)	at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:88)	at io.smallrye.reactive.messaging.providers.extension.MutinyEmitterImpl.sendMessageAndForget(MutinyEmitterImpl.java:50)	at org.kie.kogito.events.process.ReactiveMessagingEventPublisher$ReactiveMessageEmitter.accept(ReactiveMessagingEventPublisher.java:164)	at org.kie.kogito.events.process.ReactiveMessagingEventPublisher$ReactiveMessageEmitter.accept(ReactiveMessagingEventPublisher.java:161)	at org.kie.kogito.events.process.ReactiveMessagingEventPublisher.publishToTopic(ReactiveMessagingEventPublisher.java:135)	at org.kie.kogito.events.process.ReactiveMessagingEventPublisher.publish(ReactiveMessagingEventPublisher.java:98)	at org.kie.kogito.events.process.ReactiveMessagingEventPublisher.publish(ReactiveMessagingEventPublisher.java:120)	at org.kie.kogito.event.impl.BaseEventManager.lambda$publish$0(BaseEventManager.java:58)	at java.base/java.lang.Iterable.forEach(Iterable.java:75)	at org.kie.kogito.event.impl.BaseEventManager.publish(BaseEventManager.java:58)	at org.kie.kogito.services.uow.CollectingUnitOfWork.end(CollectingUnitOfWork.java:69)	at org.kie.kogito.services.uow.ManagedUnitOfWork.end(ManagedUnitOfWork.java:54)	at org.kie.kogito.services.uow.UnitOfWorkExecutor.executeInUnitOfWork(UnitOfWorkExecutor.java:41)	at org.kie.kogito.services.jobs.impl.TriggerJobCommand.execute(TriggerJobCommand.java:59)	at org.kie.kogito.jobs.quarkus.common.CallbackJobsServiceResource.triggerTimer(CallbackJobsServiceResource.java:88)	at jdk.internal.reflect.GeneratedMethodAccessor139.invoke(Unknown Source)	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)	at java.base/java.lang.reflect.Method.invoke(Method.java:568)	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:154)	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:118)	at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:560)	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:452)	at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:413)	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:321)	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:415)	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:378)	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:356)	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:70)	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:429)	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:240)	at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:154)	at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:321)	at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:157)	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:229)	at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:82)	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:147)	at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:93)	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:576)	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)	at java.base/java.lang.Thread.run(Thread.java:840)
      2024-02-22 13:44:23,029 WARN  [org.jbo.res.cli.jax.i18n] (Cleaner-0) RESTEASY004687: Closing a class org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine$CleanupAction instance for you. Please close clients yourself.2024-02-22 13:44:23,041 WARN  [org.jbo.res.cli.jax.i18n] (Cleaner-0) RESTEASY004687: Closing a class org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine$CleanupAction instance for you. Please close clients yourself.2024-02-22 13:44:23,042 WARN  [org.jbo.res.cli.jax.i18n] (Cleaner-0) RESTEASY004687: Closing a class org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine$CleanupAction instance for you. Please close clients yourself.2024-02-22 13:44:23,042 WARN  [org.jbo.res.cli.jax.i18n] (Cleaner-0) RESTEASY004687: Closing a class org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine$CleanupAction instance for you. Please close clients yourself. 

            rh-ee-gfarache Gabriel Farache
            yfirst Yona First
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: