Uploaded image for project: 'Thorntail'
  1. Thorntail
  2. THORN-2004

Document some fallback internals in Microprofile implementation when using @Asynchronous and @Bulkhead

    Details

      Description

      Internal implementation in Wildfly Swarm microprofile when using @Asynchronous together with @Bulkhead can lead to some unwanted responses if expecting fallback instead.

      When using Asynchronous services, Wildfly Swarm creates a pool that will deal with requests. Additionally, it uses other pool for bulkhead operations.

      Some of the non explicit behavior is that, when having enough number of requests, the first pool (dealing with asynchronous requests) can be full, leading to some unwanted errors, instead of getting the fallback response:

      2018-04-27 10:01:29,508 ERROR [io.undertow.request] (default task-17) UT005023: Exception handling request to /async: com.netflix.hystrix.exception.HystrixRuntimeException: CompositeCommand#org_wildfly_swarm_ts_microprofile_fault_tolerance_v10_AsyncHelloService#bulkhead(boolean) could not be queued for execution and fallback disabled.
      	at com.netflix.hystrix.AbstractCommand.handleFallbackDisabledByEmittingError(AbstractCommand.java:1052)
      	at com.netflix.hystrix.AbstractCommand.getFallbackOrThrowException(AbstractCommand.java:878)
      	at com.netflix.hystrix.AbstractCommand.handleThreadPoolRejectionViaFallback(AbstractCommand.java:993)
      	at com.netflix.hystrix.AbstractCommand.access$400(AbstractCommand.java:60)
      	at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:608)
      	at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:601)
      	at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)
      	at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
      	at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
      	at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$2.onError(AbstractCommand.java:1194)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:54)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
      	at rx.Observable.unsafeSubscribe(Observable.java:10144)
      	at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
      	at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
      	at rx.Observable.unsafeSubscribe(Observable.java:10144)
      	at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
      	at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
      	at rx.Observable.unsafeSubscribe(Observable.java:10144)
      	at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
      	at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
      	at rx.Observable.unsafeSubscribe(Observable.java:10144)
      	at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
      	at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
      	at rx.Observable.unsafeSubscribe(Observable.java:10144)
      	at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
      	at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
      	at rx.Observable.unsafeSubscribe(Observable.java:10144)
      	at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
      	at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
      	at rx.Observable.unsafeSubscribe(Observable.java:10144)
      	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
      	at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
      	at rx.Observable.unsafeSubscribe(Observable.java:10144)
      	at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
      	at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
      	at rx.Observable.unsafeSubscribe(Observable.java:10144)
      	at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
      	at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
      	at rx.Observable.unsafeSubscribe(Observable.java:10144)
      	at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
      	at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
      	at rx.Observable.subscribe(Observable.java:10240)
      	at rx.Observable.subscribe(Observable.java:10207)
      	at rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:51)
      	at rx.observables.BlockingObservable.toFuture(BlockingObservable.java:411)
      	at com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:378)
      	at org.wildfly.swarm.microprofile.faulttolerance.deployment.CompositeCommand.createAndQueue(CompositeCommand.java:40)
      	at org.wildfly.swarm.microprofile.faulttolerance.deployment.HystrixCommandInterceptor.interceptCommand(HystrixCommandInterceptor.java:141)
      	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73)
      	at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84)
      	at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72)
      	at org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56)
      	at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79)
      	at org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68)
      	at org.wildfly.swarm.ts.microprofile.fault.tolerance.v10.AsyncHelloService$Proxy$_$$_WeldSubclass.bulkhead(Unknown Source)
      	at org.wildfly.swarm.ts.microprofile.fault.tolerance.v10.AsyncHelloService$Proxy$_$$_WeldClientProxy.bulkhead(Unknown Source)
      	at org.wildfly.swarm.ts.microprofile.fault.tolerance.v10.AsyncHelloServlet.doGet(AsyncHelloServlet.java:34)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
      	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
      	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
      	at org.wildfly.swarm.generated.FaviconErrorHandler.handleRequest(FaviconErrorHandler.java:61)
      	at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:94)
      	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
      	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
      	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
      	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
      	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
      	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
      	at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
      	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
      	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
      	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
      	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
      	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
      	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
      	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
      	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:326)
      	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:812)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@34720146 rejected from java.util.concurrent.ThreadPoolExecutor@1149b77[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
      	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
      	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
      	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
      	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
      	at com.netflix.hystrix.strategy.concurrency.HystrixContextScheduler$ThreadPoolWorker.schedule(HystrixContextScheduler.java:172)
      	at com.netflix.hystrix.strategy.concurrency.HystrixContextScheduler$HystrixContextSchedulerWorker.schedule(HystrixContextScheduler.java:106)
      	at rx.internal.operators.OperatorSubscribeOn.call(OperatorSubscribeOn.java:45)
      	at rx.internal.operators.OperatorSubscribeOn.call(OperatorSubscribeOn.java:30)
      	at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
      	... 93 more
      

      So, in order to ensure fallback responses are being sent instead of unwanted errors, it's necessary to tweak this parameter in project-defaults.yml:

      swarm:
        hystrix:
          threadpool:
             default:
                maximumSize: 40
                allowMaximumSizeToDivergeFromCoreSize: true
      

      maximumSize should be high enough as the number of concurrent requests to ensure all fallback responses are being sent.

      To check this behavior use the attached Maven project test-microprofile-async-bulkhead.zip , unzip into some location, and then

      mvn clean verify

      . To make it fail, just change maximumSize to a value less than concurrent requests (40) and you would see the unwanted error instead of the expected fallback response.

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                mkouba Martin Kouba
                Reporter:
                juagonza Juan Gonzalez
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: