-
Bug
-
Resolution: Done
-
Major
-
None
-
None
-
None
-
False
-
None
-
False
-
MGDOBR - Sprint 221, MGDOBR - Sprint 222
Issue Description:
If users create one Bridge per second then Manager is not able to provision those Bridges properly, resulting in Bridges with failure state.
This seems caused by long Kafka reaction time in case of batch processing. As work is executed every 30 seconds it means that ~30 Bridges are processed at the same time (experiments shows that such problems can start happening when processing 10-20 Bridges at once), causing parallel Kafka requests. Such parallel request results in Kafka responses to be delayed, reaching defined 15 second timeout for RHOAS.
For example (see timestamps):
{"timestamp":"2022-05-19T07:21:00.002Z","loggerName":"com.redhat.service.smartevents.manager.workers.WorkManagerImpl","level":"INFO","message":"Executing work for 'aebf14bf-e22d-415f-919a-413d12875eee' [com.redhat.service.smartevents.manager.models.Bridge]","threadName":"executor-thread-217","hostName":"event-bridge-84d5f4f9f5-95zgn","workerId":"event-bridge-84d5f4f9f5-95zgn"}
...
{"timestamp":"2022-05-19T07:21:00.018Z","loggerName":"com.redhat.service.smartevents.manager.workers.resources.BridgeWorker","level":"INFO","message":"Creating dependencies for 'perf-95a8cfc5-f566-3af7-aeec-0325cdae8fbd' [aebf14bf-e22d-415f-919a-413d12875eee]","threadName":"executor-thread-232","hostName":"event-bridge-84d5f4f9f5-95zgn","workerId":"event-bridge-84d5f4f9f5-95zgn"}
...
{"timestamp":"2022-05-19T07:21:15.027Z","loggerName":"com.redhat.service.smartevents.manager.workers.resources.AbstractWorker","level":"ERROR","message":"Failed to create dependencies for 'perf-95a8cfc5-f566-3af7-aeec-0325cdae8fbd' [aebf14bf-e22d-415f-919a-413d12875eee].\nWork status: Work{id='4d0e7c34-2ca3-4b30-b0fa-ddef9e757c1c', managedResourceId='aebf14bf-e22d-415f-919a-413d12875eee', type='com.redhat.service.smartevents.manager.models.Bridge', workerId='event-bridge-84d5f4f9f5-95zgn', submittedAt=2022-05-19T07:20:46.086431Z[GMT], modifiedAt=2022-05-19T07:21:00.002897Z[GMT], attempts=0}\n{}","threadName":"executor-thread-232","hostName":"event-bridge-84d5f4f9f5-95zgn","stackTrace":"com.redhat.service.smartevents.infra.exceptions.definitions.platform.InternalPlatformException: Timeout reached while creating topic and granting access to topic 'ob-dev-brdg-aebf14bf-e22d-415f-919a-413d12875eee'\n\tat com.redhat.service.smartevents.manager.RhoasServiceImpl.createTopicAndGrantAccessFor(RhoasServiceImpl.java:53)\n\tat com.redhat.service.smartevents.manager.RhoasServiceImpl_ClientProxy.createTopicAndGrantAccessFor(Unknown Source)\n\tat com.redhat.service.smartevents.manager.workers.resources.BridgeWorker.createDependencies(BridgeWorker.java:59)\n\tat com.redhat.service.smartevents.manager.workers.resources.BridgeWorker.createDependencies(BridgeWorker.java:20)\n\tat com.redhat.service.smartevents.manager.workers.resources.AbstractWorker.handleWork(AbstractWorker.java:71)\n\tat com.redhat.service.smartevents.manager.workers.resources.BridgeWorker.handleWork(BridgeWorker.java:42)\n\tat com.redhat.service.smartevents.manager.workers.resources.BridgeWorker.handleWork(BridgeWorker.java:20)\n\tat com.redhat.service.smartevents.manager.workers.resources.BridgeWorker_ClientProxy.handleWork(Unknown Source)\n\tat com.redhat.service.smartevents.manager.workers.resources.BridgeWorker_VertxInvoker_handleWork_2d8766c2224ec4eeb8e66e7dc13b88a672cbf8ad.invokeBean(Unknown Source)\n\tat io.quarkus.vertx.runtime.EventConsumerInvoker.invoke(EventConsumerInvoker.java:41)\n\tat io.quarkus.vertx.runtime.VertxRecorder$3$1$1.handle(VertxRecorder.java:111)\n\tat io.quarkus.vertx.runtime.VertxRecorder$3$1$1.handle(VertxRecorder.java:107)\n\tat io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)\n\tat io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)\n\tat io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)\n\tat io.quarkus.vertx.core.runtime.VertxCoreRecorder$13.runWith(VertxCoreRecorder.java:543)\n\tat org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)\n\tat org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)\n\tat org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)\n\tat org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: io.smallrye.mutiny.TimeoutException\n\tat io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:64)\n\tat io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:65)\n\tat com.redhat.service.smartevents.manager.RhoasServiceImpl.createTopicAndGrantAccessFor(RhoasServiceImpl.java:49)\n\t... 21 more\n","errorType":"com.redhat.service.smartevents.infra.exceptions.definitions.platform.InternalPlatformException","errorMessage":"Timeout reached while creating topic and granting access to topic 'ob-dev-brdg-aebf14bf-e22d-415f-919a-413d12875eee'","workerId":"event-bridge-84d5f4f9f5-95zgn"}
...
{"timestamp":"2022-05-19T07:21:22.399Z","loggerName":"com.redhat.service.smartevents.rhoas.RhoasClientImpl","level":"INFO","message":"Created consumer and producer ACLs for user='srvc-acct-3a46e8f6-5a39-42d4-959f-857b5e0f944b' and topic='ob-dev-brdg-aebf14bf-e22d-415f-919a-413d12875eee'","threadName":"vert.x-eventloop-thread-2","hostName":"event-bridge-84d5f4f9f5-95zgn","workerId":"event-bridge-84d5f4f9f5-95zgn"}
...
{"timestamp":"2022-05-19T07:21:28.485Z","loggerName":"com.redhat.service.smartevents.rhoas.RhoasClientImpl","level":"INFO","message":"Created topic 'ob-dev-brdg-aebf14bf-e22d-415f-919a-413d12875eee'","threadName":"vert.x-eventloop-thread-2","hostName":"event-bridge-84d5f4f9f5-95zgn","workerId":"event-bridge-84d5f4f9f5-95zgn"}
Acceptance Criteria:
Manager is able to properly process Bridges when one Bridge is created every second.
Out of Scope:
Additional Information:
- relates to
-
MGDOBR-640 Coordinate and provide performance testing for RHOSE
-
- Closed
-
-
MGDOBR-461 Workers: Reduce WorkManager poll interval
-
- Closed
-
- mentioned on
