Loading...

Type: Bug
Resolution: Unresolved
Priority: Normal
Fix Version/s: None
Affects Version/s: None
Component/s: swatch-tally
Labels:
- refinable
- refineable

Story Points:
8
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
AssignedTeam:
subs-swatch-thunder
Intelligence Requested:
Market:

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

Splunk query: "index=rh_rhsm namespace=rhsm-prod "Error executing task: TaskDescriptor[groupId: platform.rhsm-conduit.tasks, taskType: UPDATE_ORG_INVENTORY""

Error:

2025-01-24T02:16:22.870+0000 [thread=rhsm-conduit-task-processor-0-C-1] [level=ERROR] [category=org.candlepin.subscriptions.task.queue.kafka.KafkaTaskProcessor]  - Failed to execute task: JsonTaskMessage(groupId=platform.rhsm-conduit.tasks, type=UPDATE_ORG_INVENTORY, args={offset=[null], org_id=[376200]})
org.candlepin.subscriptions.task.TaskExecutionException: Error executing task: TaskDescriptor[groupId: platform.rhsm-conduit.tasks, taskType: UPDATE_ORG_INVENTORY, args: [offset: [null], org_id: [376200]]]
	at org.candlepin.subscriptions.task.TaskWorker.executeTask(TaskWorker.java:47)
	at org.candlepin.subscriptions.task.queue.kafka.KafkaTaskProcessor.receive(KafkaTaskProcessor.java:60)
	at jdk.internal.reflect.GeneratedMethodAccessor168.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:169)
	at org.springframework.kafka.listener.adapter.KotlinAwareInvocableHandlerMethod.doInvoke(KotlinAwareInvocableHandlerMethod.java:45)
	at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:119)
	at org.springframework.kafka.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:70)
	at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:420)
	at org.springframework.kafka.listener.adapter.MessagingMessageListenerAdapter.invoke(MessagingMessageListenerAdapter.java:384)
	at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:85)
	at org.springframework.kafka.listener.adapter.RecordMessagingMessageListenerAdapter.onMessage(RecordMessagingMessageListenerAdapter.java:50)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:2800)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2778)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.lambda$doInvokeRecordListener$53(KafkaMessageListenerContainer.java:2701)
	at io.micrometer.observation.Observation.observe(Observation.java:565)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2699)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:2541)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2430)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:2085)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1461)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1426)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1296)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: jakarta.ws.rs.ProcessingException: RESTEASY004655: Unable to invoke request: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
	at org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine.invoke(ManualClosingApacheHttpClient43Engine.java:365)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:427)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:71)
	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocationBuilder.get(ClientInvocationBuilder.java:175)
	at org.candlepin.subscriptions.conduit.rhsm.client.ApiClient.invoke(ApiClient.java:700)
	at org.candlepin.subscriptions.conduit.rhsm.client.ApiClient.invokeAPI(ApiClient.java:665)
	at org.candlepin.subscriptions.conduit.rhsm.client.resources.RhsmApi.getConsumersForOrg(RhsmApi.java:87)
	at org.candlepin.subscriptions.conduit.rhsm.RhsmService.lambda$getPageOfConsumers$0(RhsmService.java:115)
	at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:344)
	at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:217)
	at org.candlepin.subscriptions.conduit.rhsm.RhsmService.getPageOfConsumers(RhsmService.java:100)
	at jdk.internal.reflect.GeneratedMethodAccessor170.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:355)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768)
	at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:174)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:720)
	at org.candlepin.subscriptions.conduit.rhsm.RhsmService$$SpringCGLIB$$0.getPageOfConsumers(<generated>)
	at org.candlepin.subscriptions.conduit.InventoryController.getConsumerFeed(InventoryController.java:615)
	at org.candlepin.subscriptions.conduit.InventoryController.updateInventoryForOrg(InventoryController.java:630)
	at org.candlepin.subscriptions.conduit.tasks.UpdateOrgInventoryTask.execute(UpdateOrgInventoryTask.java:52)
	at org.candlepin.subscriptions.task.TaskWorker.executeTask(TaskWorker.java:44)
	... 25 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
	at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1719)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1518)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1425)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine.invoke(ManualClosingApacheHttpClient43Engine.java:348)
	... 51 common frames omitted
Caused by: java.io.EOFException: SSL peer shut down incorrectly
	at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
	at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
	at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1510)
	... 67 common frames omitted

This error is not too often (it happened 12 times in the last 10 days) but it needs to be addressed.

At the moment, we have configured a bulkhead contingency that only allows 5 concurrent calls per replica with a max duration of 3 minutes (note we have 3 replicas running).
RHSM API: https://api.rhsm.redhat.com/v1/candlepin/consumers/feeds

Acceptance Criteria

Agree with the RHSM team the course of action for this issue (either investigate the RHSM limits in their side or better configure the bulkhead settings in our side).

Details

Description

Acceptance Criteria

Attachments

Easy Agile Planning Poker

Activity

People

Dates