Loading...

XML

Word

Printable

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
- sendgrid

Story Points:
5
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Discussed with Team:
Yes
Git Pull Request:
https://gitlab.cee.redhat.com/service/ocm-sendgrid-service/-/merge_requests/84, https://gitlab.cee.redhat.com/service/ocm-sendgrid-service/-/merge_requests/85
Intelligence Requested:
Market:

Sprint:
RHOAM Sprint 34, RHOAM Sprint 37

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

WHAT
There is a memory leak in sendgrid service that is preventing jobs from being processed correctly after a certain point.

This might be the cause of MGDAPI-4930

In the attached screenshot, the memory started increasing from 04:00 - 10:00 and finally dropped after the container was restarted. When this pod was restarted, another pod was elected the leader and jobs were processed again.

Jobs looks to fail to list the subusers for checking sub user existance, but no error was logged or whether the subuser creation was successfull

I1216 04:17:47.093594       1 logger.go:84]  executing job asynchronously, attempt 1 [scheduler=create_scheduler] [cluster_id=20kphn816rd9kjphf0hhcrfsoj8852ju] [cluster_uuid=58e86929-c92d-4b62-a7e4-06deb1a6d62b] [addon=Red Hat OpenShift Data Science]
I1216 04:17:47.093614       1 logger.go:84]  creating sendgrid smtp credentials [addon=Red Hat OpenShift Data Science] [scheduler=create_scheduler] [cluster_id=20kphn816rd9kjphf0hhcrfsoj8852ju] [cluster_uuid=58e86929-c92d-4b62-a7e4-06deb1a6d62b]
I1216 04:17:47.096628       1 logger.go:84]  checking if sub user 20kphn816rd9kjphf0hhcrfsoj8852ju exists [sendgrid=smtp_service_detail_provider]
I1216 04:17:47.096644       1 logger.go:84]  getting request with details endpoint=/v3/subusers host=https://api.sendgrid.com [sendgrid=sendgrid_service_api_client]
I1216 04:17:47.096653       1 logger.go:84]  performing api request with details, url=https://api.sendgrid.com/v3/subusers method=GET [sendgrid=sendgrid_service_api_client]
I1216 04:17:47.494238       1 logger.go:84] [opid=2Iyqeka7ESYwjb8Yj5whCjOypnb] {"request_method":"GET","request_url":"/api/ocm-sendgrid-service","request_remote_ip":"10.129.16.1:45940"}
I1216 04:17:47.495390       1 logger.go:84]  getting request with details endpoint=/v3/subusers host=https://api.sendgrid.com [sendgrid=sendgrid_service_api_client]
I1216 04:17:47.495408       1 logger.go:84]  performing api request with details, url=https://api.sendgrid.com/v3/subusers method=GET [sendgrid=sendgrid_service_api_client]
I1216 04:17:47.504409       1 logger.go:84] [opid=2Iyqeka7ESYwjb8Yj5whCjOypnb][tx_id=89209291] {"elapsed":"10.143628ms"}
I1216 04:17:52.494358       1 logger.go:84] [opid=2IyqfJvIA8XVZLpp5xS8H6WFVL7] {"request_method":"GET","request_url":"/api/ocm-sendgrid-service","request_remote_ip":"10.129.16.1:49310"}
I1216 04:17:52.497901       1 logger.go:84] [opid=2IyqfJvIA8XVZLpp5xS8H6WFVL7][tx_id=89209297] {"elapsed":"3.484438ms"}
I1216 04:17:53.083448       1 logger.go:84]  found no new jobs [scheduler=create_scheduler]