https://sentry.io/organizations/project-koku/issues/2291221704/?project=5183023&referrer=slack
This alert is raised because we are taking longer than 18 minutes to process a kafka message in the listener. 18 minutes is more than enough time to process the message we receive.
Looking through some logs, I see this:
[2021-03-21 21:27:32,527] INFO None Found Authentication ID 230224 for Cluster ID: ...%4|1616363132.646|MAXPOLL|rdkafka#consumer-1| [thrd:main]: Application maximum poll interval (1080000ms) exceeded by 411ms (adjust max.poll.interval.ms for long-running message processing): leaving group [2021-03-21 22:08:07,923] INFO None Found provider: ...
The second log message here is output directly from Kafka telling us we missed a poll within our 18 minute window, so we left the group. This message is fine; it's just telling us we're taking too long. We rejoin the group the next time we poll for messages.
The real issue here is the amount of time between the `INFO None Found Authentication` and the `INFO None Found provider`. These logs come from here. The code seems to hang when fetching the provider_uuid.