Uploaded image for project: 'Cost Management'
  1. Cost Management
  2. COST-1207

Investigate slowness in Listener code

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Done
    • Icon: Normal Normal
    • 2021Q2
    • None
    • None
    • False
    • False
    • Undefined

      https://sentry.io/organizations/project-koku/issues/2291221704/?project=5183023&referrer=slack

       
      This alert is raised because we are taking longer than 18 minutes to process a kafka message in the listener. 18 minutes is more than enough time to process the message we receive.

      Looking through some logs, I see this:

      [2021-03-21 21:27:32,527] INFO None Found Authentication ID 230224 for Cluster ID: 
      
      ...%4|1616363132.646|MAXPOLL|rdkafka#consumer-1| [thrd:main]: Application maximum poll interval (1080000ms) exceeded by 411ms (adjust max.poll.interval.ms for long-running message processing): leaving group
      
      [2021-03-21 22:08:07,923] INFO None Found provider: ...
      

      The second log message here is output directly from Kafka telling us we missed a poll within our 18 minute window, so we left the group. This message is fine; it's just telling us we're taking too long. We rejoin the group the next time we poll for messages.

      The real issue here is the amount of time between the `INFO None Found Authentication` and the `INFO None Found provider`. These logs come from here. The code seems to hang when fetching the provider_uuid.

              aberglun@redhat.com Andrew Berglund (Inactive)
              mskarbek Michael Skarbek
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: