Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-3875

Clair Crashing with error "fatal error: concurrent map iteration and map write"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Normal Normal
    • clair-4.4.3
    • clair-4.4.1
    • clair
    • True
    • The client is not able to use Clair as Clair pods are restarting frequently
    • False
    • Compatibility/Configuration, User Experience
    • Medium
    • Critical
    • 75% (Medium)

      As per the customer, Clair pods are restarting every 3 to 5 h.

      While checking the logs, we found this error

      ~~~
      2022-05-25T06:34:48.161870559Z {"level":"info","deliverer":"webhook","component":"notifier/webhook/Deliverer.Deliver","notification_id":"ecce9f14-a4a4-4dd6-894c-7b996a0256c5","callback":"http://quay-clair-app/notifier/api/v1/notifications/ecce9f14-a4a4-4dd6-894c-7b996a0256c5","target":"https://quay.lab.apps.cloud.internal/secscan/notification","time":"2022-05-25T06:34:48Z","message":"dispatching webhook"}
      2022-05-25T06:34:48.161912559Z fatal error: concurrent map iteration and map write
      2022-05-25T06:34:48.165187391Z 
      2022-05-25T06:34:48.165187391Z goroutine 6968 [running]:
      2022-05-25T06:34:48.165187391Z runtime.throw({0xed0143, 0x0}2022-05-25T06:34:48.165227591Z )
      2022-05-25T06:34:48.165227591Z  /usr/lib/golang/src/runtime/panic.go:1198 +0x71 fp=0xc000884930 sp=0xc000884900 pc=0x43bef1
      2022-05-25T06:34:48.165227591Z runtime.mapiternext(0xeaf5822022-05-25T06:34:48.165248691Z )
      2022-05-25T06:34:48.165248691Z  /usr/lib/golang/src/runtime/map.go:858 +0x4eb fp=0xc0008849a0 sp=0xc000884930 pc=2022-05-25T06:34:48.165268392Z 0x415bab
      2022-05-25T06:34:48.165268392Z net/http.(*http2ClientConn).encodeHeaders.func1(0xc000884b58)
      2022-05-25T06:34:48.165268392Z  /usr/lib/golang/src/net/http/h2_bundle.go:2022-05-25T06:34:48.165288592Z 8311 +0x1dc fp=0xc000884ab0 sp=0xc0008849a0 pc=0x69323c
      2022-05-25T06:34:48.165307292Z net/http.(*http2ClientConn).encodeHeaders(0xc0007d0f00, 0xc000046900, 0x1, {0x02022-05-25T06:34:48.165325792Z , 0x0}, 0xffffffffffffffff)
      ~~~
      

      I have attached the logs where you can see this error

        1. Clair_logs
          708 kB
          Raunak Borkar

              Unassigned Unassigned
              rhn-support-rborkar Raunak Borkar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: