Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-1954

Clair v4 notifier crashes on multiple intervals

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Can't Do
    • Icon: Major Major
    • None
    • None
    • clair
    • False
    • False
    • Quay Enterprise
    • Undefined

      If we let Notifier run for longer time , we see clair v4  pod being restarted multiple time and on checking the logs we observe notifier is crashing

      Attaching the logs

       

      {"level":"info","component":"notifier/service/Init","component":"notifier/service/webhookInit","component":"notifier/webhook/deliverer.Deliver","notification_id":"4fe8cf67-2a3e-421c-a258-a31f48a71ec1","notification_id":"4fe8cf67-2a3e-421c-a258-a31f48a71ec1","callback":"http://quay-clair-app/notifier/api/v1/notifications/4fe8cf67-2a3e-421c-a258-a31f48a71ec1","target":"https://quay-quay-quay-ss.apps.made-ss.513h.p1.openshiftapps.com/secscan/notification","time":"2021-04-28T05:54:42Z","time":"2021-04-28T05:54:42Z","message":"dispatching webhook"}
      fatal error: concurrent map iteration and map writegoroutine 7215 [running]:
      runtime.throw(0xed43db, 0x26)
              /opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/runtime/panic.go:1116 +0x72 fp=0xc0001c2060 sp=0xc0001c2030 pc=0x43e1f2
      runtime.mapiternext(0xc0001c2188)
              /opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/runtime/map.go:853 +0x554 fp=0xc0001c20e0 sp=0xc0001c2060 pc=0x417694
      net/http.(*http2ClientConn).encodeHeaders.func1(0xc0012220a0)
              /opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/http/h2_bundle.go:7971 +0x20e fp=0xc0001c21f8 sp=0xc0001c20e0 pc=0x7d710e
      net/http.(*http2ClientConn).encodeHeaders(0xc000879e00, 0xc0001d4a00, 0x1, 0x0, 0x0, 0xffffffffffffffff, 0x0, 0x1492, 0xc0001c2498, 0x41f285, ...)
              /opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/http/h2_bundle.go:8055 +0x637 fp=0xc0001c23e0 sp=0xc0001c21f8 pc=0x797097
      net/http.(*http2ClientConn).roundTrip(0xc000879e00, 0xc0001d4a00, 0x0, 0x0, 0x0, 0x0)
              /opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/http/h2_bundle.go:7524 +0x2bb fp=0xc0001c2788 sp=0xc0001c23e0 pc=0x7941fb
      net/http.(*http2Transport).RoundTripOpt(0xc0029386c0, 0xc0001d4a00, 0xc000150800, 0xc000cea2c0, 0xc0001508f0, 0xf00cf8)
              /opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/http/h2_bundle.go:6948 +0x170 fp=0xc0001c28b0 sp=0xc0001c2788 pc=0x790270
      net/http.(*http2Transport).RoundTrip(0xc0029386c0, 0xc0001d4a00, 0x0, 0xc00007c720, 0x5)
              /opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/http/h2_bundle.go:6909 +0x3a fp=0xc0001c28f0 sp=0xc0001c28b0 pc=0x78fd1a
      net/http.(*Transport).roundTrip(0xc0001508c0, 0xc0001d4900, 0xc000878a80, 0xc0001c2b68, 0xc0023fa000)
              /opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/http/transport.go:580 +0x7f3 fp=0xc0001c2b38 sp=0xc0001c28f0 pc=0x7c66d3
      net/http.(*Transport).RoundTrip(0xc0001508c0, 0xc0001d4900, 0xc0019e7400, 0xd, 0xc00fdee0b8)
              /opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/http/roundtrip.go:17 +0x35 fp=0xc0001c2b70 sp=0xc0001c2b38 pc=0x7ab715
      github.com/quay/clair/v4/config.(*transport).RoundTrip(0xc00021e000, 0xc0001d4900, 0xc00021e000, 0x0, 0x0)
              /remote-source/app/config/httpclient.go:78 +0x12c fp=0xc0001c2c18 sp=0xc0001c2b70 pc=0x85658c
      net/http.send(0xc0001d4900, 0xfcf000, 0xc00021e000, 0x0, 0x0, 0x0, 0xc0002081c8, 0x4800, 0x1, 0x0)
              /opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/http/client.go:252 +0x453 fp=0xc0001c2dc0 sp=0xc0001c2c18 pc=0x76cb33
      net/http.(*Client).send(0xc06a57e030, 0xc0001d4900, 0x0, 0x0, 0x0, 0xc0002081c8, 0x0, 0x1, 0x226)
              /opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/http/client.go:176 +0xff fp=0xc0001c2e40 sp=0xc0001c2dc0 pc=0x76c51f
      net/http.(*Client).do(0xc06a57e030, 0xc0001d4900, 0x0, 0x0, 0x0)
              /opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/http/client.go:718 +0x45f fp=0xc0001c3018 sp=0xc0001c2e40 pc=0x76e53f
      net/http.(*Client).Do(...)
              /opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/http/client.go:586
      github.com/quay/clair/v4/notifier/webhook.(*Deliverer).Deliver(0xc0006b0140, 0xfeadc0, 0xc000ec0030, 0x1c479fc63d63e3e1, 0x31069d20a45fee93, 0x0, 0x0)
              /remote-source/app/notifier/webhook/deliverer.go:125 +0x8cb fp=0xc0001c3530 sp=0xc0001c3018 pc=0x854e6b
      github.com/quay/clair/v4/notifier.(*Delivery).do(0xc018d28140, 0xfeadc0, 0xc000ec0030, 0x1c479fc63d63e3e1, 0x31069d20a45fee93, 0x1, 0x0)
      

       

              Unassigned Unassigned
              rhn-support-dgangaia Dixit Gangaiah (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: