Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-24491

cert-manager logs show `fatal error: concurrent map read and map write`

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Undefined Undefined
    • None
    • 4.14
    • cert-manager
    • Moderate
    • No
    • CFE Sprint 254
    • 1
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      cert-manager logs show `fatal error: concurrent map read and map write`

      Version-Release number of selected component (if applicable):

      OCP 4.14.1
      cert-manager-operator-bundle:v1.13.0-3

      How reproducible:

      Not sure. Hit once

      Steps to Reproduce:

      1. Install cert-manager from console
      2. Check pods:
      $ oc get po -n cert-manager
      NAME                                       READY   STATUS    RESTARTS        AGE
      cert-manager-b94b579f-m6j9v                1/1     Running   1 (5m28s ago)   6m20s
      cert-manager-cainjector-849684c4cd-ldqrw   1/1     Running   0               6m57s
      cert-manager-webhook-7d946df88f-4lscp      1/1     Running   0               6m59s
      3. oc logs --previous cert-manager-b94b579f-m6j9v -n cert-manager > logs/cert-manager-b94b579f-m6j9v_previous.log
      I1206 13:52:20.156106       1 controller.go:251] "cert-manager/controller/build-context: configured acme dns01 nameservers" nameservers=["172.30.0.10:53"]
      W1206 13:52:20.156232       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
      I1206 13:52:20.157341       1 controller.go:72] "cert-manager/controller: enabled controllers: [certificaterequests-approver certificaterequests-issuer-acme certificaterequests-issuer-ca certificaterequests-issuer-selfsigned certificaterequests-issuer-vault certificaterequests-issuer-venafi certificates-issuing certificates-key-manager certificates-metrics certificates-readiness certificates-request-manager certificates-revision-manager certificates-trigger challenges clusterissuers ingress-shim issuers orders]"
      I1206 13:52:20.157954       1 controller.go:145] "cert-manager/controller: starting leader election"
      I1206 13:52:20.157980       1 controller.go:93] "cert-manager/controller: starting metrics server" address="[::]:9402"
      I1206 13:52:20.158088       1 controller.go:138] "cert-manager/controller: starting healthz server" address="[::]:9403"
      I1206 13:52:20.158571       1 leaderelection.go:250] attempting to acquire leader lease kube-system/cert-manager-controller...
      I1206 13:52:20.177614       1 leaderelection.go:260] successfully acquired lease kube-system/cert-manager-controller
      I1206 13:52:20.178751       1 controller.go:215] "cert-manager/controller: starting controller" controller="challenges"
      I1206 13:52:20.179185       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificaterequests-approver"
      I1206 13:52:20.179380       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-ca"
      I1206 13:52:20.179636       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-metrics"
      I1206 13:52:20.179909       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-readiness"
      I1206 13:52:20.180172       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-revision-manager"
      I1206 13:52:20.180277       1 controller.go:192] "cert-manager/controller: not starting controller as it's disabled" controller="gateway-shim"
      I1206 13:52:20.180389       1 controller.go:215] "cert-manager/controller: starting controller" controller="issuers"
      fatal error: concurrent map read and map write
      
      goroutine 91 [running]:
      k8s.io/apimachinery/pkg/runtime.(*Scheme).ObjectKinds(0xc0001c59d0, {0x2c883f0?, 0xc000b92140})
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:267 +0x1ca
      k8s.io/apimachinery/pkg/runtime.WithVersionEncoder.Encode({{0x2c8ed40, 0xc000342540}, {0x2c8a470, 0xc00013ce10}, {0x2c8a3d0, 0xc0001c59d0}}, {0x2c883f0, 0xc000b92140}, {0x2c79640, 0xc000ad16e0})
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/runtime/helper.go:223 +0x82
      k8s.io/apimachinery/pkg/runtime.Encode({0x2c8ecc8, 0xc000ad16b0}, {0x2c883f0, 0xc000b92140})
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:49 +0x64
      k8s.io/client-go/rest.(*Request).Body(0xc0009b7c20, {0x26cef20?, 0xc000b92140?})
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/rest/request.go:470 +0x48b
      k8s.io/client-go/kubernetes/typed/coordination/v1.(*leases).Update(0xc0003433e0, {0x2ca0dc0, 0xc00045a690}, 0xc000b92140, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0, ...}, ...})
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/kubernetes/typed/coordination/v1/lease.go:136 +0x15f
      k8s.io/client-go/tools/leaderelection/resourcelock.(*LeaseLock).Update(0xc0009b6a20, {0x2ca0dc0, 0xc00045a690}, {{0xc0009e27c0, 0x3c}, 0x3c, {{0x973b938, 0xedd027394, 0x409a240}}, {{0xc1543d050a96cc89, ...}}, ...})
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/leaderelection/resourcelock/leaselock.go:75 +0x28b
      k8s.io/client-go/tools/leaderelection.(*LeaderElector).tryAcquireOrRenew(0xc0009b6b40, {0x2ca0dc0, 0xc00045a690})
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:368 +0x42f
      k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew.func1.1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:275 +0x25
      k8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1({0x7f024f546a68, 0x18})
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x1b
      k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x2ca1d48?, 0xc00042e360?}, 0xc000adfa30?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:154 +0x57
      k8s.io/apimachinery/pkg/util/wait.poll({0x2ca1d48, 0xc00042e360}, 0x70?, 0x1243f85?, 0x7f024f546a68?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:245 +0x38
      k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x2ca1d48, 0xc00042e360}, 0x40e027?, 0x20?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x49
      k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x2ca0d50?, 0xc0006de190?, 0x40dc4a?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x3f
      k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:274 +0x10d
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x3e
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00042e300?, {0x2c7f1e0, 0xc00045a660}, 0x1, 0xc00042e300)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xb6
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0006de190?, 0x37e11d600, 0x0, 0x20?, 0x20?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x89
      k8s.io/apimachinery/pkg/util/wait.Until(...)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
      k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew(0xc0009b6b40, {0x2ca0d50?, 0xc0006de140?})
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:271 +0x125
      k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc0009b6b40, {0x2ca0d50, 0xc0004a9450})
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:214 +0x119
      github.com/cert-manager/cert-manager/controller-binary/app.startLeaderElection({0x2ca0d50, 0xc0004a9450}, 0xc0009ad340, {0x2cc26e8, 0xc000503380}, {0x2c9f6e0?, 0xc000061240}, {0xc000a07e50, 0xc000342f40, 0x0}, ...)
      	/remote-source/app/cmd/controller/app/controller.go:373 +0x328
      github.com/cert-manager/cert-manager/controller-binary/app.Run.func6()
      	/remote-source/app/cmd/controller/app/controller.go:151 +0x20c
      golang.org/x/sync/errgroup.(*Group).Go.func1()
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
      created by golang.org/x/sync/errgroup.(*Group).Go
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
      
      goroutine 1 [runnable]:
      type:.eq.k8s.io/apimachinery/pkg/runtime/schema.GroupVersionKind(0xc00087f260?, 0xc00087f308?)
      	<autogenerated>:1 +0xac
      k8s.io/apimachinery/pkg/runtime.(*Scheme).AddKnownTypeWithName(0xc0001c59d0, {{0x272e6d2, 0x19}, {0x270850b, 0x7}, {0x2076db2, 0xd}}, {0x2c8a150?, 0x4099d60})
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:177 +0x32a
      k8s.io/apimachinery/pkg/runtime.(*Scheme).AddKnownTypes(0xc0001c59d0, {{0x272e6d2?, 0x19?}, {0x270850b?, 0xa?}}, {0x406f980?, 0x6, 0x2c8a1a0?})
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:148 +0x171
      k8s.io/apimachinery/pkg/apis/meta/v1.AddToGroupVersion(0xc0001c59d0?, {{0x272e6d2?, 0x2c8da30?}, {0x270850b?, 0x2c8da58?}})
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/register.go:73 +0x20c
      sigs.k8s.io/gateway-api/apis/v1beta1.addKnownTypes(0x2705a10?)
      	/remote-source/app/cmd/controller/vendor/sigs.k8s.io/gateway-api/apis/v1beta1/zz_generated.register.go:71 +0x20b
      k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme(...)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/runtime/scheme_builder.go:29
      k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme(...)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/runtime/scheme_builder.go:29
      github.com/cert-manager/cert-manager/pkg/controller.(*ContextFactory).Build(0xc0009e1800, {0xc00087f8d8?, 0x0?, 0xc00087f8d0?})
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/context.go:334 +0x122
      github.com/cert-manager/cert-manager/pkg/controller.(*Builder).Complete(0xc00087f908)
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/builder.go:83 +0x6a
      github.com/cert-manager/cert-manager/pkg/controller/certificate-shim/ingresses.init.0.func1(0xc0004a92c0?)
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/certificate-shim/ingresses/controller.go:150 +0x8c
      github.com/cert-manager/cert-manager/controller-binary/app.Run(0xc0009ad340, 0xc0009ad468?)
      	/remote-source/app/cmd/controller/app/controller.go:202 +0xe2a
      github.com/cert-manager/cert-manager/controller-binary/app.NewServerCommand.func1({0x2ca0df8?, 0xc0009a8030?}, 0x0?)
      	/remote-source/app/cmd/controller/app/start.go:59 +0x39
      github.com/cert-manager/cert-manager/controller-binary/app.newServerCommand.func1(0xc000004600?, {0x2703631?, 0x5?, 0x5?})
      	/remote-source/app/cmd/controller/app/start.go:106 +0x1f8
      github.com/spf13/cobra.(*Command).execute(0xc000004600, {0xc000130130, 0x5, 0x5})
      	/remote-source/app/cmd/controller/vendor/github.com/spf13/cobra/command.go:940 +0x862
      github.com/spf13/cobra.(*Command).ExecuteC(0xc000004600)
      	/remote-source/app/cmd/controller/vendor/github.com/spf13/cobra/command.go:1068 +0x3bd
      github.com/spf13/cobra.(*Command).Execute(0xc000280700?)
      	/remote-source/app/cmd/controller/vendor/github.com/spf13/cobra/command.go:992 +0x19
      main.main()
      	/remote-source/app/cmd/controller/main.go:38 +0x8f
      
      goroutine 24 [select]:
      go.opencensus.io/stats/view.(*worker).start(0xc0000caa00)
      	/remote-source/app/cmd/controller/vendor/go.opencensus.io/stats/view/worker.go:292 +0xad
      created by go.opencensus.io/stats/view.init.0
      	/remote-source/app/cmd/controller/vendor/go.opencensus.io/stats/view/worker.go:34 +0x96
      
      goroutine 53 [syscall]:
      os/signal.signal_recv()
      	/usr/lib/golang/src/runtime/sigqueue.go:152 +0x2f
      os/signal.loop()
      	/usr/lib/golang/src/os/signal/signal_unix.go:23 +0x19
      created by os/signal.Notify.func1.1
      	/usr/lib/golang/src/os/signal/signal.go:151 +0x2a
      
      goroutine 54 [chan receive]:
      github.com/cert-manager/cert-manager/internal/cmd/util.SetupExitHandler.func1()
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/internal/cmd/util/signal.go:58 +0x45
      created by github.com/cert-manager/cert-manager/internal/cmd/util.SetupExitHandler
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/internal/cmd/util/signal.go:56 +0xd7
      
      goroutine 57 [select]:
      k8s.io/klog/v2.(*flushDaemon).run.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/klog/v2/klog.go:1141 +0x11e
      created by k8s.io/klog/v2.(*flushDaemon).run
      	/remote-source/app/cmd/controller/vendor/k8s.io/klog/v2/klog.go:1137 +0x17b
      
      goroutine 56 [select]:
      github.com/cert-manager/cert-manager/internal/cmd/util.ContextWithStopCh.func1()
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/internal/cmd/util/context.go:30 +0x87
      created by github.com/cert-manager/cert-manager/internal/cmd/util.ContextWithStopCh
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/internal/cmd/util/context.go:28 +0xba
      
      goroutine 58 [select]:
      github.com/cert-manager/cert-manager/internal/cmd/util.ContextWithStopCh.func1()
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/internal/cmd/util/context.go:30 +0x87
      created by github.com/cert-manager/cert-manager/internal/cmd/util.ContextWithStopCh
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/internal/cmd/util/context.go:28 +0xba
      
      goroutine 59 [chan receive]:
      k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc00034dd10)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
      created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116
      
      goroutine 60 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 61 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 88 [chan receive]:
      github.com/cert-manager/cert-manager/controller-binary/app.Run.func1()
      	/remote-source/app/cmd/controller/app/controller.go:82 +0x4d
      golang.org/x/sync/errgroup.(*Group).Go.func1()
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
      created by golang.org/x/sync/errgroup.(*Group).Go
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
      
      goroutine 89 [IO wait]:
      internal/poll.runtime_pollWait(0x7f02281c18d8, 0x72)
      	/usr/lib/golang/src/runtime/netpoll.go:306 +0x89
      internal/poll.(*pollDesc).wait(0xc000308980?, 0xc000b04ca0?, 0x0)
      	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
      internal/poll.(*pollDesc).waitRead(...)
      	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
      internal/poll.(*FD).Accept(0xc000308980)
      	/usr/lib/golang/src/internal/poll/fd_unix.go:614 +0x2bd
      net.(*netFD).accept(0xc000308980)
      	/usr/lib/golang/src/net/fd_unix.go:172 +0x35
      net.(*TCPListener).accept(0xc000011d10)
      	/usr/lib/golang/src/net/tcpsock_posix.go:148 +0x25
      net.(*TCPListener).Accept(0xc000011d10)
      	/usr/lib/golang/src/net/tcpsock.go:297 +0x3d
      net/http.(*Server).Serve(0xc000a90000, {0x2c9fa70, 0xc000011d10})
      	/usr/lib/golang/src/net/http/server.go:3059 +0x385
      github.com/cert-manager/cert-manager/controller-binary/app.Run.func2()
      	/remote-source/app/cmd/controller/app/controller.go:94 +0x119
      golang.org/x/sync/errgroup.(*Group).Go.func1()
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
      created by golang.org/x/sync/errgroup.(*Group).Go
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
      
      goroutine 90 [semacquire]:
      sync.runtime_Semacquire(0xc0002f0030?)
      	/usr/lib/golang/src/runtime/sema.go:62 +0x27
      sync.(*WaitGroup).Wait(0x24750a0?)
      	/usr/lib/golang/src/sync/waitgroup.go:116 +0x4b
      golang.org/x/sync/errgroup.(*Group).Wait(0xc0004a4000)
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:53 +0x27
      github.com/cert-manager/cert-manager/pkg/healthz.(*Server).Start(0xc000a079f0, {0x2ca0d50?, 0xc0004a9450}, {0x2c9fa70?, 0xc000011fb0})
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/healthz/healthz.go:86 +0x12f
      github.com/cert-manager/cert-manager/controller-binary/app.Run.func5()
      	/remote-source/app/cmd/controller/app/controller.go:139 +0x138
      golang.org/x/sync/errgroup.(*Group).Go.func1()
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
      created by golang.org/x/sync/errgroup.(*Group).Go
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
      
      goroutine 46 [IO wait]:
      internal/poll.runtime_pollWait(0x7f02281c17e8, 0x72)
      	/usr/lib/golang/src/runtime/netpoll.go:306 +0x89
      internal/poll.(*pollDesc).wait(0xc000308a00?, 0xc000b0dcf8?, 0x0)
      	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
      internal/poll.(*pollDesc).waitRead(...)
      	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
      internal/poll.(*FD).Accept(0xc000308a00)
      	/usr/lib/golang/src/internal/poll/fd_unix.go:614 +0x2bd
      net.(*netFD).accept(0xc000308a00)
      	/usr/lib/golang/src/net/fd_unix.go:172 +0x35
      net.(*TCPListener).accept(0xc000011fb0)
      	/usr/lib/golang/src/net/tcpsock_posix.go:148 +0x25
      net.(*TCPListener).Accept(0xc000011fb0)
      	/usr/lib/golang/src/net/tcpsock.go:297 +0x3d
      net/http.(*Server).Serve(0xc000a900f0, {0x2c9fa70, 0xc000011fb0})
      	/usr/lib/golang/src/net/http/server.go:3059 +0x385
      github.com/cert-manager/cert-manager/pkg/healthz.(*Server).Start.func1()
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/healthz/healthz.go:74 +0x28
      golang.org/x/sync/errgroup.(*Group).Go.func1()
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
      created by golang.org/x/sync/errgroup.(*Group).Go
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
      
      goroutine 47 [chan receive]:
      github.com/cert-manager/cert-manager/pkg/healthz.(*Server).Start.func2()
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/healthz/healthz.go:80 +0x4d
      golang.org/x/sync/errgroup.(*Group).Go.func1()
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
      created by golang.org/x/sync/errgroup.(*Group).Go
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
      
      goroutine 92 [chan receive]:
      k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc0000ae960)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
      created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116
      
      goroutine 93 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 94 [runnable]:
      golang.org/x/net/http2.(*ClientConn).RoundTrip(0xc00017f500, 0xc0000c6b00)
      	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:1296 +0x491
      golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0009ce1b0, 0xc0000c6b00, {0xc0?})
      	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:564 +0x1c5
      golang.org/x/net/http2.(*Transport).RoundTrip(...)
      	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:513
      golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip({0xc0009e4000?}, 0xc0000c6b00?)
      	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:3136 +0x1b
      net/http.(*Transport).roundTrip(0xc0009e4000, 0xc0000c6a00)
      	/usr/lib/golang/src/net/http/transport.go:548 +0x3ca
      net/http.(*Transport).RoundTrip(0x250cba0?, 0xc0004513e0?)
      	/usr/lib/golang/src/net/http/roundtrip.go:17 +0x19
      k8s.io/client-go/transport.(*bearerAuthRoundTripper).RoundTrip(0xc0005431d0, 0xc0000c6900)
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/transport/round_trippers.go:317 +0x3e5
      k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip(0xc0003e1f80, 0xc0000c6800)
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/transport/round_trippers.go:168 +0x350
      net/http.send(0xc0000c6800, {0x2c7f960, 0xc0003e1f80}, {0x8?, 0x2687e80?, 0x0?})
      	/usr/lib/golang/src/net/http/client.go:252 +0x5f7
      net/http.(*Client).send(0xc000543200, 0xc0000c6800, {0x7f024f546108?, 0x30?, 0x0?})
      	/usr/lib/golang/src/net/http/client.go:176 +0x9b
      net/http.(*Client).do(0xc000543200, 0xc0000c6800)
      	/usr/lib/golang/src/net/http/client.go:716 +0x8fb
      net/http.(*Client).Do(...)
      	/usr/lib/golang/src/net/http/client.go:582
      k8s.io/client-go/rest.(*Request).request(0xc0000f90e0, {0x2ca0d88, 0xc000134008}, 0x1?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/rest/request.go:1023 +0x3a5
      k8s.io/client-go/rest.(*Request).Do(0xc0000f90e0, {0x2ca0d88, 0xc000134008})
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/rest/request.go:1063 +0xc9
      k8s.io/client-go/kubernetes/typed/core/v1.(*events).CreateWithEventNamespace(0xc000342ea0, 0xc0002ef400)
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/kubernetes/typed/core/v1/event_expansion.go:59 +0x10c
      k8s.io/client-go/kubernetes/typed/core/v1.(*EventSinkImpl).Create(0x23f71a0?, 0xc0003774b0?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/kubernetes/typed/core/v1/event_expansion.go:160 +0x22
      k8s.io/client-go/tools/record.recordEvent({0x2c9f1d0?, 0xc000a07e20?}, 0xc0002ef400, {0x0?, 0x0?, 0x0?}, 0x0?, 0xc0003d6138)
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:261 +0x16a
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).recordToSink(0xc000308d00, {0x2c9f1d0, 0xc000a07e20}, 0xc000b94000, 0x1?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:223 +0x16b
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartRecordingToSink.func1(0x0?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:200 +0x2f
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:327 +0xa2
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 77 [runnable]:
      golang.org/x/net/http2.(*clientStream).writeRequest(0xc00017f800, 0xc0000c6b00)
      	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:1464 +0xb27
      golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0x0?)
      	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:1326 +0x1e
      created by golang.org/x/net/http2.(*ClientConn).RoundTrip
      	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:1232 +0x34a
      
      goroutine 78 [select]:
      k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x1cd
      created by k8s.io/apimachinery/pkg/util/wait.poller.func1
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xca
      
      goroutine 75 [IO wait]:
      internal/poll.runtime_pollWait(0x7f02281c16f8, 0x72)
      	/usr/lib/golang/src/runtime/netpoll.go:306 +0x89
      internal/poll.(*pollDesc).wait(0xc000308f00?, 0xc0003a7000?, 0x0)
      	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
      internal/poll.(*pollDesc).waitRead(...)
      	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
      internal/poll.(*FD).Read(0xc000308f00, {0xc0003a7000, 0xc00, 0xc00})
      	/usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x299
      net.(*netFD).Read(0xc000308f00, {0xc0003a7000?, 0xc0003a704c?, 0x155?})
      	/usr/lib/golang/src/net/fd_posix.go:55 +0x29
      net.(*conn).Read(0xc0002fcc28, {0xc0003a7000?, 0xc000b2d878?, 0xc00042da30?})
      	/usr/lib/golang/src/net/net.go:183 +0x45
      crypto/tls.(*atLeastReader).Read(0xc0002f1800, {0xc0003a7000?, 0xc0002f1800?, 0x0?})
      	/usr/lib/golang/src/crypto/tls/conn.go:788 +0x3d
      bytes.(*Buffer).ReadFrom(0xc00042db10, {0x2c79820, 0xc0002f1800})
      	/usr/lib/golang/src/bytes/buffer.go:202 +0x98
      crypto/tls.(*Conn).readFromUntil(0xc00042d880, {0x2c7fb80?, 0xc0002fcc28}, 0xbb9?)
      	/usr/lib/golang/src/crypto/tls/conn.go:810 +0xe5
      crypto/tls.(*Conn).readRecordOrCCS(0xc00042d880, 0x0)
      	/usr/lib/golang/src/crypto/tls/conn.go:617 +0x116
      crypto/tls.(*Conn).readRecord(...)
      	/usr/lib/golang/src/crypto/tls/conn.go:583
      crypto/tls.(*Conn).Read(0xc00042d880, {0xc000b24000, 0x1000, 0x0?})
      	/usr/lib/golang/src/crypto/tls/conn.go:1316 +0x16f
      bufio.(*Reader).Read(0xc000b18e40, {0xc00018a660, 0x9, 0xc00017f888?})
      	/usr/lib/golang/src/bufio/bufio.go:237 +0x1bb
      io.ReadAtLeast({0x2c795c0, 0xc000b18e40}, {0xc00018a660, 0x9, 0x9}, 0x9)
      	/usr/lib/golang/src/io/io.go:332 +0x9a
      io.ReadFull(...)
      	/usr/lib/golang/src/io/io.go:351
      golang.org/x/net/http2.readFrameHeader({0xc00018a660?, 0x9?, 0xc000000000?}, {0x2c795c0?, 0xc000b18e40?})
      	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
      golang.org/x/net/http2.(*Framer).ReadFrame(0xc00018a620)
      	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/frame.go:498 +0x95
      golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000b2df98)
      	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:2275 +0x12e
      golang.org/x/net/http2.(*ClientConn).readLoop(0xc00017f500)
      	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:2170 +0x6f
      created by golang.org/x/net/http2.(*Transport).newClientConn
      	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:821 +0xc1f
      
      goroutine 25 [chan receive]:
      k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc000294370)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
      created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116
      
      goroutine 26 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 27 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 28 [select]:
      k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc00060c780)
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
      created by k8s.io/client-go/util/workqueue.newDelayingQueue
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x22a
      
      goroutine 29 [select]:
      k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x2ca1d48, 0xc000114de0}, 0xc0003de348, 0x124324a?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0xeb
      k8s.io/apimachinery/pkg/util/wait.poll({0x2ca1d48, 0xc000114de0}, 0x70?, 0x1243f85?, 0x7f024f546a68?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x9a
      k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x2ca1d48, 0xc000114de0}, 0x40e027?, 0x20?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x49
      k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x10?, 0x233fd00?, 0x40ca900?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x3f
      k8s.io/client-go/tools/cache.WaitForCacheSync(0x2ca5c00?, {0xc00060ca80, 0x7, 0xc})
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/cache/shared_informer.go:326 +0x88
      github.com/cert-manager/cert-manager/pkg/controller.(*controller).Run(0xc000c02480, 0x5, 0xc000114de0)
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/controller.go:104 +0xf9
      github.com/cert-manager/cert-manager/controller-binary/app.Run.func7()
      	/remote-source/app/cmd/controller/app/controller.go:217 +0xa5
      golang.org/x/sync/errgroup.(*Group).Go.func1()
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
      created by golang.org/x/sync/errgroup.(*Group).Go
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
      
      goroutine 114 [select]:
      k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x1cd
      created by k8s.io/apimachinery/pkg/util/wait.poller.func1
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xca
      
      goroutine 30 [chan receive]:
      k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc000295630)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
      created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116
      
      goroutine 31 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 32 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 129 [select]:
      k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc00060d200)
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
      created by k8s.io/client-go/util/workqueue.newDelayingQueue
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x22a
      
      goroutine 130 [select]:
      k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x2ca1d48, 0xc000114de0}, 0xc00005ad38, 0x124324a?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0xeb
      k8s.io/apimachinery/pkg/util/wait.poll({0x2ca1d48, 0xc000114de0}, 0x70?, 0x1243f85?, 0x7f024f546108?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x9a
      k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x2ca1d48, 0xc000114de0}, 0x40e027?, 0x20?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x49
      k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x10?, 0x233fd00?, 0x40ca900?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x3f
      k8s.io/client-go/tools/cache.WaitForCacheSync(0x2ca5c00?, {0xc0006d6150, 0x1, 0x1})
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/cache/shared_informer.go:326 +0x88
      github.com/cert-manager/cert-manager/pkg/controller.(*controller).Run(0xc000c50990, 0x5, 0xc000114de0)
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/controller.go:104 +0xf9
      github.com/cert-manager/cert-manager/controller-binary/app.Run.func7()
      	/remote-source/app/cmd/controller/app/controller.go:217 +0xa5
      golang.org/x/sync/errgroup.(*Group).Go.func1()
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
      created by golang.org/x/sync/errgroup.(*Group).Go
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
      
      goroutine 131 [chan receive]:
      k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc000295d10)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
      created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116
      
      goroutine 132 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 133 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 134 [select]:
      k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc00060d860)
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
      created by k8s.io/client-go/util/workqueue.newDelayingQueue
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x22a
      
      goroutine 135 [select]:
      k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x2ca1d48, 0xc000114de0}, 0xc00005ad50, 0x124324a?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0xeb
      k8s.io/apimachinery/pkg/util/wait.poll({0x2ca1d48, 0xc000114de0}, 0x70?, 0x1243f85?, 0x7f024f546108?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x9a
      k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x2ca1d48, 0xc000114de0}, 0x40e027?, 0x20?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x49
      k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x10?, 0x233fd00?, 0x40ca900?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x3f
      k8s.io/client-go/tools/cache.WaitForCacheSync(0x2ca5c00?, {0xc000cc2060, 0x4, 0x6})
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/cache/shared_informer.go:326 +0x88
      github.com/cert-manager/cert-manager/pkg/controller.(*controller).Run(0xc000c92fc0, 0x5, 0xc000114de0)
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/controller.go:104 +0xf9
      github.com/cert-manager/cert-manager/controller-binary/app.Run.func7()
      	/remote-source/app/cmd/controller/app/controller.go:217 +0xa5
      golang.org/x/sync/errgroup.(*Group).Go.func1()
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
      created by golang.org/x/sync/errgroup.(*Group).Go
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
      
      goroutine 79 [select]:
      k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x1cd
      created by k8s.io/apimachinery/pkg/util/wait.poller.func1
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xca
      
      goroutine 136 [chan receive]:
      k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc0002c8500)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
      created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116
      
      goroutine 137 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 138 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 139 [select]:
      k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000cfc000)
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
      created by k8s.io/client-go/util/workqueue.newDelayingQueue
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x22a
      
      goroutine 140 [select]:
      k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x2ca1d48, 0xc000114de0}, 0xc0003de360, 0x124324a?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0xeb
      k8s.io/apimachinery/pkg/util/wait.poll({0x2ca1d48, 0xc000114de0}, 0x70?, 0x1243f85?, 0x7f024f546a68?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x9a
      k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x2ca1d48, 0xc000114de0}, 0x40e027?, 0x20?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x49
      k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x10?, 0x233fd00?, 0x40ca900?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x3f
      k8s.io/client-go/tools/cache.WaitForCacheSync(0x2ca5c00?, {0xc0006d7198, 0x1, 0x1})
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/cache/shared_informer.go:326 +0x88
      github.com/cert-manager/cert-manager/pkg/controller.(*controller).Run(0xc000ce14d0, 0x5, 0xc000114de0)
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/controller.go:104 +0xf9
      github.com/cert-manager/cert-manager/controller-binary/app.Run.func7()
      	/remote-source/app/cmd/controller/app/controller.go:217 +0xa5
      golang.org/x/sync/errgroup.(*Group).Go.func1()
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
      created by golang.org/x/sync/errgroup.(*Group).Go
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
      
      goroutine 115 [select]:
      k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x1cd
      created by k8s.io/apimachinery/pkg/util/wait.poller.func1
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xca
      
      goroutine 141 [chan receive]:
      k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc0002c8c80)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
      created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116
      
      goroutine 142 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 143 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 144 [select]:
      k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000cfc660)
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
      created by k8s.io/client-go/util/workqueue.newDelayingQueue
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x22a
      
      goroutine 145 [select]:
      k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x2ca1d48, 0xc000114de0}, 0xc00005ad68, 0x124324a?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0xeb
      k8s.io/apimachinery/pkg/util/wait.poll({0x2ca1d48, 0xc000114de0}, 0x70?, 0x1243f85?, 0x7f024f546108?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x9a
      k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x2ca1d48, 0xc000114de0}, 0x40e027?, 0x20?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x49
      k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x10?, 0x233fd00?, 0x40ca900?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x3f
      k8s.io/client-go/tools/cache.WaitForCacheSync(0x2ca5c00?, {0xc000964b28, 0x3, 0x3})
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/cache/shared_informer.go:326 +0x88
      github.com/cert-manager/cert-manager/pkg/controller.(*controller).Run(0xc000d1fb90, 0x5, 0xc000114de0)
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/controller.go:104 +0xf9
      github.com/cert-manager/cert-manager/controller-binary/app.Run.func7()
      	/remote-source/app/cmd/controller/app/controller.go:217 +0xa5
      golang.org/x/sync/errgroup.(*Group).Go.func1()
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
      created by golang.org/x/sync/errgroup.(*Group).Go
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
      
      goroutine 80 [select]:
      k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x1cd
      created by k8s.io/apimachinery/pkg/util/wait.poller.func1
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xca
      
      goroutine 146 [chan receive]:
      k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc0002c9400)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
      created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116
      
      goroutine 147 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 148 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 149 [select]:
      k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000cfcd80)
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
      created by k8s.io/client-go/util/workqueue.newDelayingQueue
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x22a
      
      goroutine 150 [select]:
      k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x2ca1d48, 0xc000114de0}, 0xc00005ad80, 0x124324a?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0xeb
      k8s.io/apimachinery/pkg/util/wait.poll({0x2ca1d48, 0xc000114de0}, 0x70?, 0x1243f85?, 0x7f024f546108?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x9a
      k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x2ca1d48, 0xc000114de0}, 0x40e027?, 0x20?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x49
      k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x10?, 0x233fd00?, 0x40ca900?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x3f
      k8s.io/client-go/tools/cache.WaitForCacheSync(0x2ca5c00?, {0xc0004a1990, 0x2, 0x2})
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/cache/shared_informer.go:326 +0x88
      github.com/cert-manager/cert-manager/pkg/controller.(*controller).Run(0xc000d7c1b0, 0x5, 0xc000114de0)
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/controller.go:104 +0xf9
      github.com/cert-manager/cert-manager/controller-binary/app.Run.func7()
      	/remote-source/app/cmd/controller/app/controller.go:217 +0xa5
      golang.org/x/sync/errgroup.(*Group).Go.func1()
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
      created by golang.org/x/sync/errgroup.(*Group).Go
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
      
      goroutine 161 [select]:
      k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x1cd
      created by k8s.io/apimachinery/pkg/util/wait.poller.func1
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xca
      
      goroutine 151 [chan receive]:
      k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc0002c9b30)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
      created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116
      
      goroutine 152 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 153 [chan receive]:
      k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
      created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d
      
      goroutine 154 [select]:
      k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000cfd380)
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
      created by k8s.io/client-go/util/workqueue.newDelayingQueue
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x22a
      
      goroutine 155 [select]:
      k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x2ca1d48, 0xc000114de0}, 0xc00005ad98, 0x124324a?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0xeb
      k8s.io/apimachinery/pkg/util/wait.poll({0x2ca1d48, 0xc000114de0}, 0x70?, 0x1243f85?, 0x7f024f546108?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x9a
      k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x2ca1d48, 0xc000114de0}, 0x40e027?, 0x20?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x49
      k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x10?, 0x233fd00?, 0x40ca900?)
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x3f
      k8s.io/client-go/tools/cache.WaitForCacheSync(0x2ca5c00?, {0xc0003da060, 0x2, 0x2})
      	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/cache/shared_informer.go:326 +0x88
      github.com/cert-manager/cert-manager/pkg/controller.(*controller).Run(0xc000db6870, 0x5, 0xc000114de0)
      	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/controller.go:104 +0xf9
      github.com/cert-manager/cert-manager/controller-binary/app.Run.func7()
      	/remote-source/app/cmd/controller/app/controller.go:217 +0xa5
      golang.org/x/sync/errgroup.(*Group).Go.func1()
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
      created by golang.org/x/sync/errgroup.(*Group).Go
      	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5
      
      goroutine 162 [select]:
      k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x1cd
      created by k8s.io/apimachinery/pkg/util/wait.poller.func1
      	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xca
      

      Actual results:

      cert-manager logs show `fatal error: concurrent map read and map write`

      Expected results:

      No such fatal error

      Additional info:

          

              rh-ee-aagnihot Anirudh Agnihotri
              xxia-1 Xingxing Xia
              Yuedong Wu Yuedong Wu
              Subhashini T K Subhashini T K
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: