Loading...

XML

Word

Printable

Type: Bug
Resolution: Unresolved
Priority: Undefined
Fix Version/s: None
Affects Version/s: 4.14
Component/s: cert-manager
Labels:
None

Severity:
Moderate
Regression:
No
Blocked:
False
Blocked Reason:

Hide

None

Show
None

SFDC Cases Counter:
SFDC Cases Links:

Description of problem:

cert-manager logs show `fatal error: concurrent map read and map write`

Version-Release number of selected component (if applicable):

OCP 4.14.1
cert-manager-operator-bundle:v1.13.0-3

How reproducible:

Not sure. Hit once

Steps to Reproduce:

1. Install cert-manager from console
2. Check pods:
$ oc get po -n cert-manager
NAME                                       READY   STATUS    RESTARTS        AGE
cert-manager-b94b579f-m6j9v                1/1     Running   1 (5m28s ago)   6m20s
cert-manager-cainjector-849684c4cd-ldqrw   1/1     Running   0               6m57s
cert-manager-webhook-7d946df88f-4lscp      1/1     Running   0               6m59s
3. oc logs --previous cert-manager-b94b579f-m6j9v -n cert-manager > logs/cert-manager-b94b579f-m6j9v_previous.log
I1206 13:52:20.156106       1 controller.go:251] "cert-manager/controller/build-context: configured acme dns01 nameservers" nameservers=["172.30.0.10:53"]
W1206 13:52:20.156232       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1206 13:52:20.157341       1 controller.go:72] "cert-manager/controller: enabled controllers: [certificaterequests-approver certificaterequests-issuer-acme certificaterequests-issuer-ca certificaterequests-issuer-selfsigned certificaterequests-issuer-vault certificaterequests-issuer-venafi certificates-issuing certificates-key-manager certificates-metrics certificates-readiness certificates-request-manager certificates-revision-manager certificates-trigger challenges clusterissuers ingress-shim issuers orders]"
I1206 13:52:20.157954       1 controller.go:145] "cert-manager/controller: starting leader election"
I1206 13:52:20.157980       1 controller.go:93] "cert-manager/controller: starting metrics server" address="[::]:9402"
I1206 13:52:20.158088       1 controller.go:138] "cert-manager/controller: starting healthz server" address="[::]:9403"
I1206 13:52:20.158571       1 leaderelection.go:250] attempting to acquire leader lease kube-system/cert-manager-controller...
I1206 13:52:20.177614       1 leaderelection.go:260] successfully acquired lease kube-system/cert-manager-controller
I1206 13:52:20.178751       1 controller.go:215] "cert-manager/controller: starting controller" controller="challenges"
I1206 13:52:20.179185       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificaterequests-approver"
I1206 13:52:20.179380       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-ca"
I1206 13:52:20.179636       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-metrics"
I1206 13:52:20.179909       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-readiness"
I1206 13:52:20.180172       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-revision-manager"
I1206 13:52:20.180277       1 controller.go:192] "cert-manager/controller: not starting controller as it's disabled" controller="gateway-shim"
I1206 13:52:20.180389       1 controller.go:215] "cert-manager/controller: starting controller" controller="issuers"
fatal error: concurrent map read and map write

goroutine 91 [running]:
k8s.io/apimachinery/pkg/runtime.(*Scheme).ObjectKinds(0xc0001c59d0, {0x2c883f0?, 0xc000b92140})
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:267 +0x1ca
k8s.io/apimachinery/pkg/runtime.WithVersionEncoder.Encode({{0x2c8ed40, 0xc000342540}, {0x2c8a470, 0xc00013ce10}, {0x2c8a3d0, 0xc0001c59d0}}, {0x2c883f0, 0xc000b92140}, {0x2c79640, 0xc000ad16e0})
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/runtime/helper.go:223 +0x82
k8s.io/apimachinery/pkg/runtime.Encode({0x2c8ecc8, 0xc000ad16b0}, {0x2c883f0, 0xc000b92140})
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/runtime/codec.go:49 +0x64
k8s.io/client-go/rest.(*Request).Body(0xc0009b7c20, {0x26cef20?, 0xc000b92140?})
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/rest/request.go:470 +0x48b
k8s.io/client-go/kubernetes/typed/coordination/v1.(*leases).Update(0xc0003433e0, {0x2ca0dc0, 0xc00045a690}, 0xc000b92140, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, 0x0, ...}, ...})
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/kubernetes/typed/coordination/v1/lease.go:136 +0x15f
k8s.io/client-go/tools/leaderelection/resourcelock.(*LeaseLock).Update(0xc0009b6a20, {0x2ca0dc0, 0xc00045a690}, {{0xc0009e27c0, 0x3c}, 0x3c, {{0x973b938, 0xedd027394, 0x409a240}}, {{0xc1543d050a96cc89, ...}}, ...})
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/leaderelection/resourcelock/leaselock.go:75 +0x28b
k8s.io/client-go/tools/leaderelection.(*LeaderElector).tryAcquireOrRenew(0xc0009b6b40, {0x2ca0dc0, 0xc00045a690})
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:368 +0x42f
k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew.func1.1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:275 +0x25
k8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1({0x7f024f546a68, 0x18})
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109 +0x1b
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x2ca1d48?, 0xc00042e360?}, 0xc000adfa30?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:154 +0x57
k8s.io/apimachinery/pkg/util/wait.poll({0x2ca1d48, 0xc00042e360}, 0x70?, 0x1243f85?, 0x7f024f546a68?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:245 +0x38
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x2ca1d48, 0xc00042e360}, 0x40e027?, 0x20?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x49
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x2ca0d50?, 0xc0006de190?, 0x40dc4a?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x3f
k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:274 +0x10d
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00042e300?, {0x2c7f1e0, 0xc00045a660}, 0x1, 0xc00042e300)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0006de190?, 0x37e11d600, 0x0, 0x20?, 0x20?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/leaderelection.(*LeaderElector).renew(0xc0009b6b40, {0x2ca0d50?, 0xc0006de140?})
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:271 +0x125
k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc0009b6b40, {0x2ca0d50, 0xc0004a9450})
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:214 +0x119
github.com/cert-manager/cert-manager/controller-binary/app.startLeaderElection({0x2ca0d50, 0xc0004a9450}, 0xc0009ad340, {0x2cc26e8, 0xc000503380}, {0x2c9f6e0?, 0xc000061240}, {0xc000a07e50, 0xc000342f40, 0x0}, ...)
	/remote-source/app/cmd/controller/app/controller.go:373 +0x328
github.com/cert-manager/cert-manager/controller-binary/app.Run.func6()
	/remote-source/app/cmd/controller/app/controller.go:151 +0x20c
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5

goroutine 1 [runnable]:
type:.eq.k8s.io/apimachinery/pkg/runtime/schema.GroupVersionKind(0xc00087f260?, 0xc00087f308?)
	<autogenerated>:1 +0xac
k8s.io/apimachinery/pkg/runtime.(*Scheme).AddKnownTypeWithName(0xc0001c59d0, {{0x272e6d2, 0x19}, {0x270850b, 0x7}, {0x2076db2, 0xd}}, {0x2c8a150?, 0x4099d60})
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:177 +0x32a
k8s.io/apimachinery/pkg/runtime.(*Scheme).AddKnownTypes(0xc0001c59d0, {{0x272e6d2?, 0x19?}, {0x270850b?, 0xa?}}, {0x406f980?, 0x6, 0x2c8a1a0?})
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/runtime/scheme.go:148 +0x171
k8s.io/apimachinery/pkg/apis/meta/v1.AddToGroupVersion(0xc0001c59d0?, {{0x272e6d2?, 0x2c8da30?}, {0x270850b?, 0x2c8da58?}})
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/apis/meta/v1/register.go:73 +0x20c
sigs.k8s.io/gateway-api/apis/v1beta1.addKnownTypes(0x2705a10?)
	/remote-source/app/cmd/controller/vendor/sigs.k8s.io/gateway-api/apis/v1beta1/zz_generated.register.go:71 +0x20b
k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme(...)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/runtime/scheme_builder.go:29
k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme(...)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/runtime/scheme_builder.go:29
github.com/cert-manager/cert-manager/pkg/controller.(*ContextFactory).Build(0xc0009e1800, {0xc00087f8d8?, 0x0?, 0xc00087f8d0?})
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/context.go:334 +0x122
github.com/cert-manager/cert-manager/pkg/controller.(*Builder).Complete(0xc00087f908)
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/builder.go:83 +0x6a
github.com/cert-manager/cert-manager/pkg/controller/certificate-shim/ingresses.init.0.func1(0xc0004a92c0?)
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/certificate-shim/ingresses/controller.go:150 +0x8c
github.com/cert-manager/cert-manager/controller-binary/app.Run(0xc0009ad340, 0xc0009ad468?)
	/remote-source/app/cmd/controller/app/controller.go:202 +0xe2a
github.com/cert-manager/cert-manager/controller-binary/app.NewServerCommand.func1({0x2ca0df8?, 0xc0009a8030?}, 0x0?)
	/remote-source/app/cmd/controller/app/start.go:59 +0x39
github.com/cert-manager/cert-manager/controller-binary/app.newServerCommand.func1(0xc000004600?, {0x2703631?, 0x5?, 0x5?})
	/remote-source/app/cmd/controller/app/start.go:106 +0x1f8
github.com/spf13/cobra.(*Command).execute(0xc000004600, {0xc000130130, 0x5, 0x5})
	/remote-source/app/cmd/controller/vendor/github.com/spf13/cobra/command.go:940 +0x862
github.com/spf13/cobra.(*Command).ExecuteC(0xc000004600)
	/remote-source/app/cmd/controller/vendor/github.com/spf13/cobra/command.go:1068 +0x3bd
github.com/spf13/cobra.(*Command).Execute(0xc000280700?)
	/remote-source/app/cmd/controller/vendor/github.com/spf13/cobra/command.go:992 +0x19
main.main()
	/remote-source/app/cmd/controller/main.go:38 +0x8f

goroutine 24 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0000caa00)
	/remote-source/app/cmd/controller/vendor/go.opencensus.io/stats/view/worker.go:292 +0xad
created by go.opencensus.io/stats/view.init.0
	/remote-source/app/cmd/controller/vendor/go.opencensus.io/stats/view/worker.go:34 +0x96

goroutine 53 [syscall]:
os/signal.signal_recv()
	/usr/lib/golang/src/runtime/sigqueue.go:152 +0x2f
os/signal.loop()
	/usr/lib/golang/src/os/signal/signal_unix.go:23 +0x19
created by os/signal.Notify.func1.1
	/usr/lib/golang/src/os/signal/signal.go:151 +0x2a

goroutine 54 [chan receive]:
github.com/cert-manager/cert-manager/internal/cmd/util.SetupExitHandler.func1()
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/internal/cmd/util/signal.go:58 +0x45
created by github.com/cert-manager/cert-manager/internal/cmd/util.SetupExitHandler
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/internal/cmd/util/signal.go:56 +0xd7

goroutine 57 [select]:
k8s.io/klog/v2.(*flushDaemon).run.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/klog/v2/klog.go:1141 +0x11e
created by k8s.io/klog/v2.(*flushDaemon).run
	/remote-source/app/cmd/controller/vendor/k8s.io/klog/v2/klog.go:1137 +0x17b

goroutine 56 [select]:
github.com/cert-manager/cert-manager/internal/cmd/util.ContextWithStopCh.func1()
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/internal/cmd/util/context.go:30 +0x87
created by github.com/cert-manager/cert-manager/internal/cmd/util.ContextWithStopCh
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/internal/cmd/util/context.go:28 +0xba

goroutine 58 [select]:
github.com/cert-manager/cert-manager/internal/cmd/util.ContextWithStopCh.func1()
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/internal/cmd/util/context.go:30 +0x87
created by github.com/cert-manager/cert-manager/internal/cmd/util.ContextWithStopCh
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/internal/cmd/util/context.go:28 +0xba

goroutine 59 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc00034dd10)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116

goroutine 60 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 61 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 88 [chan receive]:
github.com/cert-manager/cert-manager/controller-binary/app.Run.func1()
	/remote-source/app/cmd/controller/app/controller.go:82 +0x4d
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5

goroutine 89 [IO wait]:
internal/poll.runtime_pollWait(0x7f02281c18d8, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc000308980?, 0xc000b04ca0?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000308980)
	/usr/lib/golang/src/internal/poll/fd_unix.go:614 +0x2bd
net.(*netFD).accept(0xc000308980)
	/usr/lib/golang/src/net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc000011d10)
	/usr/lib/golang/src/net/tcpsock_posix.go:148 +0x25
net.(*TCPListener).Accept(0xc000011d10)
	/usr/lib/golang/src/net/tcpsock.go:297 +0x3d
net/http.(*Server).Serve(0xc000a90000, {0x2c9fa70, 0xc000011d10})
	/usr/lib/golang/src/net/http/server.go:3059 +0x385
github.com/cert-manager/cert-manager/controller-binary/app.Run.func2()
	/remote-source/app/cmd/controller/app/controller.go:94 +0x119
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5

goroutine 90 [semacquire]:
sync.runtime_Semacquire(0xc0002f0030?)
	/usr/lib/golang/src/runtime/sema.go:62 +0x27
sync.(*WaitGroup).Wait(0x24750a0?)
	/usr/lib/golang/src/sync/waitgroup.go:116 +0x4b
golang.org/x/sync/errgroup.(*Group).Wait(0xc0004a4000)
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:53 +0x27
github.com/cert-manager/cert-manager/pkg/healthz.(*Server).Start(0xc000a079f0, {0x2ca0d50?, 0xc0004a9450}, {0x2c9fa70?, 0xc000011fb0})
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/healthz/healthz.go:86 +0x12f
github.com/cert-manager/cert-manager/controller-binary/app.Run.func5()
	/remote-source/app/cmd/controller/app/controller.go:139 +0x138
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5

goroutine 46 [IO wait]:
internal/poll.runtime_pollWait(0x7f02281c17e8, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc000308a00?, 0xc000b0dcf8?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000308a00)
	/usr/lib/golang/src/internal/poll/fd_unix.go:614 +0x2bd
net.(*netFD).accept(0xc000308a00)
	/usr/lib/golang/src/net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc000011fb0)
	/usr/lib/golang/src/net/tcpsock_posix.go:148 +0x25
net.(*TCPListener).Accept(0xc000011fb0)
	/usr/lib/golang/src/net/tcpsock.go:297 +0x3d
net/http.(*Server).Serve(0xc000a900f0, {0x2c9fa70, 0xc000011fb0})
	/usr/lib/golang/src/net/http/server.go:3059 +0x385
github.com/cert-manager/cert-manager/pkg/healthz.(*Server).Start.func1()
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/healthz/healthz.go:74 +0x28
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5

goroutine 47 [chan receive]:
github.com/cert-manager/cert-manager/pkg/healthz.(*Server).Start.func2()
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/healthz/healthz.go:80 +0x4d
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5

goroutine 92 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc0000ae960)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116

goroutine 93 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 94 [runnable]:
golang.org/x/net/http2.(*ClientConn).RoundTrip(0xc00017f500, 0xc0000c6b00)
	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:1296 +0x491
golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0009ce1b0, 0xc0000c6b00, {0xc0?})
	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:564 +0x1c5
golang.org/x/net/http2.(*Transport).RoundTrip(...)
	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:513
golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip({0xc0009e4000?}, 0xc0000c6b00?)
	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:3136 +0x1b
net/http.(*Transport).roundTrip(0xc0009e4000, 0xc0000c6a00)
	/usr/lib/golang/src/net/http/transport.go:548 +0x3ca
net/http.(*Transport).RoundTrip(0x250cba0?, 0xc0004513e0?)
	/usr/lib/golang/src/net/http/roundtrip.go:17 +0x19
k8s.io/client-go/transport.(*bearerAuthRoundTripper).RoundTrip(0xc0005431d0, 0xc0000c6900)
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/transport/round_trippers.go:317 +0x3e5
k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip(0xc0003e1f80, 0xc0000c6800)
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/transport/round_trippers.go:168 +0x350
net/http.send(0xc0000c6800, {0x2c7f960, 0xc0003e1f80}, {0x8?, 0x2687e80?, 0x0?})
	/usr/lib/golang/src/net/http/client.go:252 +0x5f7
net/http.(*Client).send(0xc000543200, 0xc0000c6800, {0x7f024f546108?, 0x30?, 0x0?})
	/usr/lib/golang/src/net/http/client.go:176 +0x9b
net/http.(*Client).do(0xc000543200, 0xc0000c6800)
	/usr/lib/golang/src/net/http/client.go:716 +0x8fb
net/http.(*Client).Do(...)
	/usr/lib/golang/src/net/http/client.go:582
k8s.io/client-go/rest.(*Request).request(0xc0000f90e0, {0x2ca0d88, 0xc000134008}, 0x1?)
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/rest/request.go:1023 +0x3a5
k8s.io/client-go/rest.(*Request).Do(0xc0000f90e0, {0x2ca0d88, 0xc000134008})
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/rest/request.go:1063 +0xc9
k8s.io/client-go/kubernetes/typed/core/v1.(*events).CreateWithEventNamespace(0xc000342ea0, 0xc0002ef400)
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/kubernetes/typed/core/v1/event_expansion.go:59 +0x10c
k8s.io/client-go/kubernetes/typed/core/v1.(*EventSinkImpl).Create(0x23f71a0?, 0xc0003774b0?)
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/kubernetes/typed/core/v1/event_expansion.go:160 +0x22
k8s.io/client-go/tools/record.recordEvent({0x2c9f1d0?, 0xc000a07e20?}, 0xc0002ef400, {0x0?, 0x0?, 0x0?}, 0x0?, 0xc0003d6138)
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:261 +0x16a
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).recordToSink(0xc000308d00, {0x2c9f1d0, 0xc000a07e20}, 0xc000b94000, 0x1?)
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:223 +0x16b
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartRecordingToSink.func1(0x0?)
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:200 +0x2f
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:327 +0xa2
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 77 [runnable]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc00017f800, 0xc0000c6b00)
	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:1464 +0xb27
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0x0?)
	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:1326 +0x1e
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:1232 +0x34a

goroutine 78 [select]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x1cd
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xca

goroutine 75 [IO wait]:
internal/poll.runtime_pollWait(0x7f02281c16f8, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc000308f00?, 0xc0003a7000?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000308f00, {0xc0003a7000, 0xc00, 0xc00})
	/usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc000308f00, {0xc0003a7000?, 0xc0003a704c?, 0x155?})
	/usr/lib/golang/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc0002fcc28, {0xc0003a7000?, 0xc000b2d878?, 0xc00042da30?})
	/usr/lib/golang/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc0002f1800, {0xc0003a7000?, 0xc0002f1800?, 0x0?})
	/usr/lib/golang/src/crypto/tls/conn.go:788 +0x3d
bytes.(*Buffer).ReadFrom(0xc00042db10, {0x2c79820, 0xc0002f1800})
	/usr/lib/golang/src/bytes/buffer.go:202 +0x98
crypto/tls.(*Conn).readFromUntil(0xc00042d880, {0x2c7fb80?, 0xc0002fcc28}, 0xbb9?)
	/usr/lib/golang/src/crypto/tls/conn.go:810 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc00042d880, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:617 +0x116
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0xc00042d880, {0xc000b24000, 0x1000, 0x0?})
	/usr/lib/golang/src/crypto/tls/conn.go:1316 +0x16f
bufio.(*Reader).Read(0xc000b18e40, {0xc00018a660, 0x9, 0xc00017f888?})
	/usr/lib/golang/src/bufio/bufio.go:237 +0x1bb
io.ReadAtLeast({0x2c795c0, 0xc000b18e40}, {0xc00018a660, 0x9, 0x9}, 0x9)
	/usr/lib/golang/src/io/io.go:332 +0x9a
io.ReadFull(...)
	/usr/lib/golang/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc00018a660?, 0x9?, 0xc000000000?}, {0x2c795c0?, 0xc000b18e40?})
	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/frame.go:237 +0x6e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00018a620)
	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/frame.go:498 +0x95
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000b2df98)
	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:2275 +0x12e
golang.org/x/net/http2.(*ClientConn).readLoop(0xc00017f500)
	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:2170 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
	/remote-source/app/cmd/controller/vendor/golang.org/x/net/http2/transport.go:821 +0xc1f

goroutine 25 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc000294370)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116

goroutine 26 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 27 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 28 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc00060c780)
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
created by k8s.io/client-go/util/workqueue.newDelayingQueue
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x22a

goroutine 29 [select]:
k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x2ca1d48, 0xc000114de0}, 0xc0003de348, 0x124324a?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0xeb
k8s.io/apimachinery/pkg/util/wait.poll({0x2ca1d48, 0xc000114de0}, 0x70?, 0x1243f85?, 0x7f024f546a68?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x9a
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x2ca1d48, 0xc000114de0}, 0x40e027?, 0x20?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x49
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x10?, 0x233fd00?, 0x40ca900?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x3f
k8s.io/client-go/tools/cache.WaitForCacheSync(0x2ca5c00?, {0xc00060ca80, 0x7, 0xc})
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/cache/shared_informer.go:326 +0x88
github.com/cert-manager/cert-manager/pkg/controller.(*controller).Run(0xc000c02480, 0x5, 0xc000114de0)
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/controller.go:104 +0xf9
github.com/cert-manager/cert-manager/controller-binary/app.Run.func7()
	/remote-source/app/cmd/controller/app/controller.go:217 +0xa5
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5

goroutine 114 [select]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x1cd
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xca

goroutine 30 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc000295630)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116

goroutine 31 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 32 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 129 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc00060d200)
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
created by k8s.io/client-go/util/workqueue.newDelayingQueue
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x22a

goroutine 130 [select]:
k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x2ca1d48, 0xc000114de0}, 0xc00005ad38, 0x124324a?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0xeb
k8s.io/apimachinery/pkg/util/wait.poll({0x2ca1d48, 0xc000114de0}, 0x70?, 0x1243f85?, 0x7f024f546108?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x9a
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x2ca1d48, 0xc000114de0}, 0x40e027?, 0x20?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x49
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x10?, 0x233fd00?, 0x40ca900?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x3f
k8s.io/client-go/tools/cache.WaitForCacheSync(0x2ca5c00?, {0xc0006d6150, 0x1, 0x1})
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/cache/shared_informer.go:326 +0x88
github.com/cert-manager/cert-manager/pkg/controller.(*controller).Run(0xc000c50990, 0x5, 0xc000114de0)
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/controller.go:104 +0xf9
github.com/cert-manager/cert-manager/controller-binary/app.Run.func7()
	/remote-source/app/cmd/controller/app/controller.go:217 +0xa5
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5

goroutine 131 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc000295d10)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116

goroutine 132 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 133 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 134 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc00060d860)
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
created by k8s.io/client-go/util/workqueue.newDelayingQueue
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x22a

goroutine 135 [select]:
k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x2ca1d48, 0xc000114de0}, 0xc00005ad50, 0x124324a?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0xeb
k8s.io/apimachinery/pkg/util/wait.poll({0x2ca1d48, 0xc000114de0}, 0x70?, 0x1243f85?, 0x7f024f546108?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x9a
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x2ca1d48, 0xc000114de0}, 0x40e027?, 0x20?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x49
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x10?, 0x233fd00?, 0x40ca900?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x3f
k8s.io/client-go/tools/cache.WaitForCacheSync(0x2ca5c00?, {0xc000cc2060, 0x4, 0x6})
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/cache/shared_informer.go:326 +0x88
github.com/cert-manager/cert-manager/pkg/controller.(*controller).Run(0xc000c92fc0, 0x5, 0xc000114de0)
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/controller.go:104 +0xf9
github.com/cert-manager/cert-manager/controller-binary/app.Run.func7()
	/remote-source/app/cmd/controller/app/controller.go:217 +0xa5
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5

goroutine 79 [select]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x1cd
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xca

goroutine 136 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc0002c8500)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116

goroutine 137 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 138 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 139 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000cfc000)
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
created by k8s.io/client-go/util/workqueue.newDelayingQueue
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x22a

goroutine 140 [select]:
k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x2ca1d48, 0xc000114de0}, 0xc0003de360, 0x124324a?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0xeb
k8s.io/apimachinery/pkg/util/wait.poll({0x2ca1d48, 0xc000114de0}, 0x70?, 0x1243f85?, 0x7f024f546a68?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x9a
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x2ca1d48, 0xc000114de0}, 0x40e027?, 0x20?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x49
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x10?, 0x233fd00?, 0x40ca900?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x3f
k8s.io/client-go/tools/cache.WaitForCacheSync(0x2ca5c00?, {0xc0006d7198, 0x1, 0x1})
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/cache/shared_informer.go:326 +0x88
github.com/cert-manager/cert-manager/pkg/controller.(*controller).Run(0xc000ce14d0, 0x5, 0xc000114de0)
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/controller.go:104 +0xf9
github.com/cert-manager/cert-manager/controller-binary/app.Run.func7()
	/remote-source/app/cmd/controller/app/controller.go:217 +0xa5
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5

goroutine 115 [select]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x1cd
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xca

goroutine 141 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc0002c8c80)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116

goroutine 142 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 143 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 144 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000cfc660)
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
created by k8s.io/client-go/util/workqueue.newDelayingQueue
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x22a

goroutine 145 [select]:
k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x2ca1d48, 0xc000114de0}, 0xc00005ad68, 0x124324a?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0xeb
k8s.io/apimachinery/pkg/util/wait.poll({0x2ca1d48, 0xc000114de0}, 0x70?, 0x1243f85?, 0x7f024f546108?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x9a
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x2ca1d48, 0xc000114de0}, 0x40e027?, 0x20?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x49
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x10?, 0x233fd00?, 0x40ca900?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x3f
k8s.io/client-go/tools/cache.WaitForCacheSync(0x2ca5c00?, {0xc000964b28, 0x3, 0x3})
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/cache/shared_informer.go:326 +0x88
github.com/cert-manager/cert-manager/pkg/controller.(*controller).Run(0xc000d1fb90, 0x5, 0xc000114de0)
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/controller.go:104 +0xf9
github.com/cert-manager/cert-manager/controller-binary/app.Run.func7()
	/remote-source/app/cmd/controller/app/controller.go:217 +0xa5
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5

goroutine 80 [select]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x1cd
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xca

goroutine 146 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc0002c9400)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116

goroutine 147 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 148 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 149 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000cfcd80)
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
created by k8s.io/client-go/util/workqueue.newDelayingQueue
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x22a

goroutine 150 [select]:
k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x2ca1d48, 0xc000114de0}, 0xc00005ad80, 0x124324a?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0xeb
k8s.io/apimachinery/pkg/util/wait.poll({0x2ca1d48, 0xc000114de0}, 0x70?, 0x1243f85?, 0x7f024f546108?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x9a
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x2ca1d48, 0xc000114de0}, 0x40e027?, 0x20?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x49
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x10?, 0x233fd00?, 0x40ca900?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x3f
k8s.io/client-go/tools/cache.WaitForCacheSync(0x2ca5c00?, {0xc0004a1990, 0x2, 0x2})
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/cache/shared_informer.go:326 +0x88
github.com/cert-manager/cert-manager/pkg/controller.(*controller).Run(0xc000d7c1b0, 0x5, 0xc000114de0)
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/controller.go:104 +0xf9
github.com/cert-manager/cert-manager/controller-binary/app.Run.func7()
	/remote-source/app/cmd/controller/app/controller.go:217 +0xa5
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5

goroutine 161 [select]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x1cd
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xca

goroutine 151 [chan receive]:
k8s.io/apimachinery/pkg/watch.(*Broadcaster).loop(0xc0002c9b30)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:268 +0x65
created by k8s.io/apimachinery/pkg/watch.NewLongQueueBroadcaster
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/watch/mux.go:93 +0x116

goroutine 152 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 153 [chan receive]:
k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1()
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:320 +0x73
created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/record/event.go:318 +0x13d

goroutine 154 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000cfd380)
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
created by k8s.io/client-go/util/workqueue.newDelayingQueue
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x22a

goroutine 155 [select]:
k8s.io/apimachinery/pkg/util/wait.waitForWithContext({0x2ca1d48, 0xc000114de0}, 0xc00005ad98, 0x124324a?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:205 +0xeb
k8s.io/apimachinery/pkg/util/wait.poll({0x2ca1d48, 0xc000114de0}, 0x70?, 0x1243f85?, 0x7f024f546108?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:260 +0x9a
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext({0x2ca1d48, 0xc000114de0}, 0x40e027?, 0x20?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:200 +0x49
k8s.io/apimachinery/pkg/util/wait.PollImmediateUntil(0x10?, 0x233fd00?, 0x40ca900?)
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:187 +0x3f
k8s.io/client-go/tools/cache.WaitForCacheSync(0x2ca5c00?, {0xc0003da060, 0x2, 0x2})
	/remote-source/app/cmd/controller/vendor/k8s.io/client-go/tools/cache/shared_informer.go:326 +0x88
github.com/cert-manager/cert-manager/pkg/controller.(*controller).Run(0xc000db6870, 0x5, 0xc000114de0)
	/remote-source/app/cmd/controller/vendor/github.com/cert-manager/cert-manager/pkg/controller/controller.go:104 +0xf9
github.com/cert-manager/cert-manager/controller-binary/app.Run.func7()
	/remote-source/app/cmd/controller/app/controller.go:217 +0xa5
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/remote-source/app/cmd/controller/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa5

goroutine 162 [select]:
k8s.io/apimachinery/pkg/util/wait.poller.func1.1()
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:297 +0x1cd
created by k8s.io/apimachinery/pkg/util/wait.poller.func1
	/remote-source/app/cmd/controller/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:280 +0xca

Actual results:

cert-manager logs show `fatal error: concurrent map read and map write`

Expected results:

No such fatal error

Additional info:

is related to

OCPBUGS-30068 "fatal error: concurrent map read and map write" on Scheme typeToGVK

OCPBUGS-30146 fatal error: concurrent map read and map write between Scheme New() and AddKnownTypeWithName()

Verified

Assignee:: Trilok Geer

Reporter:: Xingxing Xia

QA Contact:: Xingxing Xia

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Created:: 2023/12/06 2:00 PM

Updated:: 2024/03/01 9:03 PM

Details

Description

Attachments

Issue Links

Activity

People

Dates