Details
-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
4.10
-
Moderate
-
Unspecified
-
If docs needed, set a value
Description
Description of problem:
fatal error: concurrent map iteration and map write
Version-Release number of the following components:
ClusterVersionOperator 4.10.0-202203242126.p0.gf0fb0fa.assembly.stream-f0fb0fa
From a CI run for Hypershift, though I don't think there is anything Hypershift specific about this error
How reproducible:
Rare
Steps to Reproduce:
1. Unknown
2.
3.
Actual results:
fatal error: concurrent map iteration and map write
goroutine 5622 [running]:
runtime.throw(
)
/usr/lib/golang/src/runtime/panic.go:1198 +0x71 fp=0xc002c24850 sp=0xc002c24820 pc=0x43a611
runtime.mapiternext(0xc002c24c50)
/usr/lib/golang/src/runtime/map.go:858 +0x4eb fp=0xc002c248c0 sp=0xc002c24850 pc=0x41596b
github.com/openshift/cluster-version-operator/pkg/cvo.(*operatorMetrics).Collect(0xc00013e910, 0xc0003a0ae0)
/go/src/github.com/openshift/cluster-version-operator/pkg/cvo/metrics.go:493 +0x87f fp=0xc002c24f30 sp=0xc002c248c0 pc=0x16e9f5f
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
/go/src/github.com/openshift/cluster-version-operator/vendor/github.com/prometheus/client_golang/prometheus/registry.go:446 +0x102 fp=0xc002c24fe0 sp=0xc002c24f30 pc=0x1209302
runtime.goexit()
/usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc002c24fe8 sp=0xc002c24fe0 pc=0x46aac1
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
/go/src/github.com/openshift/cluster-version-operator/vendor/github.com/prometheus/client_golang/prometheus/registry.go:538 +0xb4d
https://github.com/openshift/cluster-version-operator/blob/master/pkg/cvo/metrics.go#L493
Looks like concurrent read/write on the conditions map between metrics and some other path.
Attaching full log.