Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-9193

fatal error: concurrent map iteration and map write

XMLWordPrintable

    • Quality / Stability / Reliability
    • None
    • None
    • None
    • Moderate
    • None
    • Unspecified
    • None
    • None
    • None
    • None
    • None
    • If docs needed, set a value
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      fatal error: concurrent map iteration and map write

      Version-Release number of the following components:

      ClusterVersionOperator 4.10.0-202203242126.p0.gf0fb0fa.assembly.stream-f0fb0fa

      From a CI run for Hypershift, though I don't think there is anything Hypershift specific about this error

      How reproducible:
      Rare

      Steps to Reproduce:
      1. Unknown
      2.
      3.

      Actual results:

      fatal error: concurrent map iteration and map write

      goroutine 5622 [running]:
      runtime.throw(

      {0x1b54b74, 0x1a3a5c0}

      )
      /usr/lib/golang/src/runtime/panic.go:1198 +0x71 fp=0xc002c24850 sp=0xc002c24820 pc=0x43a611
      runtime.mapiternext(0xc002c24c50)
      /usr/lib/golang/src/runtime/map.go:858 +0x4eb fp=0xc002c248c0 sp=0xc002c24850 pc=0x41596b
      github.com/openshift/cluster-version-operator/pkg/cvo.(*operatorMetrics).Collect(0xc00013e910, 0xc0003a0ae0)
      /go/src/github.com/openshift/cluster-version-operator/pkg/cvo/metrics.go:493 +0x87f fp=0xc002c24f30 sp=0xc002c248c0 pc=0x16e9f5f
      github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
      /go/src/github.com/openshift/cluster-version-operator/vendor/github.com/prometheus/client_golang/prometheus/registry.go:446 +0x102 fp=0xc002c24fe0 sp=0xc002c24f30 pc=0x1209302
      runtime.goexit()
      /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc002c24fe8 sp=0xc002c24fe0 pc=0x46aac1
      created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
      /go/src/github.com/openshift/cluster-version-operator/vendor/github.com/prometheus/client_golang/prometheus/registry.go:538 +0xb4d

      https://github.com/openshift/cluster-version-operator/blob/master/pkg/cvo/metrics.go#L493

      Looks like concurrent read/write on the conditions map between metrics and some other path.

      Attaching full log.

              trking W. Trevor King
              sjenning Seth Jennings
              None
              None
              Jia Liu Jia Liu
              None
              Red Hat Employee
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: