Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-9193

fatal error: concurrent map iteration and map write

    XMLWordPrintable

Details

    • Moderate
    • Unspecified
    • If docs needed, set a value

    Description

      Description of problem:

      fatal error: concurrent map iteration and map write

      Version-Release number of the following components:

      ClusterVersionOperator 4.10.0-202203242126.p0.gf0fb0fa.assembly.stream-f0fb0fa

      From a CI run for Hypershift, though I don't think there is anything Hypershift specific about this error

      How reproducible:
      Rare

      Steps to Reproduce:
      1. Unknown
      2.
      3.

      Actual results:

      fatal error: concurrent map iteration and map write

      goroutine 5622 [running]:
      runtime.throw(

      {0x1b54b74, 0x1a3a5c0}

      )
      /usr/lib/golang/src/runtime/panic.go:1198 +0x71 fp=0xc002c24850 sp=0xc002c24820 pc=0x43a611
      runtime.mapiternext(0xc002c24c50)
      /usr/lib/golang/src/runtime/map.go:858 +0x4eb fp=0xc002c248c0 sp=0xc002c24850 pc=0x41596b
      github.com/openshift/cluster-version-operator/pkg/cvo.(*operatorMetrics).Collect(0xc00013e910, 0xc0003a0ae0)
      /go/src/github.com/openshift/cluster-version-operator/pkg/cvo/metrics.go:493 +0x87f fp=0xc002c24f30 sp=0xc002c248c0 pc=0x16e9f5f
      github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
      /go/src/github.com/openshift/cluster-version-operator/vendor/github.com/prometheus/client_golang/prometheus/registry.go:446 +0x102 fp=0xc002c24fe0 sp=0xc002c24f30 pc=0x1209302
      runtime.goexit()
      /usr/lib/golang/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc002c24fe8 sp=0xc002c24fe0 pc=0x46aac1
      created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
      /go/src/github.com/openshift/cluster-version-operator/vendor/github.com/prometheus/client_golang/prometheus/registry.go:538 +0xb4d

      https://github.com/openshift/cluster-version-operator/blob/master/pkg/cvo/metrics.go#L493

      Looks like concurrent read/write on the conditions map between metrics and some other path.

      Attaching full log.

      Attachments

        Activity

          People

            trking W. Trevor King
            sjenning Seth Jennings
            Jia Liu Jia Liu
            Red Hat Employee
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: