Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-60166

GarbageCollection encountered "fatal error: concurrent map writes" [4.19]

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • 4.18
    • Node / Kubelet
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • None
    • None
    • None
    • None
    • In Progress
    • Release Note Not Required
    • None
    • None
    • None
    • None
    • None

      Nodes in the cluster started indicating DiskPressure.
      Draining/rebooting the nodes did not help.
      Following cri-o storage clean up procedure worked.

      The sosreport indicates that when the nodes started having DiskPressure GarbageCollection started but encountered a fatal error.
      ~~~
      May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: I0524 08:08:22.152327 2695232 image_gc_manager.go:383] "Disk usage on image filesystem is over the high threshold, trying to free bytes down to the low threshold" usage=88 highThreshold=85 amountToFree=19975571865 lowThreshold=80
      May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: I0524 08:08:22.152405 2695232 image_gc_manager.go:487] "Removing image to free bytes" imageID="bd2fb45ae0cb2e8c4001f84c513f907f7960cdac97e01fdb7214688e4288efa2" size=826271787 runtimeHandler=""
      May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: I0524 08:08:22.498929 2695232 image_gc_manager.go:487] "Removing image to free bytes" imageID="76f6b69fced95c010be36bffeeb722ce144b905c267c99248f79c9defa65a962" size=227488986 runtimeHandler=""
      May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: E0524 08:08:22.499604 2695232 log.go:32] "RemoveImage from image service failed" err="rpc error: code = Unknown desc = delete image: image used by e4c5e303adf168384679d6c1d967349d8086ff2ffbf4bf21a3e60b5cbac5ceda: image is in use by a container" image="bd2fb45ae0cb2e8c4001f84c513f907f7960cdac97e01fdb7214688e4288efa2"
      May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: E0524 08:08:22.499668 2695232 kuberuntime_image.go:150] "Failed to remove image" err="rpc error: code = Unknown desc = delete image: image used by e4c5e303adf168384679d6c1d967349d8086ff2ffbf4bf21a3e60b5cbac5ceda: image is in use by a container" image="bd2fb45ae0cb2e8c4001f84c513f907f7960cdac97e01fdb7214688e4288efa2"
      May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: I0524 08:08:22.499703 2695232 image_gc_manager.go:487] "Removing image to free bytes" imageID="76f6b69fced95c010be36bffeeb722ce144b905c267c99248f79c9defa65a962" size=227488986 runtimeHandler=""
      May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: fatal error: concurrent map writes
      May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: fatal error: concurrent map writes
      May 24 08:08:22 worker14.wdp-perf-scale.cp.fyre.ibm.com kubenswrapper[2695232]: goroutine 262 [running]:
      ....
      ~~~

              skunkerk Sohan Kunkerkar
              rhn-support-dseals Daniel Seals
              None
              None
              Min Li Min Li
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: