Uploaded image for project: 'OpenShift Node'
  1. OpenShift Node
  2. OCPNODE-2080

Unable to build images in aarch64 nodes

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Normal Normal
    • None
    • None
    • None
    • None
    • False
    • None
    • False

      Summary

      We have a multi-arch cluster. This cluster have both amd64 nodes (m6a.4xlg) and aarch64 nodes (m6g.4xlg).

      We noticed, for Builds, if scheduled on amd64 nodes, are completed successfully. If scheduled on aarch64 nodes, are mostly failed. This is independent to the specific node the pod is scheduled to (i.e., we tried multiple aarch64 nodes).

      The error messages look like:

      error: mkdir /var/lib/containers/storage/overlay/metacopy-check1398259516/l2: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: changing permissions on file for metacopy check: chmod /var/lib/containers/storage/overlay/metacopy-check271055033/merged/f: no space left on device
error: changing permissions on file for metacopy check: chmod /var/lib/containers/storage/overlay/metacopy-check3798390120/merged/f: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: changing permissions on file for metacopy check: chmod /var/lib/containers/storage/overlay/metacopy-check2121835199/merged/f: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device
error: recording metacopy-being-used status: open /var/run/containers/storage/overlay/metacopy()-false: no space left on device

      Goal

      Identify the root cause and find solutions to the issue.

      Ref: https://redhat-internal.slack.com/archives/CK1AE4ZCK/p1708356005736639

              rh-ee-kehannon Kevin Hannon
              bechen@redhat.com Bear Chen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: