Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-18277

Failed to install 4.19 spoke with "mkdirat: Read-only" error from node-image-pull.sh when fetching release image

XMLWordPrintable

    • False
    • None
    • False
    • None

      Description of problem:

      Attempt to install 4.19 nightly build on spoke using latest ACM 2.13 build via gitops workflow, installation failed with following error repeated in spoke journal log. 

      Seems to be related to this PR: https://github.com/openshift/assisted-installer/pull/899

      Feb 27 19:15:14 localhost.localdomain installer[5965]: time="2025-02-27T19:15:14Z" level=info msg="node-image-pull.service status: activating"
Feb 27 19:15:14 localhost.localdomain assisted-installer[5963]: time="2025-02-27T19:15:14Z" level=info msg="node-image-pull.service status: activating"
Feb 27 19:15:19 localhost.localdomain installer[5965]: time="2025-02-27T19:15:19Z" level=info msg="node-image-pull.service status: activating"
Feb 27 19:15:19 localhost.localdomain assisted-installer[5963]: time="2025-02-27T19:15:19Z" level=info msg="node-image-pull.service status: activating"
Feb 27 19:15:20 localhost.localdomain ironic-agent[3813]: 2025-02-27 19:15:20.438 1 DEBUG ironic_coreos_install [-] Assisted Agent is in state active (result success) _is_assisted_running /usr/lib/python3.9/site-packages/ironic_coreos_install.py:136
Feb 27 19:15:20 localhost.localdomain ironic-agent[3813]: 2025-02-27 19:15:20.438 1 DEBUG ironic_coreos_install [-] Still waiting for the assisted agent to finish start_assisted_install /usr/lib/python3.9/site-packages/ironic_coreos_install.py:157
Feb 27 19:15:20 localhost.localdomain podman[3794]: 2025-02-27 19:15:20.438 1 DEBUG ironic_coreos_install [-] Assisted Agent is in state active (result success) _is_assisted_running /usr/lib/python3.9/site-packages/ironic_coreos_install.py:136
Feb 27 19:15:20 localhost.localdomain podman[3794]: 2025-02-27 19:15:20.438 1 DEBUG ironic_coreos_install [-] Still waiting for the assisted agent to finish start_assisted_install /usr/lib/python3.9/site-packages/ironic_coreos_install.py:157
Feb 27 19:15:22 localhost.localdomain ostree-containe[12742]: Fetching ostree-unverified-registry:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:60b021d870d9243875776b5314c61a2dc7efd5f0b8f31633ba5a5755ed224096
Feb 27 19:15:22 localhost.localdomain node-image-pull.sh[12742]: layers already present: 8; layers needed: 43 (1.2 GB)
Feb 27 19:15:22 localhost.localdomain ostree-containe[12742]: layers already present: 8; layers needed: 43 (1.2 GB)
Feb 27 19:15:22 localhost.localdomain node-image-pull.sh[12742]: error: Importing: Unencapsulating base: Layer sha256:d06832d17d6e4203201a2aa56f0cf41e0d3db2f9b17fc241f77e65e0925342fc: mkdirat: Read-only file system
Feb 27 19:15:22 localhost.localdomain node-image-pull.sh[6447]: Failed to fetch release image; retrying...

       

      Version-Release number of selected component (if applicable):

      hub OCP: 4.19.0-ec.2

      ACM/MCE: 2.13.0-85/2.8.0-201 

      OS image: https://rhcos.mirror.openshift.com/art/storage/prod/streams/rhel-9.6/builds/9.6.20250121-0/x86_64/rhcos-9.6.20250121-0-live-iso.x86_64.iso

      Spoke OCP: 4.19.0-0.nightly-2025-02-26-050226

      How reproducible: 

      100%

      Steps to Reproduce:

      1.  Attempt to install 4.19 nightly on spoke using ACM 2.13/MCE 2.8
      2.  
      3. ...

      Actual results:

      • agent stuck at installing state.
      • repeated failed to fetch release image error from node-image-pull.sh on spoke journal log

      Expected results:

       installation succeeds

      Additional info:

              lgamliel liat gamliel
              rhn-support-yliu1 Yang Liu
              Vladislav Kolodny Vladislav Kolodny
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: