Uploaded image for project: 'CoreOS OCP'
  1. CoreOS OCP
  2. COS-3915 Pipeline monitoring w.s 23022026
  3. COS-3966

build #3463 - rhel-9.6 x86_64 - kola test ext.config.shared.multipath.custom-partition failed

XMLWordPrintable

    • Icon: Sub-task Sub-task
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • CoreOS East - Sprint 284

      Build Details

      Field Value
      Job build
      Build Number #3463
      Stream rhel-9.6
      Arch x86_64
      Build ID 9.6.20260225-0
      Timestamp 2026-02-25T10:30:15
      Jenkins URL build/3463

      Root Cause Analysis

      ROOT CAUSE IDENTIFIED: Ignition package regression

      The build failed because the RHAOS 4.19 Ignition package was picked up instead of the RHEL 9.6 version. The RHAOS 4.19 build is missing a critical fix for device mapper partitioning.

      Package Diff Between Working and Failing Builds

      Package Build 3399 (SUCCESS - Feb 20) Build 3463 (FAILURE - Feb 25)
      ignition 2.21.0-6.el9_6 2.21.0-6.rhaos4.19.el9
      ignition-grub 2.21.0-6.el9_6 2.21.0-6.rhaos4.19.el9
      kernel 5.14.0-570.93.1.el9_6 5.14.0-570.94.1.el9_6

      Ignition Build Details

      Build Source Commit Built Date Key Difference
      ignition-2.21.0-6.el9_6 (Working) d2f4405253ba Nov 18, 2025 Includes backport fix for device mapper partitioning
      ignition-2.21.0-6.rhaos4.19.el9 (Failing) c36d00439f7a Feb 24, 2026 Rebuilt with newer golang, MISSING device mapper fix

      Upstream Issue

      The fix for coreos/ignition#2128 was backported to ignition-2.21.0-6.el9_6 but NOT to ignition-2.21.0-6.rhaos4.19.el9.

      Issue summary: Ignition fails on change partitions on device mappers - when a device mapper (multipath) needs to be modified by Ignition, it fails with:

      Ignition failed: create partitions failed: refusing to operate on directly active disk
      

      This is exactly the error we see in the kola test console output.

      Why Did This Happen?

      The rhel-9.6 stream started picking up the RHAOS 4.19 Ignition package instead of the RHEL 9.6 version because:

      1. The RHAOS 4.19 build (Feb 24, 2026) is newer than the RHEL 9.6 build (Nov 18, 2025)
      2. Both packages are tagged for rhaos-4.19-rhel-9-candidate, so rpm-ostree selected the newer build

      Resolution

      Option 1: Backport the device mapper fix (PR #2129) to ignition-2.21.0-X.rhaos4.19.el9

      Option 2: Ensure rhel-9.6 stream uses ignition-2.21.0-6.el9_6 instead of ignition-2.21.0-6.rhaos4.19.el9

      Related Issues

              Unassigned Unassigned
              cverna@redhat.com Clément Verna
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: