Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-34657

After upgrading from 4.14.16 to 4.14.25 metal3-ironic and metal3-ironic-inspector pods are crashlooping

    • Important
    • Yes
    • 1
    • Metal Platform 254, Metal Platform 255
    • 2
    • False
    • Hide

      None

      Show
      None
    • Hide
      Cause: A recent change in RHEL python packages tightened FIPS compliance rules
      Consequence: Containerised installation of python packages containing precompiled files no longer works, preventing metal3/ironic from starting up
      Fix: Python package installation method was changed for ironic-container so that files are compiled locally
      Result: Metal3/ironic is back to operational state
      Show
      Cause: A recent change in RHEL python packages tightened FIPS compliance rules Consequence: Containerised installation of python packages containing precompiled files no longer works, preventing metal3/ironic from starting up Fix: Python package installation method was changed for ironic-container so that files are compiled locally Result: Metal3/ironic is back to operational state
    • Bug Fix
    • In Progress
    • Customer Escalated

      Description of problem:

          After upgrading from 4.14.16 to 4.14.25, there are two failing pods, but the upgrade has been completed.
      
      metal3-ironic container logs
      
      
      + ironic-dbsync --config-file /etc/ironic/ironic.conf create_schema
      Traceback (most recent call last):
      File "/usr/bin/ironic-dbsync", line 6, in <module>
      from ironic.cmd.dbsync import main
      File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
      File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 846, in exec_module
      File "<frozen importlib._bootstrap_external>", line 989, in get_code
      TypeError: source_hash() missing required argument 'source' (pos 2)
      metal3-ironic-inspector container logs
      
      
      + ironic-inspector-dbsync --config-file /etc/ironic-inspector/ironic-inspector.conf upgrade
      Traceback (most recent call last):
      File "/usr/bin/ironic-inspector-dbsync", line 6, in <module>
      from ironic_inspector.cmd.dbsync import main
      File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
      File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 846, in exec_module
      File "<frozen importlib._bootstrap_external>", line 989, in get_code
      TypeError: source_hash() missing required argument 'source' (pos 2)
       
      
       
      
      Version-Release number of selected component (if applicable):
      {code:none}
          

      How reproducible:

          always in 4.14.25 on FIPS enabled cluster 

      Steps to Reproduce:

          1. upgrade from 4.14.16 to 4.14.25.
          2.
          3.
          

      Actual results:

          pods should be up

      Expected results:

          2/4 pods are failing 
      metal3-7d8ff4f7f-4ntlm                               2/4    Running  279       23h
      

      Additional info:

          ironic logs and must-gather is available in the comment

       

            [OCPBUGS-34657] After upgrading from 4.14.16 to 4.14.25 metal3-ironic and metal3-ironic-inspector pods are crashlooping

            Errata Tool added a comment -

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory (Moderate: OpenShift Container Platform 4.14.30 bug fix and security update), and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHSA-2024:3881

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Moderate: OpenShift Container Platform 4.14.30 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:3881

            Pedro Jose Amoedo Martinez added a comment - - edited Verified with version " 4.14.0-0.nightly-2024-06-12-030043 ": [prow-CI DS profile] https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/pr-logs/pull/openshift_release/53093/rehearse-53093-periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-nightly-baremetalds-ipi-ovn-ipv4-fips-f14/1800816816440741888 [RDU IPI BM profile] https://mastern-jenkins-csb-openshift-qe.apps.ocp-c1.prod.psi.redhat.com/job/ocp-common/job/Flexy-install/289980/console [root@openshift-qe-059 /]# fips-mode-setup --check FIPS mode is enabled. Regards.

            Reporter (sothomps@redhat.com) does not have permission to create attachments in project OCPBUGS. Following attachments found in the email have been discarded:

            • noname

            Sonny Thompson added a comment - Reporter (sothomps@redhat.com) does not have permission to create attachments in project OCPBUGS. Following attachments found in the email have been discarded: noname

            Great news! Thanks for staying close to this, Jacob.

            @James Buckley <jabuckle@redhat.com> I think we have what we need for you
            to send a reliable update to the customer, at your convenience. Please
            include me, @Tim Ryan <tiryan@redhat.com>, @Kurt Phillips
            <kuphilli@redhat.com>, @Scott Hamilton <scott.hamilton@redhat.com> on Cc.

            Thanks,
            Sonny

            sonny thompson

            Customer success executive, na public sector

            defense agencies

            sothomps@redhat.com

            O: 919.647.8863

            m: 915.443.5036

            @RedHat <https://twitter.com/redhat> Red Hat
            <https://www.linkedin.com/company/red-hat> Red Hat
            <https://www.facebook.com/RedHatInc>
            <https://www.redhat.com/>

            On Wed, Jun 12, 2024 at 6:44 AM Jacob Anders (Jira) <jira-issues@redhat.com>

            Sonny Thompson added a comment - Great news! Thanks for staying close to this, Jacob. @James Buckley <jabuckle@redhat.com> I think we have what we need for you to send a reliable update to the customer, at your convenience. Please include me, @Tim Ryan <tiryan@redhat.com>, @Kurt Phillips <kuphilli@redhat.com>, @Scott Hamilton <scott.hamilton@redhat.com> on Cc. Thanks, Sonny sonny thompson Customer success executive, na public sector defense agencies sothomps@redhat.com O: 919.647.8863 m: 915.443.5036 @RedHat < https://twitter.com/redhat > Red Hat < https://www.linkedin.com/company/red-hat > Red Hat < https://www.facebook.com/RedHatInc > < https://www.redhat.com/ > On Wed, Jun 12, 2024 at 6:44 AM Jacob Anders (Jira) <jira-issues@redhat.com>

            I'm not entirely clear on the relationship between this bug and OCPBUGS-34534. But assuming they're part of the same series, I'm going to drop the UpgradeBlocker and ImpactStatementRequested labels from this one, because METAL-1023 is already tracking an impact statement for all bugs in the OCPBUGS-34534 series. If you think this bug is different enough to need an independent impact statement assessment, please restore the UpgradeBlocker label.

            W. Trevor King added a comment - I'm not entirely clear on the relationship between this bug and OCPBUGS-34534 . But assuming they're part of the same series, I'm going to drop the UpgradeBlocker and ImpactStatementRequested labels from this one, because METAL-1023 is already tracking an impact statement for all bugs in the OCPBUGS-34534 series. If you think this bug is different enough to need an independent impact statement assessment, please restore the UpgradeBlocker label.

              janders@redhat.com Jacob Anders
              rhn-support-sambekar Shubham Ambekar (Inactive)
              Pedro Jose Amoedo Martinez Pedro Jose Amoedo Martinez
              Daniel Chong
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: