Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-34657

After upgrading from 4.14.16 to 4.14.25 metal3-ironic and metal3-ironic-inspector pods are crashlooping

XMLWordPrintable

    • Important
    • Yes
    • 1
    • Metal Platform 254, Metal Platform 255
    • 2
    • False
    • Hide

      None

      Show
      None
    • Hide
      Cause: A recent change in RHEL python packages tightened FIPS compliance rules
      Consequence: Containerised installation of python packages containing precompiled files no longer works, preventing metal3/ironic from starting up
      Fix: Python package installation method was changed for ironic-container so that files are compiled locally
      Result: Metal3/ironic is back to operational state
      Show
      Cause: A recent change in RHEL python packages tightened FIPS compliance rules Consequence: Containerised installation of python packages containing precompiled files no longer works, preventing metal3/ironic from starting up Fix: Python package installation method was changed for ironic-container so that files are compiled locally Result: Metal3/ironic is back to operational state
    • Bug Fix
    • In Progress
    • Customer Escalated

      Description of problem:

          After upgrading from 4.14.16 to 4.14.25, there are two failing pods, but the upgrade has been completed.
      
      metal3-ironic container logs
      
      
      + ironic-dbsync --config-file /etc/ironic/ironic.conf create_schema
      Traceback (most recent call last):
      File "/usr/bin/ironic-dbsync", line 6, in <module>
      from ironic.cmd.dbsync import main
      File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
      File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 846, in exec_module
      File "<frozen importlib._bootstrap_external>", line 989, in get_code
      TypeError: source_hash() missing required argument 'source' (pos 2)
      metal3-ironic-inspector container logs
      
      
      + ironic-inspector-dbsync --config-file /etc/ironic-inspector/ironic-inspector.conf upgrade
      Traceback (most recent call last):
      File "/usr/bin/ironic-inspector-dbsync", line 6, in <module>
      from ironic_inspector.cmd.dbsync import main
      File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
      File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 846, in exec_module
      File "<frozen importlib._bootstrap_external>", line 989, in get_code
      TypeError: source_hash() missing required argument 'source' (pos 2)
       
      
       
      
      Version-Release number of selected component (if applicable):
      {code:none}
          

      How reproducible:

          always in 4.14.25 on FIPS enabled cluster 

      Steps to Reproduce:

          1. upgrade from 4.14.16 to 4.14.25.
          2.
          3.
          

      Actual results:

          pods should be up

      Expected results:

          2/4 pods are failing 
      metal3-7d8ff4f7f-4ntlm                               2/4    Running  279       23h
      

      Additional info:

          ironic logs and must-gather is available in the comment

       

            janders@redhat.com Jacob Anders
            rhn-support-sambekar Shubham Ambekar
            Pedro Jose Amoedo Martinez Pedro Jose Amoedo Martinez
            Daniel Chong
            Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: