Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-22542

leapp breaks in reboot phase when "jbcs-httpd24-brotli-xxx.el7jbcs" is installed

    • None
    • Important
    • rhel-sst-upgrades
    • 3
    • False
    • Hide

      None

      Show
      None
    • None
    • Red Hat Enterprise Linux
    • None
    • None
    • None
    • None

      What were you trying to do that didn't work?

      When jbcs-httpd24-brotli-xxx.el7jbcs package (delivered by jb-coreservices-1-for-rhel-7-server-rpms repository) is installed on the system, leapp breaks during reboot phase while upgrading with the following error:

      [  264.418399] upgrade[682]: 2024-01-23 17:11:55.624891 [ERROR] Actor: dnf_upgrade_transaction
      [  264.418972] upgrade[682]: Message: Cannot obtain data about the DNF configuration
      [  264.419501] upgrade[682]: Summary:
      [  264.419823] upgrade[682]:     Stderr: Traceback (most recent call last):
      [  264.420346] upgrade[682]:               File "/usr/lib64/python3.6/site-packages/libdnf/error.py", line 14, in swig_import_helper
      [  264.421081] upgrade[682]:                 return importlib.import_module(mname)
      [  264.421596] upgrade[682]:               File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
      [  264.422275] upgrade[682]:                 return _bootstrap._gcd_import(name[level:], package, level)
      [  264.422882] upgrade[682]:               File "<frozen importlib._bootstrap>", line 994, in _gcd_import
      [  264.423496] upgrade[682]:               File "<frozen importlib._bootstrap>", line 971, in _find_and_load
      [  264.424323] upgrade[682]:               File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
      [  264.425048] upgrade[682]:               File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
      [  264.425721] upgrade[682]:               File "<frozen importlib._bootstrap>", line 571, in module_from_spec
      [  264.426524] upgrade[682]:               File "<frozen importlib._bootstrap_external>", line 922, in create_module
      [  264.427257] upgrade[682]:               File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      [  264.427969] upgrade[682]:             ImportError: libbrotlidec.so.1: cannot open shared object file: No such file or directory
      [  264.428734] upgrade[682]:             
      [  264.429095] upgrade[682]:             During handling of the above exception, another exception occurred:
      [  264.429751] upgrade[682]:             
      [  264.430112] upgrade[682]:             Traceback (most recent call last):
      [  264.430617] upgrade[682]:               File "/usr/bin/dnf", line 57, in <module>
      [  264.431174] upgrade[682]:                 from dnf.cli import main
      [  264.431850] upgrade[682]:               File "/usr/lib/python3.6/site-packages/dnf/__init__.py", line 30, in <module>
      [  264.432715] upgrade[682]:                 import dnf.base
      [  264.433162] upgrade[682]:               File "/usr/lib/python3.6/site-packages/dnf/base.py", line 29, in <module>
      [  264.433847] upgrade[682]:                 import libdnf.transaction
      [  264.434336] upgrade[682]:               File "/usr/lib64/python3.6/site-packages/libdnf/__init__.py", line 8, in <module>
      [  264.435056] upgrade[682]:                 from . import error
      [  264.435528] upgrade[682]:               File "/usr/lib64/python3.6/site-packages/libdnf/error.py", line 17, in <module>
      [  264.436240] upgrade[682]:                 _error = swig_import_helper()
      [  264.436734] upgrade[682]:               File "/usr/lib64/python3.6/site-packages/libdnf/error.py", line 16, in swig_import_helper
      [  264.437502] upgrade[682]:                 return importlib.import_module('_error')
      [  264.438036] upgrade[682]:               File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
      [  264.438735] upgrade[682]:                 return _bootstrap._gcd_import(name[level:], package, level)
      [  264.439359] upgrade[682]:             ModuleNotFoundError: No module named '_error'
      

      The root cause for this is brotli package doesn't get installed on the system, because jbcs-httpd24-brotli-xxx.el7jbcs package claims to deliver libbrotli*.so libraries, which is true, but not system libraries, SCL libraries instead:

      # rpm -q --provides -p ./jbcs-httpd24-brotli-1.0.9-2.el7jbcs.x86_64.rpm 
      [...]
      libbrotlicommon.so.1()(64bit)
      libbrotlidec.so.1()(64bit)
      libbrotlienc.so.1()(64bit)
      scl-package(jbcs-httpd24)
      
      # rpm -qlp ./jbcs-httpd24-brotli-1.0.9-2.el7jbcs.x86_64.rpm
      [...]
      /opt/rh/jbcs-httpd24/root/usr/lib64/libbrotlicommon.so.1
      /opt/rh/jbcs-httpd24/root/usr/lib64/libbrotlicommon.so.1.0.9
      /opt/rh/jbcs-httpd24/root/usr/lib64/libbrotlidec.so.1
      /opt/rh/jbcs-httpd24/root/usr/lib64/libbrotlidec.so.1.0.9
      /opt/rh/jbcs-httpd24/root/usr/lib64/libbrotlienc.so.1
      /opt/rh/jbcs-httpd24/root/usr/lib64/libbrotlienc.so.1.0.9
      

      Due to this, brotli package, which is implicitly required by dnf, is not part of the transaction, letting dnf and the whole system in an unrecoverable state.

      brotli being implicitly required by dnf:

      (rhel8) # yum remove brotli
      [...]
      Error: 
       Problem: The operation would result in removing the following protected packages: dnf
      

      We can detect the issue prior to rebooting through checking the leapp upgrade log:

      • When upgrade will be ok
        # grep dnf_dry_run /var/log/leapp/leapp-upgrade.log  | grep brotli
        2024-01-23 16:30:37.135 DEBUG    PID: 13317 leapp.workflow.InterimPreparation.dnf_dry_run:  brotli                                           x86_64  1.0.6-3.el8                                 os8           323 k
        #
      • When upgrade will fail, there is no brotli package in the transaction
        # grep dnf_dry_run /var/log/leapp/leapp-upgrade.log  | grep brotli
        #

      The workaround consists in making sure brotli is always part of the transaction:

      # echo brotli >> /etc/leapp/transaction/to_install
      

      It would hence nice to harden leapp to avoid post-reboot failure, which leads the system in an unrecoverable state.

      See also jbcs-httpd24-brotli claims to deliver "libbrotlienc.so.1()", which breaks Leapp upgrades from 7 to 8.

      Please provide the package NVR for which bug is seen:

      leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch
      jbcs-httpd24-brotli-xxx.el7jbcs (e.g. jbcs-httpd24-brotli-1.0.6-21.jbcs.el7.x86_64)

      How reproducible:

      Always

      Steps to reproduce

      1. Install jbcs-httpd24-brotli from jb-coreservices-1-for-rhel-7-server-rpms repository
      2. Upgrade

      Expected results

      No error or error detected before reboot

      Actual results

      Upgrade happens but leaves system unrecoverable

              leapp-notifications leapp-notifications
              rhn-support-rmetrich Renaud Métrich
              leapp-notifications leapp-notifications
              RHEL Upgrades QE Team RHEL Upgrades QE Team
              Miriam Portman Miriam Portman
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: