Uploaded image for project: 'RHEL Conversions'
  1. RHEL Conversions
  2. RHELC-1180

When firewalld is running it fails after point of no return

XMLWordPrintable

    • False
    • False
    • Hide

      None

      Show
      None
    • Testable

      Affected versions:

      • first observed on copr build from PR 715
      • applicable also for build from main
      • tested also with brew build RC 1.4.1

      How reproduce:

      • install, enable and start firewalld
      • run the conversion

      Affected systems:

      • Alma, Oracle, Rocky Linux 8.8

      Caused by:
      firewalld-0.9.11-1.el8_8.noarch.rpm and/or its backend/dependencies

      On 8.8 systems with firewalld installed, enabled and firewalld.service running, the conversion fails during the Kernel Replacement task after the PONR leaving the system in a broken state.

      [2023-10-03T14:15:41+0000] TASK - [Convert: Prepare kernel] ******************************************
              Installing RHEL kernel ...
              [2023-10-03T14:15:41+0000] DEBUG - Calling command 'yum install -y --disablerepo=* --releasever=8.8 --setopt=module_platform_id=platform:el8 --enablerepo=rhel-8-for-x86_64-baseos-rpms --enablerepo=rhel-8-for-x86_64-appstream-rpms kernel'
              Updating Subscription Management repositories.
              Red Hat Enterprise Linux 8 for x86_64 - BaseOS  0.0  B/s |   0  B     00:00    
              Errors during downloading metadata for repository 'rhel-8-for-x86_64-baseos-rpms':
                - Curl error (6): Couldn't resolve host name for https://cdn.redhat.com/content/dist/rhel8/8.8/x86_64/baseos/os/repodata/repomd.xml [Could not resolve host: cdn.redhat.com]
              Error: Failed to download metadata for repo 'rhel-8-for-x86_64-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
              CRITICAL - Error occured while attempting to install the RHEL kernel
              Writing breadcrumbs to '/etc/migration-results'.
              Writing RHSM custom facts to '/etc/rhsm/facts/convert2rhel.facts'.
              [2023-10-03T14:15:42+0000] DEBUG - Traceback (most recent call last):
                File "/usr/lib/python3.6/site-packages/convert2rhel/main.py", line 146, in main_locked
                  post_ponr_changes()
                File "/usr/lib/python3.6/site-packages/convert2rhel/main.py", line 271, in post_ponr_changes
                  post_ponr_conversion()
                File "/usr/lib/python3.6/site-packages/convert2rhel/main.py", line 297, in post_ponr_conversion
                  pkghandler.preserve_only_rhel_kernel()
                File "/usr/lib/python3.6/site-packages/convert2rhel/pkghandler.py", line 890, in preserve_only_rhel_kernel
                  kernel_update_needed = install_rhel_kernel()
                File "/usr/lib/python3.6/site-packages/convert2rhel/pkghandler.py", line 911, in install_rhel_kernel
                  loggerinst.critical("Error occured while attempting to install the RHEL kernel")
                File "/usr/lib/python3.6/site-packages/convert2rhel/logger.py", line 162, in _critical
                  sys.exit(msg)
              SystemExit: Error occured while attempting to install the RHEL kernel
      

      This bug seems to be not introduced by convert2rhel, as it seems that it was rather discovered by it, however the issue is happening somewhere during the Package Replacement task, and I was unable to reproduce it on RHEL 8.8 yet.
      First observed on Oracle Linux 8.8 as it has the firewalld running on the base image by default.
      However, installing the package on any other 8.8 system yields the same results.
      An error can be observed in logs from `journalctl`, or specifically `systemctl status firewalld`.

      ERROR: 'python-nftables' failed: internal:0:0-0: Error: Could not process rule: Operation not supported

      If the firewalld.service is stopped before the conversion, the conversion finishes successfully, however the `systemctl status firewalld` yields the same result if the service gets restarted after the conversion.
      Additionally, the network gets messed up as well if the conversion fails

      [root@ip-10-31-9-168 ~]# ping google.com
      ping: google.com: Name or service not known
      [root@ip-10-31-9-168 ~]# ping 8.8.8.8
      PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
      ping: sendmsg: Operation not permitted
      ping: sendmsg: Operation not permitted
      ping: sendmsg: Operation not permitted
      ^C
      --- 8.8.8.8 ping statistics ---
      3 packets transmitted, 0 received, 100% packet loss, time 2040ms
      

      Digging through the internet a similar issue is described in this BZ
      And all the findings point to the firewalld backend (either nftables (python3-nftables) or iptables)

      Full output of `systemctl status firewalld` with different DefaultBacked in /etc/firewalld/firewalld.conf:

      • nftables
        [root@ip-10-31-40-203 ~]# systemctl status firewalld
        ● firewalld.service - firewalld - dynamic firewall daemon
           Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
           Active: active (running) since Wed 2023-10-04 14:06:31 GMT; 9s ago
             Docs: man:firewalld(1)
         Main PID: 25633 (firewalld)
            Tasks: 2 (limit: 22344)
           Memory: 24.1M
           CGroup: /system.slice/firewalld.service
                   └─25633 /usr/libexec/platform-python -s /usr/sbin/firewalld --nofork --nopid
        
        Oct 04 14:06:31 ip-10-31-40-203.us-east-1.aws.redhat.com systemd[1]: Starting firewalld - dynamic firewall daemon...
        Oct 04 14:06:31 ip-10-31-40-203.us-east-1.aws.redhat.com systemd[1]: Started firewalld - dynamic firewall daemon.
        Oct 04 14:06:31 ip-10-31-40-203.us-east-1.aws.redhat.com firewalld[25633]: WARNING: AllowZoneDrifting is enabled. This is considered an insecure configurat>
        Oct 04 14:06:32 ip-10-31-40-203.us-east-1.aws.redhat.com firewalld[25633]: ERROR: 'python-nftables' failed: internal:0:0-0: Error: Could not process rule: >
                                                                                   
                                                                                   
                                                                                   JSON blob:
                                                                                   {"nftables": [{"metainfo": {"json_schema_version": 1}}, {"add": {"chain": {"fami>
        Oct 04 14:06:32 ip-10-31-40-203.us-east-1.aws.redhat.com firewalld[25633]: ERROR: COMMAND_FAILED: 'python-nftables' failed: internal:0:0-0: Error: Could no>
                                                                                   
                                                                                   
                                                                                   JSON blob:
                                                                                   {"nftables": [{"metainfo": {"json_schema_version": 1}}, {"add": {"chain": {"fami>
        Oct 04 14:06:32 ip-10-31-40-203.us-east-1.aws.redhat.com firewalld[25633]: ERROR: 'python-nftables' failed: internal:0:0-0: Error: Could not process rule: >
                                                                                   
                                                                                   internal:0:0-0: Error: Could not process rule: No such file or directory
                                                                                   
                                                                                   internal:0:0-0: Error: Could not process rule: No such file or directory
                                                                                   
                                                                                   internal:0:0-0: Error: Could not process rule: No such file or directory
                                                                                   
                                                                                   internal:0:0-0: Error: Could not process rule: No such file or directory
                                                                                   
                                                                                   internal:0:0-0: Error: Could not process rule: No such file or directory
                                                                                   
                                                                                   internal:0:0-0: Error: Could not process rule: No such file or directory
                                                                                   
                                                                                   internal:0:0-0: Error: Could not process rule: No such file or directory
                                                                                   
                                                                                   
                                                                                   JSON blob:
                                                                                   {"nftables": [{"metainfo": {"json_schema_version": 1}}, {"insert": {"rule": {"fa>
        Oct 04 14:06:32 ip-10-31-40-203.us-east-1.aws.redhat.com firewalld[25633]: ERROR: COMMAND_FAILED: 'python-nftables' failed: internal:0:0-0: Error: Could no>
                                                                                   
                                                                                   internal:0:0-0: Error: Could not process rule: No such file or directory
                                                                                   
                                                                                   internal:0:0-0: Error: Could not process rule: No such file or directory
                                                                                   
                                                                                   internal:0:0-0: Error: Could not process rule: No such file or directory
                                                                                   
                                                                                   internal:0:0-0: Error: Could not process rule: No such file or directory
                                                                                   
                                                                                   internal:0:0-0: Error: Could not process rule: No such file or directory
                                                                                   
                                                                                   internal:0:0-0: Error: Could not process rule: No such file or directory
                                                                                   
                                                                                   internal:0:0-0: Error: Could not process rule: No such file or directory
                                                                                   
                                                                                   
                                                                                   JSON blob:
                                                                                   {"nftables": [{"metainfo": {"json_schema_version": 1}}, {"insert": {"rule": {"fa
        
      • iptables
        Oct 04 14:10:14 ip-10-31-40-203.us-east-1.aws.redhat.com systemd[1]: Starting firewalld - dynamic firewall daemon...
        Oct 04 14:10:14 ip-10-31-40-203.us-east-1.aws.redhat.com systemd[1]: Started firewalld - dynamic firewall daemon.
        Oct 04 14:10:14 ip-10-31-40-203.us-east-1.aws.redhat.com firewalld[26138]: WARNING: AllowZoneDrifting is enabled. This is considered an insecure configuration option. It will be removed in a future release. Please consider disabling it now.
        Oct 04 14:10:15 ip-10-31-40-203.us-east-1.aws.redhat.com firewalld[26138]: ERROR: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore v1.8.4 (nf_tables): 
                                                                                   line 3: RULE_APPEND failed (No such file or directory): rule in chain INPUT
                                                                                   line 5: RULE_APPEND failed (No such file or directory): rule in chain OUTPUT
                                                                                   line 7: RULE_APPEND failed (No such file or directory): rule in chain FORWARD
        Oct 04 14:10:15 ip-10-31-40-203.us-east-1.aws.redhat.com firewalld[26138]: ERROR: '/usr/sbin/ip6tables-restore -w -n' failed: ip6tables-restore v1.8.4 (nf_tables): 
                                                                                   line 3: RULE_APPEND failed (No such file or directory): rule in chain INPUT
                                                                                   line 5: RULE_APPEND failed (No such file or directory): rule in chain OUTPUT
                                                                                   line 7: RULE_APPEND failed (No such file or directory): rule in chain FORWARD
        Oct 04 14:10:15 ip-10-31-40-203.us-east-1.aws.redhat.com firewalld[26138]: ERROR: '/usr/sbin/ebtables-restore --noflush' failed: ebtables-restore v1.8.4 (nf_tables): 
                                                                                   line 3: RULE_INSERT failed (No such file or directory): rule in chain PREROUTING
                                                                                   line 4: RULE_INSERT failed (No such file or directory): rule in chain PREROUTING_direct
                                                                                   line 6: RULE_INSERT failed (No such file or directory): rule in chain POSTROUTING
                                                                                   line 7: RULE_INSERT failed (No such file or directory): rule in chain POSTROUTING_direct
                                                                                   line 9: RULE_INSERT failed (No such file or directory): rule in chain OUTPUT
                                                                                   line 10: RULE_INSERT failed (No such file or directory): rule in chain OUTPUT_direct
                                                                                   line 13: RULE_INSERT failed (No such file or directory): rule in chain INPUT
                                                                                   line 14: RULE_INSERT failed (No such file or directory): rule in chain INPUT_direct
                                                                                   line 16: RULE_INSERT failed (No such file or directory): rule in chain OUTPUT
                                                                                   line 17: RULE_INSERT failed (No such file or directory): rule in chain OUTPUT_direct
                                                                                   line 19: RULE_INSERT failed (No such file or directory): rule in chain FORWARD
                                                                                   line 20: RULE_INSERT failed (No such file or directory): rule in chain FORWARD_direct
                                                                                   line 20: RULE_APPEND failed (No
        Oct 04 14:10:15 ip-10-31-40-203.us-east-1.aws.redhat.com firewalld[26138]: ERROR: COMMAND_FAILED: '/usr/sbin/ebtables-restore --noflush' failed: ebtables-restore v1.8.4 (nf_tables): 
                                                                                   line 3: RULE_INSERT failed (No such file or directory): rule in chain PREROUTING
                                                                                   line 4: RULE_INSERT failed (No such file or directory): rule in chain PREROUTING_direct
                                                                                   line 6: RULE_INSERT failed (No such file or directory): rule in chain POSTROUTING
                                                                                   line 7: RULE_INSERT failed (No such file or directory): rule in chain POSTROUTING_direct
                                                                                   line 9: RULE_INSERT failed (No such file or directory): rule in chain OUTPUT
                                                                                   line 10: RULE_INSERT failed (No such file or directory): rule in chain OUTPUT_direct
                                                                                   line 13: RULE_INSERT failed (No such file or directory): rule in chain INPUT
                                                                                   line 14: RULE_INSERT failed (No such file or directory): rule in chain INPUT_direct
                                                                                   line 16: RULE_INSERT failed (No such file or directory): rule in chain OUTPUT
                                                                                   line 17: RULE_INSERT failed (No such file or directory): rule in chain OUTPUT_direct
                                                                                   line 19: RULE_INSERT failed (No such file or directory): rule in chain FORWARD
                                                                                   line 20: RULE_INSERT failed (No such file or directory): rule in chain FORWARD_direct
                                                                                   line 20: RULE_APPEND failed (No
        Oct 04 14:10:15 ip-10-31-40-203.us-east-1.aws.redhat.com firewalld[26138]: ERROR: '/usr/sbin/iptables-restore -w -n' failed: iptables-restore v1.8.4 (nf_tables): Chain 'IN_public' does not exist
                                                                                   Error occurred at line: 2
                                                                                   Try `iptables-restore -h' or 'iptables-restore --help' for more information.
        Oct 04 14:10:15 ip-10-31-40-203.us-east-1.aws.redhat.com firewalld[26138]: ERROR: '/usr/sbin/ip6tables-restore -w -n' failed: ip6tables-restore v1.8.4 (nf_tables): Chain 'IN_public' does not exist
                                                                                   Error occurred at line: 2
                                                                                   Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.
        Oct 04 14:10:15 ip-10-31-40-203.us-east-1.aws.redhat.com firewalld[26138]: ERROR: COMMAND_FAILED: '/usr/sbin/ip6tables-restore -w -n' failed: ip6tables-restore v1.8.4 (nf_tables): Chain 'IN_public' does not exist
                                                                                   Error occurred at line: 2
                                                                                   Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.
        

          There are no Sub-Tasks for this issue.

              rolivier@redhat.com Rodolfo Olivieri
              ddiblik@redhat.com Daniel Diblik
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: