Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-76812

leapp preupgrade/upgrade is awfully slow when audit configuration has "--loginuid-immutable" stanza

Linking RHIVOS CVEs to...Migration: Automation ...SWIFT: POC ConversionSync from "Extern...XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • rhel-8.10, rhel-9.5
    • systemd
    • No
    • Low
    • rhel-systemd
    • ssg_core_services
    • 5
    • False
    • False
    • Hide

      None

      Show
      None
    • None
    • Red Hat Enterprise Linux
    • None
    • None
    • None
    • None

      What were you trying to do that didn't work?

      When the audit configuration has --loginuid-immutable, a simple leapp preupgrade takes up to 1 hour to complete.
      This occurs because every systemd-nspawn instance waits for 5 seconds due to not being able to reset the /proc/self/loginuid from user's AUID to "-1" (4294967295).
      Below is the related systemd-nspawn code:

      2120 static int reset_audit_loginuid(void) {
      2121         _cleanup_free_ char *p = NULL;
      2122         int r;
      2123 
      2124         if ((arg_clone_ns_flags & CLONE_NEWPID) == 0)
      2125                 return 0;
      2126 
      2127         r = read_one_line_file("/proc/self/loginuid", &p);
      2128         if (r == -ENOENT)
      2129                 return 0;
      2130         if (r < 0)
      2131                 return log_error_errno(r, "Failed to read /proc/self/loginuid: %m");
      2132 
      2133         /* Already reset? */
      2134         if (streq(p, "4294967295"))
      2135                 return 0;
      2136 
      2137         r = write_string_file("/proc/self/loginuid", "4294967295", 0);
      2138         if (r < 0) {
      2139                 log_error_errno(r,
      2140                                 "Failed to reset audit login UID. This probably means that your kernel is too\n"
      2141                                 "old and you have audit enabled. Note that the auditing subsystem is known to\n"
      2142                                 "be incompatible with containers on old kernels. Please make sure to upgrade\n"
      2143                                 "your kernel or to off auditing with 'audit=0' on the kernel command line before\n"
      2144                                 "using systemd-nspawn. Sleeping for 5s... (%m)");
      2145 
      2146 >>>>            sleep(5);
      2147         }
      2148 
      2149         return 0;
      2150 }
      

      Here above, the line 2137 fails because of not having the permission (EPERM) due to the audit configuration, this leads to getting the error message + a nice delay on line 2146.

      In the end, because leapp preupgrade executes systemd-nspawn numerous times (more than 400 times), this leads to a huge delay.

      What is the impact of this issue to you?

      Cannot execute leapp preupgrade in a timely manner.

      Please provide the package NVR for which the bug is seen:

      leapp-0.18.0-1.el8_10.noarch
      Probably RHEL9 leapp as well but didn't test

      How reproducible is this bug?:

      Always

      Steps to reproduce

      1. Create audit rule and restart audit service
        # echo "--loginuid-immutable" > /etc/audit/rules.d/leapp_failure.rules
        # service auditd restart
        
      2. Execute leapp preupgrade

      Expected results

      Fast

      Actual results

      Super Slow + messages in the log:

       $ grep "leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator" /var/log/leapp/leapp-preupgrade.log | grep -c "Sleeping for 5s"
      476
      

      Additional infos

      On RHEL9, adding --inaccessible=/proc/self/loginuid to systemd-nspawn arguments may work, but this has to be tested.
      For sure on RHEL8, adding a bind mount --bind=/proc/1/loginuid=/proc/self/loginuid DOESN'T work, probably because of /proc/self symlink handling.

      The best way to fix this would be to remove that sleep(5) from systemd code, in case for example there is a some special variable in the environment.

              systemd-maint systemd maint mailing list
              rhn-support-rmetrich Renaud Métrich
              systemd maint mailing list systemd maint mailing list
              Frantisek Sumsal Frantisek Sumsal
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated: