Description of problem:
Cu is experciencing high memory usage in IdM/IPA acceptance environment with an AD trust.

Investigation of SAR data shows high memory and ps aux --sort=%mem use by rpcd:
root 8744 22.3 63.4 7938416 7663860 ? R Mar14 6461:50 /usr/libexec/samba/rpcd_lsad --configfile=/etc/samba/smb.conf --worker-group=4 --worker-index=5 --debuglevel=1

Version-Release number of selected component (if applicable):
samba-libs-4.16.4-2.el8.x86_64
samba-common-libs-4.16.4-2.el8.x86_64
samba-winbind-4.16.4-2.el8.x86_64
samba-winbind-modules-4.16.4-2.el8.x86_64
samba-common-tools-4.16.4-2.el8.x86_64
samba-common-4.16.4-2.el8.noarch
python3-samba-4.16.4-2.el8.x86_64
samba-client-libs-4.16.4-2.el8.x86_64
samba-4.16.4-2.el8.x86_64
idmp1.idmp.obfuscated.domain | CHANGED | rc=0 >>
samba-winbind-modules-4.16.4-2.el8.x86_64
samba-common-tools-4.16.4-2.el8.x86_64
samba-common-4.16.4-2.el8.noarch
samba-libs-4.16.4-2.el8.x86_64
python3-samba-4.16.4-2.el8.x86_64
samba-client-libs-4.16.4-2.el8.x86_64
samba-4.16.4-2.el8.x86_64
samba-common-libs-4.16.4-2.el8.x86_64
samba-winbind-4.16.4-2.el8.x86_64
idma1.idma.obfuscated.domain | CHANGED | rc=0 >>
samba-common-4.16.4-2.el8.noarch
samba-client-libs-4.16.4-2.el8.x86_64
samba-common-tools-4.16.4-2.el8.x86_64
samba-common-libs-4.16.4-2.el8.x86_64
samba-winbind-modules-4.16.4-2.el8.x86_64
samba-4.16.4-2.el8.x86_64
python3-samba-4.16.4-2.el8.x86_

How reproducible:
Issue is periodic in Cu environment. It is alleviated after restarting IPA but always seems to come back.

Actual results:
Very high period memory consumption by rpcd

Expected results:
No memory pressure issues from rpcd_lsad process.

Additional info:
Mostly, I am trying to understand how best to collect information and investigate this for a potential memory leak.

I have attempted:
Grabbing a pstack would definitely help here. But also grabbing Samba debugging to know what is going on in the logs:

1. Edit /etc/samba/smb.conf and add debugging to [global]:

log level = 10
debug pid = true
max log size = 0

2. IPA:

1. ipactl restart

3. Wait for the memory growth to occur.

4. Grab a pstack of the process ID which shows the memory growth:

1. pstack <processID> > /tmp/samba-memory-pstack.out

4. Archive and attach Samba/Winbind debugging:

1. tar czvpf /tmp/samba-debug_$(hostname -s)$(date +%F%H%M%S).tar.gz /var/log/ {samba,sssd,secure,messages}

   /etc/

   {samba,sssd,ssh/sshd_config,pam.d,nsswitch.conf,krb5.conf}

suggestion from our backline, but we aren't really getting usable data. The pstack they collected was done improperly and contained no data, and the debug can't be left at this level as they don't want to impact the usability of the environment.

How should we go about this?