Description of problem:

Customer is unable to show group information on IPA, due to some failures related to Domain Users, message shown is not helpful here to troubleshoot the issue.

Version-Release number of selected component (if applicable):

RHEL 8.5 (Ootpa)

389-ds-base-1.4.3.23-10.module+el8.5.0+12398+47000435.x86_64
389-ds-base-libs-1.4.3.23-10.module+el8.5.0+12398+47000435.x86_64
adcli-0.8.2-12.el8.x86_64
ipa-client-4.9.6-6.module+el8.5.0+12660+88e16a2c.x86_64
ipa-client-common-4.9.6-6.module+el8.5.0+12660+88e16a2c.noarch
ipa-common-4.9.6-6.module+el8.5.0+12660+88e16a2c.noarch
ipa-healthcheck-core-0.7-6.module+el8.5.0+11410+91a33fe4.noarch
ipa-selinux-4.9.6-6.module+el8.5.0+12660+88e16a2c.noarch
ipa-server-4.9.6-6.module+el8.5.0+12660+88e16a2c.x86_64
ipa-server-common-4.9.6-6.module+el8.5.0+12660+88e16a2c.noarch
ipa-server-dns-4.9.6-6.module+el8.5.0+12660+88e16a2c.noarch
ipa-server-trust-ad-4.9.6-6.module+el8.5.0+12660+88e16a2c.x86_64

How reproducible:

After some analysis, I could replicate the issue in my internal lab(RHEL 8.3) as well.

Steps to Reproduce:

1. Check AD Domain user and add to a specific group

1. id dcamilo@example.net
   uid=227401122(dcamilo@EXAMPLE.NET) gid=227401122(dcamilo@EXAMPLE.NET) groups=227401122(dcamilo@EXAMPLE.NET),227400513(domain users@EXAMPLE.NET)

1. ipa group-add-member testgroup --idoverrideuser=dcamilo@example.net

2. Check if the group-show works as expected.

1. ipa group-show testgroup
   Group name: testgroup
   GID: 1712000005
   Member users: admin
   Member ID user overrides: dcamilo@EXAMPLE.NET

3. In AD environment, remove the specific user, in my case was dcamilo@example.net

4. Clean the SSSD cache and try to fetch this user.

1. sss_cache -E

1. id dcamilo@example.net
   id: ‘dcamilo@example.net’: no such user

5. Try to run ipa group-show against this group again.

1. ipa group-show testgroup
   ipa: DEBUG: Loading Index file from '/var/lib/ipa-client/sysrestore/sysrestore.index'
   ipa: DEBUG: Loading StateFile from '/var/lib/ipa-client/sysrestore/sysrestore.state'
   ipa: DEBUG: Loading StateFile from '/var/lib/ipa-client/sysrestore/sysrestore.state'
   ipa: DEBUG: importing all plugin modules in ipaclient.remote_plugins.schema$3f71e6ba...
   ipa: DEBUG: importing plugin module ipaclient.remote_plugins.schema$3f71e6ba.plugins
   ipa: DEBUG: importing all plugin modules in ipaclient.plugins...
   ipa: DEBUG: importing plugin module ipaclient.plugins.automember
   ipa: DEBUG: importing plugin module ipaclient.plugins.automount
   ipa: DEBUG: importing plugin module ipaclient.plugins.ca
   ipa: DEBUG: importing plugin module ipaclient.plugins.cert
   ipa: DEBUG: importing plugin module ipaclient.plugins.certmap
   ipa: DEBUG: importing plugin module ipaclient.plugins.certprofile
   ipa: DEBUG: importing plugin module ipaclient.plugins.dns
   ipa: DEBUG: importing plugin module ipaclient.plugins.hbacrule
   ipa: DEBUG: importing plugin module ipaclient.plugins.hbactest
   ipa: DEBUG: importing plugin module ipaclient.plugins.host
   ipa: DEBUG: importing plugin module ipaclient.plugins.idrange
   ipa: DEBUG: importing plugin module ipaclient.plugins.internal
   ipa: DEBUG: importing plugin module ipaclient.plugins.location
   ipa: DEBUG: importing plugin module ipaclient.plugins.migration
   ipa: DEBUG: importing plugin module ipaclient.plugins.misc
   ipa: DEBUG: importing plugin module ipaclient.plugins.otptoken
   ipa: DEBUG: importing plugin module ipaclient.plugins.otptoken_yubikey
   ipa: DEBUG: importing plugin module ipaclient.plugins.passwd
   ipa: DEBUG: importing plugin module ipaclient.plugins.permission
   ipa: DEBUG: importing plugin module ipaclient.plugins.rpcclient
   ipa: DEBUG: importing plugin module ipaclient.plugins.server
   ipa: DEBUG: importing plugin module ipaclient.plugins.service
   ipa: DEBUG: importing plugin module ipaclient.plugins.sudorule
   ipa: DEBUG: importing plugin module ipaclient.plugins.topology
   ipa: DEBUG: importing plugin module ipaclient.plugins.trust
   ipa: DEBUG: importing plugin module ipaclient.plugins.user
   ipa: DEBUG: importing plugin module ipaclient.plugins.vault
   ipa: DEBUG: found session_cookie in persistent storage for principal 'admin@LAB.EXAMPLE.NET', cookie: 'ipa_session=MagBearerToken=FpBNHdtXENljIubuGjiQjfORW6AfWA9j4GDvsdJp34NGtysbW%2f%2bTfo7ZoXfaXJUJ0NS%2fPjw7OWyw41tIslMS%2f3J0A3juoiViLAMDbF2X2MpSOia2t6XRAp%2bmMhlvuEfROO4cuMV%2bNt18oeK8wEiOtMpJFiv4RQMlusp9d72aIN48DvRByW3gltsuw%2fhzOa8TmWEAMzu7GNunoSMYv4BXyA%3d%3d'
   ipa: DEBUG: setting session_cookie into context 'ipa_session=MagBearerToken=FpBNHdtXENljIubuGjiQjfORW6AfWA9j4GDvsdJp34NGtysbW%2f%2bTfo7ZoXfaXJUJ0NS%2fPjw7OWyw41tIslMS%2f3J0A3juoiViLAMDbF2X2MpSOia2t6XRAp%2bmMhlvuEfROO4cuMV%2bNt18oeK8wEiOtMpJFiv4RQMlusp9d72aIN48DvRByW3gltsuw%2fhzOa8TmWEAMzu7GNunoSMYv4BXyA%3d%3d;'
   ipa: DEBUG: trying https://ipa-master.lab.example.net/ipa/session/json
   ipa: DEBUG: New HTTP connection (ipa-master.lab.example.net)
   ipa: DEBUG: Created connection context.rpcclient_139807819720856
   ipa: DEBUG: raw: group_show('testgroup', version='2.245')
   ipa: DEBUG: group_show('testgroup', version='2.245')
   ipa: DEBUG: [try 1]: Forwarding 'group_show/1' to json server 'https://ipa-master.lab.example.net/ipa/session/json'
   ipa: DEBUG: HTTP connection keep-alive (ipa-master.lab.example.net)
   ipa: DEBUG: Destroyed connection context.rpcclient_139807819720856
   ipa: ERROR: trusted domain object not found --> That is the issue.

/var/log/httpd/error_log

[Tue Jun 07 12:21:24.349429 2022] [:warn] [pid 6096:tid 140347409671936] [client 192.168.122.249:55846] failed to set perms (3140) on file (/run/ipa/ccaches/admin@LAB.EXAMPLE.NET-Z1sYMM)!, referer: https://ipa-master.lab.example.net/ipa/xml
[Tue Jun 07 12:21:24.350039 2022] [wsgi:error] [pid 5459:tid 140347472983808] [remote 192.168.122.249:55846] ipa: DEBUG: WSGI wsgi_dispatch._call_:
[Tue Jun 07 12:21:24.350090 2022] [wsgi:error] [pid 5459:tid 140347472983808] [remote 192.168.122.249:55846] ipa: DEBUG: WSGI jsonserver_session._call_:
[Tue Jun 07 12:21:24.356748 2022] [wsgi:error] [pid 5459:tid 140347472983808] [remote 192.168.122.249:55846] ipa: DEBUG: Created connection context.ldap2_140347501990240
[Tue Jun 07 12:21:24.356797 2022] [wsgi:error] [pid 5459:tid 140347472983808] [remote 192.168.122.249:55846] ipa: DEBUG: WSGI jsonserver._call_:
[Tue Jun 07 12:21:24.356823 2022] [wsgi:error] [pid 5459:tid 140347472983808] [remote 192.168.122.249:55846] ipa: DEBUG: WSGI WSGIExecutioner._call_:
[Tue Jun 07 12:21:24.357098 2022] [wsgi:error] [pid 5459:tid 140347472983808] [remote 192.168.122.249:55846] ipa: DEBUG: raw: ping(version='2.245')
[Tue Jun 07 12:21:24.357165 2022] [wsgi:error] [pid 5459:tid 140347472983808] [remote 192.168.122.249:55846] ipa: DEBUG: ping(version='2.245')
[Tue Jun 07 12:21:24.357251 2022] [wsgi:error] [pid 5459:tid 140347472983808] [remote 192.168.122.249:55846] ipa: INFO: [jsonserver_session] admin@LAB.EXAMPLE.NET: ping(): SUCCESS
[Tue Jun 07 12:21:24.357280 2022] [wsgi:error] [pid 5459:tid 140347472983808] [remote 192.168.122.249:55846] ipa: DEBUG: [jsonserver_session] admin@LAB.EXAMPLE.NET: ping(): SUCCESS etime=388580
[Tue Jun 07 12:21:24.357636 2022] [wsgi:error] [pid 5459:tid 140347472983808] [remote 192.168.122.249:55846] ipa: DEBUG: FINAL: Hits 0 Misses 0 Size 0
[Tue Jun 07 12:21:24.357682 2022] [wsgi:error] [pid 5459:tid 140347472983808] [remote 192.168.122.249:55846] ipa: DEBUG: Destroyed connection context.ldap2_140347501990240
[Tue Jun 07 12:21:24.359567 2022] [:warn] [pid 6096:tid 140347401279232] [client 192.168.122.249:55846] failed to set perms (3140) on file (/run/ipa/ccaches/admin@LAB.EXAMPLE.NET-Z1sYMM)!, referer: https://ipa-master.lab.example.net/ipa/xml
[Tue Jun 07 12:21:24.360020 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: WSGI wsgi_dispatch._call_:
[Tue Jun 07 12:21:24.360068 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: WSGI jsonserver_session._call_:
[Tue Jun 07 12:21:24.367157 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: Created connection context.ldap2_140347501990128
[Tue Jun 07 12:21:24.367212 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: WSGI jsonserver._call_:
[Tue Jun 07 12:21:24.367244 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: WSGI WSGIExecutioner._call_:
[Tue Jun 07 12:21:24.367503 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: raw: group_show('testgroup', version='2.245')
[Tue Jun 07 12:21:24.367618 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: group_show('testgroup', rights=False, all=False, raw=False, version='2.245', no_members=False)
[Tue Jun 07 12:21:24.367799 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: Cache lookup: cn=testgroup,cn=groups,cn=accounts,dc=lab,dc=example,dc=net
[Tue Jun 07 12:21:24.367843 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: Requested attrs_list ['memberofindirect', 'membermanager', 'description', 'memberindirect', 'cn', 'ipaexternalmember', 'memberof', 'gidnumber', 'member']
[Tue Jun 07 12:21:24.370391 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: DROP: cn=testgroup,cn=groups,cn=accounts,dc=lab,dc=example,dc=net
[Tue Jun 07 12:21:24.370461 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: DROP: not in cache cn=testgroup,cn=groups,cn=accounts,dc=lab,dc=example,dc=net
[Tue Jun 07 12:21:24.370604 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: ADD: cn=testgroup,cn=groups,cn=accounts,dc=lab,dc=example,dc=net:

all=False
[Tue Jun 07 12:21:24.370649 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: MISS: Hits 0 Misses 1 Size 1
[Tue Jun 07 12:21:24.372157 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: Cache lookup: cn=lab.example.net,cn=ad,cn=etc,dc=lab,dc=example,dc=net
[Tue Jun 07 12:21:24.372223 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: Requested attrs_list ['ipantflatname', 'ipantsecurityidentifier']
[Tue Jun 07 12:21:24.372952 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: DROP: cn=lab.example.net,cn=ad,cn=etc,dc=lab,dc=example,dc=net
[Tue Jun 07 12:21:24.373006 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: DROP: not in cache cn=lab.example.net,cn=ad,cn=etc,dc=lab,dc=example,dc=net
[Tue Jun 07 12:21:24.373147 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: ADD: cn=lab.example.net,cn=ad,cn=etc,dc=lab,dc=example,dc=net:

all=False
[Tue Jun 07 12:21:24.373182 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: MISS: Hits 0 Misses 2 Size 2
[Tue Jun 07 12:21:24.373259 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: Converting SID to object name: S-1-5-21-1435538835-437086063-3443703549-1122 – Fails here
[Tue Jun 07 12:21:24.381163 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: Searching AD DC LDAP
[Tue Jun 07 12:21:24.402218 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: WSGI wsgi_execute PublicError: Traceback (most recent call last):
[Tue Jun 07 12:21:24.402242 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] File "/usr/lib/python3.6/site-packages/ipaserver/rpcserver.py", line 407, in wsgi_execute
[Tue Jun 07 12:21:24.402246 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] result = command(*args, **options)
[Tue Jun 07 12:21:24.402247 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] File "/usr/lib/python3.6/site-packages/ipalib/frontend.py", line 471, in _call_
[Tue Jun 07 12:21:24.402250 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] return self.__do_call(*args, **options)
[Tue Jun 07 12:21:24.402251 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] File "/usr/lib/python3.6/site-packages/ipalib/frontend.py", line 499, in __do_call
[Tue Jun 07 12:21:24.402253 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ret = self.run(*args, **options)
[Tue Jun 07 12:21:24.402254 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] File "/usr/lib/python3.6/site-packages/ipalib/frontend.py", line 821, in run
[Tue Jun 07 12:21:24.402259 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] return self.execute(*args, **options)
[Tue Jun 07 12:21:24.402261 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] File "/usr/lib/python3.6/site-packages/ipaserver/plugins/baseldap.py", line 1438, in execute
[Tue Jun 07 12:21:24.402263 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] self.obj.convert_attribute_members(entry_attrs, *keys, **options)
[Tue Jun 07 12:21:24.402264 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] File "/usr/lib/python3.6/site-packages/ipaserver/plugins/baseldap.py", line 754, in convert_attribute_members
[Tue Jun 07 12:21:24.402266 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] new_value = ldap_obj.get_primary_key_from_dn(memberdn)
[Tue Jun 07 12:21:24.402267 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] File "/usr/lib/python3.6/site-packages/ipaserver/plugins/idviews.py", line 878, in get_primary_key_from_dn
[Tue Jun 07 12:21:24.402269 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] dn[0].value)
[Tue Jun 07 12:21:24.402270 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] File "/usr/lib/python3.6/site-packages/ipaserver/plugins/idviews.py", line 678, in resolve_anchor_to_object_name
[Tue Jun 07 12:21:24.402272 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] name = domain_validator.get_trusted_domain_object_from_sid(sid)
[Tue Jun 07 12:21:24.402273 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] File "/usr/lib/python3.6/site-packages/ipaserver/dcerpc.py", line 521, in get_trusted_domain_object_from_sid
[Tue Jun 07 12:21:24.402275 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] attrs=attrs)
[Tue Jun 07 12:21:24.402276 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] File "/usr/lib/python3.6/site-packages/ipaserver/dcerpc.py", line 411, in get_trusted_domain_objects
[Tue Jun 07 12:21:24.402277 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] raise errors.NotFound(reason=_('trusted domain object not found'))
[Tue Jun 07 12:21:24.402279 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipalib.errors.NotFound: trusted domain object not found
[Tue Jun 07 12:21:24.402284 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846]
[Tue Jun 07 12:21:24.402394 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: INFO: [jsonserver_session] admin@LAB.EXAMPLE.NET: group_show/1('testgroup', version='2.245'): NotFound
[Tue Jun 07 12:21:24.402436 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: [jsonserver_session] admin@LAB.EXAMPLE.NET: group_show/1('testgroup', version='2.245'): NotFound etime=35075442
[Tue Jun 07 12:21:24.402957 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: FINAL: Hits 0 Misses 2 Size 2
[Tue Jun 07 12:21:24.403054 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: Destroyed connection context.ldap2_140347501990128

[WORKAROUND]

1. We should remove from IPA this override of the deleted domain user.

[Tue Jun 07 12:21:24.373259 2022] [wsgi:error] [pid 5462:tid 140347472721664] [remote 192.168.122.249:55846] ipa: DEBUG: Converting SID to object name: S-1-5-21-1435538835-437086063-3443703549-1122 – Fails here

2. Check this SID on ipa idoverrideuser-find.

1. ipa idoverrideuser-find 'Default Trust View' --all

--------------------------
1 User ID override matched
--------------------------
dn: ipaanchoruuid=:SID:S-1-5-21-1435538835-437086063-3443703549-1122,cn=Default Trust View,cn=views,cn=accounts,dc=lab,dc=example,dc=net
Anchor to override: :SID:S-1-5-21-1435538835-437086063-3443703549-1122
Member of groups: testgroup
ipaoriginaluid: dcamilo@EXAMPLE.NET
objectclass: ipaOverrideAnchor, top, ipaUserOverride, ipasshuser, ipaSshGroupOfPubKeys, nsmemberof
----------------------------
Number of entries returned 1
----------------------------

3. Remove this override.

1. ipa idoverrideuser-del 'Default Trust View' :SID:S-1-5-21-1435538835-437086063-3443703549-1122
   -----------------------------------------------------------------------------
   Deleted User ID override ":SID:S-1-5-21-1435538835-437086063-3443703549-1122"
   -----------------------------------------------------------------------------

4. Check if the group-show will work again as expected.

Group is shown again without issues and the removed user.

[root@ipa-master ~]# ipa group-show testgroup
Group name: testgroup
GID: 1712000005
Member users: admin

Actual results:

ipa group-show is presenting only the below message.

'ipa: ERROR: trusted domain object not found'

Expected results:

That ipa group-show point to the correct override domain user not found when checking the trust, specifically (ipaoriginaluid). So the customer can take action and check if the user was deleted or moved to another OU on AD.