Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-14760

ns-slapd crash in vlvIndex_delete

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Normal Normal
    • rhel-9.5
    • rhel-9.2.0
    • 389-ds-base
    • 389-ds-base-2.5.2-1.el9
    • None
    • Important
    • rhel-sst-idm-ds
    • ssg_idm
    • 25
    • 0
    • QE ack, Dev ack
    • False
    • Hide

      None

      Show
      None
    • Yes
    • None
    • Release Note Not Required
    • Hide
      Cause: The dse.ldif config file was somehow corrupted.

      Consequence: The directory server instance fails to start because it crashs in vlvIndex_delete function.

      Workaround: Restore a proper config file by copying
      /etc/dirsrv/slapd-<InstanceName>/dse.ldif.startOK into
      /etc/dirsrv/slapd-<InstanceName>/dse.ldif
      before restarting the instance
      Show
      Cause: The dse.ldif config file was somehow corrupted. Consequence: The directory server instance fails to start because it crashs in vlvIndex_delete function. Workaround: Restore a proper config file by copying /etc/dirsrv/slapd-<InstanceName>/dse.ldif.startOK into /etc/dirsrv/slapd-<InstanceName>/dse.ldif before restarting the instance
    • Proposed
    • x86_64
    • None

      What were you trying to do that didn't work?

      Customer was upgrading a RHEL 9 system and IPA LDAP server was failing to start.

      From the dnf log:

      $ grep 389-ds var/log/dnf.rpm.log 
      2023-08-09T09:14:59+0100 SUBDEBUG Installed: 389-ds-base-libs-2.2.4-3.el9.x86_64
      2023-08-09T09:15:22+0100 SUBDEBUG Installed: 389-ds-base-2.2.4-3.el9.x86_64
      2023-10-25T08:02:13+0100 SUBDEBUG Upgrade: 389-ds-base-libs-2.2.4-5.el9_2.x86_64
      2023-10-25T08:02:13+0100 SUBDEBUG Upgrade: 389-ds-base-2.2.4-5.el9_2.x86_64
      2023-10-25T08:02:51+0100 SUBDEBUG Upgraded: 389-ds-base-2.2.4-3.el9.x86_64
      2023-10-25T08:02:51+0100 SUBDEBUG Upgraded: 389-ds-base-libs-2.2.4-3.el9.x86_64
      $
      

      The core statcktrace is

                  Stack trace of thread 9575:
                  #0  0x00007f0b8db30f13 vlvIndex_delete (libback-ldbm.so + 0x8bf13)
                  #1  0x00007f0b8db320e7 vlv_init_index_entry (libback-ldbm.so + 0x8d0e7)
                  #2  0x00007f0b919a1f52 dse_call_callback.isra.0 (libslapd.so.0 + 0x1a1f52)
                  #3  0x00007f0b9190c1fb dse_search (libslapd.so.0 + 0x10c1fb)
                  #4  0x00007f0b9194df68 op_shared_search (libslapd.so.0 + 0x14df68)
                  #5  0x00007f0b9195be5d search_internal_callback_pb (libslapd.so.0 + 0x15be5d)
                  #6  0x00007f0b9195c12f search_internal_pb (libslapd.so.0 + 0x15c12f)
                  #7  0x00007f0b9195c321 slapi_search_internal (libslapd.so.0 + 0x15c321)
                  #8  0x00007f0b8db314f8 vlv_init (libback-ldbm.so + 0x8c4f8)
                  #9  0x00007f0b8db044ac ldbm_instance_startall (libback-ldbm.so + 0x5f4ac)
                  #10 0x00007f0b8db2fb03 ldbm_back_start (libback-ldbm.so + 0x8ab03)
                  #11 0x00007f0b91957117 plugin_call_func.lto_priv.0 (libslapd.so.0 + 0x157117)
                  #12 0x00007f0b91958e20 plugin_startall (libslapd.so.0 + 0x158e20)
                  #13 0x0000556b1d870639 main (ns-slapd + 0x14639)
                  #14 0x00007f0b9143feb0 __libc_start_call_main (libc.so.6 + 0x3feb0)
                  #15 0x00007f0b9143ff60 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3ff60)
                  #16 0x0000556b1d8719b5 _start (ns-slapd + 0x159b5)
                  
      

      Please provide the package NVR for which bug is seen:

      $ cat etc/redhat-release 
      Red Hat Enterprise Linux release 9.2 (Plow)
      $
      $ grep 389-ds installed-rpms 
      389-ds-base-2.2.4-5.el9_2.x86_64                            Wed Oct 25 08:02:13 2023
      389-ds-base-libs-2.2.4-5.el9_2.x86_64                       Wed Oct 25 08:02:13 2023
      $
      

      How reproducible:

      The crash happened at a customer site.

      Steps to reproduce

      1) There was some DB corruption ( unsure about what caused it ). LDAP errors log was showing these messages:

      [25/Oct/2023:08:36:24.902261899 +0100] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
      [25/Oct/2023:08:36:25.810690300 +0100] - WARN - vlv_init_index_entry - Parent doesn't exist for entry cn=allRevokedOrRevokedExpiredCaCerts-pki-tomcatIndex,cn=allRevokedOrRevokedExpiredCaCerts-pki-tomcat,cn=ipaca,cn=ldbm database,cn=plugins,cn=config.
      ...
      [25/Oct/2023:08:47:51.097498463 +0100] - NOTICE - bdb_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
      [25/Oct/2023:08:47:52.009212186 +0100] - WARN - vlv_init_index_entry - Parent doesn't exist for entry cn=allRevokedOrRevokedExpiredCaCerts-pki-tomcatIndex,cn=allRevokedOrRevokedExpiredCaCerts-pki-tomcat,cn=ipaca,cn=ldbm database,cn=plugins,cn=config.
      

      2) Any attempt to start the LDAP server was failing
      3) The parent entry was at the very end of the dse.ldif file and it was corrupt:

      $ tail -14 etc/dirsrv/slapd-<INSTANCE>/dse.ldif
      dn: cn=allRevokedOrRevokedExpiredCaCerts-pki-tomcat,cn=ipaca,cn=ldbm databas
       omcat,cn=ipaca,cn=ldbm database,cn=plugins,cn=config
      cn: allRevokedOrRevokedExpiredCaCerts-pki-tomcat
      createTimestamp: 20230809082308Z
      creatorsName: cn=directory manager
      modifiersName: cn=directory manager
      modifyTimestamp: 20230809082308Z
      objectClass: top
      objectClass: vlvSearch
      vlvBase: ou=certificateRepository,ou=ca,o=ipaca
      vlvFilter: (&(|(certSD_EXPIRED))(extension=2.5.29.19;*isCA=true*))
      vlvScope: 1
      
      $
      

      Expected results

      Running LDAP server.

      Actual results

      LDAP server is crashing.

              progier Pierre Rogier
              rhn-support-tmihinto Têko Mihinto
              IdM DS Dev IdM DS Dev
              IdM DS QE IdM DS QE
              Evgenia Martyniuk Evgenia Martyniuk
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: