Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-42415

With one CPU, named failed to start with "hp.c:156: INSIST(tid() < isc__hp_max_threads) failed, back trace"

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • rhel-9.4
    • bind
    • None
    • None
    • Moderate
    • rhel-sst-cs-net-perf-services
    • ssg_core_services
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • Red Hat Enterprise Linux
    • None
    • None
    • None
    • None

      What were you trying to do that didn't work?

      Installing IDM server fails on server with only one single-core CPU.
      It failed when restarting named. Corresponding code.

      Restarting named
      Named service failed to start (CalledProcessError(Command ['/bin/systemctl', 'restart', 'named.service'] returned non-zero exit status 1: 'Job for named.service failed because the control process exited with error code.\nSee "systemctl status named.service" and "journalctl -xeu named.service" for details.\n'))
      named service failed to start
      ...
      The ipa-server-install command failed. See /var/log/ipaserver-install.log for more information
      Named service failed to start
      

      Code dump

      Core was generated by `/usr/sbin/named -u named -c /etc/named.conf -E pkcs11'.
      Program terminated with signal SIGABRT, Aborted.
      #0  0x00007f101f28b94c in __pthread_kill_implementation () from /lib64/libc.so.6
      [Current thread is 1 (Thread 0x7f10178ed640 (LWP 47094))]
      Missing separate debuginfos, use: dnf debuginfo-install bind-dyndb-ldap-11.9-8.el9_3.3.x86_64 cyrus-sasl-gssapi-2.1.27-21.el9.x86_64 cyrus-sasl-lib-2.1.27-21.el9.x86_64 cyrus-sasl-md5-2.1.27-21.el9.x86_64 cyrus-sasl-plain-2.1.27-21.el9.x86_64 fstrm-0.6.1-3.el9.x86_64 gdbm-libs-1.19-4.el9.x86_64 glibc-2.34-100.el9.x86_64 keyutils-libs-1.6.3-1.el9.x86_64 krb5-libs-1.21.1-1.el9.x86_64 libcap-2.48-9.el9_2.x86_64 libcom_err-1.46.5-5.el9.x86_64 libevent-2.1.12-6.el9.x86_64 libffi-3.4.2-8.el9.x86_64 libgcc-11.4.1-3.el9.x86_64 libmaxminddb-1.5.2-3.el9.x86_64 libselinux-3.6-1.el9.x86_64 libuuid-2.37.4-18.el9.x86_64 libuv-1.42.0-1.el9.x86_64 libxcrypt-4.4.18-3.el9.x86_64 libxml2-2.9.13-5.el9_3.x86_64 lmdb-libs-0.9.29-3.el9.x86_64 openssl-libs-3.0.7-27.el9.x86_64 openssl-pkcs11-0.4.11-9.el9.x86_64 p11-kit-0.25.3-2.el9.x86_64 pcre2-10.40-5.el9.x86_64 protobuf-c-1.3.3-13.el9.x86_64 sssd-client-2.9.4-2.el9.x86_64 systemd-libs-252-32.el9_4.x86_64 zlib-1.2.11-40.el9.x86_64
      
      (gdb) bt full
      #0  0x00007f101f28b94c in __pthread_kill_implementation () from /lib64/libc.so.6
      No symbol table info available.
      #1  0x00007f101f23e646 in raise () from /lib64/libc.so.6
      No symbol table info available.
      #2  0x00007f101f2287f3 in abort () from /lib64/libc.so.6
      No symbol table info available.
      #3  0x000055aa382de5b5 in assertion_failed (file=<optimized out>, line=<optimized out>, type=<optimized out>, cond=<optimized out>) at ../../../bin/named/main.c:270
              tracebuf = {0x55aa382e1621 <assertion_failed+97>, 0x7f101fd8b4e0 <isc_assertion_failed+16>, 0x7f101fd8fa4d, 0x7f101fdbc99c <isc_queue_enqueue+60>, 0x7f101fda7d15 <isc__nm_enqueue_ievent+229>,
                0x7f101fdc50fc <isc_task_sendto+204>, 0x7f101c01fb93 <syncrepl_update+435>, 0x7f101c020354 <ldap_sync_search_entry+1220>, 0x7f1017a0cc5a <ldap_sync_search_entry+410>, 0x7f1017a0d9a3 <ldap_sync_init+867>,
                0x7f101c021585 <ldap_sync_doit+1013>, 0x7f101c021998 <ldap_syncrepl_watcher.lto_priv+360>, 0x7f101fdc468a <isc__trampoline_run+26>, 0x7f101f289c02 <start_thread+722>, 0x7f101f30ec40 <clone3+48>,
                0x7f101f30ec40 <clone3+48>, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7f10183b9cf0, 0x1, 0x42756621, 0x7f10178ebc98, 0x3000000000, 0x0, 0xffffffffffffffff, 0xffffffffffffffff, 0x1, 0x7f10183b9cf0, 0x4f713d47a8975400,
                0x312d653663322d63, 0x7f101e39b248, 0x7f101e3ab630, 0x7f101e3b4970, 0x7f10183b9cf0, 0x7f101e39b248, 0x7f101e3ab630, 0x7f101fec745c <detachnode+172>, 0x7f10178ebd20, 0x7f101fda17b3 <isc_log_doit+211>,
                0x7f10200341e0 <dns_modules+480>, 0x14ffffffff, 0x7f1020034230 <dns_categories+16>, 0x0, 0x7f101c02df88, 0x0, 0x7f10178ebff0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x7f10183b9cf0, 0x1, 0x0, 0x0, 0x0, 0x30, 0x3,
                0x7f101e3ab630, 0x7f10178ebff0, 0x30002e, 0x3, 0x7f101fec235e <zone_findrdataset+350>, 0x7f10178ebf20, 0x0, 0x7f10178ebf20, 0x7f101fe57630 <dns_db_findrdataset+112>, 0x7f101e3ae640, 0x4f713d47a8975400,
                0xffffffffffffffff, 0x7f101e3ab630, 0x7f101e3ab630, 0x7f10178ebff0, 0x7f10178ec210, 0x7f101c09ab10, 0x7f101e3ae640, 0x7f101febff7e <iszonesecure+94>, 0x7f10178ebff0, 0x0, 0x0, 0x0, 0x10001, 0x4f713d4700000000,
                0x7f10178ebf20, 0x7f10178ebf20, 0xffffffffffffffff, 0xffffffffffffffff, 0xebebebebebebebea, 0xebebebebebebebeb, 0xebebebebebebebeb, 0xebebebebebebebeb, 0x7f10178ebf08, 0x4f713d47a8975400, 0x7f10178ebf20, 0x7f101c08a160,
                0x7f101c0b5fd0, 0x7f101e39b230, 0x7f10178ebf08, 0x7f101c022c0a <mldap_entry_create+474>, 0x7f10178ec1a0, 0x7f10178ebff0, 0x7f10178ec0c8, 0x7f101e39b230, 0x7f10178ebff0, 0x7f1000000008, 0x7f10178ebff0, 0x1000100000008,
                0x0, 0x0, 0x0, 0x4f713d47a8975400, 0x7f1042756621, 0x7f10178ec000, 0x30000000ff, 0x0, 0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0x4f713d47a8975400}
              i = <optimized out>
              nframes = 15
              result = <optimized out>
              logsuffix = <optimized out>
              fname = 0x0
      #4  0x00007f101fd8b4e0 in isc_assertion_failed (file=file@entry=0x7f101fddb0ea "../../../lib/isc/hp.c", line=line@entry=156, type=type@entry=isc_assertiontype_insist, cond=cond@entry=0x7f101fddb0ce "tid() < isc__hp_max_threads")
          at ../../../lib/isc/assertions.c:46
      No locals.
      #5  0x00007f101fd8fa4d in isc_hp_protect (hp=0x7f101e3a0100, ihp=<optimized out>, atom=0x7f101e39f400) at ../../../lib/isc/hp.c:156
              n = 0
              ret = <optimized out>
              __atomic_store_ptr = <optimized out>
              __atomic_store_tmp = <optimized out>
              __atomic_load_ptr = <optimized out>
              __atomic_load_tmp = <optimized out>
      #6  0x00007f101fdbc99c in isc_queue_enqueue (item=<optimized out>, queue=<optimized out>) at ../../../lib/isc/queue.c:140
              lt = 0x0
              idx = <optimized out>
              n = 0
      #7  isc_queue_enqueue (queue=0x7f101e39f380, item=139707203456544) at ../../../lib/isc/queue.c:132
              lt = <optimized out>
              idx = <optimized out>
              n = <optimized out>
              lnext = <optimized out>
              __atomic_load_ptr = <optimized out>
              __atomic_load_tmp = <optimized out>
              __atomic_load_ptr = <optimized out>
              __atomic_load_tmp = <optimized out>
              newnode = <optimized out>
              __atomic_compare_exchange_ptr = <optimized out>
              __atomic_compare_exchange_tmp = <optimized out>
      #8  0x00007f101fda7d15 in isc__nm_enqueue_ievent (worker=0x55aa3a25da20, event=<optimized out>) at netmgr/../../../../lib/isc/netmgr/netmgr.c:1143
      No locals.
      #9  0x00007f101fdc50fc in task_ready (task=0x7f101e3b7010) at ../../../lib/isc/task.c:351
              manager = 0x7f101e3a7010
              manager = <optimized out>
              __v = <optimized out>
      #10 isc_task_sendto (task=0x7f101e3b7010, eventp=<optimized out>, c=-1) at ../../../lib/isc/task.c:500
              was_idle = <optimized out>
      #11 0x00007f101c01fb93 in syncrepl_update () from /usr/lib64/bind/ldap.so
      No symbol table info available.
      #12 0x00007f101c020354 in ldap_sync_search_entry () from /usr/lib64/bind/ldap.so
      No symbol table info available.
      #13 0x00007f1017a0cc5a in ldap_sync_search_entry () from /lib64/libldap.so.2
      No symbol table info available.
      #14 0x00007f1017a0d9a3 in ldap_sync_init () from /lib64/libldap.so.2
      No symbol table info available.
      #15 0x00007f101c021585 in ldap_sync_doit () from /usr/lib64/bind/ldap.so
      No symbol table info available.
      #16 0x00007f101c021998 in ldap_syncrepl_watcher.lto_priv () from /usr/lib64/bind/ldap.so
      No symbol table info available.
      #17 0x00007f101fdc468a in isc__trampoline_run (arg=0x7f10183c3d90) at ../../../lib/isc/trampoline.c:196
              trampoline = 0x7f10183c3d90
              result = <optimized out>
      #18 0x00007f101f289c02 in start_thread () from /lib64/libc.so.6
      No symbol table info available.
      #19 0x00007f101f30ec40 in clone3 () from /lib64/libc.so.6
      No symbol table info available.
      

      Please provide the package NVR for which bug is seen:

      bind-9.16.23-18.el9_4.1.x86_64
      bind-dyndb-ldap-11.9-9.el9_4.x86_64
      

      How reproducible:

      Always with exactly one CPU.

      Steps to reproduce

      1. ipa-server-install -n ipa.test -r IPA.TEST -p Secret.123 -a Secret.123 -U --setup-dns --auto-forwarder
      2. edit /et/named/ipa-logging-ext.conf and set severity to debug
      3. restart named with systemctl restart named.service

      Expected results

      named restart normally

      Actual results

      Named failed to start with

      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: command channel listening on ::1#953
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: managed-keys-zone: loaded serial 1
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: ../../../lib/isc/hp.c:156: INSIST(tid() < isc__hp_max_threads) failed, back trace
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: #0 0x55aa382e1621 in ??
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: #1 0x7f101fd8b4e0 in ??
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: #2 0x7f101fd8fa4d in ??
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: #3 0x7f101fdbc99c in ??
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: #4 0x7f101fda7d15 in ??
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: #5 0x7f101fdc50fc in ??
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: #6 0x7f101c01fb93 in ??
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: #7 0x7f101c020354 in ??
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: #8 0x7f1017a0cc5a in ??
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: #9 0x7f1017a0d9a3 in ??
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: #10 0x7f101c021585 in ??
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: #11 0x7f101c021998 in ??
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: #12 0x7f101fdc468a in ??
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: #13 0x7f101f289c02 in ??
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: #14 0x7f101f30ec40 in ??
      Jun 17 16:45:11 ipa9-single-node.example9.com named[47089]: exiting (due to assertion failure)
      Jun 17 16:45:11 ipa9-single-node.example9.com systemd[1]: named.service: Control process exited, code=exited, status=1/FAILURE
      Jun 17 16:45:11 ipa9-single-node.example9.com systemd[1]: named.service: Failed with result 'exit-code'.
      

              pemensik@redhat.com Petr Mensik
              rhn-support-dchen Ding Yi Chen
              Petr Mensik Petr Mensik
              rhel-cs-infra-services-qe rhel-cs-infra-services-qe rhel-cs-infra-services-qe rhel-cs-infra-services-qe
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: