It appears the customer is hitting the below Bugzilla:

https://bz.apache.org/bugzilla/show_bug.cgi?id=60947 ^{[Segfault on startup when using mod_ssl with APR-crypto]}

mod_ssl's use of CRYPTO_THREADID_set_callback() is unfortunately bugged if another module has loaded OpenSSL, either via the apr_crypto routines or some other pathway.

When mod_ssl loads, it calls CRYPTO_THREADID_set_callback() as part of setting up the thread-safety routines in OpenSSL. On unload, it attempts to unset this callback by calling the registration function with a NULL argument. This always fails, and the function returns zero, because unfortunately this API is write-once.

If mod_ssl is the only OpenSSL consumer, then libcrypto will be unloaded and reloaded and the OpenSSL static initialization will reset the callback, masking any issue. But if something else has loaded OpenSSL as well (say, by using the apr_crypto_* API), that threadid callback will now be pointing at junk.

Even after this, there's still a way to "succeed" – if mod_ssl is reloaded into exactly the same position, everything will proceed as normal. But if not, we'll segfault the next time we call into OpenSSL.

Note: jbcs-httpd24-apr-util-ldap is calling apr_crypto routines.

The provided corefile shows the above symptomps where we can see the callback routine 'threadid_callback' pointing to improper address 0x7f953b8ab9b0:

(gdb) bt
#0  0x00007f953b8ab9b4 in ?? () from /opt/rh/jbcs-httpd24/root/usr/lib64/httpd/modules/mod_ssl.so
#1  0x000055afb9dbe188 in ?? ()
#2  0x000055afb9deb3a8 in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x00007f953efe4425 in ERR_get_state () at err.c:1030
#5  0x00007f953efe466f in ERR_clear_error () at err.c:746
#6  0x00007f953efd576f in ENGINE_load_dynamic () at eng_dyn.c:327
#7  0x00007f953efd30f3 in ENGINE_load_builtin_engines () at eng_all.c:82
#8  0x00007f953b88b775 in ssl_hook_pre_config (pconf=0x55afb9dbe188, plog=<optimized out>, ptemp=<optimized out>) at mod_ssl.c:404
#9  0x000055afb9503f9e in ap_run_pre_config (pconf=pconf@entry=0x55afb9dbe188, plog=0x55afb9deb3a8, ptemp=0x55afb9de5338) at config.c:89
#10 0x000055afb94dedc6 in main (argc=2, argv=0x7fff57a37a68) at main.c:751
(gdb)
(gdb) f 4
#4  0x00007f953efe4425 in ERR_get_state () at err.c:1030
1030        CRYPTO_THREADID_current(&tid);
(gdb)
(gdb) list CRYPTO_THREADID_current
488         return threadid_callback;
489     }
490
491     void CRYPTO_THREADID_current(CRYPTO_THREADID *id)
492     {
493         if (threadid_callback) {
494             threadid_callback(id);   <<<=== crashes here executing improper address 0x7f953b8ab9b0
495             return;
496         }
497     #ifndef OPENSSL_NO_DEPRECATED
(gdb)
(gdb) p /x threadid_callback
$1 = 0x7f953b8ab9b0
(gdb)
(gdb) disassemble 0x7f953b8ab9b0
No function contains specified address.   <<<=== no code at 0x7f953b8ab9b0
(gdb)

no longer pointing to the previously set callback routine, but now to a .rodata section for mod_ssl.so which holds constant variables and not code to which 'threadid_callback' should have pointed to. 0x7f953b8ab9b0 is in address range of memory region mapped for .rodata section of mod_ssl.so:

(gdb) i files
...
        0x00007f953b8a66c0 - 0x00007f953b8ae530 is .rodata in /opt/rh/jbcs-httpd24/root/usr/lib64/httpd/modules/mod_ssl.so