Corrupt heap is detected on shutdown:

Crash details:

On shutdown the qd_link_t alloc_pool is corrupt. Backtrace:

(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007ff4ec1ff5b9 in __GI_abort () at abort.c:79
#2 0x00007ff4ec2577b7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ff4ec3611ac "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3 0x00007ff4ec25dc0c in malloc_printerr (str=str@entry=0x7ff4ec35f3c2 "free(): invalid pointer") at malloc.c:5337
#4 0x00007ff4ec261a3e in free_check (mem=<optimized out>, caller=<optimized out>) at hooks.c:274
#5 0x00007ff4ed6ae585 in qd_alloc_finalize () at /home/kgiusti/work/dispatch/qpid-dispatch/src/alloc_pool.c:488
#6 0x00007ff4ed6b8530 in qd_dispatch_free (qd=0xd5b1c0) at /home/kgiusti/work/dispatch/qpid-dispatch/src/dispatch.c:363
#7 0x0000000000401896 in main_process (
{{ config_path=0x7ffed82ceae2 "/home/kgiusti/work/dispatch/dispatch-tester/benchmarks/test-configurations/two-hop/qdrouterd-B.conf", python_pkgdir=<optimized out>, }}
test_hooks=<optimized out>, fd=2) at /home/kgiusti/work/dispatch/qpid-dispatch/router/src/main.c:116
#8 0x00000000004015d9 in main (argc=3, argv=0x7ffed82ce4a8) at /home/kgiusti/work/dispatch/qpid-dispatch/router/src/main.c:369

In the context of qd_alloc_finalize() we can see that the pool descriptor is for qd_link_t:

(gdb) p *desc
{{$6 = {header = 3735928559, type_name = 0x7ff4ed6f638b "qd_link_t", type_size = 104, additional_size = 0x0, total_size = 104, }}
{{ config = 0x7ff4ed917580 <qd_alloc_default_config_small>, stats = 0x1116180, global_pool = 0x0, lock = 0x11e4c00, tpool_list = {head = 0x12d7d40, tail = 0x12d7d40, }}
scratch = 0x11e4cc0, size = 1}, trailer = 3133065982}

And the local pool is corrupt:

(gdb) print *tpool
{{$1 = {prev = 0x0, next = 0x0, free_list = {top_chunk = 0x1815490, top = 0, size = 64, base_chunk = {prev = 0x6666666666666666, items = {0x12d7e80, 0x12d7f80, 0x12d8080, }}
{{ 0x12d8180, 0x12d8280, 0x12d8380, 0x12d8480, 0x12d8580, 0x12d8680, 0x12d8780, 0x12d8880, 0x12d8980, 0x12d8a80, 0x12d8b80, 0x12d8c80, 0x12d8d80}, }}
next = 0x12d8f50}}}

Note well: to reproduce it is necessary to enable run time checking of glibc's heap. Do this by setting the following environment variables:

$ export MALLOC_PERTURB_="153"
$ export MALLOC_CHECK_="3"