-
Story
-
Resolution: Unresolved
-
Major
-
None
-
False
-
-
False
-
-
Goal:
The instructlab app containers have this block to work around a problem with PyArrow and jemalloc:
# ImportError: /usr/lib64/libjemalloc.so.2: cannot allocate memory in static TLS block # # The recommendation is to build jemalloc with the --disable-initial-exec-tls option # A workaround is to preload jemalloc when invoking Python commands ENV LD_PRELOAD=/usr/lib64/libjemalloc.so.2
We need to find a more permanent solution that does not use LD_PRELOAD. Preloading is a short-term workaround but not a long-term solution. Amongst others Intel recommends tcmalloc for Gaudi. For RHELAI 1.3 we have to force jemalloc.
See https://github.com/jemalloc/jemalloc/issues/937 for upstream bug report.
Acceptance Criteria:
InstructLab and other consumers of PyArrow work without LD_PRELOAD env var.
Possible solutions:
- build PyArrow without jemalloc bindings
- rebuild jemalloc with option --disable-initial-exec-tls
Both options have pros and cons.
- is related to
-
RHELAI-2391 Intel Gaudi3: ImportError: /lib64/libjemalloc.so.2: cannot allocate memory in static TLS block
- Closed