Using device nvidia.com/gpu=all CUDA_VISIBLE_DEVICES: Command: podman run --rm -it --device nvidia.com/gpu=all --security-opt label=disable --net host --shm-size 10G --pids-limit -1 -v /var/home/vkadlec:/var/home/vkadlec -v /run/user/4218633/containers/auth.json:/run/containers/0/auth.json --env HF_TOKEN --env HOME --env NCCL_DEBUG --env VLLM_LOGGING_LEVEL --env CUDA_VISIBLE_DEVICES --entrypoint ilab registry.redhat.io/rhelai1/instructlab-nvidia-rhel9:1.5.1-1749157760 model evaluate --benchmark mmlu_branch --model /var/home/vkadlec/.local/share/instructlab/checkpoints/hf_format/samples_1743298 --tasks-dir /var/home/vkadlec/.local/share/instructlab/datasets/eval_mmlu_error --base-model /var/home/vkadlec/.cache/instructlab/models/granite-3.1-8b-starter-v2 INFO 2025-06-17 09:39:27,249 instructlab.model.evaluate:584: Using local model found at '/var/home/vkadlec/.local/share/instructlab/checkpoints/hf_format/samples_1743298' for '--model' INFO 2025-06-17 09:39:27,391 instructlab.model.evaluate:584: Using local model found at '/var/home/vkadlec/.cache/instructlab/models/granite-3.1-8b-starter-v2' for '--base-model' INFO 2025-06-17 09:39:27,724 numexpr.utils:146: Note: detected 96 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable. INFO 2025-06-17 09:39:27,724 numexpr.utils:149: Note: NumExpr detected 96 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16. INFO 2025-06-17 09:39:27,724 numexpr.utils:162: NumExpr defaulting to 16 threads. INFO 2025-06-17 09:39:28,741 datasets:54: PyTorch version 2.6.0 available. WARNING 2025-06-17 09:39:38,382 instructlab.model.evaluate:773: Using gpus from --gpus or config and ignoring --tensor-parallel-size configured in serve vllm_args INFO 2025-06-17 09:39:39,008 instructlab.model.backends.vllm:115: Trying to connect to model server at http://127.0.0.1:8001/v1 INFO 2025-06-17 09:39:40,426 instructlab.model.backends.vllm:332: vLLM starting up on pid 101 at http://127.0.0.1:45733/v1 INFO 2025-06-17 09:39:40,426 instructlab.model.backends.vllm:123: Starting a temporary vLLM server at http://127.0.0.1:45733/v1 INFO 2025-06-17 09:39:40,426 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 1/200 INFO 2025-06-17 09:39:43,726 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 2/200 INFO 2025-06-17 09:39:47,153 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 3/200 INFO 2025-06-17 09:39:50,546 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 4/200 INFO 2025-06-17 09:39:53,786 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 5/200 INFO 2025-06-17 09:39:56,960 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 6/200 INFO 2025-06-17 09:40:00,366 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 7/200 INFO 2025-06-17 09:40:03,700 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 8/200 INFO 2025-06-17 09:40:07,041 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 9/200 INFO 2025-06-17 09:40:10,215 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 10/200 INFO 2025-06-17 09:40:13,494 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 11/200 INFO 2025-06-17 09:40:16,873 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 12/200 INFO 2025-06-17 09:40:20,171 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 13/200 INFO 2025-06-17 09:40:23,513 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 14/200 INFO 2025-06-17 09:40:26,796 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 15/200 INFO 2025-06-17 09:40:30,228 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 16/200 INFO 2025-06-17 09:40:33,640 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 17/200 INFO 2025-06-17 09:40:36,914 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 18/200 INFO 2025-06-17 09:40:40,205 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 19/200 INFO 2025-06-17 09:40:43,387 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 20/200 INFO 2025-06-17 09:40:46,535 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 21/200 INFO 2025-06-17 09:40:49,826 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:45733/v1, this might take a moment... Attempt: 22/200 INFO 2025-06-17 09:40:49,831 instructlab.model.backends.vllm:145: vLLM engine successfully started at http://127.0.0.1:45733/v1 DEBUG 2025-06-17 09:40:52,835 lm_eval.tasks:523: File _evalita-mp_ner_adg.yaml in /opt/app-root/lib64/python3.11/site-packages/lm_eval/tasks/evalita_llm could not be loaded DEBUG 2025-06-17 09:40:52,839 lm_eval.tasks:523: File _evalita-mp_ner_fic.yaml in /opt/app-root/lib64/python3.11/site-packages/lm_eval/tasks/evalita_llm could not be loaded DEBUG 2025-06-17 09:40:52,844 lm_eval.tasks:523: File _evalita-mp_ner_wn.yaml in /opt/app-root/lib64/python3.11/site-packages/lm_eval/tasks/evalita_llm could not be loaded INFO 2025-06-17 09:41:01,251 lm_eval.evaluator:169: Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234 INFO 2025-06-17 09:41:01,252 lm_eval.evaluator:206: Initializing local-completions model, with arguments: {'base_url': 'http://127.0.0.1:45733/v1/completions', 'model': '/var/home/vkadlec/.local/share/instructlab/checkpoints/hf_format/samples_1743298', 'tokenizer_backend': 'huggingface'} WARNING 2025-06-17 09:41:01,252 lm_eval.models.api_models:103: Automatic batch size is not supported for API models. Defaulting to batch size 1. INFO 2025-06-17 09:41:01,252 lm_eval.models.api_models:115: Using max length 2048 - 1 INFO 2025-06-17 09:41:01,252 lm_eval.models.api_models:118: Concurrent requests are disabled. To enable concurrent requests, set `num_concurrent` > 1. INFO 2025-06-17 09:41:01,252 lm_eval.models.api_models:133: Using tokenizer huggingface DEBUG 2025-06-17 09:41:01,351 urllib3.connectionpool:1049: Starting new HTTPS connection (1): s3.amazonaws.com:443 DEBUG 2025-06-17 09:41:01,376 urllib3.connectionpool:544: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/datasets/json/json.py HTTP/1.1" 200 0 DEBUG 2025-06-17 09:41:01,383 filelock:331: Attempting to acquire lock 140250090717968 on /var/home/vkadlec/.cache/huggingface/datasets/_var_home_vkadlec_.cache_huggingface_datasets_json_default-83b908bb159d1f7b_0.0.0_f4e89e8750d5d5ffbef2c078bf0ddfedef29dc2faff52a6255cf513c05eb1092.lock DEBUG 2025-06-17 09:41:01,383 filelock:334: Lock 140250090717968 acquired on /var/home/vkadlec/.cache/huggingface/datasets/_var_home_vkadlec_.cache_huggingface_datasets_json_default-83b908bb159d1f7b_0.0.0_f4e89e8750d5d5ffbef2c078bf0ddfedef29dc2faff52a6255cf513c05eb1092.lock DEBUG 2025-06-17 09:41:01,383 fsspec.local:347: open file: /var/home/vkadlec/.cache/huggingface/datasets/json/default-83b908bb159d1f7b/0.0.0/f4e89e8750d5d5ffbef2c078bf0ddfedef29dc2faff52a6255cf513c05eb1092/dataset_info.json DEBUG 2025-06-17 09:41:01,383 filelock:364: Attempting to release lock 140250090717968 on /var/home/vkadlec/.cache/huggingface/datasets/_var_home_vkadlec_.cache_huggingface_datasets_json_default-83b908bb159d1f7b_0.0.0_f4e89e8750d5d5ffbef2c078bf0ddfedef29dc2faff52a6255cf513c05eb1092.lock DEBUG 2025-06-17 09:41:01,384 filelock:367: Lock 140250090717968 released on /var/home/vkadlec/.cache/huggingface/datasets/_var_home_vkadlec_.cache_huggingface_datasets_json_default-83b908bb159d1f7b_0.0.0_f4e89e8750d5d5ffbef2c078bf0ddfedef29dc2faff52a6255cf513c05eb1092.lock DEBUG 2025-06-17 09:41:01,399 filelock:331: Attempting to acquire lock 140250896736592 on /var/home/vkadlec/.cache/huggingface/datasets/json/default-83b908bb159d1f7b/0.0.0/f4e89e8750d5d5ffbef2c078bf0ddfedef29dc2faff52a6255cf513c05eb1092_builder.lock DEBUG 2025-06-17 09:41:01,399 filelock:334: Lock 140250896736592 acquired on /var/home/vkadlec/.cache/huggingface/datasets/json/default-83b908bb159d1f7b/0.0.0/f4e89e8750d5d5ffbef2c078bf0ddfedef29dc2faff52a6255cf513c05eb1092_builder.lock DEBUG 2025-06-17 09:41:01,399 fsspec.local:347: open file: /var/home/vkadlec/.cache/huggingface/datasets/json/default-83b908bb159d1f7b/0.0.0/f4e89e8750d5d5ffbef2c078bf0ddfedef29dc2faff52a6255cf513c05eb1092/dataset_info.json DEBUG 2025-06-17 09:41:01,399 filelock:364: Attempting to release lock 140250896736592 on /var/home/vkadlec/.cache/huggingface/datasets/json/default-83b908bb159d1f7b/0.0.0/f4e89e8750d5d5ffbef2c078bf0ddfedef29dc2faff52a6255cf513c05eb1092_builder.lock DEBUG 2025-06-17 09:41:01,399 filelock:367: Lock 140250896736592 released on /var/home/vkadlec/.cache/huggingface/datasets/json/default-83b908bb159d1f7b/0.0.0/f4e89e8750d5d5ffbef2c078bf0ddfedef29dc2faff52a6255cf513c05eb1092_builder.lock DEBUG 2025-06-17 09:41:01,402 lm_eval.api.task:842: No custom filters defined. Using default 'take_first' filter for handling repeats. WARNING 2025-06-17 09:41:01,402 lm_eval.api.task:327: [Task: knowledge_technology_cloud-computing_8_4__Creating_a_container_using_virtctl_guestfs] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended. WARNING 2025-06-17 09:41:01,402 lm_eval.api.task:327: [Task: knowledge_technology_cloud-computing_8_4__Creating_a_container_using_virtctl_guestfs] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended. WARNING 2025-06-17 09:41:01,407 lm_eval.evaluator:275: Overwriting default num_fewshot of knowledge_technology_cloud-computing_8_4__Creating_a_container_using_virtctl_guestfs from None to 5 WARNING 2025-06-17 09:41:01,408 lm_eval.evaluator:421: Chat template formatting change affects loglikelihood and multiple-choice tasks. See docs/chat-template-readme.md for details. INFO 2025-06-17 09:41:01,408 lm_eval.api.task:420: Building contexts for knowledge_technology_cloud-computing_8_4__Creating_a_container_using_virtctl_guestfs on rank 0... 0%| | 0/4 [00:00