To Reproduce

1. Run:

   ilab.sh model evaluate --benchmark mmlu_branch ... --tasks-dir <dir with attached yaml file>
   

1. The run ends with the following error:

   An error occurred during evaluation: Sample larger than population or is negative
   

Expected behavior

• The benchmark ends successfully.

Device Info (please complete the following information):

• Hardware Specs: x86_64, 8xNvidia A100, 
• OS Version: RHEL AI 1.5

  • registry.redhat.io/rhelai1/bootc-nvidia-rhel9:1.5-1747337172

    instructlab==0.26.1
    instructlab-dolomite==0.2.0
    instructlab-eval==0.5.1
    instructlab-quantize==0.1.0
    instructlab-schema==0.4.2
    instructlab-sdg==0.8.2
    instructlab-training==0.10.3
    
    

Bug impact

• Can't evaluate using MMLU Branch benchmark

Additional context
Attaching:

• log file

  eval_error_log.txt

  at the beginning there is the complete command used.

• yaml fie

  knowledge_technology_cloud-computing_8_4__Creating_a_container_using_virtctl_guestfs_task.yaml

  with the task. Use --tasks-dir parameter to directory with this file to reproduce

• mmlubech jsonl file – referenced from the yaml file.

Notes:

• The yaml file and jsonl file were generated by InstructLab SDG, the source data are attached in RHELAI-4280 
• I'm able to reproduce it, let me know if you need additional debugs.