INFO 2025-05-19 13:28:15,070 instructlab.model.backends.vllm:115: Trying to connect to model server at http://127.0.0.1:8001/v1
INFO 2025-05-19 13:28:16,845 instructlab.model.backends.vllm:332: vLLM starting up on pid 82 at http://127.0.0.1:50665/v1
INFO 2025-05-19 13:28:16,846 instructlab.model.backends.vllm:123: Starting a temporary vLLM server at http://127.0.0.1:50665/v1
INFO 2025-05-19 13:28:16,846 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 1/200
INFO 2025-05-19 13:28:20,044 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 2/200
INFO 2025-05-19 13:28:23,324 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 3/200
INFO 2025-05-19 13:28:26,651 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 4/200
INFO 2025-05-19 13:28:29,961 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 5/200
INFO 2025-05-19 13:28:33,316 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 6/200
INFO 2025-05-19 13:28:36,706 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 7/200
INFO 2025-05-19 13:28:40,095 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 8/200
INFO 2025-05-19 13:28:43,441 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 9/200
INFO 2025-05-19 13:28:46,740 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 10/200
INFO 2025-05-19 13:28:50,150 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 11/200
INFO 2025-05-19 13:28:53,437 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 12/200
INFO 2025-05-19 13:28:56,772 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 13/200
INFO 2025-05-19 13:29:00,052 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 14/200
INFO 2025-05-19 13:29:03,419 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 15/200
INFO 2025-05-19 13:29:06,600 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 16/200
INFO 2025-05-19 13:29:09,977 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 17/200
INFO 2025-05-19 13:29:13,295 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 18/200
INFO 2025-05-19 13:29:16,672 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 19/200
INFO 2025-05-19 13:29:19,892 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 20/200
INFO 2025-05-19 13:29:23,196 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 21/200
INFO 2025-05-19 13:29:26,455 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 22/200
INFO 2025-05-19 13:29:29,763 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 23/200
INFO 2025-05-19 13:29:33,124 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 24/200
INFO 2025-05-19 13:29:36,427 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 25/200
INFO 2025-05-19 13:29:39,774 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 26/200
INFO 2025-05-19 13:29:42,983 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 27/200
INFO 2025-05-19 13:29:46,228 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 28/200
INFO 2025-05-19 13:29:49,442 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 29/200
INFO 2025-05-19 13:29:52,623 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 30/200
INFO 2025-05-19 13:29:55,860 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 31/200
INFO 2025-05-19 13:29:59,111 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 32/200
INFO 2025-05-19 13:30:02,346 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 33/200
INFO 2025-05-19 13:30:05,780 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 34/200
INFO 2025-05-19 13:30:09,050 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 35/200
INFO 2025-05-19 13:30:12,409 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 36/200
INFO 2025-05-19 13:30:15,605 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 37/200
INFO 2025-05-19 13:30:18,940 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 38/200
INFO 2025-05-19 13:30:22,197 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 39/200
INFO 2025-05-19 13:30:25,483 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 40/200
INFO 2025-05-19 13:30:28,658 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 41/200
INFO 2025-05-19 13:30:31,893 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 42/200
INFO 2025-05-19 13:30:35,153 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 43/200
INFO 2025-05-19 13:30:38,391 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 44/200
INFO 2025-05-19 13:30:41,846 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 45/200
INFO 2025-05-19 13:30:45,030 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 46/200
INFO 2025-05-19 13:30:48,409 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 47/200
INFO 2025-05-19 13:30:51,610 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 48/200
INFO 2025-05-19 13:30:54,836 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 49/200
INFO 2025-05-19 13:30:58,117 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 50/200
INFO 2025-05-19 13:31:01,552 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 51/200
INFO 2025-05-19 13:31:04,694 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 52/200
INFO 2025-05-19 13:31:08,115 instructlab.model.backends.vllm:138: Waiting for the vLLM server to start at http://127.0.0.1:50665/v1, this might take a moment... Attempt: 53/200
INFO 2025-05-19 13:31:08,121 instructlab.model.backends.vllm:145: vLLM engine successfully started at http://127.0.0.1:50665/v1
INFO 2025-05-19 13:31:09,391 instructlab:206: Generating synthetic data using '/usr/share/instructlab/sdg/pipelines/agentic' pipeline, '/var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1' model, '/var/home/vkadlec/.local/share/instructlab/taxonomy' taxonomy, against http://127.0.0.1:50665/v1 server
INFO 2025-05-19 13:31:09,864 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:09,865 instructlab.sdg.utils.taxonomy:148: Pattern '6.2. Configuring automatic workload updates.md' matched 1 files.
INFO 2025-05-19 13:31:09,865 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_6_2_1__About_workload_updates_lfnf809f/6.2. Configuring automatic workload updates.md
INFO 2025-05-19 13:31:09,865 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_6_2_1__About_workload_updates_lfnf809f/6.2. Configuring automatic workload updates.md
INFO 2025-05-19 13:31:10,155 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:10,155 instructlab.sdg.utils.taxonomy:148: Pattern '9.1. Creating virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:10,155 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_1_6__About_RunStrategies_for_virtual_machines_ruphq17g/9.1. Creating virtual machines.md
INFO 2025-05-19 13:31:10,155 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_1_6__About_RunStrategies_for_virtual_machines_ruphq17g/9.1. Creating virtual machines.md
INFO 2025-05-19 13:31:10,467 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:10,467 instructlab.sdg.utils.taxonomy:148: Pattern '13.3. Viewing events.md' matched 1 files.
INFO 2025-05-19 13:31:10,467 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_3_1__About_virtual_machine_events_o73yk5qr/13.3. Viewing events.md
INFO 2025-05-19 13:31:10,467 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_3_1__About_virtual_machine_events_o73yk5qr/13.3. Viewing events.md
INFO 2025-05-19 13:31:10,760 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:10,760 instructlab.sdg.utils.taxonomy:148: Pattern '8.5. Libguestfs tools and virtctl guestfs.md' matched 1 files.
INFO 2025-05-19 13:31:10,760 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_8_5__Libguestfs_tools_and_virtctl_guestfs_tw30n4by/8.5. Libguestfs tools and virtctl guestfs.md
INFO 2025-05-19 13:31:10,760 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_8_5__Libguestfs_tools_and_virtctl_guestfs_tw30n4by/8.5. Libguestfs tools and virtctl guestfs.md
INFO 2025-05-19 13:31:11,046 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:11,047 instructlab.sdg.utils.taxonomy:148: Pattern '9.12. Managing config maps, secrets, and service accounts in virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:11,047 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_12_2__Removing_a_secret__config_map__or_service_account_from_a_virtual_machine__ywty_b9/9.12. Managing config maps, secrets, and service accounts in virtual machines.md
INFO 2025-05-19 13:31:11,047 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_12_2__Removing_a_secret__config_map__or_service_account_from_a_virtual_machine__ywty_b9/9.12. Managing config maps, secrets, and service accounts in virtual machines.md
INFO 2025-05-19 13:31:11,333 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:11,333 instructlab.sdg.utils.taxonomy:148: Pattern '9.16. Advanced virtual machine management.md' matched 1 files.
INFO 2025-05-19 13:31:11,333 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_16_10__Configuring_vGPU_passthrough_6xcbdytn/9.16. Advanced virtual machine management.md
INFO 2025-05-19 13:31:11,333 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_16_10__Configuring_vGPU_passthrough_6xcbdytn/9.16. Advanced virtual machine management.md
INFO 2025-05-19 13:31:11,635 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:11,635 instructlab.sdg.utils.taxonomy:148: Pattern '4.3. New and changed features.md' matched 1 files.
INFO 2025-05-19 13:31:11,635 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_3_2__Storage_438hx__n/4.3. New and changed features.md
INFO 2025-05-19 13:31:11,635 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_3_2__Storage_438hx__n/4.3. New and changed features.md
INFO 2025-05-19 13:31:11,909 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:11,909 instructlab.sdg.utils.taxonomy:148: Pattern '5.4. Installing OpenShift Virtualization using the CLI.md' matched 1 files.
INFO 2025-05-19 13:31:11,909 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_4_1__Prerequisites_i__pkhk9/5.4. Installing OpenShift Virtualization using the CLI.md
INFO 2025-05-19 13:31:11,909 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_4_1__Prerequisites_i__pkhk9/5.4. Installing OpenShift Virtualization using the CLI.md
INFO 2025-05-19 13:31:12,210 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:12,210 instructlab.sdg.utils.taxonomy:148: Pattern '9.6. Controlling virtual machine states.md' matched 1 files.
INFO 2025-05-19 13:31:12,210 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_6_2__Restarting_a_virtual_machine_v6msz7sg/9.6. Controlling virtual machine states.md
INFO 2025-05-19 13:31:12,210 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_6_2__Restarting_a_virtual_machine_v6msz7sg/9.6. Controlling virtual machine states.md
INFO 2025-05-19 13:31:12,517 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:12,517 instructlab.sdg.utils.taxonomy:148: Pattern '5.2. Specifying nodes for OpenShift Virtualization components.md' matched 1 files.
INFO 2025-05-19 13:31:12,517 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_2_2__Example_manifests_eckfezys/5.2. Specifying nodes for OpenShift Virtualization components.md
INFO 2025-05-19 13:31:12,517 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_2_2__Example_manifests_eckfezys/5.2. Specifying nodes for OpenShift Virtualization components.md
INFO 2025-05-19 13:31:12,820 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:12,820 instructlab.sdg.utils.taxonomy:148: Pattern '13.2. Viewing OpenShift Virtualization logs.md' matched 1 files.
INFO 2025-05-19 13:31:12,820 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_2_2__Viewing_virtual_machine_logs_in_the_web_console_0p2tq4g8/13.2. Viewing OpenShift Virtualization logs.md
INFO 2025-05-19 13:31:12,820 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_2_2__Viewing_virtual_machine_logs_in_the_web_console_0p2tq4g8/13.2. Viewing OpenShift Virtualization logs.md
INFO 2025-05-19 13:31:13,105 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:13,105 instructlab.sdg.utils.taxonomy:148: Pattern '9.12. Managing config maps, secrets, and service accounts in virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:13,105 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_12__Managing_config_maps__secrets__and_service_accounts_in_virtual_machines_7np9gqbn/9.12. Managing config maps, secrets, and service accounts in virtual machines.md
INFO 2025-05-19 13:31:13,105 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_12__Managing_config_maps__secrets__and_service_accounts_in_virtual_machines_7np9gqbn/9.12. Managing config maps, secrets, and service accounts in virtual machines.md
INFO 2025-05-19 13:31:13,382 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:13,382 instructlab.sdg.utils.taxonomy:148: Pattern '9.5. Managing virtual machine instances.md' matched 1 files.
INFO 2025-05-19 13:31:13,382 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_5_6__Deleting_a_standalone_virtual_machine_instance_using_the_web_console_bvrmft5p/9.5. Managing virtual machine instances.md
INFO 2025-05-19 13:31:13,382 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_5_6__Deleting_a_standalone_virtual_machine_instance_using_the_web_console_bvrmft5p/9.5. Managing virtual machine instances.md
INFO 2025-05-19 13:31:13,691 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:13,691 instructlab.sdg.utils.taxonomy:148: Pattern 'OpenShift Virtualization installation, usage, and release notes.md' matched 1 files.
INFO 2025-05-19 13:31:13,691 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_OpenShift_Virtualization_installation__usage__and_release_notes_4p57w_w1/OpenShift Virtualization installation, usage, and release notes.md
INFO 2025-05-19 13:31:13,691 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_OpenShift_Virtualization_installation__usage__and_release_notes_4p57w_w1/OpenShift Virtualization installation, usage, and release notes.md
INFO 2025-05-19 13:31:14,013 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:14,013 instructlab.sdg.utils.taxonomy:148: Pattern '4.1. Making open source more inclusive.md' matched 1 files.
INFO 2025-05-19 13:31:14,013 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_1__Making_open_source_more_inclusive_vgcc55ab/4.1. Making open source more inclusive.md
INFO 2025-05-19 13:31:14,013 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_1__Making_open_source_more_inclusive_vgcc55ab/4.1. Making open source more inclusive.md
INFO 2025-05-19 13:31:14,309 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:14,309 instructlab.sdg.utils.taxonomy:148: Pattern '9.20. Virtual machine disks.md' matched 1 files.
INFO 2025-05-19 13:31:14,309 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_20_20__Expanding_a_virtual_machine_disk_8gt0tmiq/9.20. Virtual machine disks.md
INFO 2025-05-19 13:31:14,309 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_20_20__Expanding_a_virtual_machine_disk_8gt0tmiq/9.20. Virtual machine disks.md
INFO 2025-05-19 13:31:14,584 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:14,584 instructlab.sdg.utils.taxonomy:148: Pattern '9.8. Automating Windows installation with sysprep.md' matched 1 files.
INFO 2025-05-19 13:31:14,584 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_8_3__Generalizing_a_Windows_VM_using_sysprep_acljvqsp/9.8. Automating Windows installation with sysprep.md
INFO 2025-05-19 13:31:14,584 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_8_3__Generalizing_a_Windows_VM_using_sysprep_acljvqsp/9.8. Automating Windows installation with sysprep.md
INFO 2025-05-19 13:31:14,868 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:14,868 instructlab.sdg.utils.taxonomy:148: Pattern '9.19. Virtual machine networking.md' matched 1 files.
INFO 2025-05-19 13:31:14,868 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_19_7__Viewing_the_IP_address_of_NICs_on_a_virtual_machine_u1cf0aa4/9.19. Virtual machine networking.md
INFO 2025-05-19 13:31:14,869 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_19_7__Viewing_the_IP_address_of_NICs_on_a_virtual_machine_u1cf0aa4/9.19. Virtual machine networking.md
INFO 2025-05-19 13:31:15,166 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:15,166 instructlab.sdg.utils.taxonomy:148: Pattern '5.1. Preparing your cluster for OpenShift Virtualization.md' matched 1 files.
INFO 2025-05-19 13:31:15,166 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_1_4__Restricted_network_environments_p37pv3h7/5.1. Preparing your cluster for OpenShift Virtualization.md
INFO 2025-05-19 13:31:15,167 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_1_4__Restricted_network_environments_p37pv3h7/5.1. Preparing your cluster for OpenShift Virtualization.md
INFO 2025-05-19 13:31:15,427 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:15,427 instructlab.sdg.utils.taxonomy:148: Pattern '10.2. Editing virtual machine templates.md' matched 1 files.
INFO 2025-05-19 13:31:15,428 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_10_2__Editing_virtual_machine_templates_x7x1ezkg/10.2. Editing virtual machine templates.md
INFO 2025-05-19 13:31:15,428 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_10_2__Editing_virtual_machine_templates_x7x1ezkg/10.2. Editing virtual machine templates.md
INFO 2025-05-19 13:31:15,715 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:15,715 instructlab.sdg.utils.taxonomy:148: Pattern '9.11. Viewing the QEMU guest agent information for virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:15,715 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_11_2__About_the_QEMU_guest_agent_information_in_the_web_console_zifjuxrp/9.11. Viewing the QEMU guest agent information for virtual machines.md
INFO 2025-05-19 13:31:15,715 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_11_2__About_the_QEMU_guest_agent_information_in_the_web_console_zifjuxrp/9.11. Viewing the QEMU guest agent information for virtual machines.md
INFO 2025-05-19 13:31:16,005 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:16,005 instructlab.sdg.utils.taxonomy:148: Pattern '13.8. Reviewing resource usage by virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:16,005 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_8_1__About_reviewing_top_consumers_rtdupvg0/13.8. Reviewing resource usage by virtual machines.md
INFO 2025-05-19 13:31:16,005 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_8_1__About_reviewing_top_consumers_rtdupvg0/13.8. Reviewing resource usage by virtual machines.md
INFO 2025-05-19 13:31:16,324 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:16,324 instructlab.sdg.utils.taxonomy:148: Pattern '5.3. Installing OpenShift Virtualization using the web console.md' matched 1 files.
INFO 2025-05-19 13:31:16,324 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_3__Installing_OpenShift_Virtualization_using_the_web_console_sujbbhbi/5.3. Installing OpenShift Virtualization using the web console.md
INFO 2025-05-19 13:31:16,325 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_3__Installing_OpenShift_Virtualization_using_the_web_console_sujbbhbi/5.3. Installing OpenShift Virtualization using the web console.md
INFO 2025-05-19 13:31:16,680 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:16,680 instructlab.sdg.utils.taxonomy:148: Pattern '13.3. Viewing events.md' matched 1 files.
INFO 2025-05-19 13:31:16,681 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_3_4__Viewing_resource_events_in_the_CLI_o80u6_vv/13.3. Viewing events.md
INFO 2025-05-19 13:31:16,681 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_3_4__Viewing_resource_events_in_the_CLI_o80u6_vv/13.3. Viewing events.md
INFO 2025-05-19 13:31:16,994 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:16,994 instructlab.sdg.utils.taxonomy:148: Pattern '13.1. Reviewing Virtualization Overview.md' matched 1 files.
INFO 2025-05-19 13:31:16,994 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_1_2__Resources_monitored_actively_in_the_Virtualization_Overview_page_v_cteebq/13.1. Reviewing Virtualization Overview.md
INFO 2025-05-19 13:31:16,994 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_1_2__Resources_monitored_actively_in_the_Virtualization_Overview_page_v_cteebq/13.1. Reviewing Virtualization Overview.md
INFO 2025-05-19 13:31:17,290 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:17,290 instructlab.sdg.utils.taxonomy:148: Pattern '1.1. What you can do with OpenShift Virtualization.md' matched 1 files.
INFO 2025-05-19 13:31:17,290 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_1_1__What_you_can_do_with_OpenShift_Virtualization_sss2aa5p/1.1. What you can do with OpenShift Virtualization.md
INFO 2025-05-19 13:31:17,290 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_1_1__What_you_can_do_with_OpenShift_Virtualization_sss2aa5p/1.1. What you can do with OpenShift Virtualization.md
INFO 2025-05-19 13:31:17,578 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:17,578 instructlab.sdg.utils.taxonomy:148: Pattern '9.2. Editing virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:17,578 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_2_5__Adding_a_network_interface_to_a_virtual_machine_xiirwq4z/9.2. Editing virtual machines.md
INFO 2025-05-19 13:31:17,578 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_2_5__Adding_a_network_interface_to_a_virtual_machine_xiirwq4z/9.2. Editing virtual machines.md
INFO 2025-05-19 13:31:17,905 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:17,905 instructlab.sdg.utils.taxonomy:148: Pattern '4.2. About Red Hat OpenShift Virtualization.md' matched 1 files.
INFO 2025-05-19 13:31:17,905 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_2_2__Supported_guest_operating_systems_7xy4qg3e/4.2. About Red Hat OpenShift Virtualization.md
INFO 2025-05-19 13:31:17,905 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_2_2__Supported_guest_operating_systems_7xy4qg3e/4.2. About Red Hat OpenShift Virtualization.md
INFO 2025-05-19 13:31:18,213 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:18,213 instructlab.sdg.utils.taxonomy:148: Pattern '13.6. Monitoring virtual machine health.md' matched 1 files.
INFO 2025-05-19 13:31:18,213 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_6_6__Additional_resources_vii026il/13.6. Monitoring virtual machine health.md
INFO 2025-05-19 13:31:18,213 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_6_6__Additional_resources_vii026il/13.6. Monitoring virtual machine health.md
INFO 2025-05-19 13:31:18,514 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:18,514 instructlab.sdg.utils.taxonomy:148: Pattern '14.3. Backing up virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:18,514 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_14_3_3__Additional_resources_u1bxxw8w/14.3. Backing up virtual machines.md
INFO 2025-05-19 13:31:18,514 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_14_3_3__Additional_resources_u1bxxw8w/14.3. Backing up virtual machines.md
INFO 2025-05-19 13:31:18,805 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:18,806 instructlab.sdg.utils.taxonomy:148: Pattern '13.11. Prometheus queries for virtual resources.md' matched 1 files.
INFO 2025-05-19 13:31:18,806 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_11_4__Additional_resources_d7byglr6/13.11. Prometheus queries for virtual resources.md
INFO 2025-05-19 13:31:18,806 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_11_4__Additional_resources_d7byglr6/13.11. Prometheus queries for virtual resources.md
INFO 2025-05-19 13:31:19,128 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:19,128 instructlab.sdg.utils.taxonomy:148: Pattern '10.5. Deleting virtual machine templates.md' matched 1 files.
INFO 2025-05-19 13:31:19,128 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_10_5__Deleting_virtual_machine_templates_es7q3vch/10.5. Deleting virtual machine templates.md
INFO 2025-05-19 13:31:19,128 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_10_5__Deleting_virtual_machine_templates_es7q3vch/10.5. Deleting virtual machine templates.md
INFO 2025-05-19 13:31:19,427 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:19,427 instructlab.sdg.utils.taxonomy:148: Pattern '2.7. About the tekton-tasks-operator.md' matched 1 files.
INFO 2025-05-19 13:31:19,427 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_2_7__About_the_tekton_tasks_operator_s936tkro/2.7. About the tekton-tasks-operator.md
INFO 2025-05-19 13:31:19,427 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_2_7__About_the_tekton_tasks_operator_s936tkro/2.7. About the tekton-tasks-operator.md
INFO 2025-05-19 13:31:19,732 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:19,732 instructlab.sdg.utils.taxonomy:148: Pattern '9.2. Editing virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:19,732 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_2_2__Editing_a_virtual_machine_YAML_configuration_using_the_web_console_yitmje_k/9.2. Editing virtual machines.md
INFO 2025-05-19 13:31:19,733 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_2_2__Editing_a_virtual_machine_YAML_configuration_using_the_web_console_yitmje_k/9.2. Editing virtual machines.md
INFO 2025-05-19 13:31:20,032 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:20,032 instructlab.sdg.utils.taxonomy:148: Pattern '13.5. Viewing information about virtual machine workloads.md' matched 1 files.
INFO 2025-05-19 13:31:20,032 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_5__Viewing_information_about_virtual_machine_workloads_cr7awjmt/13.5. Viewing information about virtual machine workloads.md
INFO 2025-05-19 13:31:20,032 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_5__Viewing_information_about_virtual_machine_workloads_cr7awjmt/13.5. Viewing information about virtual machine workloads.md
INFO 2025-05-19 13:31:20,325 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:20,325 instructlab.sdg.utils.taxonomy:148: Pattern '13.10. Running cluster checkups.md' matched 1 files.
INFO 2025-05-19 13:31:20,325 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_10_1__About_the_OpenShift_Container_Platform_cluster_checkup_framework_maeea3c7/13.10. Running cluster checkups.md
INFO 2025-05-19 13:31:20,325 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_10_1__About_the_OpenShift_Container_Platform_cluster_checkup_framework_maeea3c7/13.10. Running cluster checkups.md
INFO 2025-05-19 13:31:20,605 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:20,605 instructlab.sdg.utils.taxonomy:148: Pattern '13.9. OpenShift Container Platform cluster monitoring, logging, and Telemetry.md' matched 1 files.
INFO 2025-05-19 13:31:20,605 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_9_3__About_Telemetry_x0anhqr7/13.9. OpenShift Container Platform cluster monitoring, logging, and Telemetry.md
INFO 2025-05-19 13:31:20,605 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_9_3__About_Telemetry_x0anhqr7/13.9. OpenShift Container Platform cluster monitoring, logging, and Telemetry.md
INFO 2025-05-19 13:31:20,906 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:20,906 instructlab.sdg.utils.taxonomy:148: Pattern '4.4. Deprecated and removed features.md' matched 1 files.
INFO 2025-05-19 13:31:20,906 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_4_1__Deprecated_features_hugujtfi/4.4. Deprecated and removed features.md
INFO 2025-05-19 13:31:20,907 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_4_1__Deprecated_features_hugujtfi/4.4. Deprecated and removed features.md
INFO 2025-05-19 13:31:21,189 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:21,189 instructlab.sdg.utils.taxonomy:148: Pattern '9.3. Editing boot order.md' matched 1 files.
INFO 2025-05-19 13:31:21,189 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_3_1__Adding_items_to_a_boot_order_list_in_the_web_console_nar_9j2r/9.3. Editing boot order.md
INFO 2025-05-19 13:31:21,189 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_3_1__Adding_items_to_a_boot_order_list_in_the_web_console_nar_9j2r/9.3. Editing boot order.md
INFO 2025-05-19 13:31:21,477 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:21,477 instructlab.sdg.utils.taxonomy:148: Pattern '9.17. Importing virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:21,477 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_17_1__TLS_certificates_for_data_volume_imports_czp5k6li/9.17. Importing virtual machines.md
INFO 2025-05-19 13:31:21,477 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_17_1__TLS_certificates_for_data_volume_imports_czp5k6li/9.17. Importing virtual machines.md
INFO 2025-05-19 13:31:21,745 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:21,745 instructlab.sdg.utils.taxonomy:148: Pattern '11.3. Migrating a virtual machine instance to another node.md' matched 1 files.
INFO 2025-05-19 13:31:21,745 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_11_3__Migrating_a_virtual_machine_instance_to_another_node_euyhodzf/11.3. Migrating a virtual machine instance to another node.md
INFO 2025-05-19 13:31:21,745 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_11_3__Migrating_a_virtual_machine_instance_to_another_node_euyhodzf/11.3. Migrating a virtual machine instance to another node.md
INFO 2025-05-19 13:31:22,027 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:22,027 instructlab.sdg.utils.taxonomy:148: Pattern '4.2. About Red Hat OpenShift Virtualization.md' matched 1 files.
INFO 2025-05-19 13:31:22,027 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_2__About_Red_Hat_OpenShift_Virtualization_hr7gfcly/4.2. About Red Hat OpenShift Virtualization.md
INFO 2025-05-19 13:31:22,027 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_2__About_Red_Hat_OpenShift_Virtualization_hr7gfcly/4.2. About Red Hat OpenShift Virtualization.md
INFO 2025-05-19 13:31:22,325 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:22,325 instructlab.sdg.utils.taxonomy:148: Pattern '8.2. OpenShift Container Platform client commands.md' matched 1 files.
INFO 2025-05-19 13:31:22,325 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_8_2__OpenShift_Container_Platform_client_commands_xdd934lt/8.2. OpenShift Container Platform client commands.md
INFO 2025-05-19 13:31:22,325 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_8_2__OpenShift_Container_Platform_client_commands_xdd934lt/8.2. OpenShift Container Platform client commands.md
INFO 2025-05-19 13:31:22,611 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:22,611 instructlab.sdg.utils.taxonomy:148: Pattern '9.12. Managing config maps, secrets, and service accounts in virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:22,611 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_12_1__Adding_a_secret__config_map__or_service_account_to_a_virtual_machine_g90b92wp/9.12. Managing config maps, secrets, and service accounts in virtual machines.md
INFO 2025-05-19 13:31:22,611 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_12_1__Adding_a_secret__config_map__or_service_account_to_a_virtual_machine_g90b92wp/9.12. Managing config maps, secrets, and service accounts in virtual machines.md
INFO 2025-05-19 13:31:22,906 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:22,906 instructlab.sdg.utils.taxonomy:148: Pattern '13.11. Prometheus queries for virtual resources.md' matched 1 files.
INFO 2025-05-19 13:31:22,906 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_11_3__Virtualization_metrics_lchamwwy/13.11. Prometheus queries for virtual resources.md
INFO 2025-05-19 13:31:22,906 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_11_3__Virtualization_metrics_lchamwwy/13.11. Prometheus queries for virtual resources.md
INFO 2025-05-19 13:31:23,197 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:23,197 instructlab.sdg.utils.taxonomy:148: Pattern '12.3. Managing node labeling for obsolete CPU models.md' matched 1 files.
INFO 2025-05-19 13:31:23,197 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_12_3__Managing_node_labeling_for_obsolete_CPU_models_tm4jp2g0/12.3. Managing node labeling for obsolete CPU models.md
INFO 2025-05-19 13:31:23,197 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_12_3__Managing_node_labeling_for_obsolete_CPU_models_tm4jp2g0/12.3. Managing node labeling for obsolete CPU models.md
INFO 2025-05-19 13:31:23,517 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:23,517 instructlab.sdg.utils.taxonomy:148: Pattern '13.12. Exposing custom metrics for virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:23,517 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_12_5__Additional_resources_t7i67iq1/13.12. Exposing custom metrics for virtual machines.md
INFO 2025-05-19 13:31:23,517 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_12_5__Additional_resources_t7i67iq1/13.12. Exposing custom metrics for virtual machines.md
INFO 2025-05-19 13:31:23,808 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:23,808 instructlab.sdg.utils.taxonomy:148: Pattern '12.4. Preventing node reconciliation.md' matched 1 files.
INFO 2025-05-19 13:31:23,808 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_12_4_1__Using_skip_node_annotation_4ajb4j5o/12.4. Preventing node reconciliation.md
INFO 2025-05-19 13:31:23,808 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_12_4_1__Using_skip_node_annotation_4ajb4j5o/12.4. Preventing node reconciliation.md
INFO 2025-05-19 13:31:24,146 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:24,147 instructlab.sdg.utils.taxonomy:148: Pattern '13.6. Monitoring virtual machine health.md' matched 1 files.
INFO 2025-05-19 13:31:24,147 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_6__Monitoring_virtual_machine_health_u6jb78cw/13.6. Monitoring virtual machine health.md
INFO 2025-05-19 13:31:24,147 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_6__Monitoring_virtual_machine_health_u6jb78cw/13.6. Monitoring virtual machine health.md
INFO 2025-05-19 13:31:24,446 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:24,446 instructlab.sdg.utils.taxonomy:148: Pattern '10.1. Creating virtual machine templates.md' matched 1 files.
INFO 2025-05-19 13:31:24,446 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_10_1_2__About_virtual_machines_and_boot_sources_qgyi2y58/10.1. Creating virtual machine templates.md
INFO 2025-05-19 13:31:24,446 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_10_1_2__About_virtual_machines_and_boot_sources_qgyi2y58/10.1. Creating virtual machine templates.md
INFO 2025-05-19 13:31:25,510 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:25,510 instructlab.sdg.utils.taxonomy:148: Pattern '13.2. Viewing OpenShift Virtualization logs.md' matched 1 files.
INFO 2025-05-19 13:31:25,510 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_2__Viewing_OpenShift_Virtualization_logs_x_c_2axh/13.2. Viewing OpenShift Virtualization logs.md
INFO 2025-05-19 13:31:25,510 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_2__Viewing_OpenShift_Virtualization_logs_x_c_2axh/13.2. Viewing OpenShift Virtualization logs.md
INFO 2025-05-19 13:31:25,784 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:25,784 instructlab.sdg.utils.taxonomy:148: Pattern '2.8. About the virt-operator.md' matched 1 files.
INFO 2025-05-19 13:31:25,784 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_2_8__About_the_virt_operator_snxa6yio/2.8. About the virt-operator.md
INFO 2025-05-19 13:31:25,784 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_2_8__About_the_virt_operator_snxa6yio/2.8. About the virt-operator.md
INFO 2025-05-19 13:31:26,093 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:26,093 instructlab.sdg.utils.taxonomy:148: Pattern '9.14. Installing VirtIO driver on a new Windows virtual machine.md' matched 1 files.
INFO 2025-05-19 13:31:26,093 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_14_4__Adding_VirtIO_drivers_container_disk_to_a_virtual_machine_ct_5doh7/9.14. Installing VirtIO driver on a new Windows virtual machine.md
INFO 2025-05-19 13:31:26,094 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_14_4__Adding_VirtIO_drivers_container_disk_to_a_virtual_machine_ct_5doh7/9.14. Installing VirtIO driver on a new Windows virtual machine.md
INFO 2025-05-19 13:31:26,382 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:26,382 instructlab.sdg.utils.taxonomy:148: Pattern '9.5. Managing virtual machine instances.md' matched 1 files.
INFO 2025-05-19 13:31:26,382 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_5__Managing_virtual_machine_instances_a9ohhn33/9.5. Managing virtual machine instances.md
INFO 2025-05-19 13:31:26,382 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_5__Managing_virtual_machine_instances_a9ohhn33/9.5. Managing virtual machine instances.md
INFO 2025-05-19 13:31:26,661 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:26,662 instructlab.sdg.utils.taxonomy:148: Pattern '13.11. Prometheus queries for virtual resources.md' matched 1 files.
INFO 2025-05-19 13:31:26,662 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_11_1__Prerequisites_y_f0ymvz/13.11. Prometheus queries for virtual resources.md
INFO 2025-05-19 13:31:26,662 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_11_1__Prerequisites_y_f0ymvz/13.11. Prometheus queries for virtual resources.md
INFO 2025-05-19 13:31:26,971 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:26,971 instructlab.sdg.utils.taxonomy:148: Pattern '5.1. Preparing your cluster for OpenShift Virtualization.md' matched 1 files.
INFO 2025-05-19 13:31:26,971 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_1_3__Object_maximums_7e1gjgj7/5.1. Preparing your cluster for OpenShift Virtualization.md
INFO 2025-05-19 13:31:26,971 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_1_3__Object_maximums_7e1gjgj7/5.1. Preparing your cluster for OpenShift Virtualization.md
INFO 2025-05-19 13:31:27,252 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:27,252 instructlab.sdg.utils.taxonomy:148: Pattern '9.14. Installing VirtIO driver on a new Windows virtual machine.md' matched 1 files.
INFO 2025-05-19 13:31:27,252 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_14_5__Installing_VirtIO_drivers_during_Windows_installation_w6aqovba/9.14. Installing VirtIO driver on a new Windows virtual machine.md
INFO 2025-05-19 13:31:27,252 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_14_5__Installing_VirtIO_drivers_during_Windows_installation_w6aqovba/9.14. Installing VirtIO driver on a new Windows virtual machine.md
INFO 2025-05-19 13:31:27,558 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:27,558 instructlab.sdg.utils.taxonomy:148: Pattern '9.16. Advanced virtual machine management.md' matched 1 files.
INFO 2025-05-19 13:31:27,558 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_16_12__Configuring_a_watchdog_pl3trvm7/9.16. Advanced virtual machine management.md
INFO 2025-05-19 13:31:27,558 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_16_12__Configuring_a_watchdog_pl3trvm7/9.16. Advanced virtual machine management.md
INFO 2025-05-19 13:31:27,851 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:27,851 instructlab.sdg.utils.taxonomy:148: Pattern '2.4. About the cluster-network-addons-operator.md' matched 1 files.
INFO 2025-05-19 13:31:27,851 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_2_4__About_the_cluster_network_addons_operator_1oe_rw0h/2.4. About the cluster-network-addons-operator.md
INFO 2025-05-19 13:31:27,851 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_2_4__About_the_cluster_network_addons_operator_1oe_rw0h/2.4. About the cluster-network-addons-operator.md
INFO 2025-05-19 13:31:28,132 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:28,132 instructlab.sdg.utils.taxonomy:148: Pattern '9.15. Using virtual Trusted Platform Module devices.md' matched 1 files.
INFO 2025-05-19 13:31:28,132 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_15_1__About_vTPM_devices_7g1a3mxr/9.15. Using virtual Trusted Platform Module devices.md
INFO 2025-05-19 13:31:28,132 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_15_1__About_vTPM_devices_7g1a3mxr/9.15. Using virtual Trusted Platform Module devices.md
INFO 2025-05-19 13:31:28,418 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:28,418 instructlab.sdg.utils.taxonomy:148: Pattern '12.4. Preventing node reconciliation.md' matched 1 files.
INFO 2025-05-19 13:31:28,418 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_12_4__Preventing_node_reconciliation_usj_nwkc/12.4. Preventing node reconciliation.md
INFO 2025-05-19 13:31:28,418 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_12_4__Preventing_node_reconciliation_usj_nwkc/12.4. Preventing node reconciliation.md
INFO 2025-05-19 13:31:28,714 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:28,714 instructlab.sdg.utils.taxonomy:148: Pattern '9.19. Virtual machine networking.md' matched 1 files.
INFO 2025-05-19 13:31:28,714 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_19__Virtual_machine_networking_508254my/9.19. Virtual machine networking.md
INFO 2025-05-19 13:31:28,714 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_19__Virtual_machine_networking_508254my/9.19. Virtual machine networking.md
INFO 2025-05-19 13:31:28,982 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:28,982 instructlab.sdg.utils.taxonomy:148: Pattern '1.1. What you can do with OpenShift Virtualization.md' matched 1 files.
INFO 2025-05-19 13:31:28,983 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_1_1_1__OpenShift_Virtualization_supported_cluster_version_8drhdt1i/1.1. What you can do with OpenShift Virtualization.md
INFO 2025-05-19 13:31:28,983 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_1_1_1__OpenShift_Virtualization_supported_cluster_version_8drhdt1i/1.1. What you can do with OpenShift Virtualization.md
INFO 2025-05-19 13:31:29,279 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:29,280 instructlab.sdg.utils.taxonomy:148: Pattern '10.4. Deploying a virtual machine template to a custom namespace.md' matched 1 files.
INFO 2025-05-19 13:31:29,280 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_10_4_1__Creating_a_custom_namespace_for_templates_yufe6g2u/10.4. Deploying a virtual machine template to a custom namespace.md
INFO 2025-05-19 13:31:29,280 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_10_4_1__Creating_a_custom_namespace_for_templates_yufe6g2u/10.4. Deploying a virtual machine template to a custom namespace.md
INFO 2025-05-19 13:31:29,581 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:29,581 instructlab.sdg.utils.taxonomy:148: Pattern '4.5. Technology Preview features.md' matched 1 files.
INFO 2025-05-19 13:31:29,581 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_5__Technology_Preview_features_tkvjbxn0/4.5. Technology Preview features.md
INFO 2025-05-19 13:31:29,581 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_5__Technology_Preview_features_tkvjbxn0/4.5. Technology Preview features.md
INFO 2025-05-19 13:31:29,890 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:29,890 instructlab.sdg.utils.taxonomy:148: Pattern '9.10. Installing the QEMU guest agent on virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:29,890 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_10_1__Installing_QEMU_guest_agent_on_a_Linux_virtual_machine_rmtxmqyi/9.10. Installing the QEMU guest agent on virtual machines.md
INFO 2025-05-19 13:31:29,890 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_10_1__Installing_QEMU_guest_agent_on_a_Linux_virtual_machine_rmtxmqyi/9.10. Installing the QEMU guest agent on virtual machines.md
INFO 2025-05-19 13:31:30,178 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:30,178 instructlab.sdg.utils.taxonomy:148: Pattern '9.9. Triggering virtual machine failover by resolving a failed node.md' matched 1 files.
INFO 2025-05-19 13:31:30,178 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_9_2__Deleting_nodes_from_a_bare_metal_cluster_emkzknkv/9.9. Triggering virtual machine failover by resolving a failed node.md
INFO 2025-05-19 13:31:30,178 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_9_2__Deleting_nodes_from_a_bare_metal_cluster_emkzknkv/9.9. Triggering virtual machine failover by resolving a failed node.md
INFO 2025-05-19 13:31:30,463 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:30,463 instructlab.sdg.utils.taxonomy:148: Pattern '11.2. Live migration limits and timeouts.md' matched 1 files.
INFO 2025-05-19 13:31:30,463 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_11_2__Live_migration_limits_and_timeouts_pf26x2ky/11.2. Live migration limits and timeouts.md
INFO 2025-05-19 13:31:30,464 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_11_2__Live_migration_limits_and_timeouts_pf26x2ky/11.2. Live migration limits and timeouts.md
INFO 2025-05-19 13:31:30,756 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:30,756 instructlab.sdg.utils.taxonomy:148: Pattern '9.1. Creating virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:30,756 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_1_7__Additional_resources_i5cbtzpd/9.1. Creating virtual machines.md
INFO 2025-05-19 13:31:30,756 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_1_7__Additional_resources_i5cbtzpd/9.1. Creating virtual machines.md
INFO 2025-05-19 13:31:31,012 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:31,012 instructlab.sdg.utils.taxonomy:148: Pattern '14.1. Installing and configuring OADP.md' matched 1 files.
INFO 2025-05-19 13:31:31,012 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_14_1_1__Installing_the_OADP_Operator_xd0l1j1f/14.1. Installing and configuring OADP.md
INFO 2025-05-19 13:31:31,012 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_14_1_1__Installing_the_OADP_Operator_xd0l1j1f/14.1. Installing and configuring OADP.md
INFO 2025-05-19 13:31:31,295 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:31,295 instructlab.sdg.utils.taxonomy:148: Pattern '5.4. Installing OpenShift Virtualization using the CLI.md' matched 1 files.
INFO 2025-05-19 13:31:31,295 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_4_2__Subscribing_to_the_OpenShift_Virtualization_catalog_by_using_the_CLI_yaz02i1m/5.4. Installing OpenShift Virtualization using the CLI.md
INFO 2025-05-19 13:31:31,295 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_4_2__Subscribing_to_the_OpenShift_Virtualization_catalog_by_using_the_CLI_yaz02i1m/5.4. Installing OpenShift Virtualization using the CLI.md
INFO 2025-05-19 13:31:31,599 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:31,599 instructlab.sdg.utils.taxonomy:148: Pattern '4.3. New and changed features.md' matched 1 files.
INFO 2025-05-19 13:31:31,599 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_3__New_and_changed_features_r07tom5k/4.3. New and changed features.md
INFO 2025-05-19 13:31:31,599 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_3__New_and_changed_features_r07tom5k/4.3. New and changed features.md
INFO 2025-05-19 13:31:31,890 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:31,890 instructlab.sdg.utils.taxonomy:148: Pattern '9.6. Controlling virtual machine states.md' matched 1 files.
INFO 2025-05-19 13:31:31,890 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_6_4__Unpausing_a_virtual_machine_vowyh88e/9.6. Controlling virtual machine states.md
INFO 2025-05-19 13:31:31,890 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_6_4__Unpausing_a_virtual_machine_vowyh88e/9.6. Controlling virtual machine states.md
INFO 2025-05-19 13:31:32,203 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:32,204 instructlab.sdg.utils.taxonomy:148: Pattern '9.1. Creating virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:32,204 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_1_1__Using_a_Quick_Start_to_create_a_virtual_machine_qix_c6k6/9.1. Creating virtual machines.md
INFO 2025-05-19 13:31:32,204 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_1_1__Using_a_Quick_Start_to_create_a_virtual_machine_qix_c6k6/9.1. Creating virtual machines.md
INFO 2025-05-19 13:31:32,477 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:32,477 instructlab.sdg.utils.taxonomy:148: Pattern '2.6. About the ssp-operator.md' matched 1 files.
INFO 2025-05-19 13:31:32,477 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_2_6__About_the_ssp_operator_qxwkvmzq/2.6. About the ssp-operator.md
INFO 2025-05-19 13:31:32,477 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_2_6__About_the_ssp_operator_qxwkvmzq/2.6. About the ssp-operator.md
INFO 2025-05-19 13:31:32,777 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:32,777 instructlab.sdg.utils.taxonomy:148: Pattern '9.2. Editing virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:32,777 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_2_3__Editing_a_virtual_machine_YAML_configuration_using_the_CLI_7f_ysokg/9.2. Editing virtual machines.md
INFO 2025-05-19 13:31:32,777 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_2_3__Editing_a_virtual_machine_YAML_configuration_using_the_CLI_7f_ysokg/9.2. Editing virtual machines.md
INFO 2025-05-19 13:31:33,057 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:33,057 instructlab.sdg.utils.taxonomy:148: Pattern '7.3. Additional OpenShift Container Platform security context constraints and Linux capabilities for the kubevirt-controller service account.md' matched 1 files.
INFO 2025-05-19 13:31:33,057 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_7_3_1__Viewing_the_SCC_and_RBAC_definitions_for_the_kubevirt_controller_0urhgmok/7.3. Additional OpenShift Container Platform security context constraints and Linux capabilities for the kubevirt-controller service account.md
INFO 2025-05-19 13:31:33,057 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_7_3_1__Viewing_the_SCC_and_RBAC_definitions_for_the_kubevirt_controller_0urhgmok/7.3. Additional OpenShift Container Platform security context constraints and Linux capabilities for the kubevirt-controller service account.md
INFO 2025-05-19 13:31:33,336 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:33,336 instructlab.sdg.utils.taxonomy:148: Pattern '4.2. About Red Hat OpenShift Virtualization.md' matched 1 files.
INFO 2025-05-19 13:31:33,336 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_2_1__OpenShift_Virtualization_supported_cluster_version_qkhods2y/4.2. About Red Hat OpenShift Virtualization.md
INFO 2025-05-19 13:31:33,336 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_2_1__OpenShift_Virtualization_supported_cluster_version_qkhods2y/4.2. About Red Hat OpenShift Virtualization.md
INFO 2025-05-19 13:31:33,637 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:33,637 instructlab.sdg.utils.taxonomy:148: Pattern '12.3. Managing node labeling for obsolete CPU models.md' matched 1 files.
INFO 2025-05-19 13:31:33,637 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_12_3_3__Configuring_obsolete_CPU_models_nkllwccd/12.3. Managing node labeling for obsolete CPU models.md
INFO 2025-05-19 13:31:33,637 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_12_3_3__Configuring_obsolete_CPU_models_nkllwccd/12.3. Managing node labeling for obsolete CPU models.md
INFO 2025-05-19 13:31:33,930 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:33,930 instructlab.sdg.utils.taxonomy:148: Pattern '13.7. Using the OpenShift Container Platform dashboard to get cluster information.md' matched 1 files.
INFO 2025-05-19 13:31:33,930 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_7__Using_the_OpenShift_Container_Platform_dashboard_to_get_cluster_information_omxiqv0w/13.7. Using the OpenShift Container Platform dashboard to get cluster information.md
INFO 2025-05-19 13:31:33,930 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_7__Using_the_OpenShift_Container_Platform_dashboard_to_get_cluster_information_omxiqv0w/13.7. Using the OpenShift Container Platform dashboard to get cluster information.md
INFO 2025-05-19 13:31:34,228 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:34,229 instructlab.sdg.utils.taxonomy:148: Pattern '9.9. Triggering virtual machine failover by resolving a failed node.md' matched 1 files.
INFO 2025-05-19 13:31:34,229 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_9_1__Prerequisites_nnswko8u/9.9. Triggering virtual machine failover by resolving a failed node.md
INFO 2025-05-19 13:31:34,229 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_9_1__Prerequisites_nnswko8u/9.9. Triggering virtual machine failover by resolving a failed node.md
INFO 2025-05-19 13:31:34,513 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:34,513 instructlab.sdg.utils.taxonomy:148: Pattern '9.20. Virtual machine disks.md' matched 1 files.
INFO 2025-05-19 13:31:34,513 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_20_19__Re_using_persistent_volumes_k7wkjd0v/9.20. Virtual machine disks.md
INFO 2025-05-19 13:31:34,513 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_20_19__Re_using_persistent_volumes_k7wkjd0v/9.20. Virtual machine disks.md
INFO 2025-05-19 13:31:34,807 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:34,807 instructlab.sdg.utils.taxonomy:148: Pattern '5.5. Enabling the virtctl client.md' matched 1 files.
INFO 2025-05-19 13:31:34,807 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_5__Enabling_the_virtctl_client_zobfo9w5/5.5. Enabling the virtctl client.md
INFO 2025-05-19 13:31:34,807 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_5__Enabling_the_virtctl_client_zobfo9w5/5.5. Enabling the virtctl client.md
INFO 2025-05-19 13:31:35,113 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:35,113 instructlab.sdg.utils.taxonomy:148: Pattern '13.1. Reviewing Virtualization Overview.md' matched 1 files.
INFO 2025-05-19 13:31:35,113 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_1_1__Prerequisites_6ero1yid/13.1. Reviewing Virtualization Overview.md
INFO 2025-05-19 13:31:35,113 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_1_1__Prerequisites_6ero1yid/13.1. Reviewing Virtualization Overview.md
INFO 2025-05-19 13:31:35,404 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:35,404 instructlab.sdg.utils.taxonomy:148: Pattern '5.1. Preparing your cluster for OpenShift Virtualization.md' matched 1 files.
INFO 2025-05-19 13:31:35,405 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_1__Preparing_your_cluster_for_OpenShift_Virtualization_8_fxawud/5.1. Preparing your cluster for OpenShift Virtualization.md
INFO 2025-05-19 13:31:35,405 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_1__Preparing_your_cluster_for_OpenShift_Virtualization_8_fxawud/5.1. Preparing your cluster for OpenShift Virtualization.md
INFO 2025-05-19 13:31:35,669 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:35,669 instructlab.sdg.utils.taxonomy:148: Pattern '14.1. Installing and configuring OADP.md' matched 1 files.
INFO 2025-05-19 13:31:35,669 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_14_1__Installing_and_configuring_OADP_qi3_6jrw/14.1. Installing and configuring OADP.md
INFO 2025-05-19 13:31:35,669 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_14_1__Installing_and_configuring_OADP_qi3_6jrw/14.1. Installing and configuring OADP.md
INFO 2025-05-19 13:31:35,973 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:35,973 instructlab.sdg.utils.taxonomy:148: Pattern '13.9. OpenShift Container Platform cluster monitoring, logging, and Telemetry.md' matched 1 files.
INFO 2025-05-19 13:31:35,973 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_9__OpenShift_Container_Platform_cluster_monitoring__logging__and_Telemetry_2kapdq4s/13.9. OpenShift Container Platform cluster monitoring, logging, and Telemetry.md
INFO 2025-05-19 13:31:35,973 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_9__OpenShift_Container_Platform_cluster_monitoring__logging__and_Telemetry_2kapdq4s/13.9. OpenShift Container Platform cluster monitoring, logging, and Telemetry.md
INFO 2025-05-19 13:31:36,259 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:36,259 instructlab.sdg.utils.taxonomy:148: Pattern '5.3. Installing OpenShift Virtualization using the web console.md' matched 1 files.
INFO 2025-05-19 13:31:36,259 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_3_1__Installing_the_OpenShift_Virtualization_Operator_e6w3txg2/5.3. Installing OpenShift Virtualization using the web console.md
INFO 2025-05-19 13:31:36,259 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_3_1__Installing_the_OpenShift_Virtualization_Operator_e6w3txg2/5.3. Installing OpenShift Virtualization using the web console.md
INFO 2025-05-19 13:31:36,549 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:36,550 instructlab.sdg.utils.taxonomy:148: Pattern '9.19. Virtual machine networking.md' matched 1 files.
INFO 2025-05-19 13:31:36,550 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_19_1__Configuring_the_virtual_machine_for_the_default_pod_network_la8_gvb6/9.19. Virtual machine networking.md
INFO 2025-05-19 13:31:36,550 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_19_1__Configuring_the_virtual_machine_for_the_default_pod_network_la8_gvb6/9.19. Virtual machine networking.md
INFO 2025-05-19 13:31:36,827 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:36,827 instructlab.sdg.utils.taxonomy:148: Pattern '11.7. Configuring virtual machine eviction strategy.md' matched 1 files.
INFO 2025-05-19 13:31:36,827 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_11_7__Configuring_virtual_machine_eviction_strategy_evdblp7v/11.7. Configuring virtual machine eviction strategy.md
INFO 2025-05-19 13:31:36,827 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_11_7__Configuring_virtual_machine_eviction_strategy_evdblp7v/11.7. Configuring virtual machine eviction strategy.md
INFO 2025-05-19 13:31:37,139 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:37,139 instructlab.sdg.utils.taxonomy:148: Pattern '14.2. Backing up and restoring virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:37,139 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_14_2_1__Additional_resources_qdbm67gl/14.2. Backing up and restoring virtual machines.md
INFO 2025-05-19 13:31:37,139 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_14_2_1__Additional_resources_qdbm67gl/14.2. Backing up and restoring virtual machines.md
INFO 2025-05-19 13:31:37,512 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:37,513 instructlab.sdg.utils.taxonomy:148: Pattern '13.10. Running cluster checkups.md' matched 1 files.
INFO 2025-05-19 13:31:37,513 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_10__Running_cluster_checkups_cvn4p3yp/13.10. Running cluster checkups.md
INFO 2025-05-19 13:31:37,513 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_10__Running_cluster_checkups_cvn4p3yp/13.10. Running cluster checkups.md
INFO 2025-05-19 13:31:37,793 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:37,793 instructlab.sdg.utils.taxonomy:148: Pattern '13.12. Exposing custom metrics for virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:37,793 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_12_2__Configuring_a_virtual_machine_with_the_node_exporter_service_hxu6i_66/13.12. Exposing custom metrics for virtual machines.md
INFO 2025-05-19 13:31:37,793 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_12_2__Configuring_a_virtual_machine_with_the_node_exporter_service_hxu6i_66/13.12. Exposing custom metrics for virtual machines.md
INFO 2025-05-19 13:31:38,077 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:38,077 instructlab.sdg.utils.taxonomy:148: Pattern '7.3. Additional OpenShift Container Platform security context constraints and Linux capabilities for the kubevirt-controller service account.md' matched 1 files.
INFO 2025-05-19 13:31:38,077 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_7_3__Additional_OpenShift_Container_Platform_security_context_constraints_and_Linux_capabilities_for_the_kubevirt_controller_service_account_m3la3uh8/7.3. Additional OpenShift Container Platform security context constraints and Linux capabilities for the kubevirt-controller service account.md
INFO 2025-05-19 13:31:38,077 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_7_3__Additional_OpenShift_Container_Platform_security_context_constraints_and_Linux_capabilities_for_the_kubevirt_controller_service_account_m3la3uh8/7.3. Additional OpenShift Container Platform security context constraints and Linux capabilities for the kubevirt-controller service account.md
INFO 2025-05-19 13:31:38,365 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:38,366 instructlab.sdg.utils.taxonomy:148: Pattern '4.7. Known issues.md' matched 1 files.
INFO 2025-05-19 13:31:38,366 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_7__Known_issues_7tn7zg8s/4.7. Known issues.md
INFO 2025-05-19 13:31:38,366 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_4_7__Known_issues_7tn7zg8s/4.7. Known issues.md
INFO 2025-05-19 13:31:38,675 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:38,675 instructlab.sdg.utils.taxonomy:148: Pattern '11.4. Migrating a virtual machine over a dedicated additional network.md' matched 1 files.
INFO 2025-05-19 13:31:38,675 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_11_4__Migrating_a_virtual_machine_over_a_dedicated_additional_network_w0eq_tmy/11.4. Migrating a virtual machine over a dedicated additional network.md
INFO 2025-05-19 13:31:38,675 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_11_4__Migrating_a_virtual_machine_over_a_dedicated_additional_network_w0eq_tmy/11.4. Migrating a virtual machine over a dedicated additional network.md
INFO 2025-05-19 13:31:38,943 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:38,943 instructlab.sdg.utils.taxonomy:148: Pattern '10.1. Creating virtual machine templates.md' matched 1 files.
INFO 2025-05-19 13:31:38,944 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_10_1_3__Creating_a_virtual_machine_template_in_the_web_console_ig9h41by/10.1. Creating virtual machine templates.md
INFO 2025-05-19 13:31:38,944 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_10_1_3__Creating_a_virtual_machine_template_in_the_web_console_ig9h41by/10.1. Creating virtual machine templates.md
INFO 2025-05-19 13:31:39,240 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:39,240 instructlab.sdg.utils.taxonomy:148: Pattern '5.1. Preparing your cluster for OpenShift Virtualization.md' matched 1 files.
INFO 2025-05-19 13:31:39,240 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_1_1__Hardware_and_operating_system_requirements_nd8kes1q/5.1. Preparing your cluster for OpenShift Virtualization.md
INFO 2025-05-19 13:31:39,240 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_1_1__Hardware_and_operating_system_requirements_nd8kes1q/5.1. Preparing your cluster for OpenShift Virtualization.md
INFO 2025-05-19 13:31:39,526 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:39,526 instructlab.sdg.utils.taxonomy:148: Pattern '9.13. Installing VirtIO driver on an existing Windows virtual machine.md' matched 1 files.
INFO 2025-05-19 13:31:39,526 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_13_1__About_VirtIO_drivers_cfnb9wvb/9.13. Installing VirtIO driver on an existing Windows virtual machine.md
INFO 2025-05-19 13:31:39,526 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_13_1__About_VirtIO_drivers_cfnb9wvb/9.13. Installing VirtIO driver on an existing Windows virtual machine.md
INFO 2025-05-19 13:31:39,814 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:39,814 instructlab.sdg.utils.taxonomy:148: Pattern '10.5. Deleting virtual machine templates.md' matched 1 files.
INFO 2025-05-19 13:31:39,814 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_10_5_1__Deleting_a_virtual_machine_template_in_the_web_console_py7cfsws/10.5. Deleting virtual machine templates.md
INFO 2025-05-19 13:31:39,814 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_10_5_1__Deleting_a_virtual_machine_template_in_the_web_console_py7cfsws/10.5. Deleting virtual machine templates.md
INFO 2025-05-19 13:31:40,091 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:40,091 instructlab.sdg.utils.taxonomy:148: Pattern '9.7. Accessing virtual machine consoles.md' matched 1 files.
INFO 2025-05-19 13:31:40,092 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_7_1__Accessing_virtual_machine_consoles_in_the_OpenShift_Container_Platform_web_console_p4a3scwi/9.7. Accessing virtual machine consoles.md
INFO 2025-05-19 13:31:40,092 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_7_1__Accessing_virtual_machine_consoles_in_the_OpenShift_Container_Platform_web_console_p4a3scwi/9.7. Accessing virtual machine consoles.md
INFO 2025-05-19 13:31:40,363 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:40,363 instructlab.sdg.utils.taxonomy:148: Pattern '9.15. Using virtual Trusted Platform Module devices.md' matched 1 files.
INFO 2025-05-19 13:31:40,363 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_15__Using_virtual_Trusted_Platform_Module_devices_an9xtvgw/9.15. Using virtual Trusted Platform Module devices.md
INFO 2025-05-19 13:31:40,363 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_15__Using_virtual_Trusted_Platform_Module_devices_an9xtvgw/9.15. Using virtual Trusted Platform Module devices.md
INFO 2025-05-19 13:31:40,640 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:40,640 instructlab.sdg.utils.taxonomy:148: Pattern '11.2. Live migration limits and timeouts.md' matched 1 files.
INFO 2025-05-19 13:31:40,640 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_11_2_2__Cluster_wide_live_migration_limits_and_timeouts_6_x06733/11.2. Live migration limits and timeouts.md
INFO 2025-05-19 13:31:40,640 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_11_2_2__Cluster_wide_live_migration_limits_and_timeouts_6_x06733/11.2. Live migration limits and timeouts.md
INFO 2025-05-19 13:31:40,933 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:40,933 instructlab.sdg.utils.taxonomy:148: Pattern '13.8. Reviewing resource usage by virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:40,933 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_8_3__Additional_resources_2r2rvxih/13.8. Reviewing resource usage by virtual machines.md
INFO 2025-05-19 13:31:40,933 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_8_3__Additional_resources_2r2rvxih/13.8. Reviewing resource usage by virtual machines.md
INFO 2025-05-19 13:31:41,240 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:41,240 instructlab.sdg.utils.taxonomy:148: Pattern '9.14. Installing VirtIO driver on a new Windows virtual machine.md' matched 1 files.
INFO 2025-05-19 13:31:41,241 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_14_2__About_VirtIO_drivers_qtcyd7m5/9.14. Installing VirtIO driver on a new Windows virtual machine.md
INFO 2025-05-19 13:31:41,241 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_14_2__About_VirtIO_drivers_qtcyd7m5/9.14. Installing VirtIO driver on a new Windows virtual machine.md
INFO 2025-05-19 13:31:41,546 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:41,546 instructlab.sdg.utils.taxonomy:148: Pattern '9.5. Managing virtual machine instances.md' matched 1 files.
INFO 2025-05-19 13:31:41,546 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_5_4__Editing_a_standalone_virtual_machine_instance_using_the_web_console_x5vprglq/9.5. Managing virtual machine instances.md
INFO 2025-05-19 13:31:41,546 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_5_4__Editing_a_standalone_virtual_machine_instance_using_the_web_console_x5vprglq/9.5. Managing virtual machine instances.md
INFO 2025-05-19 13:31:41,813 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:41,813 instructlab.sdg.utils.taxonomy:148: Pattern '9.6. Controlling virtual machine states.md' matched 1 files.
INFO 2025-05-19 13:31:41,813 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_6_1__Starting_a_virtual_machine_ar1lka45/9.6. Controlling virtual machine states.md
INFO 2025-05-19 13:31:41,813 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_6_1__Starting_a_virtual_machine_ar1lka45/9.6. Controlling virtual machine states.md
INFO 2025-05-19 13:31:42,092 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:42,092 instructlab.sdg.utils.taxonomy:148: Pattern '9.13. Installing VirtIO driver on an existing Windows virtual machine.md' matched 1 files.
INFO 2025-05-19 13:31:42,092 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_13_5__Removing_the_VirtIO_container_disk_from_a_virtual_machine_mc7dkudb/9.13. Installing VirtIO driver on an existing Windows virtual machine.md
INFO 2025-05-19 13:31:42,092 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_13_5__Removing_the_VirtIO_container_disk_from_a_virtual_machine_mc7dkudb/9.13. Installing VirtIO driver on an existing Windows virtual machine.md
INFO 2025-05-19 13:31:42,378 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:42,378 instructlab.sdg.utils.taxonomy:148: Pattern '9.13. Installing VirtIO driver on an existing Windows virtual machine.md' matched 1 files.
INFO 2025-05-19 13:31:42,378 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_13_4__Installing_VirtIO_drivers_on_an_existing_Windows_virtual_machine_l9rudwog/9.13. Installing VirtIO driver on an existing Windows virtual machine.md
INFO 2025-05-19 13:31:42,378 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_13_4__Installing_VirtIO_drivers_on_an_existing_Windows_virtual_machine_l9rudwog/9.13. Installing VirtIO driver on an existing Windows virtual machine.md
INFO 2025-05-19 13:31:42,660 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:42,660 instructlab.sdg.utils.taxonomy:148: Pattern '2.1. How OpenShift Virtualization architecture works.md' matched 1 files.
INFO 2025-05-19 13:31:42,660 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_2_1__How_OpenShift_Virtualization_architecture_works_sptuyo3w/2.1. How OpenShift Virtualization architecture works.md
INFO 2025-05-19 13:31:42,660 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_2_1__How_OpenShift_Virtualization_architecture_works_sptuyo3w/2.1. How OpenShift Virtualization architecture works.md
INFO 2025-05-19 13:31:42,959 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:42,959 instructlab.sdg.utils.taxonomy:148: Pattern '13.7. Using the OpenShift Container Platform dashboard to get cluster information.md' matched 1 files.
INFO 2025-05-19 13:31:42,959 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_7_1__About_the_OpenShift_Container_Platform_dashboards_page_97d8g7ii/13.7. Using the OpenShift Container Platform dashboard to get cluster information.md
INFO 2025-05-19 13:31:42,959 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_7_1__About_the_OpenShift_Container_Platform_dashboards_page_97d8g7ii/13.7. Using the OpenShift Container Platform dashboard to get cluster information.md
INFO 2025-05-19 13:31:43,239 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:43,239 instructlab.sdg.utils.taxonomy:148: Pattern '10.4. Deploying a virtual machine template to a custom namespace.md' matched 1 files.
INFO 2025-05-19 13:31:43,239 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_10_4__Deploying_a_virtual_machine_template_to_a_custom_namespace_31hf_wzb/10.4. Deploying a virtual machine template to a custom namespace.md
INFO 2025-05-19 13:31:43,239 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_10_4__Deploying_a_virtual_machine_template_to_a_custom_namespace_31hf_wzb/10.4. Deploying a virtual machine template to a custom namespace.md
INFO 2025-05-19 13:31:43,532 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:43,533 instructlab.sdg.utils.taxonomy:148: Pattern '13.13. OpenShift Virtualization critical alerts.md' matched 1 files.
INFO 2025-05-19 13:31:43,533 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_13_4__Additional_resources_5rlm6vtq/13.13. OpenShift Virtualization critical alerts.md
INFO 2025-05-19 13:31:43,533 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_13_4__Additional_resources_5rlm6vtq/13.13. OpenShift Virtualization critical alerts.md
INFO 2025-05-19 13:31:43,818 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:43,818 instructlab.sdg.utils.taxonomy:148: Pattern '13.1. Reviewing Virtualization Overview.md' matched 1 files.
INFO 2025-05-19 13:31:43,818 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_1_4__Reviewing_top_consumers_for_projects__virtual_machines__and_nodes_kchr7psm/13.1. Reviewing Virtualization Overview.md
INFO 2025-05-19 13:31:43,819 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_1_4__Reviewing_top_consumers_for_projects__virtual_machines__and_nodes_kchr7psm/13.1. Reviewing Virtualization Overview.md
INFO 2025-05-19 13:31:44,117 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:44,117 instructlab.sdg.utils.taxonomy:148: Pattern '9.7. Accessing virtual machine consoles.md' matched 1 files.
INFO 2025-05-19 13:31:44,117 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_7__Accessing_virtual_machine_consoles_ri93hkby/9.7. Accessing virtual machine consoles.md
INFO 2025-05-19 13:31:44,117 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_7__Accessing_virtual_machine_consoles_ri93hkby/9.7. Accessing virtual machine consoles.md
INFO 2025-05-19 13:31:44,402 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:44,402 instructlab.sdg.utils.taxonomy:148: Pattern '2.5. About the hostpath-provisioner-operator.md' matched 1 files.
INFO 2025-05-19 13:31:44,403 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_2_5__About_the_hostpath_provisioner_operator_9hrgg0_l/2.5. About the hostpath-provisioner-operator.md
INFO 2025-05-19 13:31:44,403 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_2_5__About_the_hostpath_provisioner_operator_9hrgg0_l/2.5. About the hostpath-provisioner-operator.md
INFO 2025-05-19 13:31:44,701 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:44,701 instructlab.sdg.utils.taxonomy:148: Pattern '9.1. Creating virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:44,701 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_1_4__Using_the_CLI_to_create_a_virtual_machine_inippnh6/9.1. Creating virtual machines.md
INFO 2025-05-19 13:31:44,701 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_1_4__Using_the_CLI_to_create_a_virtual_machine_inippnh6/9.1. Creating virtual machines.md
INFO 2025-05-19 13:31:44,988 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:44,988 instructlab.sdg.utils.taxonomy:148: Pattern '13.14. Collecting data for Red Hat Support.md' matched 1 files.
INFO 2025-05-19 13:31:44,988 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_14_2__Collecting_data_about_virtual_machines_zys0x5s8/13.14. Collecting data for Red Hat Support.md
INFO 2025-05-19 13:31:44,988 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_14_2__Collecting_data_about_virtual_machines_zys0x5s8/13.14. Collecting data for Red Hat Support.md
INFO 2025-05-19 13:31:45,295 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:45,296 instructlab.sdg.utils.taxonomy:148: Pattern '9.14. Installing VirtIO driver on a new Windows virtual machine.md' matched 1 files.
INFO 2025-05-19 13:31:45,296 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_14_6__Removing_the_VirtIO_container_disk_from_a_virtual_machine_krsj_7ra/9.14. Installing VirtIO driver on a new Windows virtual machine.md
INFO 2025-05-19 13:31:45,296 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_14_6__Removing_the_VirtIO_container_disk_from_a_virtual_machine_krsj_7ra/9.14. Installing VirtIO driver on a new Windows virtual machine.md
INFO 2025-05-19 13:31:45,574 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:45,574 instructlab.sdg.utils.taxonomy:148: Pattern '13.14. Collecting data for Red Hat Support.md' matched 1 files.
INFO 2025-05-19 13:31:45,574 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_14_1__Collecting_data_about_your_environment_bsnl1wir/13.14. Collecting data for Red Hat Support.md
INFO 2025-05-19 13:31:45,574 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_14_1__Collecting_data_about_your_environment_bsnl1wir/13.14. Collecting data for Red Hat Support.md
INFO 2025-05-19 13:31:45,961 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:45,961 instructlab.sdg.utils.taxonomy:148: Pattern '3.1. Before you begin.md' matched 1 files.
INFO 2025-05-19 13:31:45,961 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_3_1__Before_you_begin_sjfq5ofm/3.1. Before you begin.md
INFO 2025-05-19 13:31:45,961 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_3_1__Before_you_begin_sjfq5ofm/3.1. Before you begin.md
INFO 2025-05-19 13:31:46,264 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:46,264 instructlab.sdg.utils.taxonomy:148: Pattern '5.4. Installing OpenShift Virtualization using the CLI.md' matched 1 files.
INFO 2025-05-19 13:31:46,264 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_4_3__Deploying_the_OpenShift_Virtualization_Operator_by_using_the_CLI_1c17ip_y/5.4. Installing OpenShift Virtualization using the CLI.md
INFO 2025-05-19 13:31:46,264 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_4_3__Deploying_the_OpenShift_Virtualization_Operator_by_using_the_CLI_1c17ip_y/5.4. Installing OpenShift Virtualization using the CLI.md
INFO 2025-05-19 13:31:46,531 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:46,531 instructlab.sdg.utils.taxonomy:148: Pattern '8.4. Creating a container using virtctl guestfs.md' matched 1 files.
INFO 2025-05-19 13:31:46,531 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_8_4__Creating_a_container_using_virtctl_guestfs_azuvdbyq/8.4. Creating a container using virtctl guestfs.md
INFO 2025-05-19 13:31:46,531 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_8_4__Creating_a_container_using_virtctl_guestfs_azuvdbyq/8.4. Creating a container using virtctl guestfs.md
INFO 2025-05-19 13:31:46,811 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:46,811 instructlab.sdg.utils.taxonomy:148: Pattern '5.1. Preparing your cluster for OpenShift Virtualization.md' matched 1 files.
INFO 2025-05-19 13:31:46,811 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_1_6__Snapshots_and_cloning_0qzmvrdx/5.1. Preparing your cluster for OpenShift Virtualization.md
INFO 2025-05-19 13:31:46,811 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_1_6__Snapshots_and_cloning_0qzmvrdx/5.1. Preparing your cluster for OpenShift Virtualization.md
INFO 2025-05-19 13:31:47,108 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:47,108 instructlab.sdg.utils.taxonomy:148: Pattern '9.2. Editing virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:47,108 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_2_1__Editing_a_virtual_machine_in_the_web_console_bxknsnny/9.2. Editing virtual machines.md
INFO 2025-05-19 13:31:47,108 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_2_1__Editing_a_virtual_machine_in_the_web_console_bxknsnny/9.2. Editing virtual machines.md
INFO 2025-05-19 13:31:47,412 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:47,413 instructlab.sdg.utils.taxonomy:148: Pattern '13.9. OpenShift Container Platform cluster monitoring, logging, and Telemetry.md' matched 1 files.
INFO 2025-05-19 13:31:47,413 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_9_1__About_OpenShift_Container_Platform_monitoring_fnipua6c/13.9. OpenShift Container Platform cluster monitoring, logging, and Telemetry.md
INFO 2025-05-19 13:31:47,413 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_9_1__About_OpenShift_Container_Platform_monitoring_fnipua6c/13.9. OpenShift Container Platform cluster monitoring, logging, and Telemetry.md
INFO 2025-05-19 13:31:47,709 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:47,709 instructlab.sdg.utils.taxonomy:148: Pattern '7.1. About workload security.md' matched 1 files.
INFO 2025-05-19 13:31:47,709 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_7_1__About_workload_security_0yct_ktp/7.1. About workload security.md
INFO 2025-05-19 13:31:47,710 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_7_1__About_workload_security_0yct_ktp/7.1. About workload security.md
INFO 2025-05-19 13:31:48,018 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:48,018 instructlab.sdg.utils.taxonomy:148: Pattern '10.1. Creating virtual machine templates.md' matched 1 files.
INFO 2025-05-19 13:31:48,018 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_10_1_5__Additional_resources_er0qv5gs/10.1. Creating virtual machine templates.md
INFO 2025-05-19 13:31:48,019 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_10_1_5__Additional_resources_er0qv5gs/10.1. Creating virtual machine templates.md
INFO 2025-05-19 13:31:48,356 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:48,356 instructlab.sdg.utils.taxonomy:148: Pattern '2.3. About the cdi-operator.md' matched 1 files.
INFO 2025-05-19 13:31:48,356 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_2_3__About_the_cdi_operator_6yyjvh_l/2.3. About the cdi-operator.md
INFO 2025-05-19 13:31:48,356 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_2_3__About_the_cdi_operator_6yyjvh_l/2.3. About the cdi-operator.md
INFO 2025-05-19 13:31:48,651 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:48,651 instructlab.sdg.utils.taxonomy:148: Pattern '9.1. Creating virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:48,651 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_1_5__Virtual_machine_storage_volume_types_a1i5k73i/9.1. Creating virtual machines.md
INFO 2025-05-19 13:31:48,651 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_1_5__Virtual_machine_storage_volume_types_a1i5k73i/9.1. Creating virtual machines.md
INFO 2025-05-19 13:31:48,961 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:48,962 instructlab.sdg.utils.taxonomy:148: Pattern '9.1. Creating virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:48,962 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_1__Creating_virtual_machines_g6oe7m_8/9.1. Creating virtual machines.md
INFO 2025-05-19 13:31:48,962 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_1__Creating_virtual_machines_g6oe7m_8/9.1. Creating virtual machines.md
INFO 2025-05-19 13:31:49,313 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:49,313 instructlab.sdg.utils.taxonomy:148: Pattern '9.4. Deleting virtual machines.md' matched 1 files.
INFO 2025-05-19 13:31:49,313 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_4_1__Deleting_a_virtual_machine_using_the_web_console_8p7g_pdc/9.4. Deleting virtual machines.md
INFO 2025-05-19 13:31:49,313 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_4_1__Deleting_a_virtual_machine_using_the_web_console_8p7g_pdc/9.4. Deleting virtual machines.md
INFO 2025-05-19 13:31:49,607 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:49,607 instructlab.sdg.utils.taxonomy:148: Pattern '12.1. About node maintenance.md' matched 1 files.
INFO 2025-05-19 13:31:49,607 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_12_1_1__About_node_maintenance_mode_obmcgorn/12.1. About node maintenance.md
INFO 2025-05-19 13:31:49,607 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_12_1_1__About_node_maintenance_mode_obmcgorn/12.1. About node maintenance.md
INFO 2025-05-19 13:31:49,910 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:49,910 instructlab.sdg.utils.taxonomy:148: Pattern '13.14. Collecting data for Red Hat Support.md' matched 1 files.
INFO 2025-05-19 13:31:49,910 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_14_3__Using_the_must_gather_tool_for_OpenShift_Virtualization_0a65lfjr/13.14. Collecting data for Red Hat Support.md
INFO 2025-05-19 13:31:49,910 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_13_14_3__Using_the_must_gather_tool_for_OpenShift_Virtualization_0a65lfjr/13.14. Collecting data for Red Hat Support.md
INFO 2025-05-19 13:31:50,173 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:50,173 instructlab.sdg.utils.taxonomy:148: Pattern '5.6. Uninstalling OpenShift Virtualization.md' matched 1 files.
INFO 2025-05-19 13:31:50,173 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_6__Uninstalling_OpenShift_Virtualization_ocuxinxi/5.6. Uninstalling OpenShift Virtualization.md
INFO 2025-05-19 13:31:50,173 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_5_6__Uninstalling_OpenShift_Virtualization_ocuxinxi/5.6. Uninstalling OpenShift Virtualization.md
INFO 2025-05-19 13:31:50,454 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:50,454 instructlab.sdg.utils.taxonomy:148: Pattern '9.14. Installing VirtIO driver on a new Windows virtual machine.md' matched 1 files.
INFO 2025-05-19 13:31:50,454 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_14_1__Prerequisites_7l3hmhm3/9.14. Installing VirtIO driver on a new Windows virtual machine.md
INFO 2025-05-19 13:31:50,454 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_14_1__Prerequisites_7l3hmhm3/9.14. Installing VirtIO driver on a new Windows virtual machine.md
INFO 2025-05-19 13:31:50,787 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:50,787 instructlab.sdg.utils.taxonomy:148: Pattern '9.3. Editing boot order.md' matched 1 files.
INFO 2025-05-19 13:31:50,787 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_3_2__Editing_a_boot_order_list_in_the_web_console_kwem12ct/9.3. Editing boot order.md
INFO 2025-05-19 13:31:50,787 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_3_2__Editing_a_boot_order_list_in_the_web_console_kwem12ct/9.3. Editing boot order.md
INFO 2025-05-19 13:31:51,065 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:51,065 instructlab.sdg.utils.taxonomy:148: Pattern '9.16. Advanced virtual machine management.md' matched 1 files.
INFO 2025-05-19 13:31:51,065 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_16_5__Configuring_PXE_booting_for_virtual_machines_n86oam70/9.16. Advanced virtual machine management.md
INFO 2025-05-19 13:31:51,065 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_9_16_5__Configuring_PXE_booting_for_virtual_machines_n86oam70/9.16. Advanced virtual machine management.md
INFO 2025-05-19 13:31:51,350 instructlab.sdg.utils.taxonomy:143: Processing files...
INFO 2025-05-19 13:31:51,351 instructlab.sdg.utils.taxonomy:148: Pattern '6.4. Monitoring update status.md' matched 1 files.
INFO 2025-05-19 13:31:51,351 instructlab.sdg.utils.taxonomy:152: Processing file: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_6_4_1__Monitoring_OpenShift_Virtualization_upgrade_status_55fyms10/6.4. Monitoring update status.md
INFO 2025-05-19 13:31:51,351 instructlab.sdg.utils.taxonomy:156: Added file path: /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09/documents/knowledge_technology_cloud-computing_6_4_1__Monitoring_OpenShift_Virtualization_upgrade_status_55fyms10/6.4. Monitoring update status.md
[2025-05-19 13:31:55,338] [INFO] [real_accelerator.py:222:get_accelerator] Setting ds_accelerator to cuda (auto detect)
INFO 2025-05-19 13:31:56,935 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:31:57,496 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:31:58,127 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:31:58,568 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:31:59,276 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:31:59,719 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:31:59,826 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:00,274 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:00,402 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:00,847 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:01,078 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:01,525 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:04,858 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:05,311 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:05,531 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:05,977 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:06,122 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:06,574 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:08,182 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:08,628 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:08,848 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:09,295 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:09,503 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:09,948 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:10,178 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:10,620 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:10,802 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:11,247 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:11,273 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:11,717 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:11,735 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:12,176 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:18,183 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:18,638 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:19,029 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:19,476 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:20,872 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:21,321 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:21,654 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:22,099 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:22,545 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:22,997 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:23,079 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:23,525 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:23,641 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:24,083 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:24,509 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:24,959 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:25,064 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:25,516 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:25,722 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:26,173 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:26,244 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:26,693 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:27,367 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:27,813 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:27,853 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:28,300 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:28,437 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:28,888 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:29,351 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:29,810 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:30,161 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:30,613 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:30,671 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:31,121 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:31,141 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:31,591 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:32,258 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:32,702 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:33,137 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:33,595 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:33,863 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:34,307 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:34,423 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:34,878 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:35,063 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:35,504 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:35,802 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:36,250 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:36,812 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:37,260 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:37,459 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:37,913 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:37,954 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:38,396 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:38,460 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:38,911 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:39,140 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:39,581 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:39,929 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:40,378 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:40,525 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:40,964 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:41,419 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:41,867 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:41,924 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:42,370 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:42,503 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:42,952 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:43,379 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:43,826 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:44,036 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:44,480 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:44,528 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:44,960 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:45,172 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:45,610 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:45,797 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:46,243 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:46,589 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:47,022 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:47,351 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:47,790 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:48,002 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:48,443 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:51,750 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:52,206 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:52,246 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:52,683 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:52,741 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:53,178 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:53,234 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:53,674 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:55,066 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:55,505 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:55,575 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:56,017 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:56,185 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:56,629 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:56,727 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:57,168 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:57,461 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:57,903 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:58,027 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:58,471 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:58,573 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:32:59,014 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:32:59,716 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:00,167 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:00,863 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:01,318 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:01,460 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:01,899 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:02,117 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:02,563 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:04,167 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:04,623 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:05,330 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:05,771 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:05,801 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:06,239 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:06,903 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:07,345 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:07,482 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:07,925 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:07,965 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:08,405 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:08,552 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:09,003 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:09,242 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:09,692 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:09,817 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:10,263 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:16,280 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:16,739 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:17,145 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:17,586 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:17,787 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:18,225 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:18,555 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:18,991 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:19,688 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:20,126 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:20,238 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:20,683 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:21,109 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:21,556 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:22,946 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:23,405 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:23,459 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:23,901 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:23,989 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:24,435 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:24,702 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:25,143 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:25,594 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:26,037 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:26,176 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:26,607 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:29,628 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:30,083 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:30,217 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:30,658 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:31,085 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:31,531 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:31,859 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:32,297 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:32,653 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:33,101 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:33,159 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:33,602 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:34,276 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:34,718 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:34,776 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:35,221 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:35,323 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:35,765 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:35,880 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:36,319 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:36,531 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:36,977 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:37,162 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:37,610 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:39,219 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:39,667 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:40,021 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:40,467 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:40,822 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:41,272 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:41,376 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:41,828 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:42,065 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:42,513 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:42,682 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:43,121 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:43,957 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:44,404 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:44,605 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:45,049 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:45,722 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:46,164 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:46,201 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:46,646 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:47,358 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:47,799 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:48,441 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:48,895 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:49,110 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:49,558 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:49,906 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:50,354 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:50,424 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:50,873 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:51,016 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:51,461 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:51,492 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:51,932 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:52,264 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:52,702 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:53,371 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:53,818 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:53,933 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:54,379 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:54,406 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:54,849 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:55,284 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:55,727 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:55,763 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:56,206 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:56,916 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:57,377 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:58,083 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:58,524 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:58,669 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:59,113 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:33:59,235 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:33:59,672 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:34:00,019 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:34:00,471 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:34:00,933 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:34:01,387 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:34:01,602 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:34:02,052 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:34:02,349 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:34:02,802 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:34:06,128 instructlab.sdg.utils.chunkers:144: Found the docling models
INFO 2025-05-19 13:34:06,587 instructlab.sdg.utils.chunkers:249: Successfully loaded tokenizer from: /var/home/vkadlec/.cache/instructlab/models/mixtral-8x7b-instruct-v0-1
INFO 2025-05-19 13:34:07,096 instructlab.sdg.generate_data:405: Taxonomy converted to samples and written to /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/preprocessed_2025-05-19T13_31_09
INFO 2025-05-19 13:34:07,136 instructlab.sdg.generate_data:441: Synthesizing new instructions. If you aren't satisfied with the generated instructions, interrupt training (Ctrl-C) and try adjusting your YAML files. Adding more examples may help.
INFO 2025-05-19 13:34:07,231 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_6_2_1__About_workload_updates, generating from scratch
INFO 2025-05-19 13:34:07,231 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 13:34:08,540 instructlab.sdg.blocks.llmblock:56: LLM server supports batched inputs: True
INFO 2025-05-19 13:34:08,540 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 13:34:10,049 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 13:34:10,070 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 13:34:10,079 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 13:34:15,342 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 13:34:23,395 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 13:34:29,015 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 13:34:29,035 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 13:34:29,051 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 13:35:23,919 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 13:36:33,245 instructlab.sdg.pipeline:199: Running block: filter_faithfulness

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 769.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1226.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1228.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 461.40 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 723.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 644.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1199.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1206.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 446.76 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 503.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1211.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1206.73 examples/s]

Map (num_proc=8):   0%|          | 0/172 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 44/172 [00:00<00:00, 330.45 examples/s]
Map (num_proc=8):  76%|#######5  | 130/172 [00:00<00:00, 501.18 examples/s]
Map (num_proc=8): 100%|##########| 172/172 [00:00<00:00, 453.13 examples/s]

Filter (num_proc=8):   0%|          | 0/172 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 88/172 [00:00<00:00, 834.86 examples/s]
Filter (num_proc=8): 100%|##########| 172/172 [00:00<00:00, 828.00 examples/s]INFO 2025-05-19 13:36:37,519 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 13:37:28,025 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 440.65 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 697.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 637.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1467.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1217.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.88 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 850.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 644.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1247.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1219.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.57 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 651.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1263.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1215.32 examples/s]
num_proc must be <= 7. Reducing num_proc to 7 for dataset of size 7.

Map (num_proc=7):   0%|          | 0/7 [00:00<?, ? examples/s]
Map (num_proc=7):  43%|####2     | 3/7 [00:00<00:00, 21.14 examples/s]
Map (num_proc=7): 100%|##########| 7/7 [00:00<00:00, 23.47 examples/s]
num_proc must be <= 7. Reducing num_proc to 7 for dataset of size 7.

Filter (num_proc=7):   0%|          | 0/7 [00:00<?, ? examples/s]
Filter (num_proc=7):  71%|#######1  | 5/7 [00:00<00:00, 47.93 examples/s]
Filter (num_proc=7): 100%|##########| 7/7 [00:00<00:00, 36.93 examples/s]INFO 2025-05-19 13:37:32,112 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 13:38:07,137 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.33 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 632.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1491.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1225.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 336.27 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 694.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1409.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1256.50 examples/s]

Map (num_proc=8):   0%|          | 0/228 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 58/228 [00:00<00:00, 391.50 examples/s]
Map (num_proc=8):  63%|######3   | 144/228 [00:00<00:00, 597.07 examples/s]
Map (num_proc=8): 100%|##########| 228/228 [00:00<00:00, 575.39 examples/s]

Filter (num_proc=8):   0%|          | 0/228 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 116/228 [00:00<00:00, 1087.64 examples/s]
Filter (num_proc=8): 100%|##########| 228/228 [00:00<00:00, 1071.26 examples/s]INFO 2025-05-19 13:38:10,364 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_6_2_1__About_workload_updates/data_checkpoint_037cf66b25d74037aa5c17654cb54280.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 30.31ba/s]INFO 2025-05-19 13:38:10,555 instructlab.sdg.generate_data:478: Generated 623 samples
INFO 2025-05-19 13:38:10,625 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_1_6__About_RunStrategies_for_virtual_machines, generating from scratch
INFO 2025-05-19 13:38:10,625 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 13:38:10,628 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 13:38:11,576 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 13:38:11,590 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 13:38:11,598 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 13:38:27,450 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 13:38:49,324 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 13:39:05,935 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 13:39:05,962 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 13:39:05,980 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 13:42:02,929 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 13:48:24,257 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.99 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 845.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1206.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1199.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 466.79 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 627.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 645.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1137.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1197.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.31 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 611.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 637.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1160.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1202.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.63 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 732.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1165.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1192.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 458.61 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 669.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1098.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.57 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 617.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1245.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 476.27 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 631.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 933.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1190.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.32 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 480.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1102.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1186.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.08 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 702.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 887.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1195.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.59 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 839.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1253.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1182.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.52 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 738.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 915.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1215.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.00 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 635.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1202.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 464.16 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 691.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 639.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1170.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 574.59 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 571.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 871.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1220.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1211.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 448.55 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 761.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 638.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1149.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1195.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.42 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 735.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 639.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1119.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1191.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.37 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 528.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 904.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 434.12 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 743.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 941.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1208.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.33 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 730.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 635.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1195.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.57 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 746.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1213.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1194.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.66 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 658.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1227.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1204.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.33 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 751.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1194.75 examples/s]

Map (num_proc=8):   0%|          | 0/143 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 36/143 [00:00<00:00, 262.72 examples/s]
Map (num_proc=8):  76%|#######5  | 108/143 [00:00<00:00, 459.34 examples/s]
Map (num_proc=8): 100%|##########| 143/143 [00:00<00:00, 379.06 examples/s]

Filter (num_proc=8):   0%|          | 0/143 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 72/143 [00:00<00:00, 689.66 examples/s]
Filter (num_proc=8): 100%|##########| 143/143 [00:00<00:00, 670.88 examples/s]INFO 2025-05-19 13:48:49,227 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 13:51:35,504 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 642.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1141.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 590.30 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 464.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1214.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1184.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 605.86 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 513.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1117.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 327.32 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 721.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 919.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.89 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 639.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1429.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 319.59 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 664.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1191.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 584.51 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 525.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 778.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1167.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 461.00 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 531.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 830.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 342.27 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 562.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.58 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 767.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1244.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.75 examples/s]

Map (num_proc=8):   0%|          | 0/122 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 32/122 [00:00<00:00, 262.73 examples/s]
Map (num_proc=8):  63%|######3   | 77/122 [00:00<00:00, 310.94 examples/s]
Map (num_proc=8): 100%|##########| 122/122 [00:00<00:00, 328.52 examples/s]

Filter (num_proc=8):   0%|          | 0/122 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 62/122 [00:00<00:00, 593.87 examples/s]
Filter (num_proc=8): 100%|##########| 122/122 [00:00<00:00, 581.10 examples/s]INFO 2025-05-19 13:51:47,798 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 13:53:40,491 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.29 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 690.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1224.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1190.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 501.85 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 621.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 932.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1186.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 515.83 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 500.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1465.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 442.46 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 665.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1098.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.50 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 611.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1387.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1200.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.54 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 484.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1199.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 525.01 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 531.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1234.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1192.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 330.72 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 748.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 913.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1183.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.52 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 602.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1090.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.61 examples/s]

Map (num_proc=8):   0%|          | 0/216 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 54/216 [00:00<00:00, 330.42 examples/s]
Map (num_proc=8):  75%|#######5  | 162/216 [00:00<00:00, 655.36 examples/s]
Map (num_proc=8): 100%|##########| 216/216 [00:00<00:00, 512.95 examples/s]

Filter (num_proc=8):   0%|          | 0/216 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 108/216 [00:00<00:00, 946.44 examples/s]
Filter (num_proc=8): 100%|##########| 216/216 [00:00<00:00, 996.91 examples/s]INFO 2025-05-19 13:53:51,688 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_1_6__About_RunStrategies_for_virtual_machines/data_checkpoint_1418880ef56d49f2befdb857ef8c39c0.jsonl


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 29.57ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 29.51ba/s]INFO 2025-05-19 13:53:52,321 instructlab.sdg.generate_data:478: Generated 2175 samples
INFO 2025-05-19 13:53:52,387 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_3_1__About_virtual_machine_events, generating from scratch
INFO 2025-05-19 13:53:52,387 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 13:53:52,391 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 13:53:52,583 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 13:53:52,596 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 13:53:52,604 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 13:53:58,252 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 13:54:02,683 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 13:54:06,557 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 13:54:06,576 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 13:54:06,591 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 13:54:44,384 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 13:54:56,510 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/185 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 47/185 [00:00<00:00, 338.94 examples/s]
Map (num_proc=8):  88%|########7 | 162/185 [00:00<00:00, 747.82 examples/s]
Map (num_proc=8): 100%|##########| 185/185 [00:00<00:00, 498.62 examples/s]

Filter (num_proc=8):   0%|          | 0/185 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 93/185 [00:00<00:00, 919.62 examples/s]
Filter (num_proc=8): 100%|##########| 185/185 [00:00<00:00, 892.81 examples/s]INFO 2025-05-19 13:54:57,573 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 13:55:05,781 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/109 [00:00<?, ? examples/s]
Map (num_proc=8):  39%|###8      | 42/109 [00:00<00:00, 268.96 examples/s]
Map (num_proc=8): 100%|##########| 109/109 [00:00<00:00, 309.47 examples/s]

Filter (num_proc=8):   0%|          | 0/109 [00:00<?, ? examples/s]
Filter (num_proc=8):  26%|##5       | 28/109 [00:00<00:00, 253.40 examples/s]
Filter (num_proc=8): 100%|##########| 109/109 [00:00<00:00, 479.74 examples/s]INFO 2025-05-19 13:55:06,842 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 13:55:13,242 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/104 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 26/104 [00:00<00:00, 216.68 examples/s]
Map (num_proc=8):  75%|#######5  | 78/104 [00:00<00:00, 354.99 examples/s]
Map (num_proc=8): 100%|##########| 104/104 [00:00<00:00, 296.52 examples/s]

Filter (num_proc=8):   0%|          | 0/104 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 52/104 [00:00<00:00, 508.43 examples/s]
Filter (num_proc=8): 100%|##########| 104/104 [00:00<00:00, 505.08 examples/s]INFO 2025-05-19 13:55:14,277 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_3_1__About_virtual_machine_events/data_checkpoint_55d963c9dd914cdc92a934b5ffbbb4d3.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 177.69ba/s]INFO 2025-05-19 13:55:14,316 instructlab.sdg.generate_data:478: Generated 95 samples
INFO 2025-05-19 13:55:14,367 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_8_5__Libguestfs_tools_and_virtctl_guestfs, generating from scratch
INFO 2025-05-19 13:55:14,367 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 13:55:14,371 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 13:55:14,639 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 13:55:14,652 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 13:55:14,659 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 13:55:20,125 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 13:55:29,205 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 13:55:34,447 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 13:55:34,465 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 13:55:34,481 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 13:56:25,245 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 13:56:57,806 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 445.53 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 524.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1220.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1193.51 examples/s]

Map (num_proc=8):   0%|          | 0/142 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 36/142 [00:00<00:00, 270.42 examples/s]
Map (num_proc=8): 100%|##########| 142/142 [00:00<00:00, 513.12 examples/s]
Map (num_proc=8): 100%|##########| 142/142 [00:00<00:00, 376.46 examples/s]

Filter (num_proc=8):   0%|          | 0/142 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 90/142 [00:00<00:00, 830.04 examples/s]
Filter (num_proc=8): 100%|##########| 142/142 [00:00<00:00, 676.01 examples/s]INFO 2025-05-19 13:56:59,964 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 13:57:19,870 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.87 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 649.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1187.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.63 examples/s]

Map (num_proc=8):   0%|          | 0/34 [00:00<?, ? examples/s]
Map (num_proc=8):  29%|##9       | 10/34 [00:00<00:00, 87.80 examples/s]
Map (num_proc=8):  88%|########8 | 30/34 [00:00<00:00, 144.19 examples/s]
Map (num_proc=8): 100%|##########| 34/34 [00:00<00:00, 100.65 examples/s]

Filter (num_proc=8):   0%|          | 0/34 [00:00<?, ? examples/s]
Filter (num_proc=8):  65%|######4   | 22/34 [00:00<00:00, 202.45 examples/s]
Filter (num_proc=8): 100%|##########| 34/34 [00:00<00:00, 167.10 examples/s]INFO 2025-05-19 13:57:22,000 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 13:57:38,840 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.33 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 662.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1200.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1204.56 examples/s]

Map (num_proc=8):   0%|          | 0/26 [00:00<?, ? examples/s]
Map (num_proc=8):  31%|###       | 8/26 [00:00<00:00, 77.13 examples/s]
Map (num_proc=8):  88%|########8 | 23/26 [00:00<00:00, 105.57 examples/s]
Map (num_proc=8): 100%|##########| 26/26 [00:00<00:00, 78.25 examples/s] 

Filter (num_proc=8):   0%|          | 0/26 [00:00<?, ? examples/s]
Filter (num_proc=8):  54%|#####3    | 14/26 [00:00<00:00, 131.27 examples/s]
Filter (num_proc=8): 100%|##########| 26/26 [00:00<00:00, 127.06 examples/s]INFO 2025-05-19 13:57:40,954 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_8_5__Libguestfs_tools_and_virtctl_guestfs/data_checkpoint_da14b0681aed4776933973761de71e6d.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 62.23ba/s]INFO 2025-05-19 13:57:41,043 instructlab.sdg.generate_data:478: Generated 260 samples
INFO 2025-05-19 13:57:41,106 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_12_2__Removing_a_secret__config_map__or_service_account_from_a_virtual_machine, generating from scratch
INFO 2025-05-19 13:57:41,106 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 13:57:41,109 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 13:57:41,372 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 13:57:41,385 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 13:57:41,393 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 13:57:48,245 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 13:57:55,538 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 13:58:02,084 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 13:58:02,104 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 13:58:02,119 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 13:59:03,860 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 14:00:52,190 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.79 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 516.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1206.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 836.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1227.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1198.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 607.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 909.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 795.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1181.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1185.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 446.77 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 786.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1187.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1195.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 631.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1152.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1192.01 examples/s]

Map (num_proc=8):   0%|          | 0/57 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 15/57 [00:00<00:00, 129.16 examples/s]
Map (num_proc=8):  63%|######3   | 36/57 [00:00<00:00, 159.56 examples/s]
Map (num_proc=8): 100%|##########| 57/57 [00:00<00:00, 164.93 examples/s]

Filter (num_proc=8):   0%|          | 0/57 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 36/57 [00:00<00:00, 331.33 examples/s]
Filter (num_proc=8): 100%|##########| 57/57 [00:00<00:00, 290.23 examples/s]INFO 2025-05-19 14:00:59,828 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 14:01:59,449 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.54 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 625.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1189.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1183.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 492.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 686.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1167.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.05 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 614.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1307.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.76 examples/s]

Map (num_proc=8):   0%|          | 0/191 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 48/191 [00:00<00:00, 360.84 examples/s]
Map (num_proc=8):  63%|######2   | 120/191 [00:00<00:00, 490.85 examples/s]
Map (num_proc=8): 100%|##########| 191/191 [00:00<00:00, 491.90 examples/s]

Filter (num_proc=8):   0%|          | 0/191 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 96/191 [00:00<00:00, 837.12 examples/s]
Filter (num_proc=8): 100%|##########| 191/191 [00:00<00:00, 894.61 examples/s]INFO 2025-05-19 14:02:03,869 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 14:02:46,839 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 343.43 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 791.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.09 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 479.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1216.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1186.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.28 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 663.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1257.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1182.25 examples/s]

Map (num_proc=8):   0%|          | 0/78 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 20/78 [00:00<00:00, 155.39 examples/s]
Map (num_proc=8):  63%|######2   | 49/78 [00:00<00:00, 210.07 examples/s]
Map (num_proc=8): 100%|##########| 78/78 [00:00<00:00, 219.86 examples/s]

Filter (num_proc=8):   0%|          | 0/78 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 40/78 [00:00<00:00, 382.66 examples/s]
Filter (num_proc=8): 100%|##########| 78/78 [00:00<00:00, 372.87 examples/s]INFO 2025-05-19 14:02:51,225 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_12_2__Removing_a_secret__config_map__or_service_account_from_a_virtual_machine/data_checkpoint_2ba60282be874462aab25e975f6fc60d.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 27.35ba/s]INFO 2025-05-19 14:02:51,436 instructlab.sdg.generate_data:478: Generated 692 samples
INFO 2025-05-19 14:02:51,512 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_16_10__Configuring_vGPU_passthrough, generating from scratch
INFO 2025-05-19 14:02:51,512 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 14:02:51,517 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 14:02:51,519 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 14:02:54,990 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 14:02:55,005 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 14:02:55,014 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 14:02:57,547 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 14:02:57,561 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 14:02:57,569 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 14:03:21,515 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 14:03:21,643 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 14:04:03,489 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 14:04:17,850 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 14:04:24,644 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 14:04:24,678 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 14:04:24,702 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 14:04:51,181 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 14:04:51,207 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 14:04:51,231 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 14:13:10,418 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 14:14:04,571 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 14:23:51,886 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 354.46 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1274.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 506.87 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 551.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 584.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1053.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 351.57 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 555.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 546.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 529.35 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 580.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 834.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1106.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.08 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 720.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1157.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.65 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 587.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 898.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 324.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 844.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 953.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 341.43 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 816.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.51 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 686.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1047.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 497.55 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 540.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1094.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 347.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 603.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 939.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.11 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 677.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1103.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.77 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 936.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1176.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 550.80 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 479.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1216.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.70 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 737.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 907.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.49 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 640.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1124.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1182.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1192.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 205.82 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 369.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 805.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 525.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1186.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.78 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 638.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 802.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1102.81 examples/s]

Map (num_proc=8):   0%|          | 0/123 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 32/123 [00:00<00:00, 241.66 examples/s]
Map (num_proc=8):  76%|#######5  | 93/123 [00:00<00:00, 396.26 examples/s]
Map (num_proc=8): 100%|##########| 123/123 [00:00<00:00, 317.14 examples/s]

Filter (num_proc=8):   0%|          | 0/123 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 63/123 [00:00<00:00, 600.37 examples/s]
Filter (num_proc=8): 100%|##########| 123/123 [00:00<00:00, 583.15 examples/s]INFO 2025-05-19 14:24:15,196 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 14:29:11,043 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 622.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 922.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.52 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 432.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 565.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 627.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1057.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 345.96 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 604.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 907.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.75 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 486.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1105.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 826.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1112.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 245.58 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 605.71 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 752.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 882.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.29 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 746.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1068.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.99 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 718.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1149.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.78 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 516.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 638.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1084.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.34 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 647.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1155.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.56 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 491.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 892.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.13 examples/s]

Map (num_proc=8):   0%|          | 0/129 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 33/129 [00:00<00:00, 313.82 examples/s]
Map (num_proc=8):  50%|#####     | 65/129 [00:00<00:00, 309.22 examples/s]
Map (num_proc=8): 100%|##########| 129/129 [00:00<00:00, 331.95 examples/s]

Filter (num_proc=8):   0%|          | 0/129 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 65/129 [00:00<00:00, 575.76 examples/s]
Filter (num_proc=8): 100%|##########| 129/129 [00:00<00:00, 580.76 examples/s]INFO 2025-05-19 14:29:25,137 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 14:33:43,775 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 594.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1036.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 716.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 623.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1147.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 777.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 872.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 339.57 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 745.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1066.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.09 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 572.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1227.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.93 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 490.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1233.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.62 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 517.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 639.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.83 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 647.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.44 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 733.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 854.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.03 examples/s]

Map (num_proc=8):   0%|          | 0/155 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 40/155 [00:00<00:00, 274.54 examples/s]
Map (num_proc=8):  88%|########7 | 136/155 [00:00<00:00, 499.50 examples/s]
Map (num_proc=8): 100%|##########| 155/155 [00:00<00:00, 391.17 examples/s]

Filter (num_proc=8):   0%|          | 0/155 [00:00<?, ? examples/s]
Filter (num_proc=8):  39%|###8      | 60/155 [00:00<00:00, 558.92 examples/s]
Filter (num_proc=8): 100%|##########| 155/155 [00:00<00:00, 705.34 examples/s]INFO 2025-05-19 14:33:56,850 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_16_10__Configuring_vGPU_passthrough/data_checkpoint_ea9d4752e57d492ca94c3139c9fca9a0.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 20.80ba/s]INFO 2025-05-19 14:39:26,279 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 564.45 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 600.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1187.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.93 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 638.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1392.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.63 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1199.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 460.88 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 548.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.74 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 855.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.81 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 743.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1225.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.15 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 763.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 859.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 564.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 622.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 463.29 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 641.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 928.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 670.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.24 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 632.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 920.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 671.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 964.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.53 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 592.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1119.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.40 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 792.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1215.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.51 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 633.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 927.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.66 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 702.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 863.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 663.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1109.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.03 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 483.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 903.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 854.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1214.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 310.30 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 432.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 881.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.38 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 681.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1196.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.68 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 671.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 925.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.71 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 703.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1160.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.02 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 700.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 868.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.32 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 735.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1185.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.65 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 659.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1138.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 633.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 630.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 509.01 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 565.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1221.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.54 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 707.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.15 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.60 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 816.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 833.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.25 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 808.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1233.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.33 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 594.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1065.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 682.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 460.44 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 463.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 935.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.63 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 797.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 907.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 481.82 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 700.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 638.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 920.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.24 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 775.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1273.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 539.35 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 569.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1151.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.89 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 798.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 637.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 484.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 637.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1113.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 288.57 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 548.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 863.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 262.56 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 564.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1254.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.56 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 687.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 625.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1230.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 453.32 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 525.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.44 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 684.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1108.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 451.09 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 647.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 333.54 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 779.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1211.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 546.88 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 659.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1076.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 346.32 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 869.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 902.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 434.39 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 615.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 922.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.05 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 638.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 555.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 601.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1131.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.10 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 760.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 635.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1159.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 480.23 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 678.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 648.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1130.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 444.91 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 607.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 635.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 882.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 979.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1256.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.97 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 808.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 928.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 432.00 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 562.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.39 examples/s]

Map (num_proc=8):   0%|          | 0/73 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 19/73 [00:00<00:00, 119.60 examples/s]
Map (num_proc=8):  88%|########7 | 64/73 [00:00<00:00, 252.87 examples/s]
Map (num_proc=8): 100%|##########| 73/73 [00:00<00:00, 192.10 examples/s]

Filter (num_proc=8):   0%|          | 0/73 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 37/73 [00:00<00:00, 357.88 examples/s]
Filter (num_proc=8): 100%|##########| 73/73 [00:00<00:00, 346.96 examples/s]INFO 2025-05-19 14:40:36,836 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 14:50:36,551 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.04 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 606.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1181.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1119.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 664.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1169.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.35 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 720.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.98 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 516.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 811.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.38 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 647.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 869.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 420.64 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 597.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.00 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 592.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 903.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1081.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.17 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 736.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.15 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 795.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.00 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 591.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1137.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 467.68 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 560.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1055.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.87 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 548.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.50 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 761.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1186.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.06 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 718.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.46 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 713.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1224.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 426.16 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 607.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 873.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 595.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1145.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 632.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.25 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 651.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 634.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.80 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 701.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 914.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 326.53 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 693.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1202.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 341.26 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 537.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 856.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 410.27 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 486.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 863.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 420.24 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 561.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1243.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.70 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 565.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1121.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 336.07 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 714.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 890.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.89 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 615.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1209.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.55 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 595.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 897.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.03 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 568.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 863.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.78 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 645.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.55 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 497.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 864.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.71 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 542.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1182.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 344.34 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 776.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1067.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.72 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 708.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1089.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.99 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 641.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 627.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 311.17 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 637.36 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 716.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1100.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.38 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 519.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.10 examples/s]

Map (num_proc=8):   0%|          | 0/198 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 50/198 [00:00<00:00, 320.35 examples/s]
Map (num_proc=8):  63%|######3   | 125/198 [00:00<00:00, 456.55 examples/s]
Map (num_proc=8): 100%|##########| 198/198 [00:00<00:00, 464.21 examples/s]

Filter (num_proc=8):   0%|          | 0/198 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 100/198 [00:00<00:00, 962.03 examples/s]
Filter (num_proc=8): 100%|##########| 198/198 [00:00<00:00, 883.38 examples/s]INFO 2025-05-19 14:51:23,352 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 14:58:29,582 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 436.69 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 577.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 905.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.30 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 617.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1087.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 315.47 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 677.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1135.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 331.94 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 620.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1080.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 329.05 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 667.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 865.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 252.25 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 515.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1178.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 410.34 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.47 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 464.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.30 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 560.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 887.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.05 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 614.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1246.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.86 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 484.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 918.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 494.10 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 505.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 908.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.97 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 654.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 920.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.21 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 575.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 926.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.54 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 560.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 919.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.70 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 644.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1242.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.46 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 553.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.96 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 580.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 869.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.60 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 600.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 330.91 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 831.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1189.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 466.33 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 579.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1166.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.13 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 523.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 632.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.76 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 549.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.77 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 585.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 914.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1077.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.17 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 436.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 933.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.37 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 642.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 542.40 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 510.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 620.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 594.52 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 577.74 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 593.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1089.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 593.21 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 582.21 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 580.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1069.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 590.60 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 542.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 915.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 484.13 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 520.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 924.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 589.90 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 498.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 636.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 607.23 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 511.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 895.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 483.39 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 531.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 914.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 318.37 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 462.40 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 779.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 891.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.55 examples/s]

Map (num_proc=8):   0%|          | 0/79 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 20/79 [00:00<00:00, 157.20 examples/s]
Map (num_proc=8):  63%|######3   | 50/79 [00:00<00:00, 182.75 examples/s]
Map (num_proc=8): 100%|##########| 79/79 [00:00<00:00, 192.32 examples/s]

Filter (num_proc=8):   0%|          | 0/79 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 30/79 [00:00<00:00, 294.82 examples/s]
Filter (num_proc=8): 100%|##########| 79/79 [00:00<00:00, 360.73 examples/s]INFO 2025-05-19 14:59:14,165 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_16_10__Configuring_vGPU_passthrough/data_checkpoint_c1c93884f3564b989b96d695cd1dd586.jsonl


Creating json from Arrow format:   0%|          | 0/8 [00:00<?, ?ba/s]
Creating json from Arrow format:  38%|###7      | 3/8 [00:00<00:00, 24.15ba/s]
Creating json from Arrow format:  75%|#######5  | 6/8 [00:00<00:00, 25.61ba/s]
Creating json from Arrow format: 100%|##########| 8/8 [00:00<00:00, 27.32ba/s]INFO 2025-05-19 14:59:16,817 instructlab.sdg.generate_data:478: Generated 9459 samples
INFO 2025-05-19 14:59:16,880 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_3_2__Storage, generating from scratch
INFO 2025-05-19 14:59:16,880 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 14:59:16,884 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 14:59:17,072 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 14:59:17,085 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 14:59:17,093 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 14:59:20,980 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 14:59:24,771 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 14:59:27,287 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 14:59:27,306 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 14:59:27,321 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 15:00:26,755 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 15:01:26,864 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.58 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 845.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 924.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.89 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 768.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 636.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1253.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 534.16 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 580.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 641.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.14 examples/s]

Map (num_proc=8):   0%|          | 0/165 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 42/165 [00:00<00:00, 298.44 examples/s]
Map (num_proc=8):  76%|#######5  | 125/165 [00:00<00:00, 497.91 examples/s]
Map (num_proc=8): 100%|##########| 165/165 [00:00<00:00, 438.61 examples/s]

Filter (num_proc=8):   0%|          | 0/165 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 84/165 [00:00<00:00, 747.61 examples/s]
Filter (num_proc=8): 100%|##########| 165/165 [00:00<00:00, 779.46 examples/s]INFO 2025-05-19 15:01:31,410 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 15:01:55,706 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 591.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1359.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1185.10 examples/s]

Map (num_proc=8):   0%|          | 0/138 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 36/138 [00:00<00:00, 278.28 examples/s]
Map (num_proc=8): 100%|##########| 138/138 [00:00<00:00, 561.45 examples/s]
Map (num_proc=8): 100%|##########| 138/138 [00:00<00:00, 378.57 examples/s]

Filter (num_proc=8):   0%|          | 0/138 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 70/138 [00:00<00:00, 670.48 examples/s]
Filter (num_proc=8): 100%|##########| 138/138 [00:00<00:00, 651.89 examples/s]INFO 2025-05-19 15:01:57,958 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 15:02:16,370 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 425.79 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 828.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1152.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.39 examples/s]

Map (num_proc=8):   0%|          | 0/110 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 28/110 [00:00<00:00, 215.06 examples/s]
Map (num_proc=8):  76%|#######6  | 84/110 [00:00<00:00, 382.75 examples/s]
Map (num_proc=8): 100%|##########| 110/110 [00:00<00:00, 302.67 examples/s]

Filter (num_proc=8):   0%|          | 0/110 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 56/110 [00:00<00:00, 514.05 examples/s]
Filter (num_proc=8): 100%|##########| 110/110 [00:00<00:00, 518.69 examples/s]INFO 2025-05-19 15:02:18,627 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_3_2__Storage/data_checkpoint_f4bb7fd006df4f758c9ef33da0168e50.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 80.00ba/s]INFO 2025-05-19 15:02:18,726 instructlab.sdg.generate_data:478: Generated 272 samples
INFO 2025-05-19 15:02:18,793 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_4_1__Prerequisites, generating from scratch
INFO 2025-05-19 15:02:18,794 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 15:02:18,797 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 15:02:19,185 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 15:02:19,198 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 15:02:19,206 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 15:02:29,862 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 15:02:56,612 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 15:03:06,199 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 15:03:06,219 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 15:03:06,234 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 15:04:15,404 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 15:06:22,812 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 442.97 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 637.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.35 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 744.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 867.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.48 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 626.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.98 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 719.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1245.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 334.84 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 739.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 909.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 446.24 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 591.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 928.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.96 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 503.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 590.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1084.90 examples/s]

Map (num_proc=8):   0%|          | 0/32 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 8/32 [00:00<00:00, 69.38 examples/s]
Map (num_proc=8):  75%|#######5  | 24/32 [00:00<00:00, 117.30 examples/s]
Map (num_proc=8): 100%|##########| 32/32 [00:00<00:00, 94.20 examples/s] 

Filter (num_proc=8):   0%|          | 0/32 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 16/32 [00:00<00:00, 150.38 examples/s]
Filter (num_proc=8): 100%|##########| 32/32 [00:00<00:00, 160.41 examples/s]INFO 2025-05-19 15:06:31,925 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 15:08:03,806 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.30 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 571.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1111.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 548.08 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 550.39 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 582.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.60 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 662.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1150.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 549.72 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 504.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1151.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 483.03 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 517.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.36 examples/s]

Map (num_proc=8):   0%|          | 0/213 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 54/213 [00:00<00:00, 344.77 examples/s]
Map (num_proc=8):  88%|########7 | 187/213 [00:00<00:00, 626.79 examples/s]
Map (num_proc=8): 100%|##########| 213/213 [00:00<00:00, 519.09 examples/s]

Filter (num_proc=8):   0%|          | 0/213 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 108/213 [00:00<00:00, 979.20 examples/s]
Filter (num_proc=8): 100%|##########| 213/213 [00:00<00:00, 986.59 examples/s]INFO 2025-05-19 15:08:10,770 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 15:09:16,958 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 343.36 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 688.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1107.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.13 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 609.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.56 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 634.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 953.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.60 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 734.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.85 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 618.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.38 examples/s]

Map (num_proc=8):   0%|          | 0/116 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 30/116 [00:00<00:00, 229.43 examples/s]
Map (num_proc=8):  88%|########7 | 102/116 [00:00<00:00, 409.37 examples/s]
Map (num_proc=8): 100%|##########| 116/116 [00:00<00:00, 303.08 examples/s]

Filter (num_proc=8):   0%|          | 0/116 [00:00<?, ? examples/s]
Filter (num_proc=8):  64%|######3   | 74/116 [00:00<00:00, 700.42 examples/s]
Filter (num_proc=8): 100%|##########| 116/116 [00:00<00:00, 550.09 examples/s]INFO 2025-05-19 15:09:23,859 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_4_1__Prerequisites/data_checkpoint_b9a5c9ab87384123bc37efa9003b06ea.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 29.22ba/s]INFO 2025-05-19 15:09:24,249 instructlab.sdg.generate_data:478: Generated 1310 samples
INFO 2025-05-19 15:09:24,313 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_6_2__Restarting_a_virtual_machine, generating from scratch
INFO 2025-05-19 15:09:24,313 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 15:09:24,317 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 15:09:24,593 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 15:09:24,606 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 15:09:24,614 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 15:09:29,712 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 15:09:38,037 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 15:09:42,992 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 15:09:43,012 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 15:09:43,027 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 15:10:31,871 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 15:10:59,257 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.44 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 690.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1258.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.91 examples/s]

Map (num_proc=8):   0%|          | 0/96 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 24/96 [00:00<00:00, 205.25 examples/s]
Map (num_proc=8):  75%|#######5  | 72/96 [00:00<00:00, 325.12 examples/s]
Map (num_proc=8): 100%|##########| 96/96 [00:00<00:00, 265.36 examples/s]

Filter (num_proc=8):   0%|          | 0/96 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 48/96 [00:00<00:00, 447.05 examples/s]
Filter (num_proc=8): 100%|##########| 96/96 [00:00<00:00, 458.27 examples/s]INFO 2025-05-19 15:11:01,500 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 15:11:16,760 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/255 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 32/255 [00:00<00:00, 301.52 examples/s]
Map (num_proc=8):  50%|#####     | 128/255 [00:00<00:00, 506.22 examples/s]
Map (num_proc=8): 100%|##########| 255/255 [00:00<00:00, 619.88 examples/s]

Filter (num_proc=8):   0%|          | 0/255 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/255 [00:00<00:00, 1234.41 examples/s]
Filter (num_proc=8): 100%|##########| 255/255 [00:00<00:00, 1160.35 examples/s]INFO 2025-05-19 15:11:17,912 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 15:11:28,056 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/234 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 60/234 [00:00<00:00, 338.36 examples/s]
Map (num_proc=8):  63%|######2   | 147/234 [00:00<00:00, 543.33 examples/s]
Map (num_proc=8): 100%|##########| 234/234 [00:00<00:00, 572.93 examples/s]

Filter (num_proc=8):   0%|          | 0/234 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 118/234 [00:00<00:00, 1125.89 examples/s]
Filter (num_proc=8): 100%|##########| 234/234 [00:00<00:00, 1088.48 examples/s]INFO 2025-05-19 15:11:29,201 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_6_2__Restarting_a_virtual_machine/data_checkpoint_2febf0991c834f70bf391eb72bd2002f.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 81.13ba/s]INFO 2025-05-19 15:11:29,275 instructlab.sdg.generate_data:478: Generated 221 samples
INFO 2025-05-19 15:11:29,337 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_2_2__Example_manifests, generating from scratch
INFO 2025-05-19 15:11:29,337 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 15:11:29,340 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 15:11:30,176 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 15:11:30,190 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 15:11:30,198 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 15:11:40,833 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 15:12:01,128 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 15:12:10,132 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 15:12:10,153 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 15:12:10,171 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 15:14:53,322 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 15:21:30,173 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.14 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 707.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.78 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 564.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1257.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.63 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 680.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1199.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.29 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1189.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.82 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 613.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1180.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.26 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 633.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.12 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 776.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.39 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 591.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1169.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.73 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 714.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1140.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 613.11 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 565.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 839.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1151.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.55 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 613.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1270.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 436.41 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 520.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 351.65 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 820.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1184.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 455.52 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 662.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1279.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 441.34 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 632.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1086.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.07 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 830.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1161.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 455.05 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 571.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1124.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.38 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 779.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1130.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 463.95 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 621.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1274.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.48 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 583.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 939.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.66 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 574.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1219.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.45 examples/s]

Map (num_proc=8):   0%|          | 0/100 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 26/100 [00:00<00:00, 193.23 examples/s]
Map (num_proc=8):  64%|######4   | 64/100 [00:00<00:00, 276.98 examples/s]
Map (num_proc=8): 100%|##########| 100/100 [00:00<00:00, 276.13 examples/s]

Filter (num_proc=8):   0%|          | 0/100 [00:00<?, ? examples/s]
Filter (num_proc=8):  39%|###9      | 39/100 [00:00<00:00, 368.37 examples/s]
Filter (num_proc=8): 100%|##########| 100/100 [00:00<00:00, 467.75 examples/s]INFO 2025-05-19 15:21:55,476 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 15:25:49,051 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.09 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 512.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1111.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 238.01 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 538.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 912.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 229.58 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 388.89 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 729.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 561.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 888.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 229.38 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 421.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 848.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 564.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1104.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 267.13 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 443.75 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 796.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 236.46 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 422.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 230.25 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 590.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 212.01 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 454.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 565.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1118.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 209.43 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 494.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1107.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 210.13 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 556.34 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 711.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.97 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 479.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.19 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 518.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1220.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.96 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 745.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1228.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 448.33 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 473.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.06 examples/s]

Map (num_proc=8):   0%|          | 0/208 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 26/208 [00:00<00:00, 259.32 examples/s]
Map (num_proc=8):  50%|#####     | 104/208 [00:00<00:00, 519.32 examples/s]
Map (num_proc=8):  88%|########7 | 182/208 [00:00<00:00, 574.32 examples/s]
Map (num_proc=8): 100%|##########| 208/208 [00:00<00:00, 484.77 examples/s]

Filter (num_proc=8):   0%|          | 0/208 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 78/208 [00:00<00:00, 749.77 examples/s]
Filter (num_proc=8): 100%|##########| 208/208 [00:00<00:00, 947.61 examples/s]INFO 2025-05-19 15:26:06,875 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 15:29:02,433 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.09 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 608.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1213.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.29 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 883.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 325.02 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 661.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1148.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 348.09 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 632.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1237.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.62 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 611.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1216.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.46 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 666.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1047.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.42 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 522.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1240.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.42 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 618.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1168.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.99 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 649.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1169.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 477.13 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 527.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1274.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 577.54 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 539.20 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 713.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 349.57 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 688.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 542.87 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 529.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 959.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.56 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 514.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1129.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.36 examples/s]

Map (num_proc=8):   0%|          | 0/112 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 28/112 [00:00<00:00, 218.96 examples/s]
Map (num_proc=8):  50%|#####     | 56/112 [00:00<00:00, 251.10 examples/s]
Map (num_proc=8): 100%|##########| 112/112 [00:00<00:00, 296.64 examples/s]

Filter (num_proc=8):   0%|          | 0/112 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 56/112 [00:00<00:00, 545.34 examples/s]
Filter (num_proc=8): 100%|##########| 112/112 [00:00<00:00, 517.03 examples/s]INFO 2025-05-19 15:29:20,051 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_2_2__Example_manifests/data_checkpoint_c4674ac72f2a41f39f67bd37360170e6.jsonl


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format:  67%|######6   | 2/3 [00:00<00:00, 19.11ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 23.95ba/s]INFO 2025-05-19 15:29:20,782 instructlab.sdg.generate_data:478: Generated 2501 samples
INFO 2025-05-19 15:29:20,837 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_2_2__Viewing_virtual_machine_logs_in_the_web_console, generating from scratch
INFO 2025-05-19 15:29:20,837 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 15:29:20,841 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 15:29:21,330 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 15:29:21,344 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 15:29:21,352 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 15:29:50,531 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 15:30:00,700 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 15:30:11,242 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 15:30:11,263 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 15:30:11,278 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 15:31:03,204 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 15:31:42,924 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 553.29 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 474.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1268.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.84 examples/s]

Map (num_proc=8):   0%|          | 0/193 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 49/193 [00:00<00:00, 364.19 examples/s]
Map (num_proc=8):  63%|######2   | 121/193 [00:00<00:00, 513.37 examples/s]
Map (num_proc=8): 100%|##########| 193/193 [00:00<00:00, 492.14 examples/s]

Filter (num_proc=8):   0%|          | 0/193 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 97/193 [00:00<00:00, 834.69 examples/s]
Filter (num_proc=8): 100%|##########| 193/193 [00:00<00:00, 891.32 examples/s]INFO 2025-05-19 15:31:45,249 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 15:32:07,010 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 534.26 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 625.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1185.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.12 examples/s]

Map (num_proc=8):   0%|          | 0/77 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 20/77 [00:00<00:00, 167.56 examples/s]
Map (num_proc=8):  88%|########8 | 68/77 [00:00<00:00, 290.46 examples/s]
Map (num_proc=8): 100%|##########| 77/77 [00:00<00:00, 214.55 examples/s]

Filter (num_proc=8):   0%|          | 0/77 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 40/77 [00:00<00:00, 386.04 examples/s]
Filter (num_proc=8): 100%|##########| 77/77 [00:00<00:00, 364.89 examples/s]INFO 2025-05-19 15:32:09,300 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 15:32:24,393 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 579.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1252.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.46 examples/s]

Map (num_proc=8):   0%|          | 0/55 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 14/55 [00:00<00:00, 110.23 examples/s]
Map (num_proc=8):  76%|#######6  | 42/55 [00:00<00:00, 182.16 examples/s]
Map (num_proc=8): 100%|##########| 55/55 [00:00<00:00, 156.77 examples/s]

Filter (num_proc=8):   0%|          | 0/55 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 28/55 [00:00<00:00, 270.79 examples/s]
Filter (num_proc=8): 100%|##########| 55/55 [00:00<00:00, 263.23 examples/s]INFO 2025-05-19 15:32:26,664 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_2_2__Viewing_virtual_machine_logs_in_the_web_console/data_checkpoint_73ea52541c6743dc80a14f6463817053.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 55.57ba/s]INFO 2025-05-19 15:32:26,752 instructlab.sdg.generate_data:478: Generated 222 samples
INFO 2025-05-19 15:32:26,811 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_12__Managing_config_maps__secrets__and_service_accounts_in_virtual_machines, generating from scratch
INFO 2025-05-19 15:32:26,811 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 15:32:26,814 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 15:32:27,071 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 15:32:27,084 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 15:32:27,092 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 15:32:33,956 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 15:32:42,478 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 15:32:49,064 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 15:32:49,084 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 15:32:49,099 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 15:33:51,104 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 15:35:45,114 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.22 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 630.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1228.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 468.62 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 512.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1150.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.14 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 702.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1331.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.20 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 753.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1049.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 333.79 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 703.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 922.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1094.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.57 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 800.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1151.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.53 examples/s]

Map (num_proc=8):   0%|          | 0/66 [00:00<?, ? examples/s]
Map (num_proc=8):  39%|###9      | 26/66 [00:00<00:00, 186.92 examples/s]
Map (num_proc=8):  88%|########7 | 58/66 [00:00<00:00, 241.96 examples/s]
Map (num_proc=8): 100%|##########| 66/66 [00:00<00:00, 189.78 examples/s]

Filter (num_proc=8):   0%|          | 0/66 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 34/66 [00:00<00:00, 307.00 examples/s]
Filter (num_proc=8): 100%|##########| 66/66 [00:00<00:00, 316.57 examples/s]INFO 2025-05-19 15:35:53,206 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 15:37:10,221 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.63 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 616.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 928.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.94 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 671.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.15 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1180.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.76 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 648.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 901.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.64 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 605.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1229.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.85 examples/s]

Map (num_proc=8):   0%|          | 0/200 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 50/200 [00:00<00:00, 352.67 examples/s]
Map (num_proc=8):  75%|#######5  | 150/200 [00:00<00:00, 633.46 examples/s]
Map (num_proc=8): 100%|##########| 200/200 [00:00<00:00, 499.27 examples/s]

Filter (num_proc=8):   0%|          | 0/200 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 100/200 [00:00<00:00, 994.72 examples/s]
Filter (num_proc=8): 100%|##########| 200/200 [00:00<00:00, 907.89 examples/s]INFO 2025-05-19 15:37:16,031 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 15:38:08,445 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.25 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 546.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 939.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.36 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 472.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1167.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 631.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1170.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.14 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 632.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 898.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.43 examples/s]

Map (num_proc=8):   0%|          | 0/85 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 22/85 [00:00<00:00, 188.58 examples/s]
Map (num_proc=8):  76%|#######6  | 65/85 [00:00<00:00, 277.76 examples/s]
Map (num_proc=8): 100%|##########| 85/85 [00:00<00:00, 236.19 examples/s]

Filter (num_proc=8):   0%|          | 0/85 [00:00<?, ? examples/s]
Filter (num_proc=8):  65%|######4   | 55/85 [00:00<00:00, 506.95 examples/s]
Filter (num_proc=8): 100%|##########| 85/85 [00:00<00:00, 397.74 examples/s]INFO 2025-05-19 15:38:14,195 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_12__Managing_config_maps__secrets__and_service_accounts_in_virtual_machines/data_checkpoint_6444f78de5a34b2387dce3f84f4aebdc.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 20.40ba/s]INFO 2025-05-19 15:38:14,482 instructlab.sdg.generate_data:478: Generated 986 samples
INFO 2025-05-19 15:38:14,544 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_5_6__Deleting_a_standalone_virtual_machine_instance_using_the_web_console, generating from scratch
INFO 2025-05-19 15:38:14,545 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 15:38:14,548 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 15:38:14,965 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 15:38:14,978 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 15:38:14,986 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 15:38:20,646 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 15:38:27,343 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 15:38:31,856 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 15:38:31,875 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 15:38:31,890 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 15:39:53,478 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 15:43:15,189 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 450.53 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 621.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1193.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1190.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.50 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 597.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1225.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1198.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 580.62 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 564.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.88 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 795.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1202.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.78 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 594.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 643.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1095.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1201.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.79 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 779.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 641.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1192.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.12 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 737.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.88 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 607.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 641.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1255.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1201.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 463.50 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 761.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 651.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1099.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 442.90 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 718.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 630.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1078.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.69 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 801.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 935.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 713.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 649.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1193.79 examples/s]

Map (num_proc=8):   0%|          | 0/16 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 4/16 [00:00<00:00, 36.25 examples/s]
Map (num_proc=8):  88%|########7 | 14/16 [00:00<00:00, 68.59 examples/s]
Map (num_proc=8): 100%|##########| 16/16 [00:00<00:00, 48.68 examples/s]

Filter (num_proc=8):   0%|          | 0/16 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 8/16 [00:00<00:00, 79.90 examples/s]
Filter (num_proc=8): 100%|##########| 16/16 [00:00<00:00, 77.55 examples/s]INFO 2025-05-19 15:43:29,859 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 15:45:15,367 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.14 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 618.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.08 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 524.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 926.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.03 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 692.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 953.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1184.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.60 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 732.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 926.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.34 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 725.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1196.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1196.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.77 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 541.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.73 examples/s]

Map (num_proc=8):   0%|          | 0/177 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 45/177 [00:00<00:00, 295.28 examples/s]
Map (num_proc=8):  50%|#####     | 89/177 [00:00<00:00, 351.82 examples/s]
Map (num_proc=8): 100%|##########| 177/177 [00:00<00:00, 461.07 examples/s]

Filter (num_proc=8):   0%|          | 0/177 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 89/177 [00:00<00:00, 836.42 examples/s]
Filter (num_proc=8): 100%|##########| 177/177 [00:00<00:00, 825.20 examples/s]INFO 2025-05-19 15:45:23,427 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 15:46:39,635 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.84 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 704.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1255.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.69 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 767.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1124.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 323.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 913.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1090.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1182.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.09 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 497.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.07 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 810.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 410.33 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 662.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1160.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.76 examples/s]

Map (num_proc=8):   0%|          | 0/101 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 26/101 [00:00<00:00, 222.38 examples/s]
Map (num_proc=8):  76%|#######6  | 77/101 [00:00<00:00, 326.73 examples/s]
Map (num_proc=8): 100%|##########| 101/101 [00:00<00:00, 280.68 examples/s]

Filter (num_proc=8):   0%|          | 0/101 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 52/101 [00:00<00:00, 477.60 examples/s]
Filter (num_proc=8): 100%|##########| 101/101 [00:00<00:00, 471.38 examples/s]INFO 2025-05-19 15:46:47,613 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_5_6__Deleting_a_standalone_virtual_machine_instance_using_the_web_console/data_checkpoint_93e212c0e4374ebcbbc474c939efb981.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 33.75ba/s]INFO 2025-05-19 15:46:48,006 instructlab.sdg.generate_data:478: Generated 1407 samples
INFO 2025-05-19 15:46:48,067 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_OpenShift_Virtualization_installation__usage__and_release_notes, generating from scratch
INFO 2025-05-19 15:46:48,067 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 15:46:48,070 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 15:46:48,190 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 15:46:48,204 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 15:46:48,211 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 15:46:53,344 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 15:46:54,929 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 15:47:03,878 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 15:47:03,897 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 15:47:03,912 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 15:47:47,187 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 15:48:07,711 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.20 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 775.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1246.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.92 examples/s]

Map (num_proc=8):   0%|          | 0/35 [00:00<?, ? examples/s]
Map (num_proc=8):  43%|####2     | 15/35 [00:00<00:00, 114.78 examples/s]
Map (num_proc=8): 100%|##########| 35/35 [00:00<00:00, 148.21 examples/s]
Map (num_proc=8): 100%|##########| 35/35 [00:00<00:00, 105.42 examples/s]

Filter (num_proc=8):   0%|          | 0/35 [00:00<?, ? examples/s]
Filter (num_proc=8):  66%|######5   | 23/35 [00:00<00:00, 206.91 examples/s]
Filter (num_proc=8): 100%|##########| 35/35 [00:00<00:00, 177.53 examples/s]INFO 2025-05-19 15:48:09,895 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 15:48:23,651 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/191 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 48/191 [00:00<00:00, 338.08 examples/s]
Map (num_proc=8):  75%|#######5  | 144/191 [00:00<00:00, 648.32 examples/s]
Map (num_proc=8): 100%|##########| 191/191 [00:00<00:00, 505.89 examples/s]

Filter (num_proc=8):   0%|          | 0/191 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 96/191 [00:00<00:00, 934.94 examples/s]
Filter (num_proc=8): 100%|##########| 191/191 [00:00<00:00, 906.19 examples/s]INFO 2025-05-19 15:48:24,762 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 15:48:32,683 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/179 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 68/179 [00:00<00:00, 357.43 examples/s]
Map (num_proc=8): 100%|##########| 179/179 [00:00<00:00, 477.66 examples/s]

Filter (num_proc=8):   0%|          | 0/179 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 113/179 [00:00<00:00, 1041.34 examples/s]
Filter (num_proc=8): 100%|##########| 179/179 [00:00<00:00, 840.14 examples/s] INFO 2025-05-19 15:48:33,792 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_OpenShift_Virtualization_installation__usage__and_release_notes/data_checkpoint_161c2b45c75a4d6c9e645fb0210a25fb.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 125.67ba/s]INFO 2025-05-19 15:48:33,841 instructlab.sdg.generate_data:478: Generated 138 samples
INFO 2025-05-19 15:48:33,907 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_1__Making_open_source_more_inclusive, generating from scratch
INFO 2025-05-19 15:48:33,907 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 15:48:33,910 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 15:48:34,015 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 15:48:34,029 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 15:48:34,036 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 15:48:35,412 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 15:48:37,151 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 15:48:38,556 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 15:48:38,575 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 15:48:38,589 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 15:49:23,919 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 15:49:53,188 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 449.68 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 864.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 662.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1193.30 examples/s]

Map (num_proc=8):   0%|          | 0/214 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 54/214 [00:00<00:00, 386.66 examples/s]
Map (num_proc=8):  76%|#######5  | 162/214 [00:00<00:00, 672.11 examples/s]
Map (num_proc=8): 100%|##########| 214/214 [00:00<00:00, 558.45 examples/s]

Filter (num_proc=8):   0%|          | 0/214 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 108/214 [00:00<00:00, 1020.75 examples/s]
Filter (num_proc=8): 100%|##########| 214/214 [00:00<00:00, 1007.00 examples/s]INFO 2025-05-19 15:49:55,404 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 15:50:17,674 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 432.14 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 647.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 644.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.77 examples/s]

Map (num_proc=8):   0%|          | 0/83 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 22/83 [00:00<00:00, 180.73 examples/s]
Map (num_proc=8): 100%|##########| 83/83 [00:00<00:00, 334.51 examples/s]
Map (num_proc=8): 100%|##########| 83/83 [00:00<00:00, 238.91 examples/s]

Filter (num_proc=8):   0%|          | 0/83 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 43/83 [00:00<00:00, 417.69 examples/s]
Filter (num_proc=8): 100%|##########| 83/83 [00:00<00:00, 396.93 examples/s]INFO 2025-05-19 15:50:19,865 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 15:50:36,085 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 457.05 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 707.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 638.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1185.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1186.89 examples/s]

Map (num_proc=8):   0%|          | 0/65 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 17/65 [00:00<00:00, 150.16 examples/s]
Map (num_proc=8):  88%|########7 | 57/65 [00:00<00:00, 244.89 examples/s]
Map (num_proc=8): 100%|##########| 65/65 [00:00<00:00, 188.11 examples/s]

Filter (num_proc=8):   0%|          | 0/65 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 33/65 [00:00<00:00, 309.91 examples/s]
Filter (num_proc=8): 100%|##########| 65/65 [00:00<00:00, 316.15 examples/s]INFO 2025-05-19 15:50:38,271 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_1__Making_open_source_more_inclusive/data_checkpoint_de066945872b4a28a3a999365bdab2ad.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 91.53ba/s]INFO 2025-05-19 15:50:38,346 instructlab.sdg.generate_data:478: Generated 245 samples
INFO 2025-05-19 15:50:38,429 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_20_20__Expanding_a_virtual_machine_disk, generating from scratch
INFO 2025-05-19 15:50:38,429 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 15:50:38,435 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 15:50:38,438 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 15:50:41,876 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 15:50:41,891 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 15:50:41,899 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 15:50:44,504 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 15:50:44,519 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 15:50:44,528 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 15:51:09,666 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 15:51:20,495 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 15:51:51,519 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 15:52:09,971 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 15:52:10,006 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 15:52:10,031 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 15:52:54,773 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 15:53:28,496 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 15:53:28,527 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 15:53:28,551 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 16:01:26,624 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 16:02:45,624 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 16:11:53,247 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 564.95 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 524.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 813.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1049.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 547.45 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 507.93 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 680.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 308.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1000.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 347.45 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 712.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1404.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.35 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 501.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 921.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.67 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 514.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1240.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.23 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 662.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1274.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.36 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 484.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.58 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 764.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 933.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 434.11 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 639.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1143.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 544.65 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 459.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1234.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 471.89 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 541.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.89 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 669.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1197.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 410.67 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 683.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 479.36 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 684.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1132.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.11 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 724.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 890.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.93 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 607.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1160.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.66 examples/s]

Map (num_proc=8):   0%|          | 0/131 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 34/131 [00:00<00:00, 280.75 examples/s]
Map (num_proc=8):  63%|######3   | 83/131 [00:00<00:00, 385.74 examples/s]
Map (num_proc=8): 100%|##########| 131/131 [00:00<00:00, 347.51 examples/s]

Filter (num_proc=8):   0%|          | 0/131 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 67/131 [00:00<00:00, 625.05 examples/s]
Filter (num_proc=8): 100%|##########| 131/131 [00:00<00:00, 606.58 examples/s]INFO 2025-05-19 16:12:13,997 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 16:17:17,107 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 556.19 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 497.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 544.48 examples/s]
Filter (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 894.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 786.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.50 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1158.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 332.07 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 708.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1121.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 346.96 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 626.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1113.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.66 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 623.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1199.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.21 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 652.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1096.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.93 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 493.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1150.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.14 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 775.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1271.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.81 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 644.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 870.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 481.94 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 537.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 820.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1080.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.47 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 801.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.57 examples/s]

Map (num_proc=8):   0%|          | 0/77 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 20/77 [00:00<00:00, 160.45 examples/s]
Map (num_proc=8):  64%|######3   | 49/77 [00:00<00:00, 225.51 examples/s]
Map (num_proc=8): 100%|##########| 77/77 [00:00<00:00, 214.67 examples/s]

Filter (num_proc=8):   0%|          | 0/77 [00:00<?, ? examples/s]
Filter (num_proc=8):  39%|###8      | 30/77 [00:00<00:00, 244.09 examples/s]
Filter (num_proc=8): 100%|##########| 77/77 [00:00<00:00, 339.89 examples/s]INFO 2025-05-19 16:17:31,958 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 16:21:42,162 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 297.92 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 470.92 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 755.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1122.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.37 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 587.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 998.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1018.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 338.26 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 598.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 484.69 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 592.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1147.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 545.06 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 530.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1027.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 531.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.62 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 785.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 629.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.37 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 738.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1120.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 349.22 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 568.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 897.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 557.77 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 588.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 850.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1224.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.45 examples/s]

Map (num_proc=8):   0%|          | 0/142 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 36/142 [00:00<00:00, 241.21 examples/s]
Map (num_proc=8):  76%|#######6  | 108/142 [00:00<00:00, 466.61 examples/s]
Map (num_proc=8): 100%|##########| 142/142 [00:00<00:00, 367.90 examples/s]

Filter (num_proc=8):   0%|          | 0/142 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 54/142 [00:00<00:00, 521.06 examples/s]
Filter (num_proc=8): 100%|##########| 142/142 [00:00<00:00, 641.63 examples/s]INFO 2025-05-19 16:21:55,720 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_20_20__Expanding_a_virtual_machine_disk/data_checkpoint_326acec802054d0e9af9937f0ae00664.jsonl
INFO 2025-05-19 16:21:55,723 instructlab.sdg.pipeline:199: Running block: router


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 30.09ba/s]INFO 2025-05-19 16:21:57,104 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 16:21:57,118 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 16:21:57,126 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 16:22:25,359 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 16:23:05,671 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 16:23:28,823 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 16:23:28,849 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 16:23:28,871 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 16:32:51,605 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 16:33:07,494 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 333.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 825.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 560.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.35 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 688.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.66 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 526.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 338.06 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 775.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1203.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.74 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 438.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 336.28 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 648.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1036.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.08 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 469.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1236.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 335.05 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 685.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 326.03 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 668.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 879.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.38 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 794.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.83 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 513.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 926.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.16 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 648.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.89 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1195.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 451.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 616.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 900.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.51 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1166.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.58 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 561.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 546.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1206.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1094.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.04 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 684.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1106.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.77 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 659.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 932.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.99 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 622.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.98 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 625.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 589.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1078.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.94 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 564.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1194.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.42 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 600.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1205.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.97 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 698.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1124.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.79 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 724.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1096.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 870.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.35 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 622.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 344.27 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 696.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1080.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1119.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.13 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 647.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1136.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.46 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 558.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 927.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 296.85 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 527.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 773.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 515.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1156.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1104.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 551.38 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 744.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1139.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.54 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 789.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.35 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 691.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1263.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 271.16 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 644.05 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 764.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1119.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 319.24 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 694.53 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 749.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1083.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.22 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 830.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 880.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 420.52 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 592.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.39 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 574.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 955.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 352.22 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 687.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 426.97 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 980.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1209.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.47 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 640.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1189.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 667.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 631.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.45 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 830.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 621.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1036.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 457.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 694.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 889.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 450.70 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 493.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 515.76 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 515.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 473.99 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 551.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1348.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 447.47 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 694.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.46 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 725.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1178.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 496.15 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 649.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1159.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 481.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 859.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1178.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 619.68 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 523.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1193.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 420.76 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 511.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 496.14 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 530.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 883.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.36 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 678.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 935.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 661.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1067.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 296.05 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 426.70 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 772.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 870.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 318.52 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 428.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 895.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 309.79 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 464.17 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 782.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 247.10 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 562.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 850.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1085.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 203.65 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 546.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 899.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 219.58 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 557.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 296.52 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 539.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 284.22 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 593.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 287.11 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 487.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 234.61 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 552.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 874.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 267.08 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 578.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1193.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 312.45 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 426.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 264.85 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 504.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 935.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 272.91 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 549.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 898.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.92 examples/s]

Map (num_proc=8):   0%|          | 0/28 [00:00<?, ? examples/s]
Map (num_proc=8):  43%|####2     | 12/28 [00:00<00:00, 73.79 examples/s]
Map (num_proc=8): 100%|##########| 28/28 [00:00<00:00, 78.76 examples/s]

Filter (num_proc=8):   0%|          | 0/28 [00:00<?, ? examples/s]
Filter (num_proc=8):  43%|####2     | 12/28 [00:00<00:00, 109.26 examples/s]
Filter (num_proc=8): 100%|##########| 28/28 [00:00<00:00, 123.96 examples/s]INFO 2025-05-19 16:34:34,899 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 16:52:34,304 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.60 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 608.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 955.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1090.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.45 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 576.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 598.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1052.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.54 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.62 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 492.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 920.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.31 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 605.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 890.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 345.28 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 664.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1074.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.51 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 488.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 855.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1092.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 444.83 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 542.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1100.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 345.74 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 607.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1076.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1065.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 461.79 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 549.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 921.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1102.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.06 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 706.28 examples/s]INFO 2025-05-19 16:52:48,150 instructlab.sdg.pipeline:199: Running block: filter_faithfulness

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 552.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 304.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 375.53 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 488.05 examples/s]

Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 142.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 392.92 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 535.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 135.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 764.23 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 228.83 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 419.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 483.48 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 327.75 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 318.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 298.10 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 198.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 449.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 387.84 examples/s]

Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 910.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 329.75 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 780.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 175.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 328.92 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 251.75 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 389.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 453.52 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 382.73 examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 384.52 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 301.96 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.03 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 306.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 551.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 858.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 331.78 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 771.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 180.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 455.63 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 255.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 500.46 examples/s]


Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 201.53 examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 307.28 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 352.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 296.50 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 331.85 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 294.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 507.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 349.78 examples/s]


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 132.55 examples/s]

Filter (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 643.37 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 459.24 examples/s]

Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 615.86 examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 454.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 524.15 examples/s]


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 153.32 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 204.32 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 247.28 examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 423.82 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 167.73 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 330.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 294.66 examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 497.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 349.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 182.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 996.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 557.33 examples/s]


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 171.16 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1046.39 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 538.36 examples/s] 

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 147.81 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 270.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 380.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 487.05 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 177.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 316.67 examples/s]


Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 254.31 examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 340.01 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 893.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 291.61 examples/s]

Filter (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 805.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 716.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 117.37 examples/s]
Filter (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 625.95 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 208.61 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 259.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 465.90 examples/s]


Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 459.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 338.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 163.93 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 330.50 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 438.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 492.47 examples/s]

Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 301.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 365.99 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 742.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 139.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 524.83 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 183.84 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 236.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 462.72 examples/s]


Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 250.12 examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 352.94 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 411.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 285.90 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 336.73 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 324.02 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 424.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 633.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 344.30 examples/s]


Filter (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 671.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 544.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 235.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.41 examples/s] 


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 146.64 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 204.82 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 228.59 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 432.88 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 135.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 308.28 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 288.14 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 347.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 352.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 195.64 examples/s]
Filter (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 974.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.23 examples/s]


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 163.73 examples/s]

Filter (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 837.06 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 536.05 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 142.31 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 263.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 331.58 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 221.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 472.43 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 270.20 examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 675.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 282.82 examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 348.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 193.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1096.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 559.10 examples/s] 


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 167.30 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 964.66 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 539.91 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 144.43 examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 213.23 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 221.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 443.64 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 182.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 319.14 examples/s]


Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 260.56 examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 407.10 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 310.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 301.68 examples/s]

Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 490.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 217.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.68 examples/s] 


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 147.86 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 310.62 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 391.95 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 181.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 334.25 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 322.16 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 429.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 414.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 876.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 308.11 examples/s]


Filter (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 716.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 556.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 238.63 examples/s]
Filter (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 1089.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.52 examples/s] 


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 150.27 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:01, 186.05 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 298.59 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.52 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 219.43 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 453.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 312.38 examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 395.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 312.65 examples/s]


Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 581.23 examples/s]

Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 426.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 220.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1052.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.96 examples/s] 


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 148.24 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 254.53 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 320.56 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 210.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 318.27 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 353.02 examples/s]
Map (num_proc=8):  79%|#######8  | 202/256 [00:00<00:00, 452.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 375.34 examples/s]

Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 905.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 303.32 examples/s]


Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 837.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 533.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 176.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:03, 62.02 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 250.28 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 289.02 examples/s]
Map (num_proc=8):  95%|#########5| 244/256 [00:00<00:00, 309.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:01<00:00, 185.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):   9%|8         | 23/256 [00:02<00:29,  7.84 examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:02, 106.52 examples/s]

Map (num_proc=8):  21%|##1       | 55/256 [00:03<00:09, 21.57 examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 259.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.71 examples/s]

Map (num_proc=8):  33%|###3      | 85/256 [00:03<00:04, 34.56 examples/s]

Map (num_proc=8):  58%|#####7    | 148/256 [00:03<00:01, 75.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 334.74 examples/s]


Map (num_proc=8):  82%|########2 | 211/256 [00:04<00:00, 90.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:04<00:00, 59.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 143.12 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 306.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 354.41 examples/s]

Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 837.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 369.34 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 772.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 117.29 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 210.05 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 418.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 458.60 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 386.04 examples/s]

Map (num_proc=8):  85%|########5 | 218/256 [00:00<00:00, 444.76 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 333.77 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 153.10 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 252.78 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 345.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 577.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 350.57 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 805.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 121.35 examples/s]
Filter (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 636.37 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 224.84 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 372.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 466.73 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 406.22 examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 471.64 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 274.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 301.75 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:01, 191.01 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 408.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 879.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 327.26 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 796.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 131.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 541.82 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 217.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 466.40 examples/s]


Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 277.27 examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 373.89 examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 388.53 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 453.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 276.30 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 316.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 410.02 examples/s]

Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 889.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 328.88 examples/s]


Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1254.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 768.12 examples/s] 

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 180.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 369.96 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 255.75 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 363.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 450.99 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 459.88 examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 499.49 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 329.51 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:02, 96.87 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 281.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 816.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 349.02 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 757.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 136.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 537.29 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 230.92 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 275.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 461.44 examples/s]


Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 278.39 examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 443.50 examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 467.25 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 295.89 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 297.94 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 305.58 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 463.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 919.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 317.62 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 762.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 116.16 examples/s]
Filter (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 728.29 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 185.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 465.82 examples/s]


Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 330.79 examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 327.24 examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 368.19 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 278.13 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 328.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 501.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 355.99 examples/s]


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 184.97 examples/s]

Filter (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 797.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 215.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 566.00 examples/s] 


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 146.02 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:01, 188.70 examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 346.54 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 281.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 331.45 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 410.53 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 390.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 344.92 examples/s]


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 243.67 examples/s]

Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 382.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 563.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 183.78 examples/s]
Filter (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 964.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 545.66 examples/s]


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 146.06 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 262.28 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 331.11 examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 492.63 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 181.68 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 382.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 302.06 examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 417.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 327.03 examples/s]


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 150.78 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 514.18 examples/s]

Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 162.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 940.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 522.74 examples/s]


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 145.70 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:01, 177.13 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 214.72 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 517.80 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 273.47 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 424.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 314.99 examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 433.28 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 476.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 353.74 examples/s]


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 158.92 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 898.52 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 507.74 examples/s]

Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 160.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 917.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 512.84 examples/s]


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 140.05 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 302.82 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 290.15 examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 454.98 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 156.71 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 344.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 298.96 examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 406.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 428.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 283.64 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 769.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 116.07 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 221.13 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 358.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 447.57 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 414.59 examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 484.07 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 295.01 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 237.39 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 286.40 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 500.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 504.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 620.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 339.25 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 747.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 115.25 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 196.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 472.75 examples/s]


Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 252.84 examples/s]

Map (num_proc=8):  68%|######8   | 175/256 [00:00<00:00, 410.58 examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 375.32 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 276.74 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 326.06 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 411.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 629.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 316.47 examples/s]


Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1348.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 752.88 examples/s] 

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 130.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 522.92 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 203.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 443.46 examples/s]


Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 311.17 examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 563.88 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 317.66 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:02, 102.33 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 240.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 614.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 342.59 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 726.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 153.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 443.50 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 224.14 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 244.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 414.01 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 381.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 340.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 168.07 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 206.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 535.03 examples/s]

Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 523.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 362.13 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 752.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 137.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 561.31 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 217.98 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 378.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 465.83 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 374.17 examples/s]

Map (num_proc=8):  85%|########4 | 217/256 [00:00<00:00, 369.65 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 275.30 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 473.16 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 488.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 368.89 examples/s]


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 155.92 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 897.96 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 504.19 examples/s]

Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 164.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1020.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 518.02 examples/s] 


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 150.56 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 218.13 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 215.45 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 334.07 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 187.19 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 340.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 311.66 examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 434.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 430.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 306.27 examples/s]


Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 520.38 examples/s]

Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 509.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 511.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 192.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1104.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 549.04 examples/s] 


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 152.08 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 219.14 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 234.42 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 362.04 examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 525.50 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 145.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 298.79 examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 501.69 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 438.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 353.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 234.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.88 examples/s] 


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 166.22 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 929.92 examples/s]
Map (num_proc=8):   0%|          | 0/231 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 519.06 examples/s]

Map (num_proc=8):  13%|#2        | 29/231 [00:00<00:01, 132.12 examples/s]
Map (num_proc=8):  38%|###7      | 87/231 [00:00<00:00, 271.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  75%|#######5  | 174/231 [00:00<00:00, 399.29 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 271.14 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 411.94 examples/s]
Map (num_proc=8): 100%|##########| 231/231 [00:00<00:00, 306.28 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 386.57 examples/s]
Filter (num_proc=8):   0%|          | 0/231 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 355.27 examples/s]

Filter (num_proc=8):  13%|#2        | 29/231 [00:00<00:01, 133.61 examples/s]
Filter (num_proc=8): 100%|##########| 231/231 [00:00<00:00, 815.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 231/231 [00:00<00:00, 455.69 examples/s]


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 168.97 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1032.75 examples/s]INFO 2025-05-19 16:54:01,553 instructlab.sdg.pipeline:199: Running block: eval_verify_question

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 673.82 examples/s] 

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.30 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 740.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1141.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 263.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 611.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 941.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.84 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 623.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1092.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.77 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 802.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 903.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1050.54 examples/s]

Map (num_proc=8):   0%|          | 0/86 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 22/86 [00:00<00:00, 164.04 examples/s]
Map (num_proc=8):  88%|########8 | 76/86 [00:00<00:00, 351.96 examples/s]
Map (num_proc=8): 100%|##########| 86/86 [00:00<00:00, 225.15 examples/s]

Filter (num_proc=8):   0%|          | 0/86 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 44/86 [00:00<00:00, 413.37 examples/s]
Filter (num_proc=8): 100%|##########| 86/86 [00:00<00:00, 377.66 examples/s]INFO 2025-05-19 16:54:08,340 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 17:01:16,017 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 354.25 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 712.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 903.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1085.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.69 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 570.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1083.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 257.78 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 391.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 558.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1091.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.54 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 471.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 561.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 889.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.44 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 644.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1024.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1096.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.03 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 484.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1078.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 352.26 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 879.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1079.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1098.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 347.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 512.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 909.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1086.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.10 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 553.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 895.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1084.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 313.94 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 414.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 915.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1088.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.49 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 560.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 845.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1089.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.38 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 865.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 909.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1094.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 298.76 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 454.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 873.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 547.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1103.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1084.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.57 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 508.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 956.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1094.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 306.66 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 507.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1096.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1093.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.33 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 590.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 941.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1101.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.74 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 513.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1119.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1090.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 574.01 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 489.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1031.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1079.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.06 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 690.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1158.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1088.01 examples/s]

Map (num_proc=8):   0%|          | 0/125 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 32/125 [00:00<00:00, 244.61 examples/s]
Map (num_proc=8):  76%|#######6  | 95/125 [00:00<00:00, 423.94 examples/s]
Map (num_proc=8): 100%|##########| 125/125 [00:00<00:00, 323.52 examples/s]

Filter (num_proc=8):   0%|          | 0/125 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 64/125 [00:00<00:00, 586.31 examples/s]
Filter (num_proc=8): 100%|##########| 125/125 [00:00<00:00, 547.08 examples/s]INFO 2025-05-19 17:01:43,070 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 17:07:52,546 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 314.83 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 671.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 558.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 895.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1064.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 316.80 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 651.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 566.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 893.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1081.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.59 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 744.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 556.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 899.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1082.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 244.82 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 319.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 747.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 425.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1206.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1100.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 548.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.98 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 622.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1094.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 483.80 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 469.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 945.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1098.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.81 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 572.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 925.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1079.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 336.66 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 677.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 945.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1090.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 330.22 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 449.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 351.35 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 632.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1132.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1091.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.00 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 628.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 627.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1083.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 309.89 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 471.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 924.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1087.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 544.43 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 494.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 559.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1082.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1092.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 420.02 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 673.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 922.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1099.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.85 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 715.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 896.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.46 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 526.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 907.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1089.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.61 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 591.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 918.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1081.86 examples/s]

Map (num_proc=8):   0%|          | 0/98 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 26/98 [00:00<00:00, 253.32 examples/s]
Map (num_proc=8):  63%|######3   | 62/98 [00:00<00:00, 300.47 examples/s]
Map (num_proc=8): 100%|##########| 98/98 [00:00<00:00, 254.98 examples/s]

Filter (num_proc=8):   0%|          | 0/98 [00:00<?, ? examples/s]
Filter (num_proc=8):  39%|###8      | 38/98 [00:00<00:00, 349.75 examples/s]
Filter (num_proc=8): 100%|##########| 98/98 [00:00<00:00, 430.05 examples/s]INFO 2025-05-19 17:08:30,214 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.19 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 480.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 867.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1057.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 352.38 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 461.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 563.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1158.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1082.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.20 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 486.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 558.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 900.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1066.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 319.36 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 431.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 872.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 551.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1160.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1064.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 318.11 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 653.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 558.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 932.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1068.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.30 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 648.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 909.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1074.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 318.03 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 689.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1143.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1072.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.20 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 489.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 550.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 931.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1078.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 344.67 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 730.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 883.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1078.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.42 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 633.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1103.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1076.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.64 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 618.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 859.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1090.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.20 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 500.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 565.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1111.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 326.62 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 638.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 549.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 891.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1084.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.83 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 538.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1078.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.48 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 542.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 564.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1106.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1088.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.15 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 484.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1086.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 653.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 623.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1081.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 335.27 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 572.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 552.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 883.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1077.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 341.79 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 670.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 920.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1084.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.78 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 593.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1087.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.14 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 634.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1089.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 346.81 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 713.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1110.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1086.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 344.83 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 723.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 619.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1077.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 326.84 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 707.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 869.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1071.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 347.55 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 667.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 898.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1084.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 339.76 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 700.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1083.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.78 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 481.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 561.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 636.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1096.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 317.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 884.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 884.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1065.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.73 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 499.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 915.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1072.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.47 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 739.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 824.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1078.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.09 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 587.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 926.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1102.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.76 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 570.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 884.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1071.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 597.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1060.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1077.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 339.92 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 660.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 871.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1075.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 312.17 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 684.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 803.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1060.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.78 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 570.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 915.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1066.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.22 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 582.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 889.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1072.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.36 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 656.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 861.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1085.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 620.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1081.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.57 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 558.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 554.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 902.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1099.48 examples/s]

Map (num_proc=8):   0%|          | 0/153 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 39/153 [00:00<00:00, 279.47 examples/s]
Map (num_proc=8):  75%|#######5  | 115/153 [00:00<00:00, 452.03 examples/s]
Map (num_proc=8): 100%|##########| 153/153 [00:00<00:00, 374.25 examples/s]

Filter (num_proc=8):   0%|          | 0/153 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 39/153 [00:00<00:00, 389.39 examples/s]
Filter (num_proc=8): 100%|##########| 153/153 [00:00<00:00, 641.19 examples/s]INFO 2025-05-19 17:09:25,047 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_20_20__Expanding_a_virtual_machine_disk/data_checkpoint_196932551ec34293ad74cc42d6f3384d.jsonl


Creating json from Arrow format:   0%|          | 0/9 [00:00<?, ?ba/s]
Creating json from Arrow format:  33%|###3      | 3/9 [00:00<00:00, 22.08ba/s]
Creating json from Arrow format:  67%|######6   | 6/9 [00:00<00:00, 24.84ba/s]
Creating json from Arrow format: 100%|##########| 9/9 [00:00<00:00, 25.11ba/s]
Creating json from Arrow format: 100%|##########| 9/9 [00:00<00:00, 24.71ba/s]INFO 2025-05-19 17:09:25,418 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_20_20__Expanding_a_virtual_machine_disk/data_checkpoint_ee250d2e1ece4cf3b685075794f2aaa7.jsonl


Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format:  75%|#######5  | 3/4 [00:00<00:00, 29.93ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 30.57ba/s]INFO 2025-05-19 17:09:29,248 instructlab.sdg.generate_data:478: Generated 15052 samples
INFO 2025-05-19 17:09:29,308 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_8_3__Generalizing_a_Windows_VM_using_sysprep, generating from scratch
INFO 2025-05-19 17:09:29,308 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 17:09:29,311 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 17:09:29,620 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 17:09:29,634 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 17:09:29,642 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 17:09:36,234 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 17:09:45,318 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 17:09:50,610 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 17:09:50,630 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 17:09:50,645 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 17:11:07,359 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 17:13:37,662 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 566.10 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 578.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 875.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1146.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.18 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 815.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 878.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1097.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.48 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 558.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.40 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 592.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1166.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 436.08 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 477.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 872.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.77 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 593.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1092.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.78 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 638.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 889.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 470.01 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 667.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 812.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1097.67 examples/s]

Map (num_proc=8):   0%|          | 0/73 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 19/73 [00:00<00:00, 146.83 examples/s]
Map (num_proc=8):  75%|#######5  | 55/73 [00:00<00:00, 248.91 examples/s]
Map (num_proc=8): 100%|##########| 73/73 [00:00<00:00, 198.86 examples/s]

Filter (num_proc=8):   0%|          | 0/73 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 28/73 [00:00<00:00, 263.30 examples/s]
Filter (num_proc=8): 100%|##########| 73/73 [00:00<00:00, 328.94 examples/s]INFO 2025-05-19 17:13:48,674 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 17:15:19,235 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 478.25 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 527.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 932.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.57 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 611.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 876.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1094.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.15 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 488.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1229.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 446.87 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 608.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.52 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 588.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 906.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.70 examples/s]

Map (num_proc=8):   0%|          | 0/150 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 38/150 [00:00<00:00, 365.65 examples/s]
Map (num_proc=8):  51%|#####     | 76/150 [00:00<00:00, 359.58 examples/s]
Map (num_proc=8): 100%|##########| 150/150 [00:00<00:00, 381.98 examples/s]

Filter (num_proc=8):   0%|          | 0/150 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 57/150 [00:00<00:00, 562.87 examples/s]
Filter (num_proc=8): 100%|##########| 150/150 [00:00<00:00, 668.63 examples/s]INFO 2025-05-19 17:15:26,645 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 17:16:32,734 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.55 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 672.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 620.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1079.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 324.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 538.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1163.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 448.68 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 634.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 924.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.93 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 613.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 898.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 666.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.14 examples/s]

Map (num_proc=8):   0%|          | 0/78 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 20/78 [00:00<00:00, 153.07 examples/s]
Map (num_proc=8):  64%|######4   | 50/78 [00:00<00:00, 209.19 examples/s]
Map (num_proc=8): 100%|##########| 78/78 [00:00<00:00, 212.83 examples/s]

Filter (num_proc=8):   0%|          | 0/78 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 30/78 [00:00<00:00, 290.75 examples/s]
Filter (num_proc=8): 100%|##########| 78/78 [00:00<00:00, 350.22 examples/s]INFO 2025-05-19 17:16:40,120 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_8_3__Generalizing_a_Windows_VM_using_sysprep/data_checkpoint_7ae68098f99b44b8b0a0238557de1351.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 36.58ba/s]INFO 2025-05-19 17:16:40,500 instructlab.sdg.generate_data:478: Generated 1283 samples
INFO 2025-05-19 17:16:40,577 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_19_7__Viewing_the_IP_address_of_NICs_on_a_virtual_machine, generating from scratch
INFO 2025-05-19 17:16:40,578 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 17:16:40,581 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 17:16:43,274 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 17:16:43,289 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 17:16:43,297 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 17:17:04,348 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 17:17:29,694 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 17:17:47,174 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 17:17:47,206 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 17:17:47,230 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 17:25:07,568 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 17:43:59,992 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 304.00 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 433.36 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 753.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 908.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.06 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 781.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 766.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1018.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 488.02 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 547.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1129.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.56 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 681.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1108.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.93 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 634.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 862.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1119.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.37 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 577.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 601.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.43 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 644.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 897.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 354.00 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 823.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 629.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1101.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 329.71 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 690.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 907.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.25 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 614.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 630.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1101.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.34 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 472.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 635.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 336.24 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 720.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1094.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 345.11 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 901.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.39 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 545.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 843.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.17 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 770.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1143.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.10 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 637.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 913.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.16 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 695.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1185.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 432.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 660.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 918.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 420.13 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 689.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 869.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 617.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1147.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.37 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 836.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 920.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1119.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.09 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 584.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1060.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.22 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 776.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1212.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 485.42 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 518.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 935.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.78 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 708.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 910.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 844.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.92 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 627.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 629.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1099.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.14 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 939.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1092.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.08 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 760.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 931.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1090.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 310.65 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 676.26 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 672.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 634.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 415.20 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 753.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 442.54 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 699.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 941.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 432.89 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 525.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.77 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 675.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 611.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1104.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 302.13 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 641.03 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 747.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 465.02 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 585.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 918.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 600.53 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 520.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1092.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 309.14 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 610.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 863.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 415.59 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 621.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 566.77 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 543.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1068.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 498.54 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 798.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 554.04 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 488.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 925.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 621.35 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 617.67 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 737.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 620.78 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 511.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 921.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 462.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 903.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 843.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 609.47 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 566.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 850.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1095.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 620.65 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 509.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 926.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.42 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 756.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.72 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 592.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 919.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.71 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 682.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.28 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 766.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 890.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1097.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 494.70 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 714.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 875.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.68 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 627.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 899.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1097.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 472.19 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 525.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 913.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.08 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 618.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1101.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.14 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 630.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.67 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 668.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 899.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.42 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 694.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 897.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1102.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.81 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 616.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1106.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 272.30 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 401.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 226.49 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 611.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 626.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 248.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 609.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 923.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 267.52 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 529.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 874.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1102.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 242.86 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 587.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 911.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.30 examples/s]

Map (num_proc=8):   0%|          | 0/117 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 30/117 [00:00<00:00, 200.32 examples/s]
Map (num_proc=8):  76%|#######6  | 89/117 [00:00<00:00, 353.83 examples/s]
Map (num_proc=8): 100%|##########| 117/117 [00:00<00:00, 305.78 examples/s]

Filter (num_proc=8):   0%|          | 0/117 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 45/117 [00:00<00:00, 412.06 examples/s]
Filter (num_proc=8): 100%|##########| 117/117 [00:00<00:00, 520.24 examples/s]INFO 2025-05-19 17:45:21,735 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 17:55:18,259 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.25 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 484.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1150.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.22 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 495.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.29 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 653.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 317.28 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 741.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1095.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.97 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 494.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1213.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 450.29 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 692.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1145.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 327.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 877.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1146.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 625.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 955.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.29 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 691.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 790.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 893.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 339.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 861.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 864.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 500.82 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 481.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 889.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.33 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 731.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 926.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.37 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 713.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1078.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 509.56 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 519.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 630.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1091.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 498.08 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 563.31 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 716.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1084.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 571.45 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 495.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 833.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1102.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.73 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 659.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1229.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.09 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 851.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 921.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.58 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 607.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1068.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.00 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 740.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1127.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.90 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 684.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1184.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.39 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 678.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 931.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 335.36 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 777.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 632.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1093.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.08 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 573.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1043.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 345.25 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 717.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1214.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.42 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 596.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 832.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.36 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 757.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 892.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.06 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 685.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1181.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 336.44 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 865.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 913.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1092.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 352.10 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 726.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1213.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.79 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 656.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 639.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 461.06 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 691.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.44 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 626.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 909.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1096.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.68 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 636.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.40 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 637.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1205.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 327.00 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 729.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 927.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.18 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 732.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1213.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.42 examples/s]

Map (num_proc=8):   0%|          | 0/202 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 52/202 [00:00<00:00, 323.54 examples/s]
Map (num_proc=8):  88%|########7 | 177/202 [00:00<00:00, 656.94 examples/s]
Map (num_proc=8): 100%|##########| 202/202 [00:00<00:00, 491.36 examples/s]

Filter (num_proc=8):   0%|          | 0/202 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 102/202 [00:00<00:00, 916.13 examples/s]
Filter (num_proc=8): 100%|##########| 202/202 [00:00<00:00, 897.44 examples/s]INFO 2025-05-19 17:56:06,465 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 18:03:26,025 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 318.57 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 422.30 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 758.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 551.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1131.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 451.71 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 535.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 783.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 553.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1096.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 540.52 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 468.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 772.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 556.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 292.63 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 635.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 552.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 945.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 510.26 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 445.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 560.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1125.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.80 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 458.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 886.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 548.10 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 511.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 559.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1145.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 556.56 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 558.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 713.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 536.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 921.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1092.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 506.54 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 512.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 554.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 505.83 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 480.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 557.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1141.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.80 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 596.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 559.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 899.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.10 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 661.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.32 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 697.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 558.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 308.63 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 493.71 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 683.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 558.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1158.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.58 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 558.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 559.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1202.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1119.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 566.26 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 504.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 781.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.17 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 466.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 563.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 926.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 543.51 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 561.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 780.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 557.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 918.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 556.31 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 510.45 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 653.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 557.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1137.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1101.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 491.35 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 501.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 768.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 558.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 448.25 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 539.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 561.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 880.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 575.84 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 466.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 857.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.73 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 707.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 852.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 523.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 316.69 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 674.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 926.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.34 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 620.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 890.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.20 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 488.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 563.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 934.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1098.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.79 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 593.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 564.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 894.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.06 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 743.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.02 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 618.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1102.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.75 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 603.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 557.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 912.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1085.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.37 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 618.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 560.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.92 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 637.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 478.25 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 465.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 566.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 884.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 317.76 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 492.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 890.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 928.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.39 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 617.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 614.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1042.53 examples/s]

Map (num_proc=8):   0%|          | 0/30 [00:00<?, ? examples/s]
Map (num_proc=8):  40%|####      | 12/30 [00:00<00:00, 79.52 examples/s]
Map (num_proc=8): 100%|##########| 30/30 [00:00<00:00, 126.69 examples/s]
Map (num_proc=8): 100%|##########| 30/30 [00:00<00:00, 84.97 examples/s] 

Filter (num_proc=8):   0%|          | 0/30 [00:00<?, ? examples/s]
Filter (num_proc=8):  53%|#####3    | 16/30 [00:00<00:00, 152.29 examples/s]
Filter (num_proc=8): 100%|##########| 30/30 [00:00<00:00, 136.37 examples/s]INFO 2025-05-19 18:04:13,751 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_19_7__Viewing_the_IP_address_of_NICs_on_a_virtual_machine/data_checkpoint_0803a3536988435681e9ab4641fa1cc2.jsonl


Creating json from Arrow format:   0%|          | 0/8 [00:00<?, ?ba/s]
Creating json from Arrow format:  38%|###7      | 3/8 [00:00<00:00, 22.72ba/s]
Creating json from Arrow format:  75%|#######5  | 6/8 [00:00<00:00, 24.85ba/s]
Creating json from Arrow format: 100%|##########| 8/8 [00:00<00:00, 25.78ba/s]INFO 2025-05-19 18:04:15,917 instructlab.sdg.generate_data:478: Generated 7808 samples
INFO 2025-05-19 18:04:15,975 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_1_4__Restricted_network_environments, generating from scratch
INFO 2025-05-19 18:04:15,976 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 18:04:15,979 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 18:04:16,648 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 18:04:16,662 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 18:04:16,669 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 18:04:25,536 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 18:04:36,777 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 18:04:45,013 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 18:04:45,035 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 18:04:45,050 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 18:06:30,928 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 18:10:44,651 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 440.66 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 566.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 637.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1100.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 436.43 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 611.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1081.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.77 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 501.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.67 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 774.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.69 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 921.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1240.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 426.53 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 613.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1208.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.18 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 758.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 410.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 660.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1105.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.54 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 686.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 924.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.16 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 625.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.78 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 736.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1239.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.58 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 643.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.71 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 537.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 885.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.19 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 699.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1065.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.21 examples/s]

Map (num_proc=8):   0%|          | 0/121 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 31/121 [00:00<00:00, 201.39 examples/s]
Map (num_proc=8):  75%|#######5  | 91/121 [00:00<00:00, 339.39 examples/s]
Map (num_proc=8): 100%|##########| 121/121 [00:00<00:00, 307.48 examples/s]

Filter (num_proc=8):   0%|          | 0/121 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 46/121 [00:00<00:00, 441.11 examples/s]
Filter (num_proc=8): 100%|##########| 121/121 [00:00<00:00, 543.98 examples/s]INFO 2025-05-19 18:11:03,017 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 18:13:31,350 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 631.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1188.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 582.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1081.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 315.98 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 475.63 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 780.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 945.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.63 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1167.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.85 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 527.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1196.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.31 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 669.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.73 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 443.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 877.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.49 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 686.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1080.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 351.08 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 760.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1073.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.04 examples/s]

Map (num_proc=8):   0%|          | 0/145 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 37/145 [00:00<00:00, 270.42 examples/s]
Map (num_proc=8):  75%|#######5  | 109/145 [00:00<00:00, 460.02 examples/s]
Map (num_proc=8): 100%|##########| 145/145 [00:00<00:00, 362.16 examples/s]

Filter (num_proc=8):   0%|          | 0/145 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 55/145 [00:00<00:00, 527.61 examples/s]
Filter (num_proc=8): 100%|##########| 145/145 [00:00<00:00, 662.77 examples/s]INFO 2025-05-19 18:13:43,714 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 18:15:31,888 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 316.61 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 663.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 921.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.60 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 486.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1099.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.88 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 712.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.84 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 557.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1080.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.75 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 727.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 923.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.14 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 648.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 932.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.56 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 659.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.00 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 605.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 932.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.99 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 820.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1073.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.40 examples/s]

Map (num_proc=8):   0%|          | 0/33 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##7       | 9/33 [00:00<00:00, 69.49 examples/s]
Map (num_proc=8):  76%|#######5  | 25/33 [00:00<00:00, 111.17 examples/s]
Map (num_proc=8): 100%|##########| 33/33 [00:00<00:00, 92.74 examples/s] 

Filter (num_proc=8):   0%|          | 0/33 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 17/33 [00:00<00:00, 153.60 examples/s]
Filter (num_proc=8): 100%|##########| 33/33 [00:00<00:00, 153.50 examples/s]INFO 2025-05-19 18:15:44,242 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_1_4__Restricted_network_environments/data_checkpoint_5e3fd8fa23f2414c8b35d14b6472ddba.jsonl


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 29.19ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 29.13ba/s]INFO 2025-05-19 18:15:44,885 instructlab.sdg.generate_data:478: Generated 2149 samples
INFO 2025-05-19 18:15:44,937 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_10_2__Editing_virtual_machine_templates, generating from scratch
INFO 2025-05-19 18:15:44,937 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 18:15:44,940 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 18:15:45,189 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 18:15:45,203 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 18:15:45,210 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 18:15:53,517 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 18:16:03,269 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 18:16:10,959 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 18:16:10,979 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 18:16:10,994 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 18:17:25,353 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 18:19:45,457 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.55 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 506.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 894.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 589.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1204.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 595.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1046.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 704.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.00 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 670.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1104.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.00 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 811.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.75 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 687.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 514.38 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 734.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 826.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.44 examples/s]

Map (num_proc=8):   0%|          | 0/103 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 26/103 [00:00<00:00, 197.48 examples/s]
Map (num_proc=8):  76%|#######5  | 78/103 [00:00<00:00, 309.96 examples/s]
Map (num_proc=8): 100%|##########| 103/103 [00:00<00:00, 281.91 examples/s]

Filter (num_proc=8):   0%|          | 0/103 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 52/103 [00:00<00:00, 455.31 examples/s]
Filter (num_proc=8): 100%|##########| 103/103 [00:00<00:00, 471.52 examples/s]INFO 2025-05-19 18:19:56,320 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 18:21:25,485 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.52 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 749.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1085.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.22 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 710.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1143.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.21 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 721.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1163.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.17 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 775.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1243.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.38 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 826.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.31 examples/s]

Map (num_proc=8):   0%|          | 0/185 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 47/185 [00:00<00:00, 327.18 examples/s]
Map (num_proc=8):  75%|#######5  | 139/185 [00:00<00:00, 524.47 examples/s]
Map (num_proc=8): 100%|##########| 185/185 [00:00<00:00, 476.33 examples/s]

Filter (num_proc=8):   0%|          | 0/185 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 93/185 [00:00<00:00, 888.85 examples/s]
Filter (num_proc=8): 100%|##########| 185/185 [00:00<00:00, 844.24 examples/s]INFO 2025-05-19 18:21:32,705 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 18:22:37,079 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.32 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 742.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 932.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 506.89 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 520.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 297.96 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 428.59 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 709.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 551.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.47 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 647.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1111.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.05 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 659.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 920.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.07 examples/s]

Map (num_proc=8):   0%|          | 0/149 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 38/149 [00:00<00:00, 285.99 examples/s]
Map (num_proc=8):  64%|######3   | 95/149 [00:00<00:00, 394.29 examples/s]
Map (num_proc=8): 100%|##########| 149/149 [00:00<00:00, 391.75 examples/s]

Filter (num_proc=8):   0%|          | 0/149 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 57/149 [00:00<00:00, 534.96 examples/s]
Filter (num_proc=8): 100%|##########| 149/149 [00:00<00:00, 685.82 examples/s]INFO 2025-05-19 18:22:44,361 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_10_2__Editing_virtual_machine_templates/data_checkpoint_62c8b7b8fb3f475dbfbb32e78a29e60c.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 35.93ba/s]INFO 2025-05-19 18:22:44,725 instructlab.sdg.generate_data:478: Generated 1298 samples
INFO 2025-05-19 18:22:44,788 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_11_2__About_the_QEMU_guest_agent_information_in_the_web_console, generating from scratch
INFO 2025-05-19 18:22:44,788 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 18:22:44,792 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 18:22:44,965 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 18:22:44,978 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 18:22:44,985 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 18:22:49,149 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 18:22:54,594 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 18:23:01,079 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 18:23:01,098 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 18:23:01,113 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 18:24:02,844 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 18:25:30,217 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.65 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 754.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1230.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 453.77 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 708.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 639.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1205.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.65 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 782.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 896.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 297.18 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 485.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 942.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 873.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.47 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 645.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1234.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.68 examples/s]
num_proc must be <= 3. Reducing num_proc to 3 for dataset of size 3.

Map (num_proc=3):   0%|          | 0/3 [00:00<?, ? examples/s]
Map (num_proc=3): 100%|##########| 3/3 [00:00<00:00, 27.79 examples/s]
Map (num_proc=3): 100%|##########| 3/3 [00:00<00:00, 14.70 examples/s]
num_proc must be <= 3. Reducing num_proc to 3 for dataset of size 3.

Filter (num_proc=3):   0%|          | 0/3 [00:00<?, ? examples/s]
Filter (num_proc=3): 100%|##########| 3/3 [00:00<00:00, 18.83 examples/s]INFO 2025-05-19 18:25:36,747 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 18:26:31,833 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 330.24 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 828.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 939.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 679.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1258.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 735.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1159.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.28 examples/s]

Map (num_proc=8):   0%|          | 0/83 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 22/83 [00:00<00:00, 179.23 examples/s]
Map (num_proc=8):  76%|#######5  | 63/83 [00:00<00:00, 263.82 examples/s]
Map (num_proc=8): 100%|##########| 83/83 [00:00<00:00, 226.42 examples/s]

Filter (num_proc=8):   0%|          | 0/83 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 43/83 [00:00<00:00, 392.16 examples/s]
Filter (num_proc=8): 100%|##########| 83/83 [00:00<00:00, 377.61 examples/s]INFO 2025-05-19 18:26:36,566 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 18:27:16,325 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.17 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 805.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 877.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 455.92 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 523.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 956.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.82 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 775.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.06 examples/s]

Map (num_proc=8):   0%|          | 0/59 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##7       | 16/59 [00:00<00:00, 129.25 examples/s]
Map (num_proc=8):  76%|#######6  | 45/59 [00:00<00:00, 201.92 examples/s]
Map (num_proc=8): 100%|##########| 59/59 [00:00<00:00, 166.20 examples/s]

Filter (num_proc=8):   0%|          | 0/59 [00:00<?, ? examples/s]
Filter (num_proc=8):  64%|######4   | 38/59 [00:00<00:00, 352.22 examples/s]
Filter (num_proc=8): 100%|##########| 59/59 [00:00<00:00, 276.33 examples/s]INFO 2025-05-19 18:27:21,029 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_11_2__About_the_QEMU_guest_agent_information_in_the_web_console/data_checkpoint_68827e8b28cb434da0c7f1ffdee5f814.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 28.35ba/s]INFO 2025-05-19 18:27:21,238 instructlab.sdg.generate_data:478: Generated 713 samples
INFO 2025-05-19 18:27:21,301 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_8_1__About_reviewing_top_consumers, generating from scratch
INFO 2025-05-19 18:27:21,301 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 18:27:21,304 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 18:27:21,486 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 18:27:21,499 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 18:27:21,507 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 18:27:26,585 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 18:27:30,711 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 18:27:34,648 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 18:27:34,668 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 18:27:34,683 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 18:28:36,018 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 18:30:11,174 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.78 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 731.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 865.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 339.96 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 704.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 956.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 354.65 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1249.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 671.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1100.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 255.59 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 709.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.55 examples/s]

Map (num_proc=8):   0%|          | 0/168 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 42/168 [00:00<00:00, 309.97 examples/s]
Map (num_proc=8):  75%|#######5  | 126/168 [00:00<00:00, 486.62 examples/s]
Map (num_proc=8): 100%|##########| 168/168 [00:00<00:00, 439.47 examples/s]

Filter (num_proc=8):   0%|          | 0/168 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 63/168 [00:00<00:00, 623.27 examples/s]
Filter (num_proc=8): 100%|##########| 168/168 [00:00<00:00, 774.73 examples/s]INFO 2025-05-19 18:30:18,325 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 18:31:10,876 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 343.18 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 857.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1140.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 446.26 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 506.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1178.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.71 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 807.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.67 examples/s]

Map (num_proc=8):   0%|          | 0/82 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 22/82 [00:00<00:00, 156.57 examples/s]
Map (num_proc=8):  88%|########7 | 72/82 [00:00<00:00, 310.89 examples/s]
Map (num_proc=8): 100%|##########| 82/82 [00:00<00:00, 226.82 examples/s]

Filter (num_proc=8):   0%|          | 0/82 [00:00<?, ? examples/s]
Filter (num_proc=8):  39%|###9      | 32/82 [00:00<00:00, 315.97 examples/s]
Filter (num_proc=8): 100%|##########| 82/82 [00:00<00:00, 363.82 examples/s]INFO 2025-05-19 18:31:15,637 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 18:31:54,656 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 746.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1226.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.79 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 657.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1172.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.59 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.77 examples/s]

Map (num_proc=8):   0%|          | 0/41 [00:00<?, ? examples/s]
Map (num_proc=8):  39%|###9      | 16/41 [00:00<00:00, 115.48 examples/s]
Map (num_proc=8): 100%|##########| 41/41 [00:00<00:00, 171.91 examples/s]
Map (num_proc=8): 100%|##########| 41/41 [00:00<00:00, 118.43 examples/s]

Filter (num_proc=8):   0%|          | 0/41 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 21/41 [00:00<00:00, 206.01 examples/s]
Filter (num_proc=8): 100%|##########| 41/41 [00:00<00:00, 189.79 examples/s]INFO 2025-05-19 18:31:59,384 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_8_1__About_reviewing_top_consumers/data_checkpoint_03f3723aace4445193f666b40a9008fb.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 29.26ba/s]INFO 2025-05-19 18:31:59,586 instructlab.sdg.generate_data:478: Generated 686 samples
INFO 2025-05-19 18:31:59,655 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_3__Installing_OpenShift_Virtualization_using_the_web_console, generating from scratch
INFO 2025-05-19 18:31:59,655 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 18:31:59,658 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 18:31:59,848 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 18:31:59,861 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 18:31:59,868 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 18:32:08,105 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 18:32:15,550 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 18:32:23,665 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 18:32:23,685 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 18:32:23,700 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 18:33:19,581 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 18:34:34,917 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.79 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 532.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 839.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1062.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 349.85 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 615.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1277.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.18 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 694.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 879.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 533.72 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 528.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1237.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.90 examples/s]

Map (num_proc=8):   0%|          | 0/26 [00:00<?, ? examples/s]
Map (num_proc=8):  31%|###       | 8/26 [00:00<00:00, 67.88 examples/s]
Map (num_proc=8):  77%|#######6  | 20/26 [00:00<00:00, 86.90 examples/s]
Map (num_proc=8): 100%|##########| 26/26 [00:00<00:00, 75.38 examples/s]

Filter (num_proc=8):   0%|          | 0/26 [00:00<?, ? examples/s]
Filter (num_proc=8):  54%|#####3    | 14/26 [00:00<00:00, 132.29 examples/s]
Filter (num_proc=8): 100%|##########| 26/26 [00:00<00:00, 128.57 examples/s]INFO 2025-05-19 18:34:40,878 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 18:35:26,432 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 579.07 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 490.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1181.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.70 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 668.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.80 examples/s]

Map (num_proc=8):   0%|          | 0/227 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 58/227 [00:00<00:00, 346.48 examples/s]
Map (num_proc=8):  75%|#######5  | 171/227 [00:00<00:00, 681.56 examples/s]
Map (num_proc=8): 100%|##########| 227/227 [00:00<00:00, 546.45 examples/s]

Filter (num_proc=8):   0%|          | 0/227 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 115/227 [00:00<00:00, 1048.74 examples/s]
Filter (num_proc=8): 100%|##########| 227/227 [00:00<00:00, 1011.12 examples/s]INFO 2025-05-19 18:35:30,095 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 18:36:02,380 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.48 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 611.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 886.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.14 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 599.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1051.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.88 examples/s]

Map (num_proc=8):   0%|          | 0/190 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 48/190 [00:00<00:00, 296.59 examples/s]
Map (num_proc=8):  76%|#######5  | 144/190 [00:00<00:00, 599.37 examples/s]
Map (num_proc=8): 100%|##########| 190/190 [00:00<00:00, 470.04 examples/s]

Filter (num_proc=8):   0%|          | 0/190 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 96/190 [00:00<00:00, 944.22 examples/s]
Filter (num_proc=8): 100%|##########| 190/190 [00:00<00:00, 866.35 examples/s]INFO 2025-05-19 18:36:06,026 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_3__Installing_OpenShift_Virtualization_using_the_web_console/data_checkpoint_d371caf753e4435fb6e69ceea4a2104d.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 25.65ba/s]INFO 2025-05-19 18:36:06,237 instructlab.sdg.generate_data:478: Generated 688 samples
INFO 2025-05-19 18:36:06,299 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_3_4__Viewing_resource_events_in_the_CLI, generating from scratch
INFO 2025-05-19 18:36:06,299 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 18:36:06,302 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 18:36:06,480 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 18:36:06,494 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 18:36:06,501 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 18:36:11,499 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 18:36:16,259 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 18:36:20,111 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 18:36:20,130 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 18:36:20,145 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 18:37:02,077 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 18:37:16,769 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/243 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 31/243 [00:00<00:00, 275.21 examples/s]
Map (num_proc=8):  51%|#####     | 123/243 [00:00<00:00, 606.94 examples/s]
Map (num_proc=8): 100%|##########| 243/243 [00:00<00:00, 619.14 examples/s]

Filter (num_proc=8):   0%|          | 0/243 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 123/243 [00:00<00:00, 1141.95 examples/s]
Filter (num_proc=8): 100%|##########| 243/243 [00:00<00:00, 1109.00 examples/s]INFO 2025-05-19 18:37:17,956 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 18:37:29,421 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/177 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 45/177 [00:00<00:00, 340.86 examples/s]
Map (num_proc=8):  75%|#######5  | 133/177 [00:00<00:00, 547.70 examples/s]
Map (num_proc=8): 100%|##########| 177/177 [00:00<00:00, 468.37 examples/s]

Filter (num_proc=8):   0%|          | 0/177 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######2   | 111/177 [00:00<00:00, 951.40 examples/s]
Filter (num_proc=8): 100%|##########| 177/177 [00:00<00:00, 820.27 examples/s]INFO 2025-05-19 18:37:30,585 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 18:37:39,026 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/168 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 42/168 [00:00<00:00, 354.89 examples/s]
Map (num_proc=8):  88%|########7 | 147/168 [00:00<00:00, 561.43 examples/s]
Map (num_proc=8): 100%|##########| 168/168 [00:00<00:00, 449.31 examples/s]

Filter (num_proc=8):   0%|          | 0/168 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 84/168 [00:00<00:00, 784.11 examples/s]
Filter (num_proc=8): 100%|##########| 168/168 [00:00<00:00, 762.56 examples/s]INFO 2025-05-19 18:37:40,190 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_3_4__Viewing_resource_events_in_the_CLI/data_checkpoint_a2c9e69c12ad497c8bdd1f87e8e89bd2.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 124.33ba/s]INFO 2025-05-19 18:37:40,244 instructlab.sdg.generate_data:478: Generated 162 samples
INFO 2025-05-19 18:37:40,313 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_1_2__Resources_monitored_actively_in_the_Virtualization_Overview_page, generating from scratch
INFO 2025-05-19 18:37:40,313 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 18:37:40,316 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 18:37:40,572 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 18:37:40,585 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 18:37:40,593 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 18:37:45,978 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 18:37:52,167 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 18:37:57,835 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 18:37:57,855 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 18:37:57,870 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 18:39:13,049 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 18:41:04,004 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 350.77 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 746.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1195.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 512.59 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 550.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1156.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.37 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1176.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 480.26 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 557.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 527.72 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 642.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 883.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.58 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 686.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1250.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.08 examples/s]

Map (num_proc=8):   0%|          | 0/135 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 34/135 [00:00<00:00, 256.76 examples/s]
Map (num_proc=8):  87%|########7 | 118/135 [00:00<00:00, 484.44 examples/s]
Map (num_proc=8): 100%|##########| 135/135 [00:00<00:00, 360.01 examples/s]

Filter (num_proc=8):   0%|          | 0/135 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 68/135 [00:00<00:00, 623.81 examples/s]
Filter (num_proc=8): 100%|##########| 135/135 [00:00<00:00, 633.84 examples/s]INFO 2025-05-19 18:41:12,357 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 18:42:26,297 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.25 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 663.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1089.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.13 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 439.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 477.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1080.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.76 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 777.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1232.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.45 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 594.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 922.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.81 examples/s]

Map (num_proc=8):   0%|          | 0/158 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 40/158 [00:00<00:00, 257.30 examples/s]
Map (num_proc=8):  88%|########7 | 139/158 [00:00<00:00, 565.34 examples/s]
Map (num_proc=8): 100%|##########| 158/158 [00:00<00:00, 405.46 examples/s]

Filter (num_proc=8):   0%|          | 0/158 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 80/158 [00:00<00:00, 749.09 examples/s]
Filter (num_proc=8): 100%|##########| 158/158 [00:00<00:00, 722.54 examples/s]INFO 2025-05-19 18:42:32,405 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 18:43:30,122 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 586.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1062.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.60 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 773.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1076.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.55 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 628.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 890.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 558.38 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 511.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1169.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.51 examples/s]

Map (num_proc=8):   0%|          | 0/121 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 31/121 [00:00<00:00, 237.05 examples/s]
Map (num_proc=8):  75%|#######5  | 91/121 [00:00<00:00, 396.09 examples/s]
Map (num_proc=8): 100%|##########| 121/121 [00:00<00:00, 322.13 examples/s]

Filter (num_proc=8):   0%|          | 0/121 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######2   | 76/121 [00:00<00:00, 653.90 examples/s]
Filter (num_proc=8): 100%|##########| 121/121 [00:00<00:00, 559.14 examples/s]INFO 2025-05-19 18:43:36,134 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_1_2__Resources_monitored_actively_in_the_Virtualization_Overview_page/data_checkpoint_342558a4ec904e60baa350513a13674d.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 23.68ba/s]INFO 2025-05-19 18:43:36,379 instructlab.sdg.generate_data:478: Generated 832 samples
INFO 2025-05-19 18:43:36,443 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_1_1__What_you_can_do_with_OpenShift_Virtualization, generating from scratch
INFO 2025-05-19 18:43:36,443 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 18:43:36,447 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 18:43:36,673 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 18:43:36,686 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 18:43:36,694 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 18:43:46,851 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 18:43:55,438 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 18:44:05,301 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 18:44:05,321 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 18:44:05,336 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 18:44:48,718 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 18:45:04,798 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/234 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 60/234 [00:00<00:00, 455.27 examples/s]
Map (num_proc=8):  50%|#####     | 118/234 [00:00<00:00, 413.65 examples/s]
Map (num_proc=8): 100%|##########| 234/234 [00:00<00:00, 544.38 examples/s]

Filter (num_proc=8):   0%|          | 0/234 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 118/234 [00:00<00:00, 1077.86 examples/s]
Filter (num_proc=8): 100%|##########| 234/234 [00:00<00:00, 1060.28 examples/s]INFO 2025-05-19 18:45:06,035 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 18:45:17,645 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/146 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 38/146 [00:00<00:00, 285.97 examples/s]
Map (num_proc=8):  63%|######3   | 92/146 [00:00<00:00, 393.50 examples/s]
Map (num_proc=8): 100%|##########| 146/146 [00:00<00:00, 368.19 examples/s]

Filter (num_proc=8):   0%|          | 0/146 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 74/146 [00:00<00:00, 717.66 examples/s]
Filter (num_proc=8): 100%|##########| 146/146 [00:00<00:00, 652.75 examples/s]INFO 2025-05-19 18:45:18,838 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 18:45:25,458 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/142 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 36/142 [00:00<00:00, 226.71 examples/s]
Map (num_proc=8):  63%|######3   | 90/142 [00:00<00:00, 364.59 examples/s]
Map (num_proc=8): 100%|##########| 142/142 [00:00<00:00, 363.32 examples/s]

Filter (num_proc=8):   0%|          | 0/142 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 72/142 [00:00<00:00, 618.74 examples/s]
Filter (num_proc=8): 100%|##########| 142/142 [00:00<00:00, 650.21 examples/s]INFO 2025-05-19 18:45:26,638 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_1_1__What_you_can_do_with_OpenShift_Virtualization/data_checkpoint_63309b8f484d4835ab5e11118e7abb16.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 93.52ba/s]INFO 2025-05-19 18:45:26,693 instructlab.sdg.generate_data:478: Generated 138 samples
INFO 2025-05-19 18:45:26,746 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_2_5__Adding_a_network_interface_to_a_virtual_machine, generating from scratch
INFO 2025-05-19 18:45:26,746 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 18:45:26,749 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 18:45:27,282 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 18:45:27,296 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 18:45:27,304 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 18:45:36,652 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 18:45:48,280 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 18:45:57,312 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 18:45:57,333 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 18:45:57,348 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 18:47:46,665 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 18:52:51,057 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 350.57 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 822.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1208.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.83 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.44 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 731.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 516.39 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 508.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 884.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 446.84 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 611.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 636.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1123.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 518.55 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 578.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1139.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.45 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 620.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1128.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.53 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 632.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1433.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.66 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 780.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.15 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.10 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 694.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 734.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 953.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.11 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 781.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1135.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 426.62 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 804.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 610.68 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 541.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 925.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1084.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.70 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 673.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 945.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 425.74 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 601.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 851.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 446.52 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 713.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 639.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1168.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.18 examples/s]

Map (num_proc=8):   0%|          | 0/42 [00:00<?, ? examples/s]
Map (num_proc=8):  29%|##8       | 12/42 [00:00<00:00, 101.24 examples/s]
Map (num_proc=8):  88%|########8 | 37/42 [00:00<00:00, 165.70 examples/s]
Map (num_proc=8): 100%|##########| 42/42 [00:00<00:00, 122.04 examples/s]

Filter (num_proc=8):   0%|          | 0/42 [00:00<?, ? examples/s]
Filter (num_proc=8):  40%|####      | 17/42 [00:00<00:00, 167.59 examples/s]
Filter (num_proc=8): 100%|##########| 42/42 [00:00<00:00, 202.19 examples/s]INFO 2025-05-19 18:53:13,956 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 18:55:56,338 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.83 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 585.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.92 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 645.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.39 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 477.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 912.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1119.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.05 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 566.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1248.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.17 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 742.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 953.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.10 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 640.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1109.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 425.97 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 680.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 893.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.90 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 785.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1125.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.20 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 730.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1157.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.57 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 653.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 911.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.59 examples/s]

Map (num_proc=8):   0%|          | 0/73 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###8      | 28/73 [00:00<00:00, 180.57 examples/s]
Map (num_proc=8): 100%|##########| 73/73 [00:00<00:00, 204.90 examples/s]

Filter (num_proc=8):   0%|          | 0/73 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 37/73 [00:00<00:00, 356.64 examples/s]
Filter (num_proc=8): 100%|##########| 73/73 [00:00<00:00, 341.95 examples/s]INFO 2025-05-19 18:56:09,653 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 18:58:00,585 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.36 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 650.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1235.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.70 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 532.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 941.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 589.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 410.42 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 427.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1189.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.64 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 823.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1231.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.30 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 777.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 893.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 349.94 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 756.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1071.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.79 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 657.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 959.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.24 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 735.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1157.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.93 examples/s]

Map (num_proc=8):   0%|          | 0/170 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 44/170 [00:00<00:00, 350.81 examples/s]
Map (num_proc=8):  75%|#######5  | 128/170 [00:00<00:00, 472.86 examples/s]
Map (num_proc=8): 100%|##########| 170/170 [00:00<00:00, 439.60 examples/s]

Filter (num_proc=8):   0%|          | 0/170 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 86/170 [00:00<00:00, 773.68 examples/s]
Filter (num_proc=8): 100%|##########| 170/170 [00:00<00:00, 767.97 examples/s]INFO 2025-05-19 18:58:12,776 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_2_5__Adding_a_network_interface_to_a_virtual_machine/data_checkpoint_85c31d8a980044d4a748c6d9b235e655.jsonl


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 32.27ba/s]INFO 2025-05-19 18:58:13,371 instructlab.sdg.generate_data:478: Generated 2122 samples
INFO 2025-05-19 18:58:13,433 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_2_2__Supported_guest_operating_systems, generating from scratch
INFO 2025-05-19 18:58:13,433 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 18:58:13,437 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 18:58:13,587 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 18:58:13,601 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 18:58:13,608 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 18:58:16,712 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 18:58:20,094 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 18:58:22,751 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 18:58:22,771 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 18:58:22,786 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 18:59:19,746 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 19:00:03,498 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.55 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 604.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1088.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.32 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 760.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 594.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 993.92 examples/s]

Map (num_proc=8):   0%|          | 0/142 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 36/142 [00:00<00:00, 207.86 examples/s]
Map (num_proc=8):  88%|########8 | 125/142 [00:00<00:00, 478.57 examples/s]
Map (num_proc=8): 100%|##########| 142/142 [00:00<00:00, 371.63 examples/s]

Filter (num_proc=8):   0%|          | 0/142 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 54/142 [00:00<00:00, 516.74 examples/s]
Filter (num_proc=8): 100%|##########| 142/142 [00:00<00:00, 640.77 examples/s]INFO 2025-05-19 19:00:07,122 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 19:00:26,138 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 611.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 941.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.22 examples/s]

Map (num_proc=8):   0%|          | 0/27 [00:00<?, ? examples/s]
Map (num_proc=8):  30%|##9       | 8/27 [00:00<00:00, 68.33 examples/s]
Map (num_proc=8):  89%|########8 | 24/27 [00:00<00:00, 109.93 examples/s]
Map (num_proc=8): 100%|##########| 27/27 [00:00<00:00, 79.14 examples/s] 

Filter (num_proc=8):   0%|          | 0/27 [00:00<?, ? examples/s]
Filter (num_proc=8):  44%|####4     | 12/27 [00:00<00:00, 107.23 examples/s]
Filter (num_proc=8): 100%|##########| 27/27 [00:00<00:00, 125.53 examples/s]INFO 2025-05-19 19:00:28,496 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 19:00:40,689 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.83 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 729.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 893.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.67 examples/s]
num_proc must be <= 3. Reducing num_proc to 3 for dataset of size 3.

Map (num_proc=3):   0%|          | 0/3 [00:00<?, ? examples/s]
Map (num_proc=3): 100%|##########| 3/3 [00:00<00:00, 29.68 examples/s]
Map (num_proc=3): 100%|##########| 3/3 [00:00<00:00, 14.65 examples/s]
num_proc must be <= 3. Reducing num_proc to 3 for dataset of size 3.

Filter (num_proc=3):   0%|          | 0/3 [00:00<?, ? examples/s]
Filter (num_proc=3): 100%|##########| 3/3 [00:00<00:00, 18.75 examples/s]INFO 2025-05-19 19:00:42,508 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_2_2__Supported_guest_operating_systems/data_checkpoint_11b0bc0a8b9c45f3bafbb650334b34f4.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 76.16ba/s]INFO 2025-05-19 19:00:42,589 instructlab.sdg.generate_data:478: Generated 243 samples
INFO 2025-05-19 19:00:42,652 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_6_6__Additional_resources, generating from scratch
INFO 2025-05-19 19:00:42,652 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 19:00:42,656 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 19:00:43,176 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 19:00:43,189 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 19:00:43,197 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 19:00:52,373 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 19:01:05,350 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 19:01:13,697 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 19:01:13,718 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 19:01:13,733 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 19:02:26,935 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 19:05:06,334 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.55 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 748.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1184.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.45 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 519.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1140.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.97 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 628.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.93 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 682.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 902.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.88 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 657.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 351.33 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 732.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 915.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.76 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 649.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1089.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.92 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 709.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.24 examples/s]

Map (num_proc=8):   0%|          | 0/237 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 60/237 [00:00<00:00, 340.23 examples/s]
Map (num_proc=8):  88%|########7 | 208/237 [00:00<00:00, 702.54 examples/s]
Map (num_proc=8): 100%|##########| 237/237 [00:00<00:00, 571.57 examples/s]

Filter (num_proc=8):   0%|          | 0/237 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 120/237 [00:00<00:00, 1144.79 examples/s]
Filter (num_proc=8): 100%|##########| 237/237 [00:00<00:00, 1058.61 examples/s]INFO 2025-05-19 19:05:17,321 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 19:07:08,730 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.40 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 681.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 924.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1104.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 354.60 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 923.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.59 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 609.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1168.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 350.62 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 694.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1097.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.88 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 603.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 850.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.96 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 693.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1199.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.80 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 604.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 866.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1095.92 examples/s]

Map (num_proc=8):   0%|          | 0/42 [00:00<?, ? examples/s]
Map (num_proc=8):  29%|##8       | 12/42 [00:00<00:00, 107.18 examples/s]
Map (num_proc=8):  76%|#######6  | 32/42 [00:00<00:00, 156.90 examples/s]
Map (num_proc=8): 100%|##########| 42/42 [00:00<00:00, 121.58 examples/s]

Filter (num_proc=8):   0%|          | 0/42 [00:00<?, ? examples/s]
Filter (num_proc=8):  40%|####      | 17/42 [00:00<00:00, 166.97 examples/s]
Filter (num_proc=8): 100%|##########| 42/42 [00:00<00:00, 195.92 examples/s]INFO 2025-05-19 19:07:18,527 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 19:08:41,274 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.91 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 605.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1155.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.65 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 705.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 845.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1159.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 348.57 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 646.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 956.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 499.95 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 537.77 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 685.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 874.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.46 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 491.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1238.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.09 examples/s]

Map (num_proc=8):   0%|          | 0/229 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 58/229 [00:00<00:00, 369.04 examples/s]
Map (num_proc=8):  63%|######2   | 144/229 [00:00<00:00, 571.91 examples/s]
Map (num_proc=8): 100%|##########| 229/229 [00:00<00:00, 540.89 examples/s]

Filter (num_proc=8):   0%|          | 0/229 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 116/229 [00:00<00:00, 1118.53 examples/s]
Filter (num_proc=8): 100%|##########| 229/229 [00:00<00:00, 1027.06 examples/s]INFO 2025-05-19 19:08:49,948 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_6_6__Additional_resources/data_checkpoint_e47d5c623f4f483094fe7ff4c89bf3d0.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 27.65ba/s]INFO 2025-05-19 19:08:50,376 instructlab.sdg.generate_data:478: Generated 1425 samples
INFO 2025-05-19 19:08:50,431 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_14_3_3__Additional_resources, generating from scratch
INFO 2025-05-19 19:08:50,432 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 19:08:50,435 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 19:08:51,188 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 19:08:51,201 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 19:08:51,209 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 19:09:02,635 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 19:09:14,351 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 19:09:26,177 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 19:09:26,198 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 19:09:26,213 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 19:11:01,211 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 19:14:44,277 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.01 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 880.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.19 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 751.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 441.70 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 721.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 457.22 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 549.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1188.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.20 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 728.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1376.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.11 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 489.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 782.35 examples/s]
Filter (num_proc=8):  75%|#######5  | 192/256 [00:02<00:00, 77.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:02<00:00, 114.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 468.71 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 509.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1231.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.45 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 608.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1070.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 810.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1195.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.97 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 634.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1102.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 420.00 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 824.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 872.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 547.38 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 518.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.40 examples/s]

Map (num_proc=8):   0%|          | 0/115 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 30/115 [00:00<00:00, 215.92 examples/s]
Map (num_proc=8): 100%|##########| 115/115 [00:00<00:00, 427.96 examples/s]
Map (num_proc=8): 100%|##########| 115/115 [00:00<00:00, 302.32 examples/s]

Filter (num_proc=8):   0%|          | 0/115 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 73/115 [00:00<00:00, 662.99 examples/s]
Filter (num_proc=8): 100%|##########| 115/115 [00:00<00:00, 536.27 examples/s]INFO 2025-05-19 19:15:02,307 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 19:17:17,610 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.94 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 517.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1166.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.61 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 525.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.45 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 648.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 945.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.04 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 513.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1139.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.30 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 808.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 587.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.47 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 624.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1091.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.31 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 590.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1105.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.77 examples/s]

Map (num_proc=8):   0%|          | 0/142 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 36/142 [00:00<00:00, 338.45 examples/s]
Map (num_proc=8):  51%|#####     | 72/142 [00:00<00:00, 349.76 examples/s]
Map (num_proc=8): 100%|##########| 142/142 [00:00<00:00, 368.59 examples/s]

Filter (num_proc=8):   0%|          | 0/142 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 72/142 [00:00<00:00, 669.92 examples/s]
Filter (num_proc=8): 100%|##########| 142/142 [00:00<00:00, 647.91 examples/s]INFO 2025-05-19 19:17:28,433 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 19:19:08,414 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.22 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 552.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 908.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.63 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 786.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1147.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 426.03 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 786.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 624.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1049.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.65 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 779.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 860.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.46 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 638.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.46 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 756.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 890.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.20 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 759.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.35 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 607.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 955.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.17 examples/s]

Map (num_proc=8):   0%|          | 0/17 [00:00<?, ? examples/s]
Map (num_proc=8):  41%|####1     | 7/17 [00:00<00:00, 48.62 examples/s]
Map (num_proc=8): 100%|##########| 17/17 [00:00<00:00, 71.70 examples/s]
Map (num_proc=8): 100%|##########| 17/17 [00:00<00:00, 50.71 examples/s]

Filter (num_proc=8):   0%|          | 0/17 [00:00<?, ? examples/s]
Filter (num_proc=8):  41%|####1     | 7/17 [00:00<00:00, 68.82 examples/s]
Filter (num_proc=8): 100%|##########| 17/17 [00:00<00:00, 80.52 examples/s]INFO 2025-05-19 19:19:19,283 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_14_3_3__Additional_resources/data_checkpoint_e6f776dc97934155b3fb540b7cd08401.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 25.37ba/s]INFO 2025-05-19 19:19:19,773 instructlab.sdg.generate_data:478: Generated 1687 samples
INFO 2025-05-19 19:19:19,837 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_11_4__Additional_resources, generating from scratch
INFO 2025-05-19 19:19:19,837 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 19:19:19,840 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 19:19:20,484 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 19:19:20,498 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 19:19:20,506 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 19:19:28,519 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 19:19:41,931 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 19:19:49,487 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 19:19:49,510 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 19:19:49,529 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 19:22:19,979 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 19:26:42,440 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 946.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 941.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 307.11 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 467.85 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 775.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1186.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 244.74 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 494.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1250.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.51 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 645.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 918.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 453.93 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 837.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1243.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.70 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 625.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 544.53 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 506.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 636.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.78 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 668.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1218.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.10 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 642.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1224.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.14 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 648.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 896.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.61 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 897.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.51 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 665.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 875.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.52 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 649.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.85 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 667.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1100.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.90 examples/s]

Map (num_proc=8):   0%|          | 0/219 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 56/219 [00:00<00:00, 344.66 examples/s]
Map (num_proc=8):  75%|#######5  | 165/219 [00:00<00:00, 644.34 examples/s]
Map (num_proc=8): 100%|##########| 219/219 [00:00<00:00, 552.37 examples/s]

Filter (num_proc=8):   0%|          | 0/219 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 111/219 [00:00<00:00, 1083.52 examples/s]
Filter (num_proc=8): 100%|##########| 219/219 [00:00<00:00, 1010.30 examples/s]INFO 2025-05-19 19:27:00,052 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 19:29:27,734 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.29 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 763.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 897.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.69 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 644.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 891.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 317.27 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 436.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1154.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.23 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 740.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1059.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 685.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.88 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 753.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1104.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.91 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 522.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 319.55 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 462.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.37 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 826.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.83 examples/s]
num_proc must be <= 5. Reducing num_proc to 5 for dataset of size 5.

Map (num_proc=5):   0%|          | 0/5 [00:00<?, ? examples/s]
Map (num_proc=5):  40%|####      | 2/5 [00:00<00:00, 18.72 examples/s]
Map (num_proc=5): 100%|##########| 5/5 [00:00<00:00, 19.65 examples/s]
num_proc must be <= 5. Reducing num_proc to 5 for dataset of size 5.

Filter (num_proc=5):   0%|          | 0/5 [00:00<?, ? examples/s]
Filter (num_proc=5): 100%|##########| 5/5 [00:00<00:00, 28.16 examples/s]INFO 2025-05-19 19:29:39,331 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 19:31:23,758 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 454.72 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 685.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 790.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.13 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 593.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1154.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.04 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 578.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1109.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.03 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 667.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 874.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.96 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1090.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.99 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 515.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 536.59 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 510.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1125.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.23 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 734.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 927.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.32 examples/s]

Map (num_proc=8):   0%|          | 0/179 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 46/179 [00:00<00:00, 294.09 examples/s]
Map (num_proc=8):  75%|#######5  | 135/179 [00:00<00:00, 508.06 examples/s]
Map (num_proc=8): 100%|##########| 179/179 [00:00<00:00, 453.59 examples/s]

Filter (num_proc=8):   0%|          | 0/179 [00:00<?, ? examples/s]
Filter (num_proc=8):  39%|###8      | 69/179 [00:00<00:00, 661.04 examples/s]
Filter (num_proc=8): 100%|##########| 179/179 [00:00<00:00, 830.96 examples/s]INFO 2025-05-19 19:31:34,546 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_11_4__Additional_resources/data_checkpoint_9576d950ab0943b788c230e6e97d3fe8.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 27.24ba/s]INFO 2025-05-19 19:31:34,996 instructlab.sdg.generate_data:478: Generated 1526 samples
INFO 2025-05-19 19:31:35,057 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_10_5__Deleting_virtual_machine_templates, generating from scratch
INFO 2025-05-19 19:31:35,057 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 19:31:35,061 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 19:31:35,174 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 19:31:35,188 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 19:31:35,195 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 19:31:37,901 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 19:31:41,250 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 19:31:44,434 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 19:31:44,453 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 19:31:44,468 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 19:32:36,598 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 19:33:47,027 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.60 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 735.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 650.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1376.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1192.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 469.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 664.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 657.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1144.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.95 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 734.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 644.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 468.90 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 812.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 655.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1196.37 examples/s]

Map (num_proc=8):   0%|          | 0/105 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 27/105 [00:00<00:00, 258.20 examples/s]
Map (num_proc=8):  75%|#######5  | 79/105 [00:00<00:00, 403.69 examples/s]
Map (num_proc=8): 100%|##########| 105/105 [00:00<00:00, 294.51 examples/s]

Filter (num_proc=8):   0%|          | 0/105 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 53/105 [00:00<00:00, 529.63 examples/s]
Filter (num_proc=8): 100%|##########| 105/105 [00:00<00:00, 502.69 examples/s]INFO 2025-05-19 19:33:52,748 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 19:34:36,917 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 449.64 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 544.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 649.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1122.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 448.43 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 618.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 651.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1147.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.78 examples/s]

Map (num_proc=8):   0%|          | 0/228 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 58/228 [00:00<00:00, 380.89 examples/s]
Map (num_proc=8):  75%|#######5  | 172/228 [00:00<00:00, 733.83 examples/s]
Map (num_proc=8): 100%|##########| 228/228 [00:00<00:00, 589.56 examples/s]

Filter (num_proc=8):   0%|          | 0/228 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 87/228 [00:00<00:00, 831.18 examples/s]
Filter (num_proc=8): 100%|##########| 228/228 [00:00<00:00, 1057.89 examples/s]INFO 2025-05-19 19:34:40,389 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 19:35:12,479 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 464.13 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 496.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1104.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1089.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 650.08 examples/s] 

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1198.77 examples/s]

Map (num_proc=8):   0%|          | 0/212 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 54/212 [00:00<00:00, 397.46 examples/s]
Map (num_proc=8):  75%|#######5  | 160/212 [00:00<00:00, 667.16 examples/s]
Map (num_proc=8): 100%|##########| 212/212 [00:00<00:00, 556.98 examples/s]

Filter (num_proc=8):   0%|          | 0/212 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 81/212 [00:00<00:00, 756.84 examples/s]
Filter (num_proc=8): 100%|##########| 212/212 [00:00<00:00, 954.32 examples/s]INFO 2025-05-19 19:35:15,951 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_10_5__Deleting_virtual_machine_templates/data_checkpoint_aa41c9cbc2cf42969300243df70cf221.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 41.84ba/s]INFO 2025-05-19 19:35:16,125 instructlab.sdg.generate_data:478: Generated 649 samples
INFO 2025-05-19 19:35:16,187 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_2_7__About_the_tekton_tasks_operator, generating from scratch
INFO 2025-05-19 19:35:16,187 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 19:35:16,190 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 19:35:16,313 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 19:35:16,326 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 19:35:16,333 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 19:35:17,586 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 19:35:44,776 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 19:35:46,063 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 19:35:46,082 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 19:35:46,097 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 19:36:24,694 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 19:36:41,118 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/131 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 34/131 [00:00<00:00, 227.29 examples/s]
Map (num_proc=8):  51%|#####1    | 67/131 [00:00<00:00, 267.16 examples/s]
Map (num_proc=8): 100%|##########| 131/131 [00:00<00:00, 336.47 examples/s]

Filter (num_proc=8):   0%|          | 0/131 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 83/131 [00:00<00:00, 779.03 examples/s]
Filter (num_proc=8): 100%|##########| 131/131 [00:00<00:00, 617.89 examples/s]INFO 2025-05-19 19:36:42,268 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 19:36:51,484 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/123 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 32/123 [00:00<00:00, 209.79 examples/s]
Map (num_proc=8):  76%|#######5  | 93/123 [00:00<00:00, 368.83 examples/s]
Map (num_proc=8): 100%|##########| 123/123 [00:00<00:00, 314.85 examples/s]

Filter (num_proc=8):   0%|          | 0/123 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 63/123 [00:00<00:00, 584.55 examples/s]
Filter (num_proc=8): 100%|##########| 123/123 [00:00<00:00, 571.05 examples/s]INFO 2025-05-19 19:36:52,635 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 19:37:00,056 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/119 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 30/119 [00:00<00:00, 222.51 examples/s]
Map (num_proc=8):  76%|#######5  | 90/119 [00:00<00:00, 340.43 examples/s]
Map (num_proc=8): 100%|##########| 119/119 [00:00<00:00, 311.62 examples/s]

Filter (num_proc=8):   0%|          | 0/119 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 45/119 [00:00<00:00, 419.31 examples/s]
Filter (num_proc=8): 100%|##########| 119/119 [00:00<00:00, 512.23 examples/s]INFO 2025-05-19 19:37:01,214 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_2_7__About_the_tekton_tasks_operator/data_checkpoint_df617777a8114089a7e0fccd079f0eff.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 99.48ba/s]INFO 2025-05-19 19:37:01,263 instructlab.sdg.generate_data:478: Generated 113 samples
INFO 2025-05-19 19:37:01,325 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_2_2__Editing_a_virtual_machine_YAML_configuration_using_the_web_console, generating from scratch
INFO 2025-05-19 19:37:01,325 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 19:37:01,329 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 19:37:01,866 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 19:37:01,880 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 19:37:01,888 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 19:37:11,233 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 19:37:26,610 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 19:37:35,648 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 19:37:35,670 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 19:37:35,685 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 19:39:24,916 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 19:44:26,931 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.02 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 701.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.16 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 475.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.55 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 704.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.86 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 702.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1166.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.04 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 779.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1236.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.24 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 746.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 639.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 941.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.70 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 696.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.48 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 740.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1115.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.15 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 677.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 876.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 452.96 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 722.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1107.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.27 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 956.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 637.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1081.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.48 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 756.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.31 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 662.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1255.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 458.46 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 766.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 635.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 552.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 660.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 642.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 898.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.92 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 780.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 892.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 425.18 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 496.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 860.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.89 examples/s]

Map (num_proc=8):   0%|          | 0/212 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 54/212 [00:00<00:00, 399.41 examples/s]
Map (num_proc=8):  75%|#######5  | 160/212 [00:00<00:00, 651.86 examples/s]
Map (num_proc=8): 100%|##########| 212/212 [00:00<00:00, 543.82 examples/s]

Filter (num_proc=8):   0%|          | 0/212 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 81/212 [00:00<00:00, 791.37 examples/s]
Filter (num_proc=8): 100%|##########| 212/212 [00:00<00:00, 965.02 examples/s]INFO 2025-05-19 19:44:47,790 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 19:47:57,220 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.62 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 916.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 899.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.79 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 766.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1117.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.35 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 694.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1110.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 634.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.48 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 571.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.65 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 620.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 918.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.41 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 666.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1159.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.52 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 825.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1090.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.38 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 613.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 945.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.28 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 731.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1197.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.16 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 732.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 778.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1185.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.41 examples/s]

Map (num_proc=8):   0%|          | 0/15 [00:00<?, ? examples/s]
Map (num_proc=8):  40%|####      | 6/15 [00:00<00:00, 47.12 examples/s]
Map (num_proc=8): 100%|##########| 15/15 [00:00<00:00, 66.64 examples/s]
Map (num_proc=8): 100%|##########| 15/15 [00:00<00:00, 45.67 examples/s]

Filter (num_proc=8):   0%|          | 0/15 [00:00<?, ? examples/s]
Filter (num_proc=8):  53%|#####3    | 8/15 [00:00<00:00, 78.43 examples/s]
Filter (num_proc=8): 100%|##########| 15/15 [00:00<00:00, 72.25 examples/s]INFO 2025-05-19 19:48:12,371 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 19:50:23,292 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 713.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1104.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.10 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 634.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 916.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 348.41 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 727.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1328.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.23 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 756.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.14 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 556.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 918.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.09 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 766.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1090.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 345.78 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 597.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.77 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1082.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 466.24 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1195.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.14 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 711.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1123.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.34 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 909.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 904.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.89 examples/s]

Map (num_proc=8):   0%|          | 0/11 [00:00<?, ? examples/s]
Map (num_proc=8):  55%|#####4    | 6/11 [00:00<00:00, 38.33 examples/s]
Map (num_proc=8): 100%|##########| 11/11 [00:00<00:00, 33.00 examples/s]

Filter (num_proc=8):   0%|          | 0/11 [00:00<?, ? examples/s]
Filter (num_proc=8):  64%|######3   | 7/11 [00:00<00:00, 65.41 examples/s]
Filter (num_proc=8): 100%|##########| 11/11 [00:00<00:00, 54.15 examples/s]INFO 2025-05-19 19:50:37,281 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_2_2__Editing_a_virtual_machine_YAML_configuration_using_the_web_console/data_checkpoint_095c597c85c54f018c2d7f80db214724.jsonl


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 30.04ba/s]INFO 2025-05-19 19:50:37,956 instructlab.sdg.generate_data:478: Generated 2434 samples
INFO 2025-05-19 19:50:38,022 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_5__Viewing_information_about_virtual_machine_workloads, generating from scratch
INFO 2025-05-19 19:50:38,022 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 19:50:38,025 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 19:50:38,165 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 19:50:38,178 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 19:50:38,186 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 19:50:42,216 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 19:50:49,173 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 19:50:54,050 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 19:50:54,069 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 19:50:54,084 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 19:51:45,563 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 19:52:30,339 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.21 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 660.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 499.82 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 602.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1153.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.80 examples/s]

Map (num_proc=8):   0%|          | 0/120 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 30/120 [00:00<00:00, 231.91 examples/s]
Map (num_proc=8):  75%|#######5  | 90/120 [00:00<00:00, 370.30 examples/s]
Map (num_proc=8): 100%|##########| 120/120 [00:00<00:00, 327.91 examples/s]

Filter (num_proc=8):   0%|          | 0/120 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 45/120 [00:00<00:00, 448.35 examples/s]
Filter (num_proc=8): 100%|##########| 120/120 [00:00<00:00, 563.29 examples/s]INFO 2025-05-19 19:52:33,768 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 19:53:02,907 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.58 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 742.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1193.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.59 examples/s]

Map (num_proc=8):   0%|          | 0/189 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 48/189 [00:00<00:00, 345.83 examples/s]
Map (num_proc=8):  76%|#######5  | 143/189 [00:00<00:00, 620.49 examples/s]
Map (num_proc=8): 100%|##########| 189/189 [00:00<00:00, 488.82 examples/s]

Filter (num_proc=8):   0%|          | 0/189 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 96/189 [00:00<00:00, 847.44 examples/s]
Filter (num_proc=8): 100%|##########| 189/189 [00:00<00:00, 874.62 examples/s]INFO 2025-05-19 19:53:05,229 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 19:53:26,594 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 567.39 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 557.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1168.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.16 examples/s]

Map (num_proc=8):   0%|          | 0/174 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 44/174 [00:00<00:00, 266.87 examples/s]
Map (num_proc=8):  88%|########7 | 153/174 [00:00<00:00, 610.51 examples/s]
Map (num_proc=8): 100%|##########| 174/174 [00:00<00:00, 452.49 examples/s]

Filter (num_proc=8):   0%|          | 0/174 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 66/174 [00:00<00:00, 650.08 examples/s]
Filter (num_proc=8): 100%|##########| 174/174 [00:00<00:00, 802.93 examples/s]INFO 2025-05-19 19:53:28,900 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_5__Viewing_information_about_virtual_machine_workloads/data_checkpoint_0c30d62df3ef44e098df6327d7e32680.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 53.91ba/s]INFO 2025-05-19 19:53:29,017 instructlab.sdg.generate_data:478: Generated 375 samples
INFO 2025-05-19 19:53:29,084 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_10_1__About_the_OpenShift_Container_Platform_cluster_checkup_framework, generating from scratch
INFO 2025-05-19 19:53:29,084 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 19:53:29,088 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 19:53:29,715 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 19:53:29,729 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 19:53:29,736 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 19:53:41,147 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 19:53:53,120 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 19:53:59,620 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 19:53:59,643 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 19:53:59,658 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 19:55:32,739 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 19:58:42,195 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.69 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 521.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1192.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.93 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 640.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1074.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.13 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 642.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 559.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.61 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 685.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1146.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 548.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1110.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.72 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 542.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 896.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.44 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 703.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1106.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 585.95 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 519.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 549.99 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 463.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.74 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 657.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.27 examples/s]

Map (num_proc=8):   0%|          | 0/99 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 26/99 [00:00<00:00, 193.33 examples/s]
Map (num_proc=8):  88%|########7 | 87/99 [00:00<00:00, 360.38 examples/s]
Map (num_proc=8): 100%|##########| 99/99 [00:00<00:00, 264.38 examples/s]

Filter (num_proc=8):   0%|          | 0/99 [00:00<?, ? examples/s]
Filter (num_proc=8):  39%|###9      | 39/99 [00:00<00:00, 365.28 examples/s]
Filter (num_proc=8): 100%|##########| 99/99 [00:00<00:00, 469.79 examples/s]INFO 2025-05-19 19:58:55,331 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 20:00:53,696 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.63 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 616.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1136.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 331.87 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 718.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1083.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 280.63 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 563.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1182.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.44 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 497.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1140.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 586.70 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 555.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 892.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.44 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 670.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1148.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 546.92 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 526.91 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 677.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.09 examples/s]

Map (num_proc=8):   0%|          | 0/122 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 32/122 [00:00<00:00, 193.68 examples/s]
Map (num_proc=8): 100%|##########| 122/122 [00:00<00:00, 511.37 examples/s]
Map (num_proc=8): 100%|##########| 122/122 [00:00<00:00, 319.19 examples/s]

Filter (num_proc=8):   0%|          | 0/122 [00:00<?, ? examples/s]
Filter (num_proc=8):  39%|###8      | 47/122 [00:00<00:00, 447.00 examples/s]
Filter (num_proc=8): 100%|##########| 122/122 [00:00<00:00, 554.40 examples/s]INFO 2025-05-19 20:01:03,283 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 20:02:31,240 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.16 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 491.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 864.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.09 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 570.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1144.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.93 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 577.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 544.29 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 436.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 341.97 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 695.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1143.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.93 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 625.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 916.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.84 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 590.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1034.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.61 examples/s]

Map (num_proc=8):   0%|          | 0/30 [00:00<?, ? examples/s]
Map (num_proc=8):  40%|####      | 12/30 [00:00<00:00, 73.24 examples/s]
Map (num_proc=8): 100%|##########| 30/30 [00:00<00:00, 86.84 examples/s]

Filter (num_proc=8):   0%|          | 0/30 [00:00<?, ? examples/s]
Filter (num_proc=8):  40%|####      | 12/30 [00:00<00:00, 117.06 examples/s]
Filter (num_proc=8): 100%|##########| 30/30 [00:00<00:00, 143.19 examples/s]INFO 2025-05-19 20:02:40,825 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_10_1__About_the_OpenShift_Container_Platform_cluster_checkup_framework/data_checkpoint_d3662477506749399402705abd6844f2.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 24.77ba/s]INFO 2025-05-19 20:02:41,272 instructlab.sdg.generate_data:478: Generated 1467 samples
INFO 2025-05-19 20:02:41,330 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_9_3__About_Telemetry, generating from scratch
INFO 2025-05-19 20:02:41,330 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 20:02:41,334 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 20:02:41,712 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 20:02:41,726 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 20:02:41,734 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 20:02:51,114 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 20:02:59,002 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 20:03:04,122 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 20:03:04,142 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 20:03:04,157 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 20:04:35,524 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 20:07:46,340 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.53 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 494.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1186.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 444.81 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 545.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 908.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.55 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 686.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.43 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 691.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1176.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 610.24 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 509.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1159.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.75 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 489.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 959.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 331.95 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 725.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1184.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1186.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.57 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 685.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1167.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.53 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 564.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1127.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.65 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 678.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1158.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.34 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 776.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.16 examples/s]

Map (num_proc=8):   0%|          | 0/97 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 25/97 [00:00<00:00, 207.04 examples/s]
Map (num_proc=8):  63%|######2   | 61/97 [00:00<00:00, 279.32 examples/s]
Map (num_proc=8): 100%|##########| 97/97 [00:00<00:00, 268.75 examples/s]

Filter (num_proc=8):   0%|          | 0/97 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 49/97 [00:00<00:00, 439.86 examples/s]
Filter (num_proc=8): 100%|##########| 97/97 [00:00<00:00, 456.50 examples/s]INFO 2025-05-19 20:08:00,495 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 20:09:47,396 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 319.16 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 503.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 910.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1224.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.82 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 550.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 955.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.45 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 719.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 330.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 701.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 893.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 523.38 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 509.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 466.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 581.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1148.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.87 examples/s]

Map (num_proc=8):   0%|          | 0/224 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 56/224 [00:00<00:00, 334.93 examples/s]
Map (num_proc=8):  62%|######2   | 140/224 [00:00<00:00, 533.63 examples/s]
Map (num_proc=8): 100%|##########| 224/224 [00:00<00:00, 532.67 examples/s]

Filter (num_proc=8):   0%|          | 0/224 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 112/224 [00:00<00:00, 1027.22 examples/s]
Filter (num_proc=8): 100%|##########| 224/224 [00:00<00:00, 1008.89 examples/s]INFO 2025-05-19 20:09:55,750 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 20:11:10,953 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 336.66 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 591.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1250.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.68 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 657.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 350.02 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 694.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1163.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 317.39 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 454.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1262.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.71 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 558.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1092.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.53 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 681.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1107.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.58 examples/s]

Map (num_proc=8):   0%|          | 0/144 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 36/144 [00:00<00:00, 309.37 examples/s]
Map (num_proc=8):  62%|######2   | 90/144 [00:00<00:00, 361.66 examples/s]
Map (num_proc=8): 100%|##########| 144/144 [00:00<00:00, 367.98 examples/s]

Filter (num_proc=8):   0%|          | 0/144 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 54/144 [00:00<00:00, 539.25 examples/s]
Filter (num_proc=8): 100%|##########| 144/144 [00:00<00:00, 654.71 examples/s]INFO 2025-05-19 20:11:19,242 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_9_3__About_Telemetry/data_checkpoint_ce391b6a551145049772f01ae95b88f1.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 27.53ba/s]INFO 2025-05-19 20:11:19,697 instructlab.sdg.generate_data:478: Generated 1581 samples
INFO 2025-05-19 20:11:19,761 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_4_1__Deprecated_features, generating from scratch
INFO 2025-05-19 20:11:19,761 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 20:11:19,764 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 20:11:19,928 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 20:11:19,942 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 20:11:19,949 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 20:11:22,743 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 20:11:26,939 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 20:11:29,611 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 20:11:29,630 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 20:11:29,645 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 20:12:18,487 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 20:12:59,852 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.06 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 862.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1154.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.46 examples/s]

Map (num_proc=8):   0%|          | 0/253 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/253 [00:00<00:00, 415.01 examples/s]
Map (num_proc=8):  88%|########7 | 222/253 [00:00<00:00, 850.21 examples/s]
Map (num_proc=8): 100%|##########| 253/253 [00:00<00:00, 636.56 examples/s]

Filter (num_proc=8):   0%|          | 0/253 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 128/253 [00:00<00:00, 1218.84 examples/s]
Filter (num_proc=8): 100%|##########| 253/253 [00:00<00:00, 1143.03 examples/s]INFO 2025-05-19 20:13:02,172 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 20:13:29,240 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 637.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 914.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.68 examples/s]

Map (num_proc=8):   0%|          | 0/155 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###8      | 59/155 [00:00<00:00, 308.07 examples/s]
Map (num_proc=8):  88%|########7 | 136/155 [00:00<00:00, 504.97 examples/s]
Map (num_proc=8): 100%|##########| 155/155 [00:00<00:00, 383.47 examples/s]

Filter (num_proc=8):   0%|          | 0/155 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 79/155 [00:00<00:00, 762.38 examples/s]
Filter (num_proc=8): 100%|##########| 155/155 [00:00<00:00, 729.51 examples/s]INFO 2025-05-19 20:13:31,559 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 20:13:51,965 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.05 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 603.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1233.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.93 examples/s]

Map (num_proc=8):   0%|          | 0/124 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 32/124 [00:00<00:00, 226.57 examples/s]
Map (num_proc=8):  76%|#######5  | 94/124 [00:00<00:00, 409.19 examples/s]
Map (num_proc=8): 100%|##########| 124/124 [00:00<00:00, 329.62 examples/s]

Filter (num_proc=8):   0%|          | 0/124 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 64/124 [00:00<00:00, 553.30 examples/s]
Filter (num_proc=8): 100%|##########| 124/124 [00:00<00:00, 572.87 examples/s]INFO 2025-05-19 20:13:54,264 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_4_1__Deprecated_features/data_checkpoint_7576157ec8354becaeb771d4bd20c35e.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 62.27ba/s]INFO 2025-05-19 20:13:54,365 instructlab.sdg.generate_data:478: Generated 327 samples
INFO 2025-05-19 20:13:54,432 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_3_1__Adding_items_to_a_boot_order_list_in_the_web_console, generating from scratch
INFO 2025-05-19 20:13:54,432 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 20:13:54,436 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 20:13:54,765 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 20:13:54,778 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 20:13:54,786 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 20:14:00,264 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 20:14:08,623 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 20:14:14,257 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 20:14:14,277 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 20:14:14,292 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 20:15:18,279 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 20:17:26,238 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 542.45 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 558.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.84 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 695.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1188.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 441.70 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 488.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1251.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.46 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 623.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1214.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 746.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1197.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.49 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 499.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1170.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.61 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 615.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1206.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.61 examples/s]

Map (num_proc=8):   0%|          | 0/99 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 26/99 [00:00<00:00, 199.33 examples/s]
Map (num_proc=8):  76%|#######5  | 75/99 [00:00<00:00, 306.06 examples/s]
Map (num_proc=8): 100%|##########| 99/99 [00:00<00:00, 274.80 examples/s]

Filter (num_proc=8):   0%|          | 0/99 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 51/99 [00:00<00:00, 483.98 examples/s]
Filter (num_proc=8): 100%|##########| 99/99 [00:00<00:00, 464.38 examples/s]INFO 2025-05-19 20:17:35,478 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 20:19:01,816 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 501.86 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 570.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1067.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.66 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 789.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1129.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.35 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 772.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1249.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.21 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 606.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1157.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.31 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 681.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.46 examples/s]

Map (num_proc=8):   0%|          | 0/129 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 33/129 [00:00<00:00, 252.65 examples/s]
Map (num_proc=8):  88%|########7 | 113/129 [00:00<00:00, 461.59 examples/s]
Map (num_proc=8): 100%|##########| 129/129 [00:00<00:00, 345.35 examples/s]

Filter (num_proc=8):   0%|          | 0/129 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 65/129 [00:00<00:00, 601.04 examples/s]
Filter (num_proc=8): 100%|##########| 129/129 [00:00<00:00, 600.35 examples/s]INFO 2025-05-19 20:19:08,766 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 20:20:14,414 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 448.35 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 660.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1215.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.69 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 736.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1097.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.12 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 740.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1176.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 483.97 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 544.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1230.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 632.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.01 examples/s]

Map (num_proc=8):   0%|          | 0/82 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 22/82 [00:00<00:00, 154.94 examples/s]
Map (num_proc=8):  88%|########7 | 72/82 [00:00<00:00, 310.56 examples/s]
Map (num_proc=8): 100%|##########| 82/82 [00:00<00:00, 228.63 examples/s]

Filter (num_proc=8):   0%|          | 0/82 [00:00<?, ? examples/s]
Filter (num_proc=8):  39%|###9      | 32/82 [00:00<00:00, 303.85 examples/s]
Filter (num_proc=8): 100%|##########| 82/82 [00:00<00:00, 382.52 examples/s]INFO 2025-05-19 20:20:21,354 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_3_1__Adding_items_to_a_boot_order_list_in_the_web_console/data_checkpoint_688ace1ed3f34b168a7e2d8daa43ce8e.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 39.43ba/s]INFO 2025-05-19 20:20:21,672 instructlab.sdg.generate_data:478: Generated 1113 samples
INFO 2025-05-19 20:20:21,734 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_17_1__TLS_certificates_for_data_volume_imports, generating from scratch
INFO 2025-05-19 20:20:21,734 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 20:20:21,737 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 20:20:22,932 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 20:20:22,946 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 20:20:22,954 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 20:20:36,213 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 20:20:50,881 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 20:21:04,672 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 20:21:04,697 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 20:21:04,715 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 20:23:01,164 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 20:25:30,123 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.14 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 629.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1099.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.31 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 913.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 301.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 331.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 900.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 914.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.25 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 785.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1163.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.15 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 779.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1275.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 420.38 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 608.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1099.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.83 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 655.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 635.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1255.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.59 examples/s]

Map (num_proc=8):   0%|          | 0/101 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 26/101 [00:00<00:00, 205.16 examples/s]
Map (num_proc=8):  64%|######4   | 65/101 [00:00<00:00, 301.99 examples/s]
Map (num_proc=8): 100%|##########| 101/101 [00:00<00:00, 278.60 examples/s]

Filter (num_proc=8):   0%|          | 0/101 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 52/101 [00:00<00:00, 451.48 examples/s]
Filter (num_proc=8): 100%|##########| 101/101 [00:00<00:00, 482.14 examples/s]INFO 2025-05-19 20:25:40,554 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 20:27:08,746 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.28 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 520.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1113.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 452.30 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 529.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1090.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 325.48 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 687.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1234.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 323.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 948.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.42 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 607.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1126.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.81 examples/s]

Map (num_proc=8):   0%|          | 0/156 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 40/156 [00:00<00:00, 299.76 examples/s]
Map (num_proc=8):  76%|#######5  | 118/156 [00:00<00:00, 547.52 examples/s]
Map (num_proc=8): 100%|##########| 156/156 [00:00<00:00, 411.55 examples/s]

Filter (num_proc=8):   0%|          | 0/156 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 80/156 [00:00<00:00, 769.93 examples/s]
Filter (num_proc=8): 100%|##########| 156/156 [00:00<00:00, 709.43 examples/s]INFO 2025-05-19 20:27:15,799 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 20:28:19,244 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.68 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 486.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1320.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.59 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 693.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 876.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 460.57 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 559.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1091.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.22 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 764.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 598.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1223.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.91 examples/s]

Map (num_proc=8):   0%|          | 0/76 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 20/76 [00:00<00:00, 169.76 examples/s]
Map (num_proc=8):  88%|########8 | 67/76 [00:00<00:00, 304.37 examples/s]
Map (num_proc=8): 100%|##########| 76/76 [00:00<00:00, 212.56 examples/s]

Filter (num_proc=8):   0%|          | 0/76 [00:00<?, ? examples/s]
Filter (num_proc=8):  53%|#####2    | 40/76 [00:00<00:00, 388.77 examples/s]
Filter (num_proc=8): 100%|##########| 76/76 [00:00<00:00, 361.89 examples/s]INFO 2025-05-19 20:28:26,300 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_17_1__TLS_certificates_for_data_volume_imports/data_checkpoint_eb1ac2a30f3c49529429ade6dec1b6cf.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 22.38ba/s]INFO 2025-05-19 20:28:26,551 instructlab.sdg.generate_data:478: Generated 835 samples
INFO 2025-05-19 20:28:26,618 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_11_3__Migrating_a_virtual_machine_instance_to_another_node, generating from scratch
INFO 2025-05-19 20:28:26,618 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 20:28:26,621 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 20:28:26,818 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 20:28:26,831 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 20:28:26,839 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 20:28:32,526 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 20:28:37,425 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 20:28:40,528 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 20:28:40,547 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 20:28:40,562 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 20:29:12,802 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 20:29:23,824 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/160 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 40/160 [00:00<00:00, 279.76 examples/s]
Map (num_proc=8):  88%|########7 | 140/160 [00:00<00:00, 568.72 examples/s]
Map (num_proc=8): 100%|##########| 160/160 [00:00<00:00, 425.06 examples/s]

Filter (num_proc=8):   0%|          | 0/160 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 80/160 [00:00<00:00, 720.63 examples/s]
Filter (num_proc=8): 100%|##########| 160/160 [00:00<00:00, 738.92 examples/s]INFO 2025-05-19 20:29:24,968 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 20:29:34,745 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/144 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 36/144 [00:00<00:00, 317.57 examples/s]
Map (num_proc=8):  75%|#######5  | 108/144 [00:00<00:00, 439.98 examples/s]
Map (num_proc=8): 100%|##########| 144/144 [00:00<00:00, 390.33 examples/s]

Filter (num_proc=8):   0%|          | 0/144 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 54/144 [00:00<00:00, 503.18 examples/s]
Filter (num_proc=8): 100%|##########| 144/144 [00:00<00:00, 660.98 examples/s]INFO 2025-05-19 20:29:35,877 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 20:29:43,706 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/138 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 36/138 [00:00<00:00, 265.99 examples/s]
Map (num_proc=8):  88%|########7 | 121/138 [00:00<00:00, 508.45 examples/s]
Map (num_proc=8): 100%|##########| 138/138 [00:00<00:00, 375.18 examples/s]

Filter (num_proc=8):   0%|          | 0/138 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 70/138 [00:00<00:00, 621.75 examples/s]
Filter (num_proc=8): 100%|##########| 138/138 [00:00<00:00, 646.15 examples/s]INFO 2025-05-19 20:29:44,830 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_11_3__Migrating_a_virtual_machine_instance_to_another_node/data_checkpoint_ece5bd428fcf4fb9899c5e904ea74e92.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 152.49ba/s]INFO 2025-05-19 20:29:44,872 instructlab.sdg.generate_data:478: Generated 115 samples
INFO 2025-05-19 20:29:44,933 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_2__About_Red_Hat_OpenShift_Virtualization, generating from scratch
INFO 2025-05-19 20:29:44,933 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 20:29:44,936 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 20:29:45,078 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 20:29:45,092 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 20:29:45,099 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 20:29:48,209 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 20:29:51,434 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 20:29:54,053 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 20:29:54,072 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 20:29:54,087 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 20:30:49,435 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 20:31:45,498 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.77 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 491.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1086.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 445.63 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 519.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.78 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 495.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1115.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.49 examples/s]

Map (num_proc=8):   0%|          | 0/138 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 36/138 [00:00<00:00, 276.23 examples/s]
Map (num_proc=8):  63%|######3   | 87/138 [00:00<00:00, 392.97 examples/s]
Map (num_proc=8): 100%|##########| 138/138 [00:00<00:00, 373.15 examples/s]

Filter (num_proc=8):   0%|          | 0/138 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 70/138 [00:00<00:00, 591.82 examples/s]
Filter (num_proc=8): 100%|##########| 138/138 [00:00<00:00, 649.33 examples/s]INFO 2025-05-19 20:31:50,113 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 20:32:15,816 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.20 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 748.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.21 examples/s]

Map (num_proc=8):   0%|          | 0/120 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 30/120 [00:00<00:00, 241.30 examples/s]
Map (num_proc=8):  75%|#######5  | 90/120 [00:00<00:00, 356.44 examples/s]
Map (num_proc=8): 100%|##########| 120/120 [00:00<00:00, 325.56 examples/s]

Filter (num_proc=8):   0%|          | 0/120 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 75/120 [00:00<00:00, 629.55 examples/s]
Filter (num_proc=8): 100%|##########| 120/120 [00:00<00:00, 559.71 examples/s]INFO 2025-05-19 20:32:18,115 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 20:32:34,793 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 331.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 861.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1189.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.69 examples/s]

Map (num_proc=8):   0%|          | 0/97 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 25/97 [00:00<00:00, 189.17 examples/s]
Map (num_proc=8):  88%|########7 | 85/97 [00:00<00:00, 398.62 examples/s]
Map (num_proc=8): 100%|##########| 97/97 [00:00<00:00, 270.42 examples/s]

Filter (num_proc=8):   0%|          | 0/97 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 49/97 [00:00<00:00, 462.68 examples/s]
Filter (num_proc=8): 100%|##########| 97/97 [00:00<00:00, 457.99 examples/s]INFO 2025-05-19 20:32:37,082 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_2__About_Red_Hat_OpenShift_Virtualization/data_checkpoint_93cd980b5eae4961966246467c507b05.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 66.19ba/s]INFO 2025-05-19 20:32:37,181 instructlab.sdg.generate_data:478: Generated 330 samples
INFO 2025-05-19 20:32:37,238 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_8_2__OpenShift_Container_Platform_client_commands, generating from scratch
INFO 2025-05-19 20:32:37,238 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 20:32:37,241 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 20:32:37,392 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 20:32:37,405 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 20:32:37,413 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 20:32:42,556 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 20:32:50,660 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 20:32:55,809 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 20:32:55,828 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 20:32:55,843 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 20:33:39,702 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 20:34:08,569 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.52 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 596.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1156.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.22 examples/s]

Map (num_proc=8):   0%|          | 0/163 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 42/163 [00:00<00:00, 327.36 examples/s]
Map (num_proc=8):  63%|######3   | 103/163 [00:00<00:00, 469.79 examples/s]
Map (num_proc=8): 100%|##########| 163/163 [00:00<00:00, 428.88 examples/s]

Filter (num_proc=8):   0%|          | 0/163 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 83/163 [00:00<00:00, 711.41 examples/s]
Filter (num_proc=8): 100%|##########| 163/163 [00:00<00:00, 755.74 examples/s]INFO 2025-05-19 20:34:10,872 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 20:34:37,477 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.24 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 485.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1050.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.75 examples/s]

Map (num_proc=8):   0%|          | 0/158 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 40/158 [00:00<00:00, 297.84 examples/s]
Map (num_proc=8):  88%|########7 | 139/158 [00:00<00:00, 515.03 examples/s]
Map (num_proc=8): 100%|##########| 158/158 [00:00<00:00, 411.77 examples/s]

Filter (num_proc=8):   0%|          | 0/158 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 80/158 [00:00<00:00, 740.97 examples/s]
Filter (num_proc=8): 100%|##########| 158/158 [00:00<00:00, 740.10 examples/s]INFO 2025-05-19 20:34:39,775 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 20:35:00,528 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.13 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 635.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1128.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.93 examples/s]

Map (num_proc=8):   0%|          | 0/153 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 39/153 [00:00<00:00, 268.16 examples/s]
Map (num_proc=8):  88%|########7 | 134/153 [00:00<00:00, 533.00 examples/s]
Map (num_proc=8): 100%|##########| 153/153 [00:00<00:00, 403.55 examples/s]

Filter (num_proc=8):   0%|          | 0/153 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 77/153 [00:00<00:00, 716.17 examples/s]
Filter (num_proc=8): 100%|##########| 153/153 [00:00<00:00, 711.11 examples/s]INFO 2025-05-19 20:35:02,838 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_8_2__OpenShift_Container_Platform_client_commands/data_checkpoint_dc50663940c4486facb7896e7b5452a1.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 47.94ba/s]INFO 2025-05-19 20:35:02,961 instructlab.sdg.generate_data:478: Generated 395 samples
INFO 2025-05-19 20:35:03,041 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_12_1__Adding_a_secret__config_map__or_service_account_to_a_virtual_machine, generating from scratch
INFO 2025-05-19 20:35:03,041 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 20:35:03,045 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 20:35:03,293 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 20:35:03,307 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 20:35:03,314 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 20:35:10,191 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 20:35:17,650 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 20:35:24,232 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 20:35:24,253 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 20:35:24,268 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 20:36:25,431 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 20:38:17,891 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.25 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 578.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1176.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.89 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 590.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1201.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 444.75 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1268.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 870.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1182.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.11 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 829.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1161.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.74 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 932.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.72 examples/s]

Map (num_proc=8):   0%|          | 0/121 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###8      | 46/121 [00:00<00:00, 263.50 examples/s]
Map (num_proc=8): 100%|##########| 121/121 [00:00<00:00, 333.51 examples/s]

Filter (num_proc=8):   0%|          | 0/121 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 61/121 [00:00<00:00, 585.38 examples/s]
Filter (num_proc=8): 100%|##########| 121/121 [00:00<00:00, 564.00 examples/s]INFO 2025-05-19 20:38:26,088 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 20:39:32,687 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.39 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 743.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.48 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.61 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 618.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1127.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.38 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 600.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1086.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.88 examples/s]

Map (num_proc=8):   0%|          | 0/37 [00:00<?, ? examples/s]
Map (num_proc=8):  41%|####      | 15/37 [00:00<00:00, 111.23 examples/s]
Map (num_proc=8):  89%|########9 | 33/37 [00:00<00:00, 144.54 examples/s]
Map (num_proc=8): 100%|##########| 37/37 [00:00<00:00, 109.08 examples/s]

Filter (num_proc=8):   0%|          | 0/37 [00:00<?, ? examples/s]
Filter (num_proc=8):  54%|#####4    | 20/37 [00:00<00:00, 178.61 examples/s]
Filter (num_proc=8): 100%|##########| 37/37 [00:00<00:00, 174.11 examples/s]INFO 2025-05-19 20:39:38,548 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 20:40:22,248 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.30 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 688.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1208.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 432.22 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 524.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 888.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.96 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 723.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.89 examples/s]

Map (num_proc=8):   0%|          | 0/141 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 36/141 [00:00<00:00, 265.05 examples/s]
Map (num_proc=8):  76%|#######5  | 107/141 [00:00<00:00, 476.63 examples/s]
Map (num_proc=8): 100%|##########| 141/141 [00:00<00:00, 372.62 examples/s]

Filter (num_proc=8):   0%|          | 0/141 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 54/141 [00:00<00:00, 509.36 examples/s]
Filter (num_proc=8): 100%|##########| 141/141 [00:00<00:00, 645.05 examples/s]INFO 2025-05-19 20:40:26,972 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_12_1__Adding_a_secret__config_map__or_service_account_to_a_virtual_machine/data_checkpoint_ce3235f623d342008d592e626ddaf149.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 25.15ba/s]INFO 2025-05-19 20:40:27,197 instructlab.sdg.generate_data:478: Generated 747 samples
INFO 2025-05-19 20:40:27,269 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_11_3__Virtualization_metrics, generating from scratch
INFO 2025-05-19 20:40:27,269 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 20:40:27,272 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 20:40:28,015 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 20:40:28,029 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 20:40:28,037 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 20:40:36,044 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 20:40:48,099 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 20:40:55,670 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 20:40:55,691 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 20:40:55,709 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 20:43:19,220 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 20:48:26,884 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 608.09 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 605.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 642.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 931.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.42 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 771.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1129.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 628.09 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 485.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 614.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1135.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.79 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 767.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 614.84 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 561.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 649.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 441.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 703.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1107.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.87 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 750.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 447.46 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 740.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 924.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 613.18 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 532.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1187.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 300.37 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 604.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 639.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.15 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 731.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 883.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.98 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 665.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 636.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 927.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 468.91 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 566.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1189.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.54 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 823.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 636.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.70 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 744.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1222.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 426.35 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 678.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1197.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 470.62 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 645.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 646.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.55 examples/s]

Map (num_proc=8):   0%|          | 0/87 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 22/87 [00:00<00:00, 183.60 examples/s]
Map (num_proc=8):  87%|########7 | 76/87 [00:00<00:00, 308.49 examples/s]
Map (num_proc=8): 100%|##########| 87/87 [00:00<00:00, 238.54 examples/s]

Filter (num_proc=8):   0%|          | 0/87 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 44/87 [00:00<00:00, 413.94 examples/s]
Filter (num_proc=8): 100%|##########| 87/87 [00:00<00:00, 405.81 examples/s]INFO 2025-05-19 20:48:48,974 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 20:51:25,636 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.92 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 479.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 897.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.33 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 812.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 935.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.16 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 767.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 624.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 613.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 877.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.35 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 664.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 499.33 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 571.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.59 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 756.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.44 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 706.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 923.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 727.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 899.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.48 examples/s]

Map (num_proc=8):   0%|          | 0/230 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 58/230 [00:00<00:00, 445.39 examples/s]
Map (num_proc=8):  50%|#####     | 116/230 [00:00<00:00, 434.94 examples/s]
Map (num_proc=8): 100%|##########| 230/230 [00:00<00:00, 580.39 examples/s]

Filter (num_proc=8):   0%|          | 0/230 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 87/230 [00:00<00:00, 825.97 examples/s]
Filter (num_proc=8): 100%|##########| 230/230 [00:00<00:00, 1042.72 examples/s]INFO 2025-05-19 20:51:37,436 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 20:53:31,888 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.27 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 553.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1181.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.87 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 681.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 861.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.46 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 949.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 900.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 683.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1239.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.56 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 764.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 908.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.27 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 642.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1240.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.31 examples/s]
Map (num_proc=8):  56%|#####5    | 143/256 [00:00<00:00, 560.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1145.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.40 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 620.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1068.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 349.66 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 829.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 945.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.22 examples/s]

Map (num_proc=8):   0%|          | 0/133 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 34/133 [00:00<00:00, 254.56 examples/s]
Map (num_proc=8):  51%|#####1    | 68/133 [00:00<00:00, 296.07 examples/s]
Map (num_proc=8): 100%|##########| 133/133 [00:00<00:00, 355.11 examples/s]

Filter (num_proc=8):   0%|          | 0/133 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 51/133 [00:00<00:00, 471.59 examples/s]
Filter (num_proc=8): 100%|##########| 133/133 [00:00<00:00, 601.64 examples/s]INFO 2025-05-19 20:53:43,679 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_11_3__Virtualization_metrics/data_checkpoint_4981fdfec5884574926ff2324e50f88d.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 33.80ba/s]INFO 2025-05-19 20:53:44,065 instructlab.sdg.generate_data:478: Generated 1391 samples
INFO 2025-05-19 20:53:44,128 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_12_3__Managing_node_labeling_for_obsolete_CPU_models, generating from scratch
INFO 2025-05-19 20:53:44,128 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 20:53:44,132 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 20:53:44,487 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 20:53:44,500 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 20:53:44,508 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 20:53:52,775 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 20:54:01,314 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 20:54:09,585 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 20:54:09,604 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 20:54:09,620 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 20:54:53,915 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 20:55:11,902 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/247 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 62/247 [00:00<00:00, 418.38 examples/s]
Map (num_proc=8):  75%|#######4  | 185/247 [00:00<00:00, 700.12 examples/s]
Map (num_proc=8): 100%|##########| 247/247 [00:00<00:00, 602.29 examples/s]

Filter (num_proc=8):   0%|          | 0/247 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######2   | 155/247 [00:00<00:00, 1349.04 examples/s]
Filter (num_proc=8): 100%|##########| 247/247 [00:00<00:00, 1128.00 examples/s]INFO 2025-05-19 20:55:13,084 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 20:55:24,133 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/163 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 41/163 [00:00<00:00, 242.40 examples/s]
Map (num_proc=8):  88%|########7 | 143/163 [00:00<00:00, 537.22 examples/s]
Map (num_proc=8): 100%|##########| 163/163 [00:00<00:00, 416.16 examples/s]

Filter (num_proc=8):   0%|          | 0/163 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 83/163 [00:00<00:00, 803.81 examples/s]
Filter (num_proc=8): 100%|##########| 163/163 [00:00<00:00, 750.29 examples/s]INFO 2025-05-19 20:55:25,301 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 20:55:34,138 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/153 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 39/153 [00:00<00:00, 277.45 examples/s]
Map (num_proc=8):  75%|#######5  | 115/153 [00:00<00:00, 431.00 examples/s]
Map (num_proc=8): 100%|##########| 153/153 [00:00<00:00, 398.73 examples/s]

Filter (num_proc=8):   0%|          | 0/153 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 77/153 [00:00<00:00, 720.95 examples/s]
Filter (num_proc=8): 100%|##########| 153/153 [00:00<00:00, 700.23 examples/s]INFO 2025-05-19 20:55:35,290 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_12_3__Managing_node_labeling_for_obsolete_CPU_models/data_checkpoint_8c3837ff58ba47769549243e2e96fe1e.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 140.35ba/s]INFO 2025-05-19 20:55:35,336 instructlab.sdg.generate_data:478: Generated 118 samples
INFO 2025-05-19 20:55:35,408 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_12_5__Additional_resources, generating from scratch
INFO 2025-05-19 20:55:35,408 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 20:55:35,412 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 20:55:36,333 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 20:55:36,347 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 20:55:36,355 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 20:55:50,778 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 20:56:25,435 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 20:56:37,410 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 20:56:37,433 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 20:56:37,448 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 20:57:39,649 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 20:59:05,661 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 552.78 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 503.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1157.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.29 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 589.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1106.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 568.51 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 496.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1263.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.93 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 597.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1181.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.44 examples/s]

Map (num_proc=8):   0%|          | 0/102 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 26/102 [00:00<00:00, 170.51 examples/s]
Map (num_proc=8):  76%|#######6  | 78/102 [00:00<00:00, 307.65 examples/s]
Map (num_proc=8): 100%|##########| 102/102 [00:00<00:00, 270.54 examples/s]

Filter (num_proc=8):   0%|          | 0/102 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 52/102 [00:00<00:00, 498.81 examples/s]
Filter (num_proc=8): 100%|##########| 102/102 [00:00<00:00, 475.59 examples/s]INFO 2025-05-19 20:59:11,628 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 21:00:08,997 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 316.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 697.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1244.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.72 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 569.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 576.53 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 470.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.78 examples/s]

Map (num_proc=8):   0%|          | 0/175 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 44/175 [00:00<00:00, 309.12 examples/s]
Map (num_proc=8):  63%|######2   | 110/175 [00:00<00:00, 471.42 examples/s]
Map (num_proc=8): 100%|##########| 175/175 [00:00<00:00, 441.51 examples/s]

Filter (num_proc=8):   0%|          | 0/175 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 66/175 [00:00<00:00, 599.34 examples/s]
Filter (num_proc=8): 100%|##########| 175/175 [00:00<00:00, 804.56 examples/s]INFO 2025-05-19 21:00:13,799 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 21:00:56,035 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.81 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 498.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.56 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 703.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 465.52 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 552.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 828.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1163.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.41 examples/s]

Map (num_proc=8):   0%|          | 0/124 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 32/124 [00:00<00:00, 234.86 examples/s]
Map (num_proc=8):  52%|#####1    | 64/124 [00:00<00:00, 278.32 examples/s]
Map (num_proc=8): 100%|##########| 124/124 [00:00<00:00, 331.23 examples/s]

Filter (num_proc=8):   0%|          | 0/124 [00:00<?, ? examples/s]
Filter (num_proc=8):  64%|######3   | 79/124 [00:00<00:00, 725.98 examples/s]
Filter (num_proc=8): 100%|##########| 124/124 [00:00<00:00, 579.76 examples/s]INFO 2025-05-19 21:01:00,808 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_12_5__Additional_resources/data_checkpoint_9e70baed836c4297b3bdd3639168d10e.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 24.44ba/s]INFO 2025-05-19 21:01:01,028 instructlab.sdg.generate_data:478: Generated 714 samples
INFO 2025-05-19 21:01:01,107 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_12_4_1__Using_skip_node_annotation, generating from scratch
INFO 2025-05-19 21:01:01,107 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 21:01:01,111 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 21:01:01,262 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 21:01:01,276 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 21:01:01,283 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 21:01:04,203 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 21:01:09,024 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 21:01:11,851 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 21:01:11,869 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 21:01:11,884 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 21:01:54,512 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 21:02:16,263 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 612.28 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 680.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 652.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1409.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.36 examples/s]

Map (num_proc=8):   0%|          | 0/64 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 16/64 [00:00<00:00, 147.49 examples/s]
Map (num_proc=8):  88%|########7 | 56/64 [00:00<00:00, 247.27 examples/s]
Map (num_proc=8): 100%|##########| 64/64 [00:00<00:00, 177.01 examples/s]

Filter (num_proc=8):   0%|          | 0/64 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 24/64 [00:00<00:00, 238.95 examples/s]
Filter (num_proc=8): 100%|##########| 64/64 [00:00<00:00, 314.55 examples/s]INFO 2025-05-19 21:02:18,516 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 21:02:31,959 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/205 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 52/205 [00:00<00:00, 469.34 examples/s]
Map (num_proc=8):  51%|#####     | 104/205 [00:00<00:00, 497.05 examples/s]
Map (num_proc=8): 100%|##########| 205/205 [00:00<00:00, 542.81 examples/s]

Filter (num_proc=8):   0%|          | 0/205 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 104/205 [00:00<00:00, 987.39 examples/s]
Filter (num_proc=8): 100%|##########| 205/205 [00:00<00:00, 951.45 examples/s]INFO 2025-05-19 21:02:33,103 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 21:02:44,224 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/181 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 46/181 [00:00<00:00, 339.58 examples/s]
Map (num_proc=8):  76%|#######5  | 137/181 [00:00<00:00, 536.08 examples/s]
Map (num_proc=8): 100%|##########| 181/181 [00:00<00:00, 484.19 examples/s]

Filter (num_proc=8):   0%|          | 0/181 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 92/181 [00:00<00:00, 915.90 examples/s]
Filter (num_proc=8): 100%|##########| 181/181 [00:00<00:00, 833.38 examples/s]INFO 2025-05-19 21:02:45,361 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_12_4_1__Using_skip_node_annotation/data_checkpoint_916404f80a4246e193873c5131f9d2cf.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 162.27ba/s]INFO 2025-05-19 21:02:45,404 instructlab.sdg.generate_data:478: Generated 126 samples
INFO 2025-05-19 21:02:45,469 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_6__Monitoring_virtual_machine_health, generating from scratch
INFO 2025-05-19 21:02:45,469 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 21:02:45,472 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 21:02:45,991 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 21:02:46,005 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 21:02:46,012 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 21:02:55,161 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 21:03:10,061 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 21:03:18,394 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 21:03:18,415 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 21:03:18,430 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 21:04:40,691 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 21:07:46,041 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 856.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1204.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 445.13 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 600.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1219.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.89 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 635.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 415.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 638.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1163.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.89 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 678.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1151.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.55 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 646.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1152.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.13 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 680.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1116.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.19 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 699.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1167.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.25 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 606.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 926.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.16 examples/s]

Map (num_proc=8):   0%|          | 0/211 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 54/211 [00:00<00:00, 365.78 examples/s]
Map (num_proc=8):  63%|######3   | 133/211 [00:00<00:00, 528.55 examples/s]
Map (num_proc=8): 100%|##########| 211/211 [00:00<00:00, 510.88 examples/s]

Filter (num_proc=8):   0%|          | 0/211 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 81/211 [00:00<00:00, 750.08 examples/s]
Filter (num_proc=8): 100%|##########| 211/211 [00:00<00:00, 943.55 examples/s]INFO 2025-05-19 21:07:57,814 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 21:09:57,932 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.30 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 839.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1194.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.34 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 458.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 915.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 466.23 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 552.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1232.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 466.62 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 640.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1113.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.61 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 688.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.11 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 782.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1185.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.27 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 672.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1266.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.32 examples/s]

Map (num_proc=8):   0%|          | 0/206 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 52/206 [00:00<00:00, 352.71 examples/s]
Map (num_proc=8):  63%|######3   | 130/206 [00:00<00:00, 506.07 examples/s]
Map (num_proc=8): 100%|##########| 206/206 [00:00<00:00, 495.58 examples/s]

Filter (num_proc=8):   0%|          | 0/206 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 78/206 [00:00<00:00, 776.74 examples/s]
Filter (num_proc=8): 100%|##########| 206/206 [00:00<00:00, 926.49 examples/s]INFO 2025-05-19 21:10:07,428 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 21:11:39,040 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 579.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 904.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.67 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 593.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1153.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.71 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 750.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1200.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1194.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.60 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 719.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.01 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 689.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.09 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 653.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1078.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 449.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 562.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1184.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.79 examples/s]

Map (num_proc=8):   0%|          | 0/130 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 34/130 [00:00<00:00, 252.81 examples/s]
Map (num_proc=8):  88%|########7 | 114/130 [00:00<00:00, 499.96 examples/s]
Map (num_proc=8): 100%|##########| 130/130 [00:00<00:00, 345.99 examples/s]

Filter (num_proc=8):   0%|          | 0/130 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 66/130 [00:00<00:00, 589.35 examples/s]
Filter (num_proc=8): 100%|##########| 130/130 [00:00<00:00, 609.94 examples/s]INFO 2025-05-19 21:11:48,518 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_6__Monitoring_virtual_machine_health/data_checkpoint_90fcbbca733b45118acea761e7dd44ff.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 26.12ba/s]INFO 2025-05-19 21:11:48,974 instructlab.sdg.generate_data:478: Generated 1557 samples
INFO 2025-05-19 21:11:49,033 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_10_1_2__About_virtual_machines_and_boot_sources, generating from scratch
INFO 2025-05-19 21:11:49,033 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 21:11:49,036 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 21:11:49,531 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 21:11:49,544 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 21:11:49,552 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 21:11:58,037 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 21:12:09,897 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 21:12:17,857 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 21:12:17,877 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 21:12:17,892 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 21:13:08,901 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 21:13:49,993 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.74 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 656.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 904.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.56 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.84 examples/s]

Map (num_proc=8):   0%|          | 0/48 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 18/48 [00:00<00:00, 120.06 examples/s]
Map (num_proc=8): 100%|##########| 48/48 [00:00<00:00, 140.11 examples/s]

Filter (num_proc=8):   0%|          | 0/48 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 18/48 [00:00<00:00, 175.63 examples/s]
Filter (num_proc=8): 100%|##########| 48/48 [00:00<00:00, 229.77 examples/s]INFO 2025-05-19 21:13:53,456 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 21:14:19,873 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 457.80 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1187.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.50 examples/s]

Map (num_proc=8):   0%|          | 0/155 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 40/155 [00:00<00:00, 280.96 examples/s]
Map (num_proc=8):  75%|#######5  | 117/155 [00:00<00:00, 507.53 examples/s]
Map (num_proc=8): 100%|##########| 155/155 [00:00<00:00, 395.15 examples/s]

Filter (num_proc=8):   0%|          | 0/155 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 79/155 [00:00<00:00, 698.27 examples/s]
Filter (num_proc=8): 100%|##########| 155/155 [00:00<00:00, 706.94 examples/s]INFO 2025-05-19 21:14:22,216 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 21:14:41,988 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.38 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.42 examples/s]

Map (num_proc=8):   0%|          | 0/140 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 36/140 [00:00<00:00, 283.61 examples/s]
Map (num_proc=8):  64%|######3   | 89/140 [00:00<00:00, 381.94 examples/s]
Map (num_proc=8): 100%|##########| 140/140 [00:00<00:00, 364.82 examples/s]

Filter (num_proc=8):   0%|          | 0/140 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 72/140 [00:00<00:00, 718.67 examples/s]
Filter (num_proc=8): 100%|##########| 140/140 [00:00<00:00, 648.70 examples/s]INFO 2025-05-19 21:14:44,331 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_10_1_2__About_virtual_machines_and_boot_sources/data_checkpoint_4d1e68107c37485c9a67f702d02c410e.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 44.98ba/s]INFO 2025-05-19 21:14:44,453 instructlab.sdg.generate_data:478: Generated 351 samples
INFO 2025-05-19 21:14:44,512 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_2__Viewing_OpenShift_Virtualization_logs, generating from scratch
INFO 2025-05-19 21:14:44,512 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 21:14:44,515 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 21:14:44,979 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 21:14:44,993 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 21:14:45,000 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 21:15:14,130 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 21:15:23,875 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 21:15:34,401 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 21:15:34,425 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 21:15:34,441 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 21:16:28,113 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 21:17:11,734 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 336.95 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 584.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1097.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.40 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 669.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1181.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.72 examples/s]

Map (num_proc=8):   0%|          | 0/49 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 13/49 [00:00<00:00, 96.67 examples/s]
Map (num_proc=8):  88%|########7 | 43/49 [00:00<00:00, 192.83 examples/s]
Map (num_proc=8): 100%|##########| 49/49 [00:00<00:00, 141.82 examples/s]

Filter (num_proc=8):   0%|          | 0/49 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 31/49 [00:00<00:00, 306.87 examples/s]
Filter (num_proc=8): 100%|##########| 49/49 [00:00<00:00, 245.42 examples/s]INFO 2025-05-19 21:17:15,198 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 21:17:37,626 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.94 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 708.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1163.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.76 examples/s]

Map (num_proc=8):   0%|          | 0/93 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 24/93 [00:00<00:00, 190.16 examples/s]
Map (num_proc=8):  88%|########8 | 82/93 [00:00<00:00, 342.57 examples/s]
Map (num_proc=8): 100%|##########| 93/93 [00:00<00:00, 258.34 examples/s]

Filter (num_proc=8):   0%|          | 0/93 [00:00<?, ? examples/s]
Filter (num_proc=8):  65%|######4   | 60/93 [00:00<00:00, 513.18 examples/s]
Filter (num_proc=8): 100%|##########| 93/93 [00:00<00:00, 434.88 examples/s]INFO 2025-05-19 21:17:39,961 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 21:17:57,037 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.66 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 621.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1097.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.89 examples/s]

Map (num_proc=8):   0%|          | 0/77 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 20/77 [00:00<00:00, 160.81 examples/s]
Map (num_proc=8):  77%|#######6  | 59/77 [00:00<00:00, 276.42 examples/s]
Map (num_proc=8): 100%|##########| 77/77 [00:00<00:00, 215.91 examples/s]

Filter (num_proc=8):   0%|          | 0/77 [00:00<?, ? examples/s]
Filter (num_proc=8):  65%|######4   | 50/77 [00:00<00:00, 431.46 examples/s]
Filter (num_proc=8): 100%|##########| 77/77 [00:00<00:00, 364.56 examples/s]INFO 2025-05-19 21:17:59,353 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_2__Viewing_OpenShift_Virtualization_logs/data_checkpoint_da9e7396a82244a1a5585b1feede289f.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 59.08ba/s]INFO 2025-05-19 21:17:59,443 instructlab.sdg.generate_data:478: Generated 254 samples
INFO 2025-05-19 21:17:59,513 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_2_8__About_the_virt_operator, generating from scratch
INFO 2025-05-19 21:17:59,513 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 21:17:59,516 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 21:17:59,662 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 21:17:59,675 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 21:17:59,683 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 21:18:05,017 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 21:18:09,501 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 21:18:14,655 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 21:18:14,674 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 21:18:14,689 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 21:19:06,881 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 21:19:48,633 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 668.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 910.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 468.09 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 687.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 651.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.07 examples/s]

Map (num_proc=8):   0%|          | 0/115 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 30/115 [00:00<00:00, 254.59 examples/s]
Map (num_proc=8):  88%|########7 | 101/115 [00:00<00:00, 411.03 examples/s]
Map (num_proc=8): 100%|##########| 115/115 [00:00<00:00, 328.60 examples/s]

Filter (num_proc=8):   0%|          | 0/115 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 73/115 [00:00<00:00, 652.13 examples/s]
Filter (num_proc=8): 100%|##########| 115/115 [00:00<00:00, 550.09 examples/s]INFO 2025-05-19 21:19:52,049 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 21:20:17,783 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.38 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 670.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1270.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.77 examples/s]

Map (num_proc=8):   0%|          | 0/160 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 40/160 [00:00<00:00, 306.07 examples/s]
Map (num_proc=8):  88%|########7 | 140/160 [00:00<00:00, 561.85 examples/s]
Map (num_proc=8): 100%|##########| 160/160 [00:00<00:00, 432.61 examples/s]

Filter (num_proc=8):   0%|          | 0/160 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 60/160 [00:00<00:00, 584.23 examples/s]
Filter (num_proc=8): 100%|##########| 160/160 [00:00<00:00, 748.62 examples/s]INFO 2025-05-19 21:20:20,080 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 21:20:40,318 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.72 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 743.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1211.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.12 examples/s]

Map (num_proc=8):   0%|          | 0/152 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 38/152 [00:00<00:00, 306.32 examples/s]
Map (num_proc=8):  75%|#######5  | 114/152 [00:00<00:00, 546.67 examples/s]
Map (num_proc=8): 100%|##########| 152/152 [00:00<00:00, 409.89 examples/s]

Filter (num_proc=8):   0%|          | 0/152 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 76/152 [00:00<00:00, 710.13 examples/s]
Filter (num_proc=8): 100%|##########| 152/152 [00:00<00:00, 706.69 examples/s]INFO 2025-05-19 21:20:42,609 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_2_8__About_the_virt_operator/data_checkpoint_1c1abc7296ef49b89b629732f0366f85.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 57.23ba/s]INFO 2025-05-19 21:20:42,722 instructlab.sdg.generate_data:478: Generated 381 samples
INFO 2025-05-19 21:20:42,781 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_14_4__Adding_VirtIO_drivers_container_disk_to_a_virtual_machine, generating from scratch
INFO 2025-05-19 21:20:42,782 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 21:20:42,785 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 21:20:43,133 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 21:20:43,147 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 21:20:43,154 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 21:20:52,391 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 21:21:00,574 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 21:21:09,007 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 21:21:09,028 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 21:21:09,043 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 21:22:26,562 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 21:25:14,404 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.77 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 683.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1135.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 436.89 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 694.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 635.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 934.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.63 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 769.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 883.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.39 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 607.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 876.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 442.04 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 500.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 931.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 595.84 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 609.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1191.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.64 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 695.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1140.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.54 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 684.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1278.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.84 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 642.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 907.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.50 examples/s]

Map (num_proc=8):   0%|          | 0/177 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 45/177 [00:00<00:00, 274.14 examples/s]
Map (num_proc=8):  63%|######2   | 111/177 [00:00<00:00, 431.16 examples/s]
Map (num_proc=8): 100%|##########| 177/177 [00:00<00:00, 461.03 examples/s]

Filter (num_proc=8):   0%|          | 0/177 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 89/177 [00:00<00:00, 838.83 examples/s]
Filter (num_proc=8): 100%|##########| 177/177 [00:00<00:00, 822.97 examples/s]INFO 2025-05-19 21:25:26,078 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 21:27:10,800 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.68 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 607.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 896.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 598.03 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 467.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1141.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 680.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1187.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.17 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 824.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1155.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 660.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.58 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 461.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1189.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.34 examples/s]

Map (num_proc=8):   0%|          | 0/169 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 43/169 [00:00<00:00, 343.59 examples/s]
Map (num_proc=8):  75%|#######5  | 127/169 [00:00<00:00, 485.00 examples/s]
Map (num_proc=8): 100%|##########| 169/169 [00:00<00:00, 437.10 examples/s]

Filter (num_proc=8):   0%|          | 0/169 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 85/169 [00:00<00:00, 754.99 examples/s]
Filter (num_proc=8): 100%|##########| 169/169 [00:00<00:00, 774.98 examples/s]INFO 2025-05-19 21:27:19,042 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 21:28:35,124 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.84 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 674.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1235.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.28 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 642.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1209.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.22 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 601.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.79 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 739.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 925.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.26 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 651.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 892.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.59 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 664.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1105.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.70 examples/s]

Map (num_proc=8):   0%|          | 0/100 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 26/100 [00:00<00:00, 210.07 examples/s]
Map (num_proc=8):  76%|#######6  | 76/100 [00:00<00:00, 364.43 examples/s]
Map (num_proc=8): 100%|##########| 100/100 [00:00<00:00, 274.12 examples/s]

Filter (num_proc=8):   0%|          | 0/100 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####2    | 52/100 [00:00<00:00, 451.19 examples/s]
Filter (num_proc=8): 100%|##########| 100/100 [00:00<00:00, 464.17 examples/s]INFO 2025-05-19 21:28:43,320 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_14_4__Adding_VirtIO_drivers_container_disk_to_a_virtual_machine/data_checkpoint_30fd4d16ffea4af6a5fc94b40e11c2c4.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 31.90ba/s]INFO 2025-05-19 21:28:43,711 instructlab.sdg.generate_data:478: Generated 1368 samples
INFO 2025-05-19 21:28:43,767 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_5__Managing_virtual_machine_instances, generating from scratch
INFO 2025-05-19 21:28:43,767 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 21:28:43,770 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 21:28:44,109 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 21:28:44,123 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 21:28:44,131 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 21:28:49,779 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 21:28:56,053 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 21:29:00,548 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 21:29:00,569 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 21:29:00,584 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 21:30:21,328 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 21:33:15,704 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 637.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1212.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.72 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 724.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1279.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 415.29 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 672.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1246.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 345.74 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 728.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1256.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.35 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 710.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1102.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 457.85 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 648.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.58 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 726.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1119.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 459.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 662.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 444.30 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 692.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1165.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.78 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 790.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1194.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.29 examples/s]

Map (num_proc=8):   0%|          | 0/125 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 32/125 [00:00<00:00, 249.66 examples/s]
Map (num_proc=8):  76%|#######6  | 95/125 [00:00<00:00, 447.99 examples/s]
Map (num_proc=8): 100%|##########| 125/125 [00:00<00:00, 343.51 examples/s]

Filter (num_proc=8):   0%|          | 0/125 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 64/125 [00:00<00:00, 628.23 examples/s]
Filter (num_proc=8): 100%|##########| 125/125 [00:00<00:00, 577.08 examples/s]INFO 2025-05-19 21:33:28,719 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 21:35:09,677 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.09 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 784.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 883.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.00 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 757.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1424.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.05 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 594.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1213.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 300.71 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 568.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1070.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.54 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 742.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1080.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.35 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 641.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1257.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.53 examples/s]

Map (num_proc=8):   0%|          | 0/103 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 39/103 [00:00<00:00, 253.49 examples/s]
Map (num_proc=8): 100%|##########| 103/103 [00:00<00:00, 436.63 examples/s]
Map (num_proc=8): 100%|##########| 103/103 [00:00<00:00, 286.09 examples/s]

Filter (num_proc=8):   0%|          | 0/103 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 65/103 [00:00<00:00, 566.26 examples/s]
Filter (num_proc=8): 100%|##########| 103/103 [00:00<00:00, 474.01 examples/s]INFO 2025-05-19 21:35:17,941 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 21:36:34,843 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 333.83 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 694.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 868.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.59 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 754.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1184.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.09 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 599.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 932.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 682.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1105.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.03 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 626.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.20 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 729.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1120.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.77 examples/s]

Map (num_proc=8):   0%|          | 0/54 [00:00<?, ? examples/s]
Map (num_proc=8):  39%|###8      | 21/54 [00:00<00:00, 113.58 examples/s]
Map (num_proc=8): 100%|##########| 54/54 [00:00<00:00, 155.92 examples/s]

Filter (num_proc=8):   0%|          | 0/54 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 28/54 [00:00<00:00, 253.64 examples/s]
Filter (num_proc=8): 100%|##########| 54/54 [00:00<00:00, 251.26 examples/s]INFO 2025-05-19 21:36:43,129 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_5__Managing_virtual_machine_instances/data_checkpoint_efa75ab595e7429a9a71f9c0d6e475d9.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 31.25ba/s]INFO 2025-05-19 21:36:43,547 instructlab.sdg.generate_data:478: Generated 1476 samples
INFO 2025-05-19 21:36:43,609 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_11_1__Prerequisites, generating from scratch
INFO 2025-05-19 21:36:43,609 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 21:36:43,612 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 21:36:44,258 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 21:36:44,272 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 21:36:44,280 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 21:36:52,292 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 21:37:05,678 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 21:37:13,228 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 21:37:13,249 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 21:37:13,267 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 21:39:33,047 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 21:44:47,050 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.31 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 648.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1163.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.92 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 706.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1243.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 946.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1134.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.30 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 791.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 447.31 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 849.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 907.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.06 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 823.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.62 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 624.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.39 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 602.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1238.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1037.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.37 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 579.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.34 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 656.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 933.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.41 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 715.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 635.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 918.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 647.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.27 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 773.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1060.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.21 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 771.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 853.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 471.83 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 708.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 635.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 933.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.54 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 678.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 884.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 630.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.26 examples/s]

Map (num_proc=8):   0%|          | 0/214 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 54/214 [00:00<00:00, 329.99 examples/s]
Map (num_proc=8):  76%|#######5  | 162/214 [00:00<00:00, 666.66 examples/s]
Map (num_proc=8): 100%|##########| 214/214 [00:00<00:00, 538.55 examples/s]

Filter (num_proc=8):   0%|          | 0/214 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 108/214 [00:00<00:00, 985.85 examples/s]
Filter (num_proc=8): 100%|##########| 214/214 [00:00<00:00, 965.75 examples/s]INFO 2025-05-19 21:45:08,432 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 21:48:39,399 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.24 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 814.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.23 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 599.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1094.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.88 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 590.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1160.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.10 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 677.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 636.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1129.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.75 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 454.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.45 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 666.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 912.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.02 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 616.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 901.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 541.67 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 512.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 879.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.10 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 653.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.04 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1239.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 836.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 898.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 500.10 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 501.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 906.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.79 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 696.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.16 examples/s]

Map (num_proc=8):   0%|          | 0/37 [00:00<?, ? examples/s]
Map (num_proc=8):  41%|####      | 15/37 [00:00<00:00, 97.51 examples/s]
Map (num_proc=8): 100%|##########| 37/37 [00:00<00:00, 107.66 examples/s]

Filter (num_proc=8):   0%|          | 0/37 [00:00<?, ? examples/s]
Filter (num_proc=8):  41%|####      | 15/37 [00:00<00:00, 145.19 examples/s]
Filter (num_proc=8): 100%|##########| 37/37 [00:00<00:00, 170.96 examples/s]INFO 2025-05-19 21:48:56,005 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 21:51:24,460 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.54 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 779.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 872.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 588.15 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 585.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1091.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 341.03 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 758.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1096.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 300.15 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 431.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 425.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 426.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 939.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.19 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 684.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1184.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.08 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 720.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1178.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.88 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 605.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 582.79 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 602.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 851.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1114.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.71 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 571.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1046.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.86 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 709.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.83 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 525.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 934.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.29 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 507.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 880.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.63 examples/s]

Map (num_proc=8):   0%|          | 0/191 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 48/191 [00:00<00:00, 316.57 examples/s]
Map (num_proc=8):  88%|########7 | 168/191 [00:00<00:00, 614.80 examples/s]
Map (num_proc=8): 100%|##########| 191/191 [00:00<00:00, 481.46 examples/s]

Filter (num_proc=8):   0%|          | 0/191 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 96/191 [00:00<00:00, 915.72 examples/s]
Filter (num_proc=8): 100%|##########| 191/191 [00:00<00:00, 861.22 examples/s]INFO 2025-05-19 21:51:40,177 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_11_1__Prerequisites/data_checkpoint_d683eb28b0454a1e9b209f2ad1f9e885.jsonl


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 32.26ba/s]INFO 2025-05-19 21:51:40,811 instructlab.sdg.generate_data:478: Generated 2315 samples
INFO 2025-05-19 21:51:40,876 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_1_3__Object_maximums, generating from scratch
INFO 2025-05-19 21:51:40,876 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 21:51:40,880 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 21:51:41,563 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 21:51:41,578 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 21:51:41,585 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 21:51:50,460 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 21:52:01,460 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 21:52:09,706 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 21:52:09,728 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 21:52:09,744 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 21:53:55,417 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 21:58:32,693 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 568.01 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 487.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1273.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.61 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 609.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 927.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 471.89 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 557.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 854.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1247.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 717.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.17 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 572.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.78 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 619.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 656.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1146.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.23 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 816.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 923.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.46 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 820.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 459.68 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 592.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 890.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.47 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 662.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1054.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.38 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 761.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1153.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 459.74 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 541.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 945.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.98 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 939.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.47 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 661.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.55 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 767.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.15 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 927.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.28 examples/s]

Map (num_proc=8):   0%|          | 0/12 [00:00<?, ? examples/s]
Map (num_proc=8):  33%|###3      | 4/12 [00:00<00:00, 35.18 examples/s]
Map (num_proc=8):  83%|########3 | 10/12 [00:00<00:00, 45.92 examples/s]
Map (num_proc=8): 100%|##########| 12/12 [00:00<00:00, 33.93 examples/s]

Filter (num_proc=8):   0%|          | 0/12 [00:00<?, ? examples/s]
Filter (num_proc=8):  75%|#######5  | 9/12 [00:00<00:00, 88.46 examples/s]
Filter (num_proc=8): 100%|##########| 12/12 [00:00<00:00, 59.96 examples/s]INFO 2025-05-19 21:58:53,015 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 22:01:30,618 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 544.01 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 477.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 777.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 900.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 317.74 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 622.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 923.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 591.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.15 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 590.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.29 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 552.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 864.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.78 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 622.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 662.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 941.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 571.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 912.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 511.25 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 511.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 810.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 931.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.45 examples/s]

Map (num_proc=8):   0%|          | 0/249 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 63/249 [00:00<00:00, 368.15 examples/s]
Map (num_proc=8):  63%|######2   | 156/249 [00:00<00:00, 587.84 examples/s]
Map (num_proc=8): 100%|##########| 249/249 [00:00<00:00, 572.16 examples/s]

Filter (num_proc=8):   0%|          | 0/249 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 125/249 [00:00<00:00, 1159.60 examples/s]
Filter (num_proc=8): 100%|##########| 249/249 [00:00<00:00, 1108.22 examples/s]INFO 2025-05-19 22:01:42,887 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 22:03:33,294 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 597.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 933.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 501.90 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 456.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1086.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.85 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 669.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 953.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 462.19 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 541.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 826.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1161.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 500.40 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 502.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 890.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 452.32 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 668.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 905.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.15 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 565.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.22 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 735.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1103.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.33 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 625.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1107.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.76 examples/s]

Map (num_proc=8):   0%|          | 0/149 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 38/149 [00:00<00:00, 261.08 examples/s]
Map (num_proc=8):  76%|#######5  | 113/149 [00:00<00:00, 405.60 examples/s]
Map (num_proc=8): 100%|##########| 149/149 [00:00<00:00, 377.30 examples/s]

Filter (num_proc=8):   0%|          | 0/149 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 76/149 [00:00<00:00, 649.56 examples/s]
Filter (num_proc=8): 100%|##########| 149/149 [00:00<00:00, 676.77 examples/s]INFO 2025-05-19 22:03:45,461 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_1_3__Object_maximums/data_checkpoint_0f9eebd8965a4797991a04b3838020c6.jsonl


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 30.14ba/s]INFO 2025-05-19 22:03:46,094 instructlab.sdg.generate_data:478: Generated 2213 samples
INFO 2025-05-19 22:03:46,151 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_14_5__Installing_VirtIO_drivers_during_Windows_installation, generating from scratch
INFO 2025-05-19 22:03:46,151 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 22:03:46,155 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 22:03:46,513 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 22:03:46,526 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 22:03:46,534 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 22:03:55,739 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 22:04:03,760 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 22:04:13,150 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 22:04:13,170 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 22:04:13,185 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 22:05:26,342 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 22:08:16,787 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.71 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 691.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.04 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 808.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.54 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 468.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.66 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 752.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 905.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.70 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 594.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1146.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.38 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 776.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 638.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.03 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 922.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 918.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1088.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 469.21 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 657.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1153.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 445.45 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 609.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.14 examples/s]

Map (num_proc=8):   0%|          | 0/218 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 56/218 [00:00<00:00, 422.83 examples/s]
Map (num_proc=8):  63%|######2   | 137/218 [00:00<00:00, 557.32 examples/s]
Map (num_proc=8): 100%|##########| 218/218 [00:00<00:00, 549.60 examples/s]

Filter (num_proc=8):   0%|          | 0/218 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 110/218 [00:00<00:00, 1047.57 examples/s]
Filter (num_proc=8): 100%|##########| 218/218 [00:00<00:00, 970.95 examples/s] INFO 2025-05-19 22:08:28,699 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 22:10:21,836 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.20 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 673.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1170.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.02 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 765.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1098.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.54 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 712.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1136.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.45 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 734.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 931.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 644.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 881.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.54 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 499.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 901.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.15 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 565.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 922.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.51 examples/s]

Map (num_proc=8):   0%|          | 0/23 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 6/23 [00:00<00:00, 53.21 examples/s]
Map (num_proc=8): 100%|##########| 23/23 [00:00<00:00, 96.87 examples/s]
Map (num_proc=8): 100%|##########| 23/23 [00:00<00:00, 68.09 examples/s]

Filter (num_proc=8):   0%|          | 0/23 [00:00<?, ? examples/s]
Filter (num_proc=8):  39%|###9      | 9/23 [00:00<00:00, 88.97 examples/s]
Filter (num_proc=8): 100%|##########| 23/23 [00:00<00:00, 109.23 examples/s]INFO 2025-05-19 22:10:31,365 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 22:11:48,700 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 506.62 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 640.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 899.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.16 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 457.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1212.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.40 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 663.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.96 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 670.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1120.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.31 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 561.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1260.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 275.81 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 459.56 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 625.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1131.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.36 examples/s]

Map (num_proc=8):   0%|          | 0/176 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 44/176 [00:00<00:00, 303.94 examples/s]
Map (num_proc=8):  88%|########7 | 154/176 [00:00<00:00, 563.52 examples/s]
Map (num_proc=8): 100%|##########| 176/176 [00:00<00:00, 452.05 examples/s]

Filter (num_proc=8):   0%|          | 0/176 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 66/176 [00:00<00:00, 657.96 examples/s]
Filter (num_proc=8): 100%|##########| 176/176 [00:00<00:00, 783.63 examples/s]INFO 2025-05-19 22:11:57,081 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_14_5__Installing_VirtIO_drivers_during_Windows_installation/data_checkpoint_7cca294e70db4976b5c622e17c92b341.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 30.91ba/s]INFO 2025-05-19 22:11:57,496 instructlab.sdg.generate_data:478: Generated 1454 samples
INFO 2025-05-19 22:11:57,565 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_16_12__Configuring_a_watchdog, generating from scratch
INFO 2025-05-19 22:11:57,565 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 22:11:57,570 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 22:11:57,573 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 22:12:01,043 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 22:12:01,058 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 22:12:01,067 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 22:12:03,591 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 22:12:03,606 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 22:12:03,615 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 22:12:27,447 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 22:12:27,574 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 22:13:08,157 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 22:13:24,237 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 22:13:33,311 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 22:13:33,341 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 22:13:33,366 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 22:13:52,279 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 22:13:52,311 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 22:13:52,335 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 22:21:40,528 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 22:22:02,918 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 22:29:45,223 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.48 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 575.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 903.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.43 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 599.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 891.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 252.03 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 444.35 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 667.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 544.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1150.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.19 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 673.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1032.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 354.53 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 608.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1256.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.59 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 718.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 886.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 824.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 803.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1031.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 352.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 601.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 858.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.93 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 623.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 575.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 884.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 334.60 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 835.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1055.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.27 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 727.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 891.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.52 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 714.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1117.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 615.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1230.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.84 examples/s]

Map (num_proc=8):   0%|          | 0/49 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 13/49 [00:00<00:00, 98.53 examples/s]
Map (num_proc=8):  63%|######3   | 31/49 [00:00<00:00, 123.15 examples/s]
Map (num_proc=8): 100%|##########| 49/49 [00:00<00:00, 133.01 examples/s]

Filter (num_proc=8):   0%|          | 0/49 [00:00<?, ? examples/s]
Filter (num_proc=8):  39%|###8      | 19/49 [00:00<00:00, 173.78 examples/s]
Filter (num_proc=8): 100%|##########| 49/49 [00:00<00:00, 224.30 examples/s]INFO 2025-05-19 22:30:03,822 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 22:34:01,553 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.97 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 564.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 807.93 examples/s]
Filter (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 961.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 777.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 315.75 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 442.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 852.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1149.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 622.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 345.69 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 554.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1137.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.53 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1100.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.69 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 693.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 870.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.64 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 674.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 941.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.49 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 783.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 941.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.00 examples/s]

Map (num_proc=8):   0%|          | 0/201 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 51/201 [00:00<00:00, 289.92 examples/s]
Map (num_proc=8):  63%|######2   | 126/201 [00:00<00:00, 488.26 examples/s]
Map (num_proc=8): 100%|##########| 201/201 [00:00<00:00, 491.43 examples/s]

Filter (num_proc=8):   0%|          | 0/201 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 76/201 [00:00<00:00, 743.26 examples/s]
Filter (num_proc=8): 100%|##########| 201/201 [00:00<00:00, 895.48 examples/s]INFO 2025-05-19 22:34:12,857 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 22:37:20,390 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.47 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 653.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 496.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 893.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1088.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.54 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1085.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.83 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 701.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.40 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 561.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1119.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 529.14 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 529.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.03 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 817.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 934.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.65 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 617.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 905.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.63 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 727.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 892.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.41 examples/s]

Map (num_proc=8):   0%|          | 0/32 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 12/32 [00:00<00:00, 71.10 examples/s]
Map (num_proc=8): 100%|##########| 32/32 [00:00<00:00, 90.72 examples/s]

Filter (num_proc=8):   0%|          | 0/32 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 16/32 [00:00<00:00, 140.94 examples/s]
Filter (num_proc=8): 100%|##########| 32/32 [00:00<00:00, 146.30 examples/s]INFO 2025-05-19 22:37:31,695 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_16_12__Configuring_a_watchdog/data_checkpoint_4610906af2eb47578653ab5bd738d0c4.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 27.18ba/s]INFO 2025-05-19 22:43:28,617 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.00 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 595.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 441.81 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 524.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1205.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.15 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 842.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.77 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 692.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1144.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 461.45 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 667.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.81 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 536.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1167.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.23 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 903.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 927.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.84 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 597.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 868.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 531.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1050.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.65 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 831.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 420.67 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 886.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 352.69 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 792.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.28 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 659.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 955.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.11 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 531.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 933.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.85 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 835.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1137.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.11 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 476.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.28 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 723.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 610.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1060.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.27 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 647.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 886.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.69 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 734.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 915.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.86 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 774.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1247.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.24 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 645.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1074.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.63 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 600.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 539.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.04 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 622.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 425.53 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 611.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1148.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.67 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 799.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1148.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 462.04 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 753.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 887.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.83 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 635.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 635.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1047.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.96 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 600.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 919.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1100.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 552.13 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 531.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 665.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.04 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 602.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 464.60 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 516.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 513.17 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 530.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.12 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 635.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1085.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.14 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 640.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 625.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 458.54 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 470.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1233.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 444.91 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 517.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 959.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.93 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 654.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 932.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.27 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 701.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.23 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 568.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 857.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1096.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.67 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 679.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 791.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1102.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 309.53 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 459.56 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 766.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 878.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 307.39 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 427.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1165.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 275.85 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 608.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 933.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 311.03 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 496.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 891.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 316.23 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 485.61 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 815.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1182.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1100.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 313.41 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 453.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 886.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 286.67 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 520.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 257.17 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 522.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 921.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1119.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 524.29 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 528.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1083.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.77 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 617.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1214.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1119.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.15 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 829.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 639.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.86 examples/s]

Map (num_proc=8):   0%|          | 0/236 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 60/236 [00:00<00:00, 410.91 examples/s]
Map (num_proc=8):  63%|######3   | 149/236 [00:00<00:00, 591.90 examples/s]
Map (num_proc=8): 100%|##########| 236/236 [00:00<00:00, 563.61 examples/s]

Filter (num_proc=8):   0%|          | 0/236 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 90/236 [00:00<00:00, 866.11 examples/s]
Filter (num_proc=8): 100%|##########| 236/236 [00:00<00:00, 1028.62 examples/s]INFO 2025-05-19 22:44:36,490 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 22:52:20,833 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.50 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 695.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 566.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 315.69 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 574.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 892.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 315.25 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 496.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 916.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 327.21 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 531.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 634.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 354.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 625.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.56 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 652.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 616.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 300.22 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 420.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 922.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 696.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 945.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.24 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 620.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 865.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1096.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.45 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 617.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 883.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1099.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 316.14 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 513.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.13 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 657.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 637.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1096.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 275.03 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 570.22 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 708.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.14 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 485.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1211.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.20 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 742.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 904.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1090.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.60 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 830.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1060.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.36 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 506.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 621.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 277.06 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 499.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1122.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 272.61 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 502.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 859.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 315.75 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 610.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 863.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 870.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.36 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 715.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 627.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1099.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 444.06 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 494.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 332.59 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 458.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.15 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 844.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1096.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 603.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 919.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1097.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 314.07 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 419.13 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 730.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 905.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 325.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 561.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1135.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 277.62 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 570.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 848.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1160.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 344.99 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 572.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 884.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1101.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.71 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 508.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 595.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1061.09 examples/s]

Map (num_proc=8):   0%|          | 0/47 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###8      | 18/47 [00:00<00:00, 105.82 examples/s]
Map (num_proc=8): 100%|##########| 47/47 [00:00<00:00, 128.99 examples/s]

Filter (num_proc=8):   0%|          | 0/47 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 18/47 [00:00<00:00, 172.40 examples/s]
Filter (num_proc=8): 100%|##########| 47/47 [00:00<00:00, 209.25 examples/s]INFO 2025-05-19 22:52:59,214 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 22:58:30,412 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 273.10 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 534.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 920.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 343.41 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 693.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 874.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1091.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 310.53 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 420.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 872.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 920.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1101.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 341.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 785.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 622.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1089.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.33 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 678.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 922.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1101.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 348.50 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 723.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1078.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1101.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.67 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 623.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1079.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1098.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.13 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 540.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 859.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1084.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.03 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 653.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 939.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1093.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 317.25 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 429.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1057.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1100.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 346.40 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 638.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1088.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 317.91 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 477.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 524.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1100.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 336.94 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 546.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1104.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.83 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 647.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1204.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 315.23 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 553.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 932.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1101.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 341.27 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 700.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1226.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.35 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 615.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1068.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.44 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 529.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 933.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1096.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.50 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 699.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1101.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 630.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 881.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1093.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.25 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 714.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1089.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.71 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 573.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 625.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1080.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.67 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 574.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 911.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1093.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 442.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 662.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 841.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1095.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.08 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 787.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 610.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1077.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 311.60 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 493.27 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 655.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 593.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1054.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.87 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 670.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1081.12 examples/s]

Map (num_proc=8):   0%|          | 0/180 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 46/180 [00:00<00:00, 320.81 examples/s]
Map (num_proc=8):  76%|#######5  | 136/180 [00:00<00:00, 518.08 examples/s]
Map (num_proc=8): 100%|##########| 180/180 [00:00<00:00, 429.23 examples/s]

Filter (num_proc=8):   0%|          | 0/180 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 69/180 [00:00<00:00, 671.42 examples/s]
Filter (num_proc=8): 100%|##########| 180/180 [00:00<00:00, 781.12 examples/s]INFO 2025-05-19 22:59:06,432 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_16_12__Configuring_a_watchdog/data_checkpoint_83a53a71c944473cbded5d1e20c2651a.jsonl


Creating json from Arrow format:   0%|          | 0/6 [00:00<?, ?ba/s]
Creating json from Arrow format:  50%|#####     | 3/6 [00:00<00:00, 23.37ba/s]
Creating json from Arrow format: 100%|##########| 6/6 [00:00<00:00, 26.33ba/s]
Creating json from Arrow format: 100%|##########| 6/6 [00:00<00:00, 25.82ba/s]INFO 2025-05-19 22:59:08,423 instructlab.sdg.generate_data:478: Generated 7161 samples
INFO 2025-05-19 22:59:08,488 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_2_4__About_the_cluster_network_addons_operator, generating from scratch
INFO 2025-05-19 22:59:08,488 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 22:59:08,492 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 22:59:08,635 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 22:59:08,649 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 22:59:08,656 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 22:59:11,962 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 22:59:18,994 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 22:59:22,301 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 22:59:22,319 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 22:59:22,334 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 23:00:08,685 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 23:00:35,513 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.04 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 633.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 904.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.04 examples/s]

Map (num_proc=8):   0%|          | 0/70 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 18/70 [00:00<00:00, 155.51 examples/s]
Map (num_proc=8):  77%|#######7  | 54/70 [00:00<00:00, 227.71 examples/s]
Map (num_proc=8): 100%|##########| 70/70 [00:00<00:00, 195.23 examples/s]

Filter (num_proc=8):   0%|          | 0/70 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 36/70 [00:00<00:00, 352.62 examples/s]
Filter (num_proc=8): 100%|##########| 70/70 [00:00<00:00, 320.92 examples/s]INFO 2025-05-19 23:00:37,868 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 23:00:58,351 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.65 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 791.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1150.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.58 examples/s]

Map (num_proc=8):   0%|          | 0/25 [00:00<?, ? examples/s]
Map (num_proc=8):  40%|####      | 10/25 [00:00<00:00, 61.07 examples/s]
Map (num_proc=8): 100%|##########| 25/25 [00:00<00:00, 73.88 examples/s]

Filter (num_proc=8):   0%|          | 0/25 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####2    | 13/25 [00:00<00:00, 114.63 examples/s]
Filter (num_proc=8): 100%|##########| 25/25 [00:00<00:00, 117.65 examples/s]INFO 2025-05-19 23:01:00,680 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 23:01:16,948 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 846.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 925.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.08 examples/s]

Map (num_proc=8):   0%|          | 0/18 [00:00<?, ? examples/s]
Map (num_proc=8):  44%|####4     | 8/18 [00:00<00:00, 53.53 examples/s]
Map (num_proc=8): 100%|##########| 18/18 [00:00<00:00, 52.63 examples/s]

Filter (num_proc=8):   0%|          | 0/18 [00:00<?, ? examples/s]
Filter (num_proc=8):  56%|#####5    | 10/18 [00:00<00:00, 96.75 examples/s]
Filter (num_proc=8): 100%|##########| 18/18 [00:00<00:00, 83.99 examples/s]INFO 2025-05-19 23:01:19,288 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_2_4__About_the_cluster_network_addons_operator/data_checkpoint_a301af5a6bdd4dc480fa2017c20ff7f8.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 75.97ba/s]INFO 2025-05-19 23:01:19,382 instructlab.sdg.generate_data:478: Generated 257 samples
INFO 2025-05-19 23:01:19,441 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_15_1__About_vTPM_devices, generating from scratch
INFO 2025-05-19 23:01:19,441 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 23:01:19,444 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 23:01:19,637 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 23:01:19,650 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 23:01:19,657 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 23:01:23,961 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 23:01:29,781 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 23:01:32,730 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 23:01:32,749 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 23:01:32,764 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 23:02:26,465 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 23:03:32,593 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 459.33 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 684.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 907.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.85 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 636.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1278.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.00 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 828.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 644.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1211.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.85 examples/s]

Map (num_proc=8):   0%|          | 0/128 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 32/128 [00:00<00:00, 228.61 examples/s]
Map (num_proc=8):  75%|#######5  | 96/128 [00:00<00:00, 417.91 examples/s]
Map (num_proc=8): 100%|##########| 128/128 [00:00<00:00, 341.46 examples/s]

Filter (num_proc=8):   0%|          | 0/128 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 64/128 [00:00<00:00, 593.67 examples/s]
Filter (num_proc=8): 100%|##########| 128/128 [00:00<00:00, 595.00 examples/s]INFO 2025-05-19 23:03:37,256 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 23:04:14,736 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.68 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 868.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.08 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 695.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.83 examples/s]

Map (num_proc=8):   0%|          | 0/58 [00:00<?, ? examples/s]
Map (num_proc=8):  28%|##7       | 16/58 [00:00<00:00, 139.39 examples/s]
Map (num_proc=8):  88%|########7 | 51/58 [00:00<00:00, 200.52 examples/s]
Map (num_proc=8): 100%|##########| 58/58 [00:00<00:00, 162.94 examples/s]

Filter (num_proc=8):   0%|          | 0/58 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 30/58 [00:00<00:00, 270.54 examples/s]
Filter (num_proc=8): 100%|##########| 58/58 [00:00<00:00, 272.96 examples/s]INFO 2025-05-19 23:04:18,198 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 23:04:44,497 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 460.34 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 697.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1112.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.16 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 626.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 927.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.44 examples/s]

Map (num_proc=8):   0%|          | 0/20 [00:00<?, ? examples/s]
Map (num_proc=8):  30%|###       | 6/20 [00:00<00:00, 54.75 examples/s]
Map (num_proc=8):  80%|########  | 16/20 [00:00<00:00, 79.93 examples/s]
Map (num_proc=8): 100%|##########| 20/20 [00:00<00:00, 59.05 examples/s]

Filter (num_proc=8):   0%|          | 0/20 [00:00<?, ? examples/s]
Filter (num_proc=8):  60%|######    | 12/20 [00:00<00:00, 108.98 examples/s]
Filter (num_proc=8): 100%|##########| 20/20 [00:00<00:00, 94.69 examples/s] INFO 2025-05-19 23:04:47,970 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_15_1__About_vTPM_devices/data_checkpoint_9b31212076dc4c8ab619723b41d73f5f.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 45.75ba/s]INFO 2025-05-19 23:04:48,118 instructlab.sdg.generate_data:478: Generated 515 samples
INFO 2025-05-19 23:04:48,176 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_12_4__Preventing_node_reconciliation, generating from scratch
INFO 2025-05-19 23:04:48,176 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 23:04:48,179 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 23:04:48,323 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 23:04:48,336 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 23:04:48,344 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 23:04:51,937 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 23:04:56,046 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 23:04:58,878 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 23:04:58,897 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 23:04:58,912 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 23:05:30,724 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 23:05:40,919 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/142 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###8      | 54/142 [00:00<00:00, 307.58 examples/s]
Map (num_proc=8): 100%|##########| 142/142 [00:00<00:00, 380.31 examples/s]

Filter (num_proc=8):   0%|          | 0/142 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 54/142 [00:00<00:00, 517.14 examples/s]
Filter (num_proc=8): 100%|##########| 142/142 [00:00<00:00, 656.06 examples/s]INFO 2025-05-19 23:05:42,067 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 23:05:50,525 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/113 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 29/113 [00:00<00:00, 224.81 examples/s]
Map (num_proc=8):  63%|######2   | 71/113 [00:00<00:00, 316.59 examples/s]
Map (num_proc=8): 100%|##########| 113/113 [00:00<00:00, 311.98 examples/s]

Filter (num_proc=8):   0%|          | 0/113 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 57/113 [00:00<00:00, 486.87 examples/s]
Filter (num_proc=8): 100%|##########| 113/113 [00:00<00:00, 517.80 examples/s]INFO 2025-05-19 23:05:51,658 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 23:05:59,118 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/105 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 27/105 [00:00<00:00, 259.03 examples/s]
Map (num_proc=8):  75%|#######5  | 79/105 [00:00<00:00, 359.07 examples/s]
Map (num_proc=8): 100%|##########| 105/105 [00:00<00:00, 296.17 examples/s]

Filter (num_proc=8):   0%|          | 0/105 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 53/105 [00:00<00:00, 525.38 examples/s]
Filter (num_proc=8): 100%|##########| 105/105 [00:00<00:00, 487.98 examples/s]INFO 2025-05-19 23:06:00,241 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_12_4__Preventing_node_reconciliation/data_checkpoint_9be2a681c02e45188a273848add7a8c5.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 223.85ba/s]INFO 2025-05-19 23:06:00,270 instructlab.sdg.generate_data:478: Generated 66 samples
INFO 2025-05-19 23:06:00,342 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_19__Virtual_machine_networking, generating from scratch
INFO 2025-05-19 23:06:00,342 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 23:06:00,346 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 23:06:03,025 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 23:06:03,040 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 23:06:03,049 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 23:06:24,103 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 23:06:50,852 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 23:07:08,342 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 23:07:08,377 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 23:07:08,401 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 23:13:50,101 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 23:28:44,558 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.82 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 488.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1195.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.25 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 516.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 899.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.49 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 624.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 710.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 540.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 617.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1078.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 324.09 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 739.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1226.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.74 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 482.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1180.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 346.60 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 757.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.48 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 699.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1058.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.61 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 730.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 565.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 935.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.25 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 623.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 933.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1119.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.25 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 621.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1143.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 345.15 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 674.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 892.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 436.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 630.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 880.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 487.15 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 565.61 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 584.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 924.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.26 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 620.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 897.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 458.82 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 540.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1123.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.13 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 631.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.20 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 794.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 912.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1098.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 597.30 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 515.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 345.62 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 558.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 913.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 330.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 832.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 885.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.80 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 569.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 894.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.54 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 622.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 928.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 598.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.70 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 609.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 639.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 634.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1184.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.58 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 743.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 839.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 448.35 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 603.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1134.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.77 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 592.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 578.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 933.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 615.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 914.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.70 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 739.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 575.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1033.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.47 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 680.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1096.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.02 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 673.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1215.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 342.43 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 719.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 931.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 449.55 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 542.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 854.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.27 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 632.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1226.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.43 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 741.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.70 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 449.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1093.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1095.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.50 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 706.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 900.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.75 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 558.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 926.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.67 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 779.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 752.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 953.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1104.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 460.93 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 548.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1125.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.53 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 590.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 896.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.30 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 784.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1212.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.35 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 623.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1104.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 467.57 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 529.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 859.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1102.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.64 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 582.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 922.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.36 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 660.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1128.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.29 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 797.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1083.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.60 examples/s]

Map (num_proc=8):   0%|          | 0/177 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 45/177 [00:00<00:00, 300.88 examples/s]
Map (num_proc=8):  75%|#######5  | 133/177 [00:00<00:00, 579.79 examples/s]
Map (num_proc=8): 100%|##########| 177/177 [00:00<00:00, 443.92 examples/s]

Filter (num_proc=8):   0%|          | 0/177 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 45/177 [00:00<00:00, 449.64 examples/s]
Filter (num_proc=8): 100%|##########| 177/177 [00:00<00:00, 780.11 examples/s]INFO 2025-05-19 23:29:48,457 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 23:39:07,226 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.23 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 443.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 555.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1109.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 306.95 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 599.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 551.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1067.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1102.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.52 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 555.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 921.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1093.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 310.07 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 831.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 556.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1097.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 351.30 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 571.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 550.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 871.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1101.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 553.08 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 543.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 741.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 556.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 895.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1087.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 630.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 555.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1098.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 317.65 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 771.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 553.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 912.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1096.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 491.15 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 484.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 786.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 561.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1165.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.34 examples/s]
Map (num_proc=8):  64%|######4   | 164/256 [00:00<00:00, 512.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 553.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1221.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 302.16 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 592.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 560.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 894.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1093.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 532.92 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 495.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 566.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 912.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1102.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 347.94 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 621.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 563.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1072.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1087.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.66 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 559.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 556.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 939.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1097.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 320.58 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 564.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 560.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 616.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.98 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 658.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 564.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1199.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1119.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 319.68 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 638.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 564.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1098.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 495.26 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 532.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 765.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 561.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1151.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1100.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 425.28 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 523.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 825.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 556.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1125.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 475.55 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 507.12 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 682.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 556.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 909.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.02 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 540.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 559.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 855.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.17 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 482.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 893.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 533.12 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 565.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 566.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 859.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.66 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 489.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 559.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 908.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1097.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.25 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 473.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 555.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 928.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1094.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 434.95 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 524.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 560.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1122.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.39 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 483.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 552.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1097.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 444.80 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 458.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 556.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1104.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 354.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 602.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 564.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1154.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1101.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 537.20 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 514.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 565.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 576.40 examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 556.01 examples/s]
Filter (num_proc=8):  75%|#######5  | 192/256 [00:02<00:00, 66.92 examples/s] 
Filter (num_proc=8): 100%|##########| 256/256 [00:02<00:00, 112.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.55 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 618.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1232.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.39 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 470.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 563.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 955.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 575.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 845.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 558.56 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 458.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 863.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 681.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 912.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.41 examples/s]

Map (num_proc=8):   0%|          | 0/112 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 28/112 [00:00<00:00, 191.35 examples/s]
Map (num_proc=8):  75%|#######5  | 84/112 [00:00<00:00, 361.73 examples/s]
Map (num_proc=8): 100%|##########| 112/112 [00:00<00:00, 284.81 examples/s]

Filter (num_proc=8):   0%|          | 0/112 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 56/112 [00:00<00:00, 483.33 examples/s]
Filter (num_proc=8): 100%|##########| 112/112 [00:00<00:00, 509.46 examples/s]INFO 2025-05-19 23:39:56,011 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 23:46:34,082 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 350.61 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 528.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 560.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1165.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 487.05 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 562.85 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 553.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 563.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 915.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 492.99 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 524.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1205.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 539.75 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 512.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 789.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 566.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 927.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.46 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 547.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1205.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 594.36 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 509.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 815.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1104.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1097.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 580.19 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 514.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1119.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.00 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 914.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1104.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 329.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 706.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 898.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 561.19 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 496.26 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 677.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 560.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 874.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1102.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.66 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 771.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1143.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.69 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 624.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1228.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 484.95 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 503.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 862.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1095.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.97 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.11 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 611.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.73 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 785.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 901.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 303.06 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 714.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 906.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 650.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1098.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 572.55 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 519.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 788.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1088.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 625.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 349.86 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 803.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 900.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 332.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 814.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1176.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 334.97 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 797.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 426.94 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 508.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 863.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.10 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 662.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 963.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1061.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.11 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 582.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.01 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 510.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.71 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 658.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 924.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 540.34 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 468.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 911.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1097.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.17 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 680.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1097.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.87 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 636.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 900.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1089.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 581.46 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 507.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 932.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1104.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.68 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 530.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1180.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.34 examples/s]

Map (num_proc=8):   0%|          | 0/218 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 56/218 [00:00<00:00, 362.63 examples/s]
Map (num_proc=8):  75%|#######5  | 164/218 [00:00<00:00, 657.89 examples/s]
Map (num_proc=8): 100%|##########| 218/218 [00:00<00:00, 526.38 examples/s]

Filter (num_proc=8):   0%|          | 0/218 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 110/218 [00:00<00:00, 972.02 examples/s]
Filter (num_proc=8): 100%|##########| 218/218 [00:00<00:00, 972.30 examples/s]INFO 2025-05-19 23:47:17,278 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_19__Virtual_machine_networking/data_checkpoint_b70320939d4b4e00b45fbbf44925b0e4.jsonl


Creating json from Arrow format:   0%|          | 0/8 [00:00<?, ?ba/s]
Creating json from Arrow format:  25%|##5       | 2/8 [00:00<00:00, 18.44ba/s]
Creating json from Arrow format:  62%|######2   | 5/8 [00:00<00:00, 20.87ba/s]
Creating json from Arrow format: 100%|##########| 8/8 [00:00<00:00, 23.63ba/s]
Creating json from Arrow format: 100%|##########| 8/8 [00:00<00:00, 22.65ba/s]INFO 2025-05-19 23:47:19,411 instructlab.sdg.generate_data:478: Generated 7327 samples
INFO 2025-05-19 23:47:19,474 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_1_1_1__OpenShift_Virtualization_supported_cluster_version, generating from scratch
INFO 2025-05-19 23:47:19,475 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 23:47:19,478 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 23:47:19,700 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 23:47:19,713 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 23:47:19,720 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 23:47:29,745 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 23:47:38,319 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 23:47:48,217 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 23:47:48,236 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 23:47:48,251 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 23:48:34,223 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 23:49:00,544 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 699.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1232.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.23 examples/s]

Map (num_proc=8):   0%|          | 0/50 [00:00<?, ? examples/s]
Map (num_proc=8):  28%|##8       | 14/50 [00:00<00:00, 112.23 examples/s]
Map (num_proc=8):  88%|########8 | 44/50 [00:00<00:00, 185.44 examples/s]
Map (num_proc=8): 100%|##########| 50/50 [00:00<00:00, 142.83 examples/s]

Filter (num_proc=8):   0%|          | 0/50 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####2    | 26/50 [00:00<00:00, 227.67 examples/s]
Filter (num_proc=8): 100%|##########| 50/50 [00:00<00:00, 235.82 examples/s]INFO 2025-05-19 23:49:02,894 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 23:49:15,775 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/201 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 51/201 [00:00<00:00, 394.37 examples/s]
Map (num_proc=8):  50%|#####     | 101/201 [00:00<00:00, 407.43 examples/s]
Map (num_proc=8): 100%|##########| 201/201 [00:00<00:00, 492.54 examples/s]

Filter (num_proc=8):   0%|          | 0/201 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 76/201 [00:00<00:00, 758.53 examples/s]
Filter (num_proc=8): 100%|##########| 201/201 [00:00<00:00, 903.48 examples/s]INFO 2025-05-19 23:49:16,981 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 23:49:25,824 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/195 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 50/195 [00:00<00:00, 365.27 examples/s]
Map (num_proc=8):  75%|#######5  | 147/195 [00:00<00:00, 540.48 examples/s]
Map (num_proc=8): 100%|##########| 195/195 [00:00<00:00, 484.75 examples/s]

Filter (num_proc=8):   0%|          | 0/195 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 99/195 [00:00<00:00, 909.58 examples/s]
Filter (num_proc=8): 100%|##########| 195/195 [00:00<00:00, 875.82 examples/s]INFO 2025-05-19 23:49:27,029 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_1_1_1__OpenShift_Virtualization_supported_cluster_version/data_checkpoint_fd68ee5fa06044e7a48e1debbe7febfa.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 78.50ba/s]INFO 2025-05-19 23:49:27,107 instructlab.sdg.generate_data:478: Generated 193 samples
INFO 2025-05-19 23:49:27,165 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_10_4_1__Creating_a_custom_namespace_for_templates, generating from scratch
INFO 2025-05-19 23:49:27,165 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 23:49:27,169 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 23:49:27,488 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 23:49:27,501 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 23:49:27,509 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 23:49:34,446 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 23:49:42,135 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 23:49:48,727 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 23:49:48,746 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 23:49:48,761 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 23:50:33,025 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 23:50:59,218 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.26 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 649.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 953.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.21 examples/s]

Map (num_proc=8):   0%|          | 0/87 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 22/87 [00:00<00:00, 174.19 examples/s]
Map (num_proc=8):  76%|#######5  | 66/87 [00:00<00:00, 301.72 examples/s]
Map (num_proc=8): 100%|##########| 87/87 [00:00<00:00, 240.12 examples/s]

Filter (num_proc=8):   0%|          | 0/87 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 44/87 [00:00<00:00, 394.88 examples/s]
Filter (num_proc=8): 100%|##########| 87/87 [00:00<00:00, 414.16 examples/s]INFO 2025-05-19 23:51:01,561 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 23:51:17,862 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.33 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 821.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 883.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.25 examples/s]
num_proc must be <= 1. Reducing num_proc to 1 for dataset of size 1.

Map:   0%|          | 0/1 [00:00<?, ? examples/s]
Map: 100%|##########| 1/1 [00:00<00:00, 184.11 examples/s]
num_proc must be <= 1. Reducing num_proc to 1 for dataset of size 1.

Filter:   0%|          | 0/1 [00:00<?, ? examples/s]
Filter: 100%|##########| 1/1 [00:00<00:00, 395.17 examples/s]INFO 2025-05-19 23:51:19,103 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 23:51:29,734 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/242 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 31/242 [00:00<00:00, 260.07 examples/s]
Map (num_proc=8):  50%|#####     | 122/242 [00:00<00:00, 570.01 examples/s]
Map (num_proc=8): 100%|##########| 242/242 [00:00<00:00, 590.16 examples/s]

Filter (num_proc=8):   0%|          | 0/242 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 122/242 [00:00<00:00, 1082.66 examples/s]
Filter (num_proc=8): 100%|##########| 242/242 [00:00<00:00, 1106.03 examples/s]INFO 2025-05-19 23:51:30,935 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_10_4_1__Creating_a_custom_namespace_for_templates/data_checkpoint_cecad59fbaf34d19b4a765a6950db1b4.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 90.90ba/s]INFO 2025-05-19 23:51:31,004 instructlab.sdg.generate_data:478: Generated 203 samples
INFO 2025-05-19 23:51:31,062 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_5__Technology_Preview_features, generating from scratch
INFO 2025-05-19 23:51:31,062 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 23:51:31,066 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 23:51:31,177 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 23:51:31,191 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 23:51:31,198 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 23:51:34,380 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 23:51:34,964 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 23:51:38,437 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 23:51:38,455 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 23:51:38,470 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 23:52:22,948 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 23:52:38,148 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/200 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 50/200 [00:00<00:00, 304.22 examples/s]
Map (num_proc=8):  62%|######2   | 125/200 [00:00<00:00, 463.52 examples/s]
Map (num_proc=8): 100%|##########| 200/200 [00:00<00:00, 485.35 examples/s]

Filter (num_proc=8):   0%|          | 0/200 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 100/200 [00:00<00:00, 988.32 examples/s]
Filter (num_proc=8): 100%|##########| 200/200 [00:00<00:00, 929.14 examples/s]INFO 2025-05-19 23:52:39,337 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 23:52:48,817 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/123 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 32/123 [00:00<00:00, 247.00 examples/s]
Map (num_proc=8):  88%|########7 | 108/123 [00:00<00:00, 407.22 examples/s]
Map (num_proc=8): 100%|##########| 123/123 [00:00<00:00, 328.82 examples/s]

Filter (num_proc=8):   0%|          | 0/123 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 63/123 [00:00<00:00, 591.61 examples/s]
Filter (num_proc=8): 100%|##########| 123/123 [00:00<00:00, 572.43 examples/s]INFO 2025-05-19 23:52:49,967 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 23:52:57,294 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/111 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 28/111 [00:00<00:00, 199.29 examples/s]
Map (num_proc=8):  87%|########7 | 97/111 [00:00<00:00, 399.19 examples/s]
Map (num_proc=8): 100%|##########| 111/111 [00:00<00:00, 301.16 examples/s]

Filter (num_proc=8):   0%|          | 0/111 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 42/111 [00:00<00:00, 407.64 examples/s]
Filter (num_proc=8): 100%|##########| 111/111 [00:00<00:00, 526.05 examples/s]INFO 2025-05-19 23:52:58,431 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_5__Technology_Preview_features/data_checkpoint_5f679729a20145d494f3d860bb028cfa.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 133.63ba/s]INFO 2025-05-19 23:52:58,472 instructlab.sdg.generate_data:478: Generated 106 samples
INFO 2025-05-19 23:52:58,533 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_10_1__Installing_QEMU_guest_agent_on_a_Linux_virtual_machine, generating from scratch
INFO 2025-05-19 23:52:58,533 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 23:52:58,536 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 23:52:58,835 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 23:52:58,848 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 23:52:58,856 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 23:53:04,860 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 23:53:13,883 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 23:53:18,431 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 23:53:18,452 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 23:53:18,467 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-19 23:54:26,647 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-19 23:56:22,065 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 463.27 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 696.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1201.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.57 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 638.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1256.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.63 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 473.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1272.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 535.99 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 579.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1393.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.72 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 608.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1138.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 292.15 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 581.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1086.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.44 examples/s]

Map (num_proc=8):   0%|          | 0/168 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 42/168 [00:00<00:00, 301.44 examples/s]
Map (num_proc=8):  62%|######2   | 105/168 [00:00<00:00, 440.65 examples/s]
Map (num_proc=8): 100%|##########| 168/168 [00:00<00:00, 434.63 examples/s]

Filter (num_proc=8):   0%|          | 0/168 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 63/168 [00:00<00:00, 609.68 examples/s]
Filter (num_proc=8): 100%|##########| 168/168 [00:00<00:00, 736.88 examples/s]INFO 2025-05-19 23:56:30,343 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-19 23:57:41,668 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.75 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 871.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1258.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 324.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 911.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1243.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.71 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 722.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1127.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.11 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 526.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1346.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.20 examples/s]

Map (num_proc=8):   0%|          | 0/110 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 28/110 [00:00<00:00, 213.54 examples/s]
Map (num_proc=8):  87%|########7 | 96/110 [00:00<00:00, 386.83 examples/s]
Map (num_proc=8): 100%|##########| 110/110 [00:00<00:00, 297.84 examples/s]

Filter (num_proc=8):   0%|          | 0/110 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 56/110 [00:00<00:00, 559.76 examples/s]
Filter (num_proc=8): 100%|##########| 110/110 [00:00<00:00, 513.28 examples/s]INFO 2025-05-19 23:57:47,624 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-19 23:58:38,725 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.11 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 526.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1169.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.23 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 755.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1181.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 456.37 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 480.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1060.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 528.45 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 511.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1221.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.99 examples/s]

Map (num_proc=8):   0%|          | 0/75 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 20/75 [00:00<00:00, 162.59 examples/s]
Map (num_proc=8): 100%|##########| 75/75 [00:00<00:00, 284.49 examples/s]
Map (num_proc=8): 100%|##########| 75/75 [00:00<00:00, 206.99 examples/s]

Filter (num_proc=8):   0%|          | 0/75 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####2    | 39/75 [00:00<00:00, 368.89 examples/s]
Filter (num_proc=8): 100%|##########| 75/75 [00:00<00:00, 351.13 examples/s]INFO 2025-05-19 23:58:44,653 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_10_1__Installing_QEMU_guest_agent_on_a_Linux_virtual_machine/data_checkpoint_99903260f15644b297b5ca42171a3e93.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 35.42ba/s]INFO 2025-05-19 23:58:44,973 instructlab.sdg.generate_data:478: Generated 1052 samples
INFO 2025-05-19 23:58:45,034 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_9_2__Deleting_nodes_from_a_bare_metal_cluster, generating from scratch
INFO 2025-05-19 23:58:45,034 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-19 23:58:45,037 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-19 23:58:45,325 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-19 23:58:45,339 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-19 23:58:45,346 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-19 23:58:52,375 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-19 23:59:00,996 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-19 23:59:07,267 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-19 23:59:07,286 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-19 23:59:07,301 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 00:00:14,393 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 00:02:22,607 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.66 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 757.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1407.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.91 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 754.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 647.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1115.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.35 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 797.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1211.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.71 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 781.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1203.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.71 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 754.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 635.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1191.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1190.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.73 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 825.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1272.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1185.27 examples/s]

Map (num_proc=8):   0%|          | 0/95 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 36/95 [00:00<00:00, 236.13 examples/s]
Map (num_proc=8):  88%|########8 | 84/95 [00:00<00:00, 337.98 examples/s]
Map (num_proc=8): 100%|##########| 95/95 [00:00<00:00, 268.99 examples/s]

Filter (num_proc=8):   0%|          | 0/95 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 48/95 [00:00<00:00, 447.11 examples/s]
Filter (num_proc=8): 100%|##########| 95/95 [00:00<00:00, 443.76 examples/s]INFO 2025-05-20 00:02:31,921 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 00:03:42,665 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.85 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 593.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1169.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.73 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 768.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 630.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.88 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 621.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1197.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 504.30 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 587.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.15 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1185.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.96 examples/s]

Map (num_proc=8):   0%|          | 0/116 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 30/116 [00:00<00:00, 226.00 examples/s]
Map (num_proc=8):  76%|#######5  | 88/116 [00:00<00:00, 382.44 examples/s]
Map (num_proc=8): 100%|##########| 116/116 [00:00<00:00, 319.45 examples/s]

Filter (num_proc=8):   0%|          | 0/116 [00:00<?, ? examples/s]
Filter (num_proc=8):  39%|###8      | 45/116 [00:00<00:00, 434.29 examples/s]
Filter (num_proc=8): 100%|##########| 116/116 [00:00<00:00, 538.91 examples/s]INFO 2025-05-20 00:03:48,541 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 00:04:43,331 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.57 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 626.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1243.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 600.11 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 550.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1217.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.29 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 709.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1089.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.19 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 724.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 836.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.45 examples/s]

Map (num_proc=8):   0%|          | 0/79 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 20/79 [00:00<00:00, 148.99 examples/s]
Map (num_proc=8):  89%|########8 | 70/79 [00:00<00:00, 329.33 examples/s]
Map (num_proc=8): 100%|##########| 79/79 [00:00<00:00, 222.37 examples/s]

Filter (num_proc=8):   0%|          | 0/79 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 50/79 [00:00<00:00, 474.51 examples/s]
Filter (num_proc=8): 100%|##########| 79/79 [00:00<00:00, 381.06 examples/s]INFO 2025-05-20 00:04:49,247 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_9_2__Deleting_nodes_from_a_bare_metal_cluster/data_checkpoint_d79d372f45d9463d81bba060b5696d04.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 40.41ba/s]INFO 2025-05-20 00:04:49,539 instructlab.sdg.generate_data:478: Generated 1006 samples
INFO 2025-05-20 00:04:49,609 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_11_2__Live_migration_limits_and_timeouts, generating from scratch
INFO 2025-05-20 00:04:49,609 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 00:04:49,613 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 00:04:49,837 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 00:04:49,851 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 00:04:49,858 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 00:04:55,061 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 00:05:02,863 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 00:05:07,794 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 00:05:07,816 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 00:05:07,831 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 00:06:03,383 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 00:07:14,979 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.34 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 818.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1208.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1185.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.24 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 766.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1087.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.84 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 637.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1139.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 516.67 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 643.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1467.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.62 examples/s]

Map (num_proc=8):   0%|          | 0/27 [00:00<?, ? examples/s]
Map (num_proc=8):  30%|##9       | 8/27 [00:00<00:00, 66.31 examples/s]
Map (num_proc=8): 100%|##########| 27/27 [00:00<00:00, 114.22 examples/s]
Map (num_proc=8): 100%|##########| 27/27 [00:00<00:00, 78.17 examples/s] 

Filter (num_proc=8):   0%|          | 0/27 [00:00<?, ? examples/s]
Filter (num_proc=8):  56%|#####5    | 15/27 [00:00<00:00, 145.18 examples/s]
Filter (num_proc=8): 100%|##########| 27/27 [00:00<00:00, 129.75 examples/s]INFO 2025-05-20 00:07:20,827 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 00:08:04,522 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.39 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 687.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1187.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.17 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 787.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1142.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.60 examples/s]

Map (num_proc=8):   0%|          | 0/222 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 56/222 [00:00<00:00, 508.39 examples/s]
Map (num_proc=8):  50%|#####     | 112/222 [00:00<00:00, 417.74 examples/s]
Map (num_proc=8): 100%|##########| 222/222 [00:00<00:00, 549.79 examples/s]

Filter (num_proc=8):   0%|          | 0/222 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 112/222 [00:00<00:00, 1021.92 examples/s]
Filter (num_proc=8): 100%|##########| 222/222 [00:00<00:00, 1014.81 examples/s]INFO 2025-05-20 00:08:08,065 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 00:08:43,319 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 447.38 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 702.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1275.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1182.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.25 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 652.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 913.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.86 examples/s]

Map (num_proc=8):   0%|          | 0/205 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 52/205 [00:00<00:00, 339.32 examples/s]
Map (num_proc=8):  76%|#######5  | 155/205 [00:00<00:00, 641.05 examples/s]
Map (num_proc=8): 100%|##########| 205/205 [00:00<00:00, 521.25 examples/s]

Filter (num_proc=8):   0%|          | 0/205 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 130/205 [00:00<00:00, 1201.77 examples/s]
Filter (num_proc=8): 100%|##########| 205/205 [00:00<00:00, 951.97 examples/s] INFO 2025-05-20 00:08:46,858 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_11_2__Live_migration_limits_and_timeouts/data_checkpoint_de76528c0a674ebdaec57747926ccf7b.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 40.91ba/s]INFO 2025-05-20 00:08:47,015 instructlab.sdg.generate_data:478: Generated 543 samples
INFO 2025-05-20 00:08:47,081 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_1_7__Additional_resources, generating from scratch
INFO 2025-05-20 00:08:47,081 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 00:08:47,084 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 00:08:48,033 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 00:08:48,047 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 00:08:48,055 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 00:09:03,842 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 00:09:28,506 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 00:09:45,002 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 00:09:45,026 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 00:09:45,044 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 00:12:35,706 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 00:19:28,674 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 480.14 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 499.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1151.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.89 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1213.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 443.31 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 694.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.54 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 604.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.29 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 469.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1137.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 325.87 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 710.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1279.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.75 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 780.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1210.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.33 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 774.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 882.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.57 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 832.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1233.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.60 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 778.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.74 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 792.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1257.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.12 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 623.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1189.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.57 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 692.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1176.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.78 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 663.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 753.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1046.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 454.43 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1201.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 343.20 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 878.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.56 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 624.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1199.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.50 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 617.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1159.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 467.69 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 615.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1158.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.80 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 565.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1240.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.68 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 700.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1228.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.39 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 600.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1241.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 320.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 893.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1144.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.58 examples/s]

Map (num_proc=8):   0%|          | 0/61 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 16/61 [00:00<00:00, 123.92 examples/s]
Map (num_proc=8):  89%|########8 | 54/61 [00:00<00:00, 248.41 examples/s]
Map (num_proc=8): 100%|##########| 61/61 [00:00<00:00, 173.53 examples/s]

Filter (num_proc=8):   0%|          | 0/61 [00:00<?, ? examples/s]
Filter (num_proc=8):  39%|###9      | 24/61 [00:00<00:00, 236.89 examples/s]
Filter (num_proc=8): 100%|##########| 61/61 [00:00<00:00, 287.63 examples/s]INFO 2025-05-20 00:19:57,746 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 00:24:33,281 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 339.96 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 576.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1169.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.40 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 763.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1216.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 454.41 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 513.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1250.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 339.08 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 739.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 415.04 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 496.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1098.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 434.41 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 522.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1255.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 455.19 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 487.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 959.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 341.44 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 655.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 425.37 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 527.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1114.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.78 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 589.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1165.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 623.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1102.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 532.70 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 507.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1242.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.92 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 562.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 910.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 516.95 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 529.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.27 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 772.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1085.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 352.22 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 712.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 331.89 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 772.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1188.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.68 examples/s]

Map (num_proc=8):   0%|          | 0/194 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 50/194 [00:00<00:00, 332.72 examples/s]
Map (num_proc=8):  63%|######2   | 122/194 [00:00<00:00, 493.64 examples/s]
Map (num_proc=8): 100%|##########| 194/194 [00:00<00:00, 455.22 examples/s]

Filter (num_proc=8):   0%|          | 0/194 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 98/194 [00:00<00:00, 950.54 examples/s]
Filter (num_proc=8): 100%|##########| 194/194 [00:00<00:00, 877.39 examples/s]INFO 2025-05-20 00:24:55,348 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 00:28:08,184 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.93 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 496.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1195.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.22 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 703.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.12 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 697.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1241.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 324.85 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 563.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.45 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 650.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1245.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 420.21 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 787.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 892.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.67 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 680.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 902.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.44 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 621.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1154.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.46 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 503.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1250.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 592.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1160.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 444.91 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 543.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1248.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.51 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 757.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1222.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 354.38 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1144.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.91 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 717.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.46 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 708.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 312.57 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 699.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1119.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.35 examples/s]

Map (num_proc=8):   0%|          | 0/164 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 42/164 [00:00<00:00, 258.52 examples/s]
Map (num_proc=8): 100%|##########| 164/164 [00:00<00:00, 590.72 examples/s]
Map (num_proc=8): 100%|##########| 164/164 [00:00<00:00, 413.68 examples/s]

Filter (num_proc=8):   0%|          | 0/164 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 84/164 [00:00<00:00, 786.35 examples/s]
Filter (num_proc=8): 100%|##########| 164/164 [00:00<00:00, 749.38 examples/s]INFO 2025-05-20 00:28:28,855 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_1_7__Additional_resources/data_checkpoint_470d2f0815e4408e8648387c15376927.jsonl


Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format:  75%|#######5  | 3/4 [00:00<00:00, 23.86ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 27.77ba/s]INFO 2025-05-20 00:28:29,826 instructlab.sdg.generate_data:478: Generated 3495 samples
INFO 2025-05-20 00:28:29,892 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_14_1_1__Installing_the_OADP_Operator, generating from scratch
INFO 2025-05-20 00:28:29,892 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 00:28:29,896 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 00:28:30,832 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 00:28:30,846 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 00:28:30,853 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 00:28:42,100 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 00:29:00,503 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 00:29:11,098 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 00:29:11,120 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 00:29:11,138 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 00:31:00,233 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 00:32:45,315 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.87 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 476.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 878.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 592.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1271.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 572.32 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 543.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1089.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 443.29 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 576.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1424.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.53 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 687.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1161.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.54 examples/s]

Map (num_proc=8):   0%|          | 0/170 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 44/170 [00:00<00:00, 262.62 examples/s]
Map (num_proc=8):  88%|########7 | 149/170 [00:00<00:00, 545.49 examples/s]
Map (num_proc=8): 100%|##########| 170/170 [00:00<00:00, 435.53 examples/s]

Filter (num_proc=8):   0%|          | 0/170 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 86/170 [00:00<00:00, 815.78 examples/s]
Filter (num_proc=8): 100%|##########| 170/170 [00:00<00:00, 765.50 examples/s]INFO 2025-05-20 00:32:52,497 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 00:34:04,850 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 344.48 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 785.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1070.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 547.07 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 499.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 925.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 342.24 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 673.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 955.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1082.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 632.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1191.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.46 examples/s]

Map (num_proc=8):   0%|          | 0/120 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 30/120 [00:00<00:00, 227.66 examples/s]
Map (num_proc=8):  75%|#######5  | 90/120 [00:00<00:00, 381.11 examples/s]
Map (num_proc=8): 100%|##########| 120/120 [00:00<00:00, 324.68 examples/s]

Filter (num_proc=8):   0%|          | 0/120 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 75/120 [00:00<00:00, 673.94 examples/s]
Filter (num_proc=8): 100%|##########| 120/120 [00:00<00:00, 560.93 examples/s]INFO 2025-05-20 00:34:10,824 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 00:35:01,018 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 434.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 671.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 902.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 315.79 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 695.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1332.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.14 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 799.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1077.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 523.99 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 509.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.98 examples/s]

Map (num_proc=8):   0%|          | 0/63 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 16/63 [00:00<00:00, 136.69 examples/s]
Map (num_proc=8):  89%|########8 | 56/63 [00:00<00:00, 253.37 examples/s]
Map (num_proc=8): 100%|##########| 63/63 [00:00<00:00, 178.84 examples/s]

Filter (num_proc=8):   0%|          | 0/63 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 40/63 [00:00<00:00, 348.20 examples/s]
Filter (num_proc=8): 100%|##########| 63/63 [00:00<00:00, 299.72 examples/s]INFO 2025-05-20 00:35:06,951 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_14_1_1__Installing_the_OADP_Operator/data_checkpoint_e4d3ccf8a2cb4b50a3ef76e011acc64d.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 23.03ba/s]INFO 2025-05-20 00:35:07,203 instructlab.sdg.generate_data:478: Generated 814 samples
INFO 2025-05-20 00:35:07,271 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_4_2__Subscribing_to_the_OpenShift_Virtualization_catalog_by_using_the_CLI, generating from scratch
INFO 2025-05-20 00:35:07,271 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 00:35:07,275 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 00:35:07,567 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 00:35:07,581 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 00:35:07,589 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 00:35:18,213 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 00:35:26,770 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 00:35:36,339 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 00:35:36,360 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 00:35:36,376 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 00:36:45,167 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 00:39:15,822 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.77 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 581.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1121.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.13 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 493.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1413.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.64 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 725.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1275.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.03 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 666.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1165.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 528.02 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 716.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 808.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1231.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.33 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 720.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1172.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.55 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 806.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 426.87 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 660.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1132.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.43 examples/s]

Map (num_proc=8):   0%|          | 0/98 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 26/98 [00:00<00:00, 200.43 examples/s]
Map (num_proc=8):  76%|#######5  | 74/98 [00:00<00:00, 295.51 examples/s]
Map (num_proc=8): 100%|##########| 98/98 [00:00<00:00, 266.30 examples/s]

Filter (num_proc=8):   0%|          | 0/98 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 50/98 [00:00<00:00, 481.76 examples/s]
Filter (num_proc=8): 100%|##########| 98/98 [00:00<00:00, 474.40 examples/s]INFO 2025-05-20 00:39:26,643 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 00:40:57,118 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 342.14 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 820.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1227.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 569.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1117.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 473.79 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 700.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1421.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.12 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 674.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1137.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.23 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 645.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1165.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.04 examples/s]

Map (num_proc=8):   0%|          | 0/206 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 52/206 [00:00<00:00, 325.91 examples/s]
Map (num_proc=8):  88%|########7 | 181/206 [00:00<00:00, 656.83 examples/s]
Map (num_proc=8): 100%|##########| 206/206 [00:00<00:00, 510.30 examples/s]

Filter (num_proc=8):   0%|          | 0/206 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 104/206 [00:00<00:00, 955.90 examples/s]
Filter (num_proc=8): 100%|##########| 206/206 [00:00<00:00, 922.85 examples/s]INFO 2025-05-20 00:41:04,399 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 00:42:12,520 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 420.99 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 782.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1243.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 443.92 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 631.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1245.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 346.98 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 651.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1146.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.16 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 871.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1241.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 467.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 544.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 909.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.73 examples/s]

Map (num_proc=8):   0%|          | 0/138 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#3        | 18/138 [00:00<00:00, 169.26 examples/s]
Map (num_proc=8):  38%|###8      | 53/138 [00:00<00:00, 209.32 examples/s]
Map (num_proc=8):  88%|########7 | 121/138 [00:00<00:00, 389.60 examples/s]
Map (num_proc=8): 100%|##########| 138/138 [00:00<00:00, 244.22 examples/s]

Filter (num_proc=8):   0%|          | 0/138 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 70/138 [00:00<00:00, 666.72 examples/s]
Filter (num_proc=8): 100%|##########| 138/138 [00:00<00:00, 639.03 examples/s]INFO 2025-05-20 00:42:19,960 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_4_2__Subscribing_to_the_OpenShift_Virtualization_catalog_by_using_the_CLI/data_checkpoint_44588233111e448d95f6b6ef6fb58528.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 32.29ba/s]INFO 2025-05-20 00:42:20,320 instructlab.sdg.generate_data:478: Generated 1221 samples
INFO 2025-05-20 00:42:20,381 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_3__New_and_changed_features, generating from scratch
INFO 2025-05-20 00:42:20,381 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 00:42:20,385 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 00:42:20,552 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 00:42:20,566 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 00:42:20,573 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 00:42:24,461 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 00:42:29,533 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 00:42:32,031 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 00:42:32,051 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 00:42:32,066 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 00:43:31,358 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 00:44:30,244 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 478.55 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 719.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 636.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1480.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.70 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 647.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 637.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1184.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 477.60 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 698.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 636.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1181.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1182.93 examples/s]

Map (num_proc=8):   0%|          | 0/143 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 36/143 [00:00<00:00, 267.51 examples/s]
Map (num_proc=8):  87%|########7 | 125/143 [00:00<00:00, 477.58 examples/s]
Map (num_proc=8): 100%|##########| 143/143 [00:00<00:00, 387.30 examples/s]

Filter (num_proc=8):   0%|          | 0/143 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 72/143 [00:00<00:00, 699.47 examples/s]
Filter (num_proc=8): 100%|##########| 143/143 [00:00<00:00, 672.60 examples/s]INFO 2025-05-20 00:44:34,910 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 00:45:08,559 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.87 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 806.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1200.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.10 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 699.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 637.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1153.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.63 examples/s]

Map (num_proc=8):   0%|          | 0/19 [00:00<?, ? examples/s]
Map (num_proc=8):  47%|####7     | 9/19 [00:00<00:00, 62.83 examples/s]
Map (num_proc=8): 100%|##########| 19/19 [00:00<00:00, 57.07 examples/s]

Filter (num_proc=8):   0%|          | 0/19 [00:00<?, ? examples/s]
Filter (num_proc=8):  68%|######8   | 13/19 [00:00<00:00, 117.20 examples/s]
Filter (num_proc=8): 100%|##########| 19/19 [00:00<00:00, 91.37 examples/s] INFO 2025-05-20 00:45:12,036 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 00:45:35,563 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.47 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 576.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1169.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.95 examples/s]

Map (num_proc=8):   0%|          | 0/230 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 58/230 [00:00<00:00, 399.59 examples/s]
Map (num_proc=8):  88%|########7 | 202/230 [00:00<00:00, 751.33 examples/s]
Map (num_proc=8): 100%|##########| 230/230 [00:00<00:00, 584.50 examples/s]

Filter (num_proc=8):   0%|          | 0/230 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 116/230 [00:00<00:00, 1100.73 examples/s]
Filter (num_proc=8): 100%|##########| 230/230 [00:00<00:00, 1056.07 examples/s]INFO 2025-05-20 00:45:37,944 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_3__New_and_changed_features/data_checkpoint_aaaea2716a99454f88a18cfacd8b003f.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 56.84ba/s]INFO 2025-05-20 00:45:38,060 instructlab.sdg.generate_data:478: Generated 394 samples
INFO 2025-05-20 00:45:38,113 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_6_4__Unpausing_a_virtual_machine, generating from scratch
INFO 2025-05-20 00:45:38,113 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 00:45:38,117 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 00:45:38,381 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 00:45:38,394 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 00:45:38,402 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 00:45:43,509 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 00:45:50,100 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 00:45:55,055 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 00:45:55,074 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 00:45:55,090 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 00:46:39,452 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 00:47:06,495 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 656.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1176.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.66 examples/s]

Map (num_proc=8):   0%|          | 0/147 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 38/147 [00:00<00:00, 309.39 examples/s]
Map (num_proc=8):  76%|#######5  | 111/147 [00:00<00:00, 465.89 examples/s]
Map (num_proc=8): 100%|##########| 147/147 [00:00<00:00, 387.01 examples/s]

Filter (num_proc=8):   0%|          | 0/147 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 75/147 [00:00<00:00, 710.17 examples/s]
Filter (num_proc=8): 100%|##########| 147/147 [00:00<00:00, 682.99 examples/s]INFO 2025-05-20 00:47:08,840 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 00:47:28,435 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.85 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 788.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1243.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.17 examples/s]

Map (num_proc=8):   0%|          | 0/38 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 10/38 [00:00<00:00, 94.89 examples/s]
Map (num_proc=8):  89%|########9 | 34/38 [00:00<00:00, 154.02 examples/s]
Map (num_proc=8): 100%|##########| 38/38 [00:00<00:00, 111.59 examples/s]

Filter (num_proc=8):   0%|          | 0/38 [00:00<?, ? examples/s]
Filter (num_proc=8):  53%|#####2    | 20/38 [00:00<00:00, 195.15 examples/s]
Filter (num_proc=8): 100%|##########| 38/38 [00:00<00:00, 181.88 examples/s]INFO 2025-05-20 00:47:30,734 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 00:47:46,130 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.89 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 652.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.80 examples/s]

Map (num_proc=8):   0%|          | 0/15 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 4/15 [00:00<00:00, 36.07 examples/s]
Map (num_proc=8):  73%|#######3  | 11/15 [00:00<00:00, 53.48 examples/s]
Map (num_proc=8): 100%|##########| 15/15 [00:00<00:00, 44.83 examples/s]

Filter (num_proc=8):   0%|          | 0/15 [00:00<?, ? examples/s]
Filter (num_proc=8):  53%|#####3    | 8/15 [00:00<00:00, 73.13 examples/s]
Filter (num_proc=8): 100%|##########| 15/15 [00:00<00:00, 72.21 examples/s]INFO 2025-05-20 00:47:48,429 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_6_4__Unpausing_a_virtual_machine/data_checkpoint_1229813ffb9b439d9ef5684226a6c78a.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 86.31ba/s]INFO 2025-05-20 00:47:48,502 instructlab.sdg.generate_data:478: Generated 227 samples
INFO 2025-05-20 00:47:48,565 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_1_1__Using_a_Quick_Start_to_create_a_virtual_machine, generating from scratch
INFO 2025-05-20 00:47:48,565 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 00:47:48,569 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 00:47:49,497 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 00:47:49,510 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 00:47:49,518 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 00:48:05,299 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 00:48:24,029 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 00:48:40,524 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 00:48:40,546 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 00:48:40,564 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 00:51:28,247 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 00:58:23,637 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.05 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 673.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1235.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.12 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 805.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1200.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.00 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 681.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1131.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.86 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 720.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1237.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.79 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 657.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1189.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.02 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 678.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 959.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.77 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 764.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1253.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.29 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 685.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 881.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 352.38 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 836.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 893.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.11 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 670.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 617.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1045.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.91 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 868.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1215.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 444.15 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 682.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1163.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.68 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 923.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.26 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 835.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1160.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 442.17 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 844.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1154.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.56 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 712.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1159.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.35 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 658.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1131.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.69 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 687.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.20 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 686.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1258.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.78 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 603.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1149.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.83 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 690.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1176.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.49 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 632.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1265.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.53 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 770.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.31 examples/s]

Map (num_proc=8):   0%|          | 0/213 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 54/213 [00:00<00:00, 485.35 examples/s]
Map (num_proc=8):  51%|#####     | 108/213 [00:00<00:00, 461.19 examples/s]
Map (num_proc=8): 100%|##########| 213/213 [00:00<00:00, 532.45 examples/s]

Filter (num_proc=8):   0%|          | 0/213 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 108/213 [00:00<00:00, 912.43 examples/s]
Filter (num_proc=8): 100%|##########| 213/213 [00:00<00:00, 964.24 examples/s]INFO 2025-05-20 00:58:52,664 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 01:03:39,619 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.63 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 485.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1149.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 595.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 916.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 236.62 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 429.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 873.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 897.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 264.61 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 459.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 228.67 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 479.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 924.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 295.01 examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 295.49 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 864.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1260.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 256.10 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 437.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 919.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 240.45 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 518.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 905.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 464.21 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 551.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.71 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 829.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.57 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 556.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1110.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 616.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1064.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.43 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 889.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1142.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.52 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 500.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.02 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 655.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 924.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 349.40 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 667.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1279.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.21 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 661.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.58 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 744.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 956.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.34 examples/s]

Map (num_proc=8):   0%|          | 0/135 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 34/135 [00:00<00:00, 282.34 examples/s]
Map (num_proc=8):  63%|######2   | 85/135 [00:00<00:00, 342.31 examples/s]
Map (num_proc=8): 100%|##########| 135/135 [00:00<00:00, 352.54 examples/s]

Filter (num_proc=8):   0%|          | 0/135 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######2   | 85/135 [00:00<00:00, 799.52 examples/s]
Filter (num_proc=8): 100%|##########| 135/135 [00:00<00:00, 626.22 examples/s]INFO 2025-05-20 01:04:03,011 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 01:07:26,376 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.31 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 658.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1209.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 885.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 274.71 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 411.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1087.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 524.23 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 526.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1040.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.75 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1261.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 685.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1148.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.96 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 743.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1206.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 269.37 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 622.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 855.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1261.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.51 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 771.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 477.12 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 549.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 799.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1180.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 272.68 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 645.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 885.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1161.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 470.98 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 653.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.14 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 710.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 316.16 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 499.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 878.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1199.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.56 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 722.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 545.79 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 489.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1122.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 576.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 507.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1267.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.50 examples/s]

Map (num_proc=8):   0%|          | 0/130 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 34/130 [00:00<00:00, 241.30 examples/s]
Map (num_proc=8):  75%|#######5  | 98/130 [00:00<00:00, 412.35 examples/s]
Map (num_proc=8): 100%|##########| 130/130 [00:00<00:00, 344.80 examples/s]

Filter (num_proc=8):   0%|          | 0/130 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 66/130 [00:00<00:00, 652.24 examples/s]
Filter (num_proc=8): 100%|##########| 130/130 [00:00<00:00, 599.51 examples/s]INFO 2025-05-20 01:07:48,393 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_1_1__Using_a_Quick_Start_to_create_a_virtual_machine/data_checkpoint_ea5ce216e4324456b8cf8bae6332ffee.jsonl


Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format:  75%|#######5  | 3/4 [00:00<00:00, 24.17ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 27.05ba/s]INFO 2025-05-20 01:07:49,418 instructlab.sdg.generate_data:478: Generated 3679 samples
INFO 2025-05-20 01:07:49,476 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_2_6__About_the_ssp_operator, generating from scratch
INFO 2025-05-20 01:07:49,476 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 01:07:49,480 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 01:07:49,619 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 01:07:49,633 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 01:07:49,640 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 01:07:51,146 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 01:07:53,318 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 01:07:54,876 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 01:07:54,894 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 01:07:54,909 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 01:08:40,309 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 01:09:09,746 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 462.44 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 545.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 639.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1180.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1192.68 examples/s]

Map (num_proc=8):   0%|          | 0/153 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 39/153 [00:00<00:00, 254.79 examples/s]
Map (num_proc=8):  88%|########7 | 134/153 [00:00<00:00, 554.15 examples/s]
Map (num_proc=8): 100%|##########| 153/153 [00:00<00:00, 411.24 examples/s]

Filter (num_proc=8):   0%|          | 0/153 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 77/153 [00:00<00:00, 660.37 examples/s]
Filter (num_proc=8): 100%|##########| 153/153 [00:00<00:00, 719.81 examples/s]INFO 2025-05-20 01:09:12,064 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 01:09:34,665 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 456.49 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 650.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 637.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1200.07 examples/s]

Map (num_proc=8):   0%|          | 0/70 [00:00<?, ? examples/s]
Map (num_proc=8):  39%|###8      | 27/70 [00:00<00:00, 198.59 examples/s]
Map (num_proc=8):  89%|########8 | 62/70 [00:00<00:00, 228.98 examples/s]
Map (num_proc=8): 100%|##########| 70/70 [00:00<00:00, 190.14 examples/s]

Filter (num_proc=8):   0%|          | 0/70 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 36/70 [00:00<00:00, 332.64 examples/s]
Filter (num_proc=8): 100%|##########| 70/70 [00:00<00:00, 341.16 examples/s]INFO 2025-05-20 01:09:36,969 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 01:09:54,399 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.61 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 662.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 651.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1086.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.59 examples/s]

Map (num_proc=8):   0%|          | 0/53 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 14/53 [00:00<00:00, 110.03 examples/s]
Map (num_proc=8):  89%|########8 | 47/53 [00:00<00:00, 215.92 examples/s]
Map (num_proc=8): 100%|##########| 53/53 [00:00<00:00, 154.52 examples/s]

Filter (num_proc=8):   0%|          | 0/53 [00:00<?, ? examples/s]
Filter (num_proc=8):  40%|###9      | 21/53 [00:00<00:00, 206.99 examples/s]
Filter (num_proc=8): 100%|##########| 53/53 [00:00<00:00, 256.50 examples/s]INFO 2025-05-20 01:09:56,671 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_2_6__About_the_ssp_operator/data_checkpoint_cf9260c160384fd08cc6040476e586cb.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 94.50ba/s]INFO 2025-05-20 01:09:56,747 instructlab.sdg.generate_data:478: Generated 234 samples
INFO 2025-05-20 01:09:56,800 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_2_3__Editing_a_virtual_machine_YAML_configuration_using_the_CLI, generating from scratch
INFO 2025-05-20 01:09:56,800 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 01:09:56,803 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 01:09:57,363 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 01:09:57,377 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 01:09:57,385 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 01:10:06,733 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 01:10:19,305 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 01:10:28,329 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 01:10:28,351 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 01:10:28,366 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 01:12:19,458 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 01:16:45,543 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 519.19 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 566.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 820.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1151.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.38 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 708.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.93 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 666.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1166.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 442.68 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1129.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.16 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 630.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1168.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 455.99 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 568.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 545.71 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 537.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1081.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.00 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 785.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1443.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.27 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 728.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1065.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 641.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1196.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.13 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 768.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1442.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.83 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 719.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.26 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 834.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.93 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 633.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 580.13 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 576.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1154.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.94 examples/s]

Map (num_proc=8):   0%|          | 0/125 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 32/125 [00:00<00:00, 254.51 examples/s]
Map (num_proc=8):  76%|#######6  | 95/125 [00:00<00:00, 406.15 examples/s]
Map (num_proc=8): 100%|##########| 125/125 [00:00<00:00, 343.92 examples/s]

Filter (num_proc=8):   0%|          | 0/125 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 64/125 [00:00<00:00, 589.36 examples/s]
Filter (num_proc=8): 100%|##########| 125/125 [00:00<00:00, 595.86 examples/s]INFO 2025-05-20 01:17:04,488 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 01:19:28,281 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.77 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 592.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.08 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 557.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1155.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.77 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 782.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1145.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 441.50 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 660.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1191.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.78 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 574.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1236.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.24 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 652.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.54 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 617.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1186.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.00 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 731.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 790.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.67 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 657.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1270.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.56 examples/s]
num_proc must be <= 5. Reducing num_proc to 5 for dataset of size 5.

Map (num_proc=5):   0%|          | 0/5 [00:00<?, ? examples/s]
Map (num_proc=5):  60%|######    | 3/5 [00:00<00:00, 23.83 examples/s]
Map (num_proc=5): 100%|##########| 5/5 [00:00<00:00, 19.44 examples/s]
num_proc must be <= 5. Reducing num_proc to 5 for dataset of size 5.

Filter (num_proc=5):   0%|          | 0/5 [00:00<?, ? examples/s]
Filter (num_proc=5): 100%|##########| 5/5 [00:00<00:00, 28.34 examples/s]INFO 2025-05-20 01:19:39,845 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 01:21:21,332 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.08 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 711.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 878.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 469.31 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 536.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1191.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.61 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 644.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1131.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.73 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 682.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.84 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 560.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1125.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 577.40 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 523.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1244.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.52 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 729.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1208.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.08 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 638.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.45 examples/s]

Map (num_proc=8):   0%|          | 0/138 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 36/138 [00:00<00:00, 249.86 examples/s]
Map (num_proc=8): 100%|##########| 138/138 [00:00<00:00, 571.07 examples/s]
Map (num_proc=8): 100%|##########| 138/138 [00:00<00:00, 368.99 examples/s]

Filter (num_proc=8):   0%|          | 0/138 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 70/138 [00:00<00:00, 647.68 examples/s]
Filter (num_proc=8): 100%|##########| 138/138 [00:00<00:00, 652.37 examples/s]INFO 2025-05-20 01:21:32,097 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_2_3__Editing_a_virtual_machine_YAML_configuration_using_the_CLI/data_checkpoint_2915002e2b7440d68b6b2dc1f2d7a32d.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 23.23ba/s]INFO 2025-05-20 01:21:32,639 instructlab.sdg.generate_data:478: Generated 1906 samples
INFO 2025-05-20 01:21:32,703 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_7_3_1__Viewing_the_SCC_and_RBAC_definitions_for_the_kubevirt_controller, generating from scratch
INFO 2025-05-20 01:21:32,703 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 01:21:32,707 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 01:21:32,895 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 01:21:32,908 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 01:21:32,915 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 01:21:37,366 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 01:21:44,650 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 01:21:50,118 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 01:21:50,137 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 01:21:50,152 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 01:22:42,028 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 01:23:26,625 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.42 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 635.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1270.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 448.84 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 736.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1273.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.16 examples/s]

Map (num_proc=8):   0%|          | 0/113 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 29/113 [00:00<00:00, 243.05 examples/s]
Map (num_proc=8):  63%|######2   | 71/113 [00:00<00:00, 332.87 examples/s]
Map (num_proc=8): 100%|##########| 113/113 [00:00<00:00, 313.46 examples/s]

Filter (num_proc=8):   0%|          | 0/113 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######2   | 71/113 [00:00<00:00, 636.45 examples/s]
Filter (num_proc=8): 100%|##########| 113/113 [00:00<00:00, 540.40 examples/s]INFO 2025-05-20 01:23:30,095 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 01:24:00,899 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.42 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 801.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1197.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.77 examples/s]

Map (num_proc=8):   0%|          | 0/244 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 62/244 [00:00<00:00, 386.45 examples/s]
Map (num_proc=8): 100%|##########| 244/244 [00:00<00:00, 830.46 examples/s]
Map (num_proc=8): 100%|##########| 244/244 [00:00<00:00, 606.89 examples/s]

Filter (num_proc=8):   0%|          | 0/244 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 124/244 [00:00<00:00, 1159.70 examples/s]
Filter (num_proc=8): 100%|##########| 244/244 [00:00<00:00, 1118.94 examples/s]INFO 2025-05-20 01:24:03,268 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 01:24:25,725 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 415.81 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 453.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 899.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.80 examples/s]

Map (num_proc=8):   0%|          | 0/228 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 58/228 [00:00<00:00, 436.29 examples/s]
Map (num_proc=8):  88%|########7 | 200/228 [00:00<00:00, 926.83 examples/s]
Map (num_proc=8): 100%|##########| 228/228 [00:00<00:00, 576.88 examples/s]

Filter (num_proc=8):   0%|          | 0/228 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 116/228 [00:00<00:00, 1037.94 examples/s]
Filter (num_proc=8): 100%|##########| 228/228 [00:00<00:00, 1040.20 examples/s]INFO 2025-05-20 01:24:28,096 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_7_3_1__Viewing_the_SCC_and_RBAC_definitions_for_the_kubevirt_controller/data_checkpoint_1ec84138e32c42c28959536f6ac717a3.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 47.25ba/s]INFO 2025-05-20 01:24:28,228 instructlab.sdg.generate_data:478: Generated 426 samples
INFO 2025-05-20 01:24:28,287 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_2_1__OpenShift_Virtualization_supported_cluster_version, generating from scratch
INFO 2025-05-20 01:24:28,287 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 01:24:28,290 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 01:24:28,428 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 01:24:28,441 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 01:24:28,449 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 01:24:31,554 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 01:24:34,667 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 01:24:37,303 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 01:24:37,322 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 01:24:37,337 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 01:25:32,289 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 01:26:12,939 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 563.97 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 529.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 895.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.21 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 749.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1458.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1183.76 examples/s]

Map (num_proc=8):   0%|          | 0/108 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 28/108 [00:00<00:00, 192.57 examples/s]
Map (num_proc=8):  88%|########7 | 95/108 [00:00<00:00, 412.54 examples/s]
Map (num_proc=8): 100%|##########| 108/108 [00:00<00:00, 296.13 examples/s]

Filter (num_proc=8):   0%|          | 0/108 [00:00<?, ? examples/s]
Filter (num_proc=8):  64%|######3   | 69/108 [00:00<00:00, 664.31 examples/s]
Filter (num_proc=8): 100%|##########| 108/108 [00:00<00:00, 521.84 examples/s]INFO 2025-05-20 01:26:16,434 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 01:26:31,649 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/245 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 31/245 [00:00<00:00, 266.73 examples/s]
Map (num_proc=8):  51%|#####     | 124/245 [00:00<00:00, 544.92 examples/s]
Map (num_proc=8): 100%|##########| 245/245 [00:00<00:00, 588.31 examples/s]

Filter (num_proc=8):   0%|          | 0/245 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 124/245 [00:00<00:00, 1152.07 examples/s]
Filter (num_proc=8): 100%|##########| 245/245 [00:00<00:00, 1147.62 examples/s]INFO 2025-05-20 01:26:32,842 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 01:26:42,822 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/222 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 56/222 [00:00<00:00, 436.15 examples/s]
Map (num_proc=8):  63%|######3   | 140/222 [00:00<00:00, 636.66 examples/s]
Map (num_proc=8): 100%|##########| 222/222 [00:00<00:00, 556.18 examples/s]

Filter (num_proc=8):   0%|          | 0/222 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 112/222 [00:00<00:00, 1008.11 examples/s]
Filter (num_proc=8): 100%|##########| 222/222 [00:00<00:00, 1022.13 examples/s]INFO 2025-05-20 01:26:43,995 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_2_1__OpenShift_Virtualization_supported_cluster_version/data_checkpoint_ec80710fceeb4373ad669d290625a372.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 95.38ba/s]INFO 2025-05-20 01:26:44,058 instructlab.sdg.generate_data:478: Generated 196 samples
INFO 2025-05-20 01:26:44,123 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_12_3_3__Configuring_obsolete_CPU_models, generating from scratch
INFO 2025-05-20 01:26:44,123 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 01:26:44,127 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 01:26:44,429 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 01:26:44,443 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 01:26:44,450 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 01:26:53,781 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 01:27:03,107 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 01:27:11,344 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 01:27:11,365 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 01:27:11,380 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 01:28:03,094 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 01:28:33,658 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.41 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 793.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1091.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.73 examples/s]

Map (num_proc=8):   0%|          | 0/152 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 38/152 [00:00<00:00, 276.18 examples/s]
Map (num_proc=8):  75%|#######5  | 114/152 [00:00<00:00, 495.39 examples/s]
Map (num_proc=8): 100%|##########| 152/152 [00:00<00:00, 402.59 examples/s]

Filter (num_proc=8):   0%|          | 0/152 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 76/152 [00:00<00:00, 748.89 examples/s]
Filter (num_proc=8): 100%|##########| 152/152 [00:00<00:00, 715.89 examples/s]INFO 2025-05-20 01:28:35,965 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 01:28:55,067 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 342.30 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 750.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1199.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.36 examples/s]

Map (num_proc=8):   0%|          | 0/11 [00:00<?, ? examples/s]
Map (num_proc=8):  36%|###6      | 4/11 [00:00<00:00, 33.58 examples/s]
Map (num_proc=8): 100%|##########| 11/11 [00:00<00:00, 47.54 examples/s]
Map (num_proc=8): 100%|##########| 11/11 [00:00<00:00, 33.10 examples/s]

Filter (num_proc=8):   0%|          | 0/11 [00:00<?, ? examples/s]
Filter (num_proc=8):  64%|######3   | 7/11 [00:00<00:00, 64.75 examples/s]
Filter (num_proc=8): 100%|##########| 11/11 [00:00<00:00, 50.86 examples/s]INFO 2025-05-20 01:28:57,348 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 01:29:12,709 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.73 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 720.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1131.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1184.67 examples/s]
num_proc must be <= 2. Reducing num_proc to 2 for dataset of size 2.

Map (num_proc=2):   0%|          | 0/2 [00:00<?, ? examples/s]
Map (num_proc=2): 100%|##########| 2/2 [00:00<00:00, 11.41 examples/s]
num_proc must be <= 2. Reducing num_proc to 2 for dataset of size 2.

Filter (num_proc=2):   0%|          | 0/2 [00:00<?, ? examples/s]
Filter (num_proc=2): 100%|##########| 2/2 [00:00<00:00, 14.11 examples/s]INFO 2025-05-20 01:29:14,353 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_12_3_3__Configuring_obsolete_CPU_models/data_checkpoint_6787a11627294d48ad68a99f0683dcac.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 104.12ba/s]INFO 2025-05-20 01:29:14,411 instructlab.sdg.generate_data:478: Generated 175 samples
INFO 2025-05-20 01:29:14,477 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_7__Using_the_OpenShift_Container_Platform_dashboard_to_get_cluster_information, generating from scratch
INFO 2025-05-20 01:29:14,477 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 01:29:14,480 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 01:29:14,627 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 01:29:14,640 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 01:29:14,648 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 01:29:19,704 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 01:29:27,388 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 01:29:31,633 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 01:29:31,652 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 01:29:31,667 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 01:30:13,421 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 01:30:30,919 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/243 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 62/243 [00:00<00:00, 578.41 examples/s]
Map (num_proc=8):  51%|#####     | 123/243 [00:00<00:00, 548.56 examples/s]
Map (num_proc=8): 100%|##########| 243/243 [00:00<00:00, 604.41 examples/s]

Filter (num_proc=8):   0%|          | 0/243 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######2   | 153/243 [00:00<00:00, 1294.39 examples/s]
Filter (num_proc=8): 100%|##########| 243/243 [00:00<00:00, 1126.55 examples/s]INFO 2025-05-20 01:30:32,090 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 01:30:44,143 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/188 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 48/188 [00:00<00:00, 330.20 examples/s]
Map (num_proc=8):  76%|#######5  | 142/188 [00:00<00:00, 615.67 examples/s]
Map (num_proc=8): 100%|##########| 188/188 [00:00<00:00, 479.10 examples/s]

Filter (num_proc=8):   0%|          | 0/188 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 96/188 [00:00<00:00, 916.07 examples/s]
Filter (num_proc=8): 100%|##########| 188/188 [00:00<00:00, 863.44 examples/s]INFO 2025-05-20 01:30:45,312 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 01:30:54,306 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/171 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 44/171 [00:00<00:00, 293.78 examples/s]
Map (num_proc=8): 100%|##########| 171/171 [00:00<00:00, 754.56 examples/s]
Map (num_proc=8): 100%|##########| 171/171 [00:00<00:00, 438.92 examples/s]

Filter (num_proc=8):   0%|          | 0/171 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 87/171 [00:00<00:00, 858.23 examples/s]
Filter (num_proc=8): 100%|##########| 171/171 [00:00<00:00, 789.80 examples/s]INFO 2025-05-20 01:30:55,465 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_7__Using_the_OpenShift_Container_Platform_dashboard_to_get_cluster_information/data_checkpoint_4b68e1cf6533491a98c42d2f462dd9c0.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 96.94ba/s]INFO 2025-05-20 01:30:55,522 instructlab.sdg.generate_data:478: Generated 158 samples
INFO 2025-05-20 01:30:55,579 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_9_1__Prerequisites, generating from scratch
INFO 2025-05-20 01:30:55,579 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 01:30:55,583 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 01:30:55,819 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 01:30:55,833 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 01:30:55,840 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 01:31:02,871 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 01:31:12,528 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 01:31:18,796 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 01:31:18,815 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 01:31:18,831 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 01:32:25,340 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 01:34:23,165 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.64 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 691.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1195.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1201.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.51 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 486.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1278.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1194.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 469.85 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 616.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1186.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1190.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.33 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 635.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1212.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.33 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 637.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1157.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 450.66 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 752.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 913.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1190.46 examples/s]

Map (num_proc=8):   0%|          | 0/195 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 50/195 [00:00<00:00, 333.51 examples/s]
Map (num_proc=8):  88%|########7 | 171/195 [00:00<00:00, 707.77 examples/s]
Map (num_proc=8): 100%|##########| 195/195 [00:00<00:00, 507.03 examples/s]

Filter (num_proc=8):   0%|          | 0/195 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 99/195 [00:00<00:00, 970.73 examples/s]
Filter (num_proc=8): 100%|##########| 195/195 [00:00<00:00, 906.27 examples/s]INFO 2025-05-20 01:34:31,253 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 01:35:48,264 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.44 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 602.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1238.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.60 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 800.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 638.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 867.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.53 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 598.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1188.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 512.55 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 521.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1199.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.72 examples/s]

Map (num_proc=8):   0%|          | 0/228 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 58/228 [00:00<00:00, 341.84 examples/s]
Map (num_proc=8):  63%|######3   | 144/228 [00:00<00:00, 570.56 examples/s]
Map (num_proc=8): 100%|##########| 228/228 [00:00<00:00, 572.32 examples/s]

Filter (num_proc=8):   0%|          | 0/228 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 116/228 [00:00<00:00, 1063.11 examples/s]
Filter (num_proc=8): 100%|##########| 228/228 [00:00<00:00, 1056.49 examples/s]INFO 2025-05-20 01:35:54,021 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 01:36:51,053 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.60 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 688.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1107.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1182.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 442.15 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 750.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 646.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1265.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 415.63 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 575.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1096.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.35 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 818.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1154.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.39 examples/s]

Map (num_proc=8):   0%|          | 0/191 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 48/191 [00:00<00:00, 341.85 examples/s]
Map (num_proc=8):  63%|######2   | 120/191 [00:00<00:00, 503.77 examples/s]
Map (num_proc=8): 100%|##########| 191/191 [00:00<00:00, 491.71 examples/s]

Filter (num_proc=8):   0%|          | 0/191 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 96/191 [00:00<00:00, 927.86 examples/s]
Filter (num_proc=8): 100%|##########| 191/191 [00:00<00:00, 890.79 examples/s]INFO 2025-05-20 01:36:56,780 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_9_1__Prerequisites/data_checkpoint_6d2f78c863f34b33b8c2a35fe88ba7d7.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 37.18ba/s]INFO 2025-05-20 01:36:57,110 instructlab.sdg.generate_data:478: Generated 1153 samples
INFO 2025-05-20 01:36:57,190 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_20_19__Re_using_persistent_volumes, generating from scratch
INFO 2025-05-20 01:36:57,190 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 01:36:57,196 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 01:36:57,198 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 01:37:00,622 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 01:37:00,638 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 01:37:00,647 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 01:37:03,232 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 01:37:03,247 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 01:37:03,256 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 01:37:28,796 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 01:37:39,469 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 01:38:10,835 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 01:38:10,918 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 01:38:35,036 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 01:38:35,104 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 01:38:35,106 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 01:38:35,155 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 01:38:35,169 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 01:38:35,251 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 01:47:42,846 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 01:47:42,865 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 01:56:47,441 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 563.04 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 505.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1250.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 344.37 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 726.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 931.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 332.47 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 799.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 837.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1090.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 338.63 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 645.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1212.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 548.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 813.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 594.79 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 534.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 829.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1279.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.51 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 642.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1271.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.65 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 715.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1141.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.26 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 541.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1150.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.44 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 680.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 955.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 440.48 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 472.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1063.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 350.20 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 706.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1205.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 327.24 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 691.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 934.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.77 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 605.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 565.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 956.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 549.68 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 510.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 910.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.83 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 630.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1227.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.96 examples/s]

Map (num_proc=8):   0%|          | 0/126 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 32/126 [00:00<00:00, 231.49 examples/s]
Map (num_proc=8):  75%|#######5  | 95/126 [00:00<00:00, 409.77 examples/s]
Map (num_proc=8): 100%|##########| 126/126 [00:00<00:00, 329.93 examples/s]

Filter (num_proc=8):   0%|          | 0/126 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 64/126 [00:00<00:00, 638.28 examples/s]
Filter (num_proc=8): 100%|##########| 126/126 [00:00<00:00, 584.93 examples/s]INFO 2025-05-20 01:57:08,593 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 02:01:32,599 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 323.97 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 699.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 544.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 891.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1098.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 339.71 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 573.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1140.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.86 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 624.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 886.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.67 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 628.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 328.97 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 722.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1418.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.35 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 631.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 955.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 351.54 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 583.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 932.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 467.38 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 642.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.38 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 507.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1248.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 607.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1169.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.35 examples/s]

Map (num_proc=8):   0%|          | 0/109 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 28/109 [00:00<00:00, 180.40 examples/s]
Map (num_proc=8):  88%|########8 | 96/109 [00:00<00:00, 418.81 examples/s]
Map (num_proc=8): 100%|##########| 109/109 [00:00<00:00, 284.76 examples/s]

Filter (num_proc=8):   0%|          | 0/109 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 56/109 [00:00<00:00, 485.52 examples/s]
Filter (num_proc=8): 100%|##########| 109/109 [00:00<00:00, 507.73 examples/s]INFO 2025-05-20 02:01:46,379 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 02:05:44,789 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.91 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 697.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 882.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 309.23 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 500.91 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 562.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.53 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 514.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 188.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 807.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 334.12 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 507.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 560.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1184.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 345.92 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 582.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1205.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.51 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 449.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 342.40 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 748.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 330.38 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 601.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1152.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 511.53 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 429.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1253.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.71 examples/s]

Map (num_proc=8):   0%|          | 0/183 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 46/183 [00:00<00:00, 272.07 examples/s]
Map (num_proc=8):  63%|######2   | 115/183 [00:00<00:00, 451.21 examples/s]
Map (num_proc=8): 100%|##########| 183/183 [00:00<00:00, 454.09 examples/s]

Filter (num_proc=8):   0%|          | 0/183 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 92/183 [00:00<00:00, 847.77 examples/s]
Filter (num_proc=8): 100%|##########| 183/183 [00:00<00:00, 835.32 examples/s]INFO 2025-05-20 02:05:57,572 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_20_19__Re_using_persistent_volumes/data_checkpoint_30ce55e61ab243b79fe4cdfb58e2bb59.jsonl
INFO 2025-05-20 02:05:57,574 instructlab.sdg.pipeline:199: Running block: router


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 20.93ba/s]INFO 2025-05-20 02:06:01,905 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 02:06:01,920 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 02:06:01,928 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 02:06:27,524 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 02:07:09,031 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 02:07:28,459 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 02:07:28,484 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 02:07:28,505 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 02:15:38,490 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 240.00 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 524.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 815.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 557.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1254.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.72 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 602.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1196.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 331.28 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 830.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1077.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.26 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 634.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1116.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 303.81 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 688.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.03 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 501.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1242.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 337.51 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 736.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1157.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 768.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1094.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.53 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 729.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1188.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 410.76 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 515.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1245.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.90 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 507.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1237.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.86 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 623.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 875.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 335.16 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 640.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1088.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.28 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 552.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1089.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 692.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.05 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 614.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1079.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.76 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 622.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 900.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.49 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 575.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.12 examples/s]INFO 2025-05-20 02:16:01,327 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair


Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 232.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 844.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.54 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 664.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.66 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 578.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1178.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 461.89 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 516.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1139.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 514.40 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 578.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 597.43 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 495.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1076.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 420.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 622.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1201.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.34 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 646.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1185.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.79 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 651.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 534.38 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 636.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.63 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 772.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1220.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.10 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 594.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 892.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.95 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 719.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1170.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.70 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 701.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 544.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 886.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.08 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 560.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1215.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 644.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1262.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 493.28 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 519.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1132.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.51 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 677.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 524.83 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 580.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 858.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1161.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.00 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 731.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1256.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.88 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 650.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1240.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 349.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 843.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1217.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.80 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 863.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.14 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 642.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1092.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 864.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1214.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 472.10 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 556.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 308.78 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 610.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1233.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 166.34 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 323.88 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 646.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 480.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1186.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 611.89 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 609.28 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 730.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 914.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 543.28 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 439.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1167.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 493.63 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 684.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1237.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 612.76 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 524.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 630.61 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 505.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 609.68 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 513.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1180.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 609.75 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 560.07 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 695.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1185.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1119.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 614.51 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 500.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 829.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 630.28 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 507.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1240.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.78 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 632.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 935.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.31 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 798.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.66 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 652.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 908.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1102.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.42 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 585.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1214.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.11 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 738.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 457.60 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 589.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 882.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.12 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 498.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 928.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.48 examples/s]

Map (num_proc=8):   0%|          | 0/85 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 22/85 [00:00<00:00, 173.83 examples/s]
Map (num_proc=8):  87%|########7 | 74/85 [00:00<00:00, 349.00 examples/s]
Map (num_proc=8): 100%|##########| 85/85 [00:00<00:00, 229.61 examples/s]

Filter (num_proc=8):   0%|          | 0/85 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 44/85 [00:00<00:00, 423.37 examples/s]
Filter (num_proc=8): 100%|##########| 85/85 [00:00<00:00, 396.68 examples/s]INFO 2025-05-20 02:16:58,261 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 02:33:47,170 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 323.85 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 641.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 555.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1219.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 319.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 959.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 926.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 498.36 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 533.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 798.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1024.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 582.90 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 482.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 901.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 508.22 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 480.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1078.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 523.54 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 507.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 907.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.91 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 583.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1172.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.42 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 580.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 887.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.11 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 592.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 599.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1048.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.63 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 475.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 556.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 928.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.72 examples/s]INFO 2025-05-20 02:34:00,607 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 248.00 examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 277.23 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 387.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.55 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 133.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 370.03 examples/s]


Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 348.60 examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 387.46 examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 404.44 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 287.98 examples/s]

Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 548.40 examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 534.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 528.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 192.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 557.62 examples/s] 


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 151.61 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 206.51 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 233.56 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 360.80 examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 534.31 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 157.15 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 353.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 305.10 examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 372.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 464.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 299.37 examples/s]


Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 609.48 examples/s]

Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 506.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 545.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 192.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 557.70 examples/s] 


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 143.23 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 280.29 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 287.02 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 378.24 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 162.09 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 332.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 301.61 examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 303.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 408.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 599.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 295.19 examples/s]


Filter (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 723.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 556.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 242.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1257.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.51 examples/s] 


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 144.89 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 253.95 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 357.69 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 154.85 examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 430.65 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 298.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 287.54 examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 345.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 333.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 202.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.99 examples/s] 


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 166.65 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1041.93 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 527.85 examples/s] 

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 146.42 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 340.77 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 334.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 473.92 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 198.50 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 548.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 296.73 examples/s]


Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 355.97 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 335.24 examples/s]

Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 167.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1016.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 526.22 examples/s] 


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 169.67 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 949.73 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 531.65 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 146.98 examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 204.54 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 324.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 478.94 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 197.09 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 396.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 296.06 examples/s]


Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 451.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 342.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 187.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1036.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.90 examples/s] 


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 167.69 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1035.50 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 526.19 examples/s] 

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 128.58 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 302.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 350.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 478.70 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 120.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 312.39 examples/s]


Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 342.18 examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 410.83 examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 358.63 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 909.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 279.51 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 785.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 177.46 examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 373.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 828.94 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 255.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 463.10 examples/s]


Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 381.46 examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 372.82 examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 424.99 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 600.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 287.47 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 306.77 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 353.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 934.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 337.52 examples/s]


Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 856.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 553.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 227.12 examples/s]
Filter (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 1050.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.16 examples/s] 


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 142.51 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 334.84 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 358.12 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 223.49 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 516.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 298.50 examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 435.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 362.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 193.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1081.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 553.68 examples/s] 


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 170.67 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 529.74 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 145.98 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 334.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 329.42 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 171.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 419.82 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 220.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 283.37 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 445.26 examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 337.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 216.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.04 examples/s] 


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 172.54 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 952.70 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 528.39 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 142.76 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 286.10 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 294.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 395.27 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 180.11 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 345.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 314.26 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 342.25 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1202.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 287.46 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 885.99 examples/s] 
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 676.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 839.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 680.29 examples/s]


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 138.15 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 248.84 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 265.38 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 300.64 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 476.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 309.85 examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 430.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 375.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 189.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1067.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 542.27 examples/s] 


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 167.86 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 939.88 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 527.42 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 146.49 examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 210.42 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 333.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 417.05 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 263.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 419.45 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 233.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 290.88 examples/s]


Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 368.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 343.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 211.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.25 examples/s] 


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 166.84 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1032.19 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 523.81 examples/s] 

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 143.90 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 332.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 317.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 482.89 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 152.95 examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 506.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 295.59 examples/s]


Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 517.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 344.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 181.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 546.99 examples/s]


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 158.12 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 958.54 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 528.81 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 142.15 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 313.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 316.25 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 181.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 301.98 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 325.82 examples/s]

Map (num_proc=8):  82%|########2 | 210/256 [00:00<00:00, 343.27 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1220.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 283.83 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 984.11 examples/s] 
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 722.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.49 examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 443.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 274.83 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 432.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 449.84 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 356.12 examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 411.84 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 328.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 294.28 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 387.04 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 573.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 342.44 examples/s]


Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 606.37 examples/s]

Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 434.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 528.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 187.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 559.69 examples/s]


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 127.10 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:01, 190.55 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 222.27 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 450.47 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 128.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 306.39 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 352.14 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 356.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 286.72 examples/s]


Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 601.34 examples/s]

Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 538.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 566.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 221.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1045.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.08 examples/s] 


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 147.76 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 207.76 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 308.75 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 454.31 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 159.35 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 358.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 292.48 examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 392.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 324.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 187.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1076.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 548.41 examples/s] 


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 164.93 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 950.20 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 528.32 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 147.29 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 266.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 377.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 522.06 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 196.77 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 381.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 322.48 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 339.22 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 306.23 examples/s]

Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 589.73 examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 503.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 540.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 227.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1075.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.47 examples/s] 


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 143.96 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 209.25 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 288.86 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 480.03 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 213.54 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 340.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 291.79 examples/s]

Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 437.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 343.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 182.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 559.92 examples/s]


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 166.90 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 952.82 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 529.78 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 141.13 examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 203.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 282.60 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 468.55 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 181.71 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 341.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 296.30 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 412.75 examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 477.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 341.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 191.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 985.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 548.36 examples/s]


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 168.02 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1039.74 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 528.22 examples/s] 

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 147.58 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 275.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 428.40 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 152.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 298.24 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 486.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 342.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 186.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 900.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 547.75 examples/s]


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 165.22 examples/s]

Filter (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 903.62 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 497.83 examples/s]

Map (num_proc=8):   9%|9         | 24/256 [00:00<00:03, 74.00 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 315.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 292.87 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 410.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 315.52 examples/s]


Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 427.47 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 359.87 examples/s]

Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 165.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1010.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 520.39 examples/s] 


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 165.45 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1028.08 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 521.15 examples/s] 

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 142.46 examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 194.56 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 276.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 456.72 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 180.06 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 357.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 309.04 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 350.82 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 343.69 examples/s]

Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 155.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 997.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 515.07 examples/s]


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 170.84 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 941.04 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 527.60 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 144.53 examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 209.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 288.16 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 455.38 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 155.58 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 388.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 295.48 examples/s]


Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 375.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 336.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 187.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1077.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 550.64 examples/s] 


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 170.49 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1043.74 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 501.09 examples/s] 

Map (num_proc=8):   3%|3         | 8/256 [00:00<00:10, 24.57 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 265.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 370.23 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 336.52 examples/s]


Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 353.28 examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 541.18 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1023.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 328.65 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 836.43 examples/s] 
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 548.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 208.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.02 examples/s] 


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 142.38 examples/s]

Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 207.73 examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 286.12 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 453.61 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 147.40 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 316.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 296.15 examples/s]

Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 346.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 323.41 examples/s]


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 157.42 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 913.65 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 511.07 examples/s]

Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 160.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1021.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 516.74 examples/s] 


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 153.76 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 284.68 examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 426.99 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 153.75 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 519.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 302.99 examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 408.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 323.97 examples/s]


Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 161.46 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 929.44 examples/s]
Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 498.05 examples/s]

Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 162.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1000.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 517.35 examples/s] 


Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 145.28 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 259.80 examples/s]

Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 490.12 examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 194.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 318.46 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 364.82 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 324.35 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 371.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]

Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 842.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 291.68 examples/s]


Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 965.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 707.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 309.00 examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 357.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 676.47 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 259.07 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 402.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 433.53 examples/s]


Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 410.07 examples/s]

Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 468.11 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 342.08 examples/s]

Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 148.80 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 248.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 381.52 examples/s]

Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 637.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 345.80 examples/s]

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 736.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 173.13 examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 376.60 examples/s]

Map (num_proc=8):   0%|          | 0/217 [00:00<?, ? examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 834.44 examples/s]

Map (num_proc=8):  13%|#2        | 28/217 [00:00<00:00, 205.39 examples/s]

Map (num_proc=8):  38%|###7      | 82/217 [00:00<00:00, 334.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 461.21 examples/s]


Map (num_proc=8):  63%|######2   | 136/217 [00:00<00:00, 305.32 examples/s]

Map (num_proc=8): 100%|##########| 217/217 [00:00<00:00, 342.53 examples/s]
Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 259.59 examples/s]
Map (num_proc=8): 100%|##########| 217/217 [00:00<00:00, 246.53 examples/s]

Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 317.81 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 425.93 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 380.90 examples/s]

Filter (num_proc=8):   0%|          | 0/217 [00:00<?, ? examples/s]

Filter (num_proc=8):  38%|###7      | 82/217 [00:00<00:00, 804.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 322.16 examples/s]

Filter (num_proc=8): 100%|##########| 217/217 [00:00<00:00, 672.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 869.85 examples/s]INFO 2025-05-20 02:35:07,309 instructlab.sdg.pipeline:199: Running block: eval_verify_question

Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1024.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 311.48 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 551.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1100.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 299.52 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 609.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1155.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.82 examples/s]

Map (num_proc=8):   0%|          | 0/19 [00:00<?, ? examples/s]
Map (num_proc=8):  47%|####7     | 9/19 [00:00<00:00, 54.61 examples/s]
Map (num_proc=8): 100%|##########| 19/19 [00:00<00:00, 73.49 examples/s]
Map (num_proc=8): 100%|##########| 19/19 [00:00<00:00, 51.94 examples/s]

Filter (num_proc=8):   0%|          | 0/19 [00:00<?, ? examples/s]
Filter (num_proc=8):  47%|####7     | 9/19 [00:00<00:00, 87.72 examples/s]
Filter (num_proc=8): 100%|##########| 19/19 [00:00<00:00, 90.74 examples/s]INFO 2025-05-20 02:35:11,373 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 02:41:29,114 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 552.26 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 507.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1196.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 478.44 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 650.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 877.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1100.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 342.38 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 565.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1119.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 465.02 examples/s]
Map (num_proc=8):  48%|####8     | 124/256 [00:00<00:00, 470.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.15 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 902.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.57 examples/s]
Map (num_proc=8):  54%|#####3    | 137/256 [00:00<00:00, 503.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 878.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1101.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 352.66 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 640.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1212.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.21 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 662.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 959.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1099.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 336.22 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 637.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 922.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1104.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.17 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 618.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1065.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.52 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 615.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.37 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 656.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1245.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 560.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 878.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.60 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 653.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1111.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.16 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 620.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1230.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.42 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 457.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 928.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.04 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 656.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 910.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1104.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.35 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 571.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1048.75 examples/s]

Map (num_proc=8):   0%|          | 0/252 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/252 [00:00<00:00, 400.46 examples/s]
Map (num_proc=8):  63%|######3   | 159/252 [00:00<00:00, 599.98 examples/s]
Map (num_proc=8): 100%|##########| 252/252 [00:00<00:00, 577.87 examples/s]

Filter (num_proc=8):   0%|          | 0/252 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 96/252 [00:00<00:00, 922.20 examples/s]
Filter (num_proc=8): 100%|##########| 252/252 [00:00<00:00, 1092.77 examples/s]INFO 2025-05-20 02:41:53,204 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 02:47:24,315 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 344.61 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 658.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 879.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1077.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.20 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 483.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 871.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1085.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 602.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1082.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 496.81 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 481.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 907.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1098.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.65 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 535.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 904.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1078.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.70 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 784.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1126.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.83 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 537.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.10 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 531.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1096.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.41 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 729.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1091.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 516.92 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 566.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 835.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1098.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 589.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 932.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1093.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.50 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 450.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1080.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1095.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 347.98 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 580.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 876.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1091.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 263.94 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 420.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 858.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 544.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1170.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.17 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 431.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1091.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.17 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 614.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1258.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.32 examples/s]

Map (num_proc=8):   0%|          | 0/232 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 58/232 [00:00<00:00, 303.84 examples/s]
Map (num_proc=8):  75%|#######5  | 174/232 [00:00<00:00, 667.05 examples/s]
Map (num_proc=8): 100%|##########| 232/232 [00:00<00:00, 537.93 examples/s]

Filter (num_proc=8):   0%|          | 0/232 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 87/232 [00:00<00:00, 804.47 examples/s]
Filter (num_proc=8): 100%|##########| 232/232 [00:00<00:00, 986.87 examples/s]INFO 2025-05-20 02:48:40,646 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.13 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 417.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1115.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1092.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.06 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 705.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 928.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1092.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 483.56 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 509.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 557.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 927.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1084.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 339.92 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 531.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 559.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 858.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1082.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 354.72 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 559.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 563.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1090.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 348.66 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 533.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 867.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 331.35 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 649.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 556.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 909.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1072.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.25 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 873.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1095.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.62 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 540.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 533.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 914.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1081.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.61 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 741.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 919.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1082.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 337.70 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 587.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 564.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1104.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1096.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 334.56 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 643.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 564.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1076.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1082.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 332.59 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 714.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 945.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 352.06 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 680.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 560.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 897.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1092.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.65 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 648.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 764.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 555.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 898.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1075.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 351.97 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 592.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 566.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1067.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1102.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 349.25 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 540.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1059.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 338.60 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 584.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1091.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 641.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 921.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1089.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.19 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 633.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 923.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1092.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.69 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 524.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 895.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1080.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 331.54 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 695.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 565.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1100.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.64 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 571.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 927.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1101.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 334.72 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 582.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 901.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1094.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.03 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 591.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1090.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.28 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 616.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1095.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.75 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 622.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 925.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 328.47 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 571.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 923.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1089.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.56 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 756.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 563.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1176.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1090.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.79 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 710.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 911.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1102.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.96 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 544.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.15 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 565.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1109.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1084.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 341.55 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 684.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 857.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1079.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 454.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 501.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 551.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 812.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1087.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.42 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 490.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 903.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1092.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.36 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 636.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1102.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 345.79 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 714.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 580.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1048.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 339.46 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 468.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 553.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1066.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1075.47 examples/s]

Map (num_proc=8):   0%|          | 0/224 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 56/224 [00:00<00:00, 285.36 examples/s]
Map (num_proc=8):  75%|#######5  | 168/224 [00:00<00:00, 597.02 examples/s]
Map (num_proc=8): 100%|##########| 224/224 [00:00<00:00, 511.76 examples/s]

Filter (num_proc=8):   0%|          | 0/224 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 84/224 [00:00<00:00, 767.93 examples/s]
Filter (num_proc=8): 100%|##########| 224/224 [00:00<00:00, 972.11 examples/s]INFO 2025-05-20 02:49:32,728 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_20_19__Re_using_persistent_volumes/data_checkpoint_9c524198119e47a0bf20d8ede99bfb18.jsonl


Creating json from Arrow format:   0%|          | 0/9 [00:00<?, ?ba/s]
Creating json from Arrow format:  33%|###3      | 3/9 [00:00<00:00, 21.47ba/s]
Creating json from Arrow format:  67%|######6   | 6/9 [00:00<00:00, 23.57ba/s]
Creating json from Arrow format: 100%|##########| 9/9 [00:00<00:00, 25.28ba/s]
Creating json from Arrow format: 100%|##########| 9/9 [00:00<00:00, 24.53ba/s]INFO 2025-05-20 02:49:33,101 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_20_19__Re_using_persistent_volumes/data_checkpoint_87bb3d99fb9f4ae89dbc3b88cc8fb14b.jsonl


Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format:  75%|#######5  | 3/4 [00:00<00:00, 27.63ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 30.79ba/s]INFO 2025-05-20 02:49:36,633 instructlab.sdg.generate_data:478: Generated 13798 samples
INFO 2025-05-20 02:49:36,694 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_5__Enabling_the_virtctl_client, generating from scratch
INFO 2025-05-20 02:49:36,694 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 02:49:36,698 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 02:49:37,056 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 02:49:37,070 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 02:49:37,078 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 02:49:44,273 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 02:49:52,191 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 02:49:59,175 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 02:49:59,195 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 02:49:59,210 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 02:51:18,090 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 02:54:24,365 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 518.47 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 515.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1254.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1116.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.18 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 490.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 935.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.09 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 645.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1210.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 345.40 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 432.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1108.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 517.61 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 549.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 532.74 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 651.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1136.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 867.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 910.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 329.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 910.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 486.79 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 482.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 956.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 604.56 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 511.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1165.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.87 examples/s]

Map (num_proc=8):   0%|          | 0/122 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 32/122 [00:00<00:00, 242.32 examples/s]
Map (num_proc=8):  88%|########7 | 107/122 [00:00<00:00, 500.99 examples/s]
Map (num_proc=8): 100%|##########| 122/122 [00:00<00:00, 327.39 examples/s]

Filter (num_proc=8):   0%|          | 0/122 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 62/122 [00:00<00:00, 587.38 examples/s]
Filter (num_proc=8): 100%|##########| 122/122 [00:00<00:00, 564.59 examples/s]INFO 2025-05-20 02:54:39,010 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 02:56:17,833 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.11 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 645.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 886.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.40 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 713.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 933.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.13 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 713.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.44 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 598.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 898.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 342.83 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 609.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.69 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 805.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1188.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.82 examples/s]

Map (num_proc=8):   0%|          | 0/82 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 22/82 [00:00<00:00, 165.90 examples/s]
Map (num_proc=8):  88%|########7 | 72/82 [00:00<00:00, 293.13 examples/s]
Map (num_proc=8): 100%|##########| 82/82 [00:00<00:00, 224.00 examples/s]

Filter (num_proc=8):   0%|          | 0/82 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 52/82 [00:00<00:00, 484.27 examples/s]
Filter (num_proc=8): 100%|##########| 82/82 [00:00<00:00, 378.16 examples/s]INFO 2025-05-20 02:56:26,416 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 02:57:40,778 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 536.13 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 474.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1208.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 354.08 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 672.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1102.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.52 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 678.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 898.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 826.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1180.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.12 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 589.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1185.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.35 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 734.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 602.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1061.51 examples/s]

Map (num_proc=8):   0%|          | 0/13 [00:00<?, ? examples/s]
Map (num_proc=8):  46%|####6     | 6/13 [00:00<00:00, 45.73 examples/s]
Map (num_proc=8):  92%|#########2| 12/13 [00:00<00:00, 51.38 examples/s]
Map (num_proc=8): 100%|##########| 13/13 [00:00<00:00, 38.47 examples/s]

Filter (num_proc=8):   0%|          | 0/13 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######1   | 8/13 [00:00<00:00, 71.88 examples/s]
Filter (num_proc=8): 100%|##########| 13/13 [00:00<00:00, 61.16 examples/s]INFO 2025-05-20 02:57:49,358 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_5__Enabling_the_virtctl_client/data_checkpoint_461d90ff347443c7a03e14f38cc9da45.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 35.08ba/s]INFO 2025-05-20 02:57:49,742 instructlab.sdg.generate_data:478: Generated 1307 samples
INFO 2025-05-20 02:57:49,799 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_1_1__Prerequisites, generating from scratch
INFO 2025-05-20 02:57:49,799 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 02:57:49,802 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 02:57:50,063 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 02:57:50,077 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 02:57:50,085 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 02:57:55,448 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 02:58:02,157 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 02:58:07,815 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 02:58:07,835 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 02:58:07,850 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 02:59:18,190 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 03:01:20,124 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.91 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 702.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1205.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.36 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 476.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1144.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.84 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 508.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1160.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.70 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 636.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1196.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.64 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 614.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.83 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 613.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1249.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.57 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 892.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1263.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.24 examples/s]

Map (num_proc=8):   0%|          | 0/165 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 42/165 [00:00<00:00, 318.04 examples/s]
Map (num_proc=8):  76%|#######5  | 125/165 [00:00<00:00, 505.27 examples/s]
Map (num_proc=8): 100%|##########| 165/165 [00:00<00:00, 427.61 examples/s]

Filter (num_proc=8):   0%|          | 0/165 [00:00<?, ? examples/s]
Filter (num_proc=8):  64%|######3   | 105/165 [00:00<00:00, 917.88 examples/s]
Filter (num_proc=8): 100%|##########| 165/165 [00:00<00:00, 780.51 examples/s]INFO 2025-05-20 03:01:29,732 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 03:02:57,667 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 351.08 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 736.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1094.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.91 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 485.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1135.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.59 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 645.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.35 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 711.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1126.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.13 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 699.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1208.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.57 examples/s]

Map (num_proc=8):   0%|          | 0/169 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 43/169 [00:00<00:00, 294.18 examples/s]
Map (num_proc=8):  63%|######2   | 106/169 [00:00<00:00, 456.34 examples/s]
Map (num_proc=8): 100%|##########| 169/169 [00:00<00:00, 427.02 examples/s]

Filter (num_proc=8):   0%|          | 0/169 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 85/169 [00:00<00:00, 849.42 examples/s]
Filter (num_proc=8): 100%|##########| 169/169 [00:00<00:00, 770.35 examples/s]INFO 2025-05-20 03:03:04,868 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 03:04:09,546 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.32 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 711.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1105.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 343.46 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 646.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1167.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.43 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 692.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1180.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.11 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 835.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1144.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 486.67 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 586.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1061.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.89 examples/s]

Map (num_proc=8):   0%|          | 0/114 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 30/114 [00:00<00:00, 218.77 examples/s]
Map (num_proc=8):  88%|########7 | 100/114 [00:00<00:00, 457.95 examples/s]
Map (num_proc=8): 100%|##########| 114/114 [00:00<00:00, 314.82 examples/s]

Filter (num_proc=8):   0%|          | 0/114 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 58/114 [00:00<00:00, 561.13 examples/s]
Filter (num_proc=8): 100%|##########| 114/114 [00:00<00:00, 540.55 examples/s]INFO 2025-05-20 03:04:17,049 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_1_1__Prerequisites/data_checkpoint_e4a00ae0857449249572db6ed97fb6cb.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 40.67ba/s]INFO 2025-05-20 03:04:17,348 instructlab.sdg.generate_data:478: Generated 1071 samples
INFO 2025-05-20 03:04:17,412 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_1__Preparing_your_cluster_for_OpenShift_Virtualization, generating from scratch
INFO 2025-05-20 03:04:17,413 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 03:04:17,416 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 03:04:18,108 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 03:04:18,122 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 03:04:18,130 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 03:04:27,000 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 03:04:37,323 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 03:04:45,584 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 03:04:45,606 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 03:04:45,622 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 03:06:35,043 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 03:11:21,348 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.76 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 672.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1147.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 558.50 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 523.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1363.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.60 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 686.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1219.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 444.45 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 534.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1204.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1185.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 337.26 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 801.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 631.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 998.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.03 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 689.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.78 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 703.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1199.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 493.71 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 618.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1216.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.02 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 792.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1211.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 319.80 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 831.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1176.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.69 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 637.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1228.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.96 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 776.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1459.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 432.75 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 638.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1154.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.81 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 732.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 436.56 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 651.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1218.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.15 examples/s]

Map (num_proc=8):   0%|          | 0/232 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 58/232 [00:00<00:00, 428.77 examples/s]
Map (num_proc=8):  62%|######2   | 145/232 [00:00<00:00, 519.06 examples/s]
Map (num_proc=8): 100%|##########| 232/232 [00:00<00:00, 576.10 examples/s]

Filter (num_proc=8):   0%|          | 0/232 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 116/232 [00:00<00:00, 1076.85 examples/s]
Filter (num_proc=8): 100%|##########| 232/232 [00:00<00:00, 1058.89 examples/s]INFO 2025-05-20 03:11:40,246 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 03:14:39,273 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 595.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.87 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 701.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1088.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 336.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 913.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1096.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.77 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 606.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1187.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1182.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.17 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 638.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1261.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 333.46 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 789.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1266.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 852.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1354.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 458.68 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 645.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 945.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 343.95 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 685.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1199.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.02 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 584.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1245.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 626.09 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 476.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.22 examples/s]

Map (num_proc=8):   0%|          | 0/160 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 40/160 [00:00<00:00, 282.42 examples/s]
Map (num_proc=8):  75%|#######5  | 120/160 [00:00<00:00, 443.47 examples/s]
Map (num_proc=8): 100%|##########| 160/160 [00:00<00:00, 406.05 examples/s]

Filter (num_proc=8):   0%|          | 0/160 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 100/160 [00:00<00:00, 926.93 examples/s]
Filter (num_proc=8): 100%|##########| 160/160 [00:00<00:00, 743.73 examples/s]INFO 2025-05-20 03:14:53,612 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 03:17:05,153 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 527.58 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 543.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 808.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1172.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 434.88 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 526.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1234.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.28 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 611.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1086.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 322.48 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 575.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1086.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.86 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 659.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1149.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 279.86 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 470.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1144.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.51 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 588.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1083.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.84 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 713.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.19 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 770.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1104.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.79 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 765.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1048.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 446.24 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 663.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.45 examples/s]

Map (num_proc=8):   0%|          | 0/9 [00:00<?, ? examples/s]
Map (num_proc=8):  44%|####4     | 4/9 [00:00<00:00, 29.39 examples/s]
Map (num_proc=8): 100%|##########| 9/9 [00:00<00:00, 37.73 examples/s]
Map (num_proc=8): 100%|##########| 9/9 [00:00<00:00, 26.65 examples/s]

Filter (num_proc=8):   0%|          | 0/9 [00:00<?, ? examples/s]
Filter (num_proc=8):  56%|#####5    | 5/9 [00:00<00:00, 49.57 examples/s]
Filter (num_proc=8): 100%|##########| 9/9 [00:00<00:00, 43.50 examples/s]INFO 2025-05-20 03:17:19,506 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_1__Preparing_your_cluster_for_OpenShift_Virtualization/data_checkpoint_60b3c6f245064691acef57c5136ef68e.jsonl


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 24.71ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 24.66ba/s]INFO 2025-05-20 03:17:20,265 instructlab.sdg.generate_data:478: Generated 2617 samples
INFO 2025-05-20 03:17:20,330 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_14_1__Installing_and_configuring_OADP, generating from scratch
INFO 2025-05-20 03:17:20,330 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 03:17:20,333 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 03:17:21,256 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 03:17:21,270 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 03:17:21,278 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 03:17:32,526 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 03:17:58,351 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 03:18:09,573 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 03:18:09,595 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 03:18:09,613 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 03:19:59,710 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 03:21:38,208 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.04 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 530.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1159.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 796.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1169.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.57 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 614.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1237.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1185.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1268.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 436.21 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 688.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1155.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.85 examples/s]

Map (num_proc=8):   0%|          | 0/116 [00:00<?, ? examples/s]
Map (num_proc=8):  39%|###8      | 45/116 [00:00<00:00, 272.53 examples/s]
Map (num_proc=8): 100%|##########| 116/116 [00:00<00:00, 443.04 examples/s]
Map (num_proc=8): 100%|##########| 116/116 [00:00<00:00, 318.75 examples/s]

Filter (num_proc=8):   0%|          | 0/116 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 60/116 [00:00<00:00, 586.04 examples/s]
Filter (num_proc=8): 100%|##########| 116/116 [00:00<00:00, 556.91 examples/s]INFO 2025-05-20 03:21:45,241 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 03:22:46,653 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 415.92 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 745.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.15 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1276.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.76 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 517.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 886.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 520.72 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 480.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.47 examples/s]

Map (num_proc=8):   0%|          | 0/249 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 63/249 [00:00<00:00, 367.27 examples/s]
Map (num_proc=8):  75%|#######5  | 187/249 [00:00<00:00, 670.84 examples/s]
Map (num_proc=8): 100%|##########| 249/249 [00:00<00:00, 593.46 examples/s]

Filter (num_proc=8):   0%|          | 0/249 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 125/249 [00:00<00:00, 1168.43 examples/s]
Filter (num_proc=8): 100%|##########| 249/249 [00:00<00:00, 1158.51 examples/s]INFO 2025-05-20 03:22:51,420 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 03:23:34,961 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 304.88 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 783.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1195.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.41 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 522.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1248.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 571.85 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 551.05 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 693.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1185.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.13 examples/s]

Map (num_proc=8):   0%|          | 0/207 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 52/207 [00:00<00:00, 369.91 examples/s]
Map (num_proc=8):  50%|#####     | 104/207 [00:00<00:00, 439.58 examples/s]
Map (num_proc=8): 100%|##########| 207/207 [00:00<00:00, 516.37 examples/s]

Filter (num_proc=8):   0%|          | 0/207 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 104/207 [00:00<00:00, 955.23 examples/s]
Filter (num_proc=8): 100%|##########| 207/207 [00:00<00:00, 956.97 examples/s]INFO 2025-05-20 03:23:39,715 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_14_1__Installing_and_configuring_OADP/data_checkpoint_267f2d1bf64b41fb9ff115d722210e8f.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 24.80ba/s]INFO 2025-05-20 03:23:39,951 instructlab.sdg.generate_data:478: Generated 760 samples
INFO 2025-05-20 03:23:40,015 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_9__OpenShift_Container_Platform_cluster_monitoring__logging__and_Telemetry, generating from scratch
INFO 2025-05-20 03:23:40,015 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 03:23:40,019 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 03:23:40,429 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 03:23:40,443 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 03:23:40,450 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 03:23:49,854 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 03:23:58,639 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 03:24:04,367 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 03:24:04,388 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 03:24:04,403 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 03:25:34,238 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 03:28:25,932 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.81 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 703.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1216.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 318.79 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 457.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1156.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 476.59 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 515.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1262.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.21 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 770.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1351.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 499.32 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 571.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1247.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.55 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 681.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1245.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 618.42 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 536.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 641.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 875.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 426.37 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 746.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1239.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.54 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 641.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 959.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 419.28 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 661.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 639.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1273.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.25 examples/s]

Map (num_proc=8):   0%|          | 0/60 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 16/60 [00:00<00:00, 135.30 examples/s]
Map (num_proc=8):  77%|#######6  | 46/60 [00:00<00:00, 220.79 examples/s]
Map (num_proc=8): 100%|##########| 60/60 [00:00<00:00, 172.17 examples/s]

Filter (num_proc=8):   0%|          | 0/60 [00:00<?, ? examples/s]
Filter (num_proc=8):  40%|####      | 24/60 [00:00<00:00, 237.91 examples/s]
Filter (num_proc=8): 100%|##########| 60/60 [00:00<00:00, 292.67 examples/s]INFO 2025-05-20 03:28:38,782 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 03:30:02,099 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.59 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 694.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1080.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.30 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 589.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1243.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.86 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 822.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 896.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.12 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 616.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1217.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 574.22 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 551.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 956.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1183.93 examples/s]

Map (num_proc=8):   0%|          | 0/72 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 18/72 [00:00<00:00, 153.97 examples/s]
Map (num_proc=8):  75%|#######5  | 54/72 [00:00<00:00, 263.30 examples/s]
Map (num_proc=8): 100%|##########| 72/72 [00:00<00:00, 200.15 examples/s]

Filter (num_proc=8):   0%|          | 0/72 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 36/72 [00:00<00:00, 322.42 examples/s]
Filter (num_proc=8): 100%|##########| 72/72 [00:00<00:00, 340.34 examples/s]INFO 2025-05-20 03:30:09,149 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 03:31:05,819 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 522.06 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 477.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 933.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 876.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1152.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.55 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 566.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1366.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 434.80 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 671.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1176.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.30 examples/s]

Map (num_proc=8):   0%|          | 0/246 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 62/246 [00:00<00:00, 398.95 examples/s]
Map (num_proc=8):  88%|########7 | 216/246 [00:00<00:00, 793.08 examples/s]
Map (num_proc=8): 100%|##########| 246/246 [00:00<00:00, 608.05 examples/s]

Filter (num_proc=8):   0%|          | 0/246 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 93/246 [00:00<00:00, 901.16 examples/s]
Filter (num_proc=8): 100%|##########| 246/246 [00:00<00:00, 1124.44 examples/s]INFO 2025-05-20 03:31:11,749 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_9__OpenShift_Container_Platform_cluster_monitoring__logging__and_Telemetry/data_checkpoint_e6e0e0d9c28d46e08eaf81fcd7a756bf.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 37.19ba/s]INFO 2025-05-20 03:31:12,082 instructlab.sdg.generate_data:478: Generated 1195 samples
INFO 2025-05-20 03:31:12,138 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_3_1__Installing_the_OpenShift_Virtualization_Operator, generating from scratch
INFO 2025-05-20 03:31:12,138 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 03:31:12,141 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 03:31:12,309 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 03:31:12,323 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 03:31:12,331 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 03:31:20,556 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 03:31:27,908 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 03:31:36,030 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 03:31:36,050 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 03:31:36,066 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 03:32:31,546 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 03:33:40,151 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 624.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 939.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.12 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 611.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1192.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.66 examples/s]

Map (num_proc=8):   0%|          | 0/207 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 52/207 [00:00<00:00, 483.69 examples/s]
Map (num_proc=8):  50%|#####     | 104/207 [00:00<00:00, 500.80 examples/s]
Map (num_proc=8): 100%|##########| 207/207 [00:00<00:00, 733.78 examples/s]
Map (num_proc=8): 100%|##########| 207/207 [00:00<00:00, 521.70 examples/s]

Filter (num_proc=8):   0%|          | 0/207 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######2   | 130/207 [00:00<00:00, 1105.38 examples/s]
Filter (num_proc=8): 100%|##########| 207/207 [00:00<00:00, 967.04 examples/s] INFO 2025-05-20 03:33:44,885 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 03:34:24,444 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.72 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 634.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 953.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.70 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 803.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1184.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.69 examples/s]

Map (num_proc=8):   0%|          | 0/114 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 30/114 [00:00<00:00, 245.15 examples/s]
Map (num_proc=8):  88%|########7 | 100/114 [00:00<00:00, 397.92 examples/s]
Map (num_proc=8): 100%|##########| 114/114 [00:00<00:00, 311.54 examples/s]

Filter (num_proc=8):   0%|          | 0/114 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 58/114 [00:00<00:00, 525.60 examples/s]
Filter (num_proc=8): 100%|##########| 114/114 [00:00<00:00, 528.04 examples/s]INFO 2025-05-20 03:34:27,980 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 03:34:57,514 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 326.34 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 722.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.49 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 687.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1158.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.88 examples/s]

Map (num_proc=8):   0%|          | 0/99 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 26/99 [00:00<00:00, 208.92 examples/s]
Map (num_proc=8):  64%|######3   | 63/99 [00:00<00:00, 250.70 examples/s]
Map (num_proc=8): 100%|##########| 99/99 [00:00<00:00, 246.43 examples/s]

Filter (num_proc=8):   0%|          | 0/99 [00:00<?, ? examples/s]
Filter (num_proc=8):  64%|######3   | 63/99 [00:00<00:00, 550.52 examples/s]
Filter (num_proc=8): 100%|##########| 99/99 [00:00<00:00, 466.06 examples/s]INFO 2025-05-20 03:35:01,084 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_3_1__Installing_the_OpenShift_Virtualization_Operator/data_checkpoint_4a0f35b91db44c3e8e6f93a45fe6334e.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 28.89ba/s]INFO 2025-05-20 03:35:01,269 instructlab.sdg.generate_data:478: Generated 576 samples
INFO 2025-05-20 03:35:01,341 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_19_1__Configuring_the_virtual_machine_for_the_default_pod_network, generating from scratch
INFO 2025-05-20 03:35:01,341 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 03:35:01,344 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 03:35:04,014 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 03:35:04,029 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 03:35:04,038 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 03:35:25,063 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 03:35:51,227 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 03:36:08,680 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 03:36:08,716 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 03:36:08,740 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 03:43:12,968 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 04:00:58,790 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.18 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 651.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 323.24 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 589.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 564.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1165.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 324.19 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 695.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1054.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 521.52 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 546.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 825.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 557.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1264.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 558.84 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 551.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 810.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 585.12 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 581.61 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 706.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1142.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 559.10 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 528.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1195.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 354.65 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 719.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 474.67 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 550.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 331.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 584.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.93 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 584.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1155.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 337.49 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 666.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1089.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 543.47 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 522.55 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 678.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1116.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 282.05 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 578.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 632.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.71 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 587.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 275.66 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 574.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1071.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.44 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 751.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.51 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 590.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 878.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 668.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1148.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.25 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 650.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 848.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 482.95 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 497.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1160.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.45 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 907.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1128.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 449.78 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 559.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 886.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.57 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 834.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 603.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1021.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.01 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 757.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1203.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.88 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 601.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1139.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.74 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 656.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 446.88 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 663.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1235.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.38 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 516.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1137.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.57 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 559.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.89 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 736.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 899.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 452.03 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.43 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 648.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1223.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 627.57 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 530.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1180.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 466.69 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 640.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 633.02 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 547.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.15 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1222.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 639.03 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 488.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 630.21 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 567.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1211.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 631.18 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 550.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1138.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.35 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 636.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.14 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 718.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1254.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.59 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 464.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1228.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.46 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 576.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1170.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.43 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 735.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 873.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 246.16 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 432.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 911.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 251.65 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 495.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 933.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 257.49 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 525.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1147.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 315.93 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 438.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 858.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1252.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 241.22 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 534.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1180.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 434.54 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 499.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1083.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.62 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 898.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 935.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.48 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 748.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.96 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 745.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 953.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.59 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 594.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1150.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 548.54 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 540.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1110.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 349.09 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 647.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 904.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.33 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 731.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1233.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.67 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 637.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 888.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 348.77 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 792.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 912.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.19 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 763.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.39 examples/s]

Map (num_proc=8):   0%|          | 0/158 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 40/158 [00:00<00:00, 278.13 examples/s]
Map (num_proc=8):  63%|######3   | 100/158 [00:00<00:00, 395.46 examples/s]
Map (num_proc=8): 100%|##########| 158/158 [00:00<00:00, 402.49 examples/s]

Filter (num_proc=8):   0%|          | 0/158 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 80/158 [00:00<00:00, 736.95 examples/s]
Filter (num_proc=8): 100%|##########| 158/158 [00:00<00:00, 715.26 examples/s]INFO 2025-05-20 04:02:14,747 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 04:12:43,949 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.92 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 546.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 879.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.38 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 872.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.69 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 621.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 564.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 953.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 485.36 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 465.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 564.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 860.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 577.57 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 485.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 628.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1106.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.12 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 555.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 560.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1118.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 479.38 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 523.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 821.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1339.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 350.08 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 605.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1172.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.22 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 610.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 564.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 867.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 568.78 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 506.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 939.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 605.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 564.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.91 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 595.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 565.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.05 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 545.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 909.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.77 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 585.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1123.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.74 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 572.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 561.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 859.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1087.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.07 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 580.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 565.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 503.92 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 483.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1219.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 297.40 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 456.25 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 683.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 549.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 879.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 311.85 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 834.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 564.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 296.60 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 744.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 875.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 464.70 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 541.10 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 682.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 560.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1138.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 483.56 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 450.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 910.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.05 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 470.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 564.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 887.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1109.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 559.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 509.14 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 506.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 894.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 542.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.13 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 512.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.17 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 645.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 931.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.40 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 489.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 339.92 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 668.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 866.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 457.93 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 470.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 563.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 928.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1119.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 546.26 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 498.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 929.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.14 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 529.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.52 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 594.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1130.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.33 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 551.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 884.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 300.28 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 657.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1170.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.21 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 446.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 564.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1217.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.84 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 503.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1212.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1113.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.64 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 488.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 561.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.79 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 542.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1201.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.79 examples/s]

Map (num_proc=8):   0%|          | 0/32 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 8/32 [00:00<00:00, 69.23 examples/s]
Map (num_proc=8):  62%|######2   | 20/32 [00:00<00:00, 96.14 examples/s]
Map (num_proc=8): 100%|##########| 32/32 [00:00<00:00, 89.87 examples/s]

Filter (num_proc=8):   0%|          | 0/32 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 20/32 [00:00<00:00, 174.98 examples/s]
Filter (num_proc=8): 100%|##########| 32/32 [00:00<00:00, 148.37 examples/s]INFO 2025-05-20 04:13:36,406 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 04:21:13,269 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.31 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 502.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 565.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.55 examples/s]
Map (num_proc=8):  59%|#####8    | 151/256 [00:00<00:00, 505.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 872.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1107.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 341.98 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 462.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 560.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1105.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.45 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 509.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1138.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1119.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 338.92 examples/s]
Map (num_proc=8):  49%|####9     | 126/256 [00:00<00:00, 451.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 945.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 320.58 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 584.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.12 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 597.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 560.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1108.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 600.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 563.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.15 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 631.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1240.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 507.90 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 450.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1065.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 322.12 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 538.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 566.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1149.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 343.89 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 885.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 624.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 338.64 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 729.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1080.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 347.53 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 647.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.82 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 482.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1138.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.36 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 450.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1155.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.75 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 620.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 898.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.70 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 509.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1153.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.54 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 714.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1132.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 315.12 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 550.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 941.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.99 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 556.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 953.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.93 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 498.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1157.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.35 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 566.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1220.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1120.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.84 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 500.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 865.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.21 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 576.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 935.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.09 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 543.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 303.62 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 807.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 566.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 443.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 533.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 348.10 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 670.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1142.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.68 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 517.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 410.47 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 665.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 774.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1060.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.63 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 613.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.54 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1165.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.79 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 664.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 898.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 286.50 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 743.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 925.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 573.19 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 458.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 916.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 321.66 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 570.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 956.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.52 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 547.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1125.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.78 examples/s]

Map (num_proc=8):   0%|          | 0/79 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 20/79 [00:00<00:00, 155.66 examples/s]
Map (num_proc=8):  89%|########8 | 70/79 [00:00<00:00, 307.25 examples/s]
Map (num_proc=8): 100%|##########| 79/79 [00:00<00:00, 212.09 examples/s]

Filter (num_proc=8):   0%|          | 0/79 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 40/79 [00:00<00:00, 376.64 examples/s]
Filter (num_proc=8): 100%|##########| 79/79 [00:00<00:00, 362.04 examples/s]INFO 2025-05-20 04:22:02,996 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_19_1__Configuring_the_virtual_machine_for_the_default_pod_network/data_checkpoint_a477bb7142674a01b520d08f485ba6d6.jsonl


Creating json from Arrow format:   0%|          | 0/9 [00:00<?, ?ba/s]
Creating json from Arrow format:  22%|##2       | 2/9 [00:00<00:00, 18.81ba/s]
Creating json from Arrow format:  56%|#####5    | 5/9 [00:00<00:00, 22.43ba/s]
Creating json from Arrow format:  89%|########8 | 8/9 [00:00<00:00, 23.02ba/s]
Creating json from Arrow format: 100%|##########| 9/9 [00:00<00:00, 24.16ba/s]INFO 2025-05-20 04:22:05,389 instructlab.sdg.generate_data:478: Generated 8462 samples
INFO 2025-05-20 04:22:05,452 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_11_7__Configuring_virtual_machine_eviction_strategy, generating from scratch
INFO 2025-05-20 04:22:05,452 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 04:22:05,456 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 04:22:05,614 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 04:22:05,627 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 04:22:05,635 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 04:22:11,539 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 04:22:17,089 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 04:22:22,526 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 04:22:22,545 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 04:22:22,560 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 04:23:14,119 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 04:24:03,088 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.95 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 595.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1182.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.82 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 649.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1358.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.93 examples/s]

Map (num_proc=8):   0%|          | 0/217 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 55/217 [00:00<00:00, 342.08 examples/s]
Map (num_proc=8):  63%|######2   | 136/217 [00:00<00:00, 508.28 examples/s]
Map (num_proc=8): 100%|##########| 217/217 [00:00<00:00, 533.67 examples/s]

Filter (num_proc=8):   0%|          | 0/217 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 109/217 [00:00<00:00, 1007.47 examples/s]
Filter (num_proc=8): 100%|##########| 217/217 [00:00<00:00, 1007.49 examples/s]INFO 2025-05-20 04:24:06,635 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 04:24:35,565 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 334.03 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 873.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1254.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.26 examples/s]

Map (num_proc=8):   0%|          | 0/195 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 50/195 [00:00<00:00, 365.32 examples/s]
Map (num_proc=8):  63%|######3   | 123/195 [00:00<00:00, 478.59 examples/s]
Map (num_proc=8): 100%|##########| 195/195 [00:00<00:00, 487.74 examples/s]

Filter (num_proc=8):   0%|          | 0/195 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 123/195 [00:00<00:00, 1057.38 examples/s]
Filter (num_proc=8): 100%|##########| 195/195 [00:00<00:00, 898.13 examples/s] INFO 2025-05-20 04:24:37,937 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 04:24:57,751 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.58 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 712.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1168.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.23 examples/s]

Map (num_proc=8):   0%|          | 0/175 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 44/175 [00:00<00:00, 308.68 examples/s]
Map (num_proc=8):  88%|########8 | 154/175 [00:00<00:00, 600.48 examples/s]
Map (num_proc=8): 100%|##########| 175/175 [00:00<00:00, 449.56 examples/s]

Filter (num_proc=8):   0%|          | 0/175 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 88/175 [00:00<00:00, 859.03 examples/s]
Filter (num_proc=8): 100%|##########| 175/175 [00:00<00:00, 798.77 examples/s]INFO 2025-05-20 04:25:00,105 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_11_7__Configuring_virtual_machine_eviction_strategy/data_checkpoint_3f611d605a7c471aa5151ab41d3f4d6a.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 44.47ba/s]INFO 2025-05-20 04:25:00,243 instructlab.sdg.generate_data:478: Generated 402 samples
INFO 2025-05-20 04:25:00,299 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_14_2_1__Additional_resources, generating from scratch
INFO 2025-05-20 04:25:00,299 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 04:25:00,303 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 04:25:00,448 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 04:25:00,461 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 04:25:00,468 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 04:25:06,233 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 04:25:12,095 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 04:25:16,415 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 04:25:16,434 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 04:25:16,449 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 04:25:53,231 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 04:26:03,928 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/154 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 40/154 [00:00<00:00, 329.52 examples/s]
Map (num_proc=8):  63%|######2   | 97/154 [00:00<00:00, 443.43 examples/s]
Map (num_proc=8): 100%|##########| 154/154 [00:00<00:00, 401.63 examples/s]

Filter (num_proc=8):   0%|          | 0/154 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 78/154 [00:00<00:00, 750.09 examples/s]
Filter (num_proc=8): 100%|##########| 154/154 [00:00<00:00, 726.36 examples/s]INFO 2025-05-20 04:26:05,089 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 04:26:13,743 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/121 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 31/121 [00:00<00:00, 222.32 examples/s]
Map (num_proc=8):  75%|#######5  | 91/121 [00:00<00:00, 402.33 examples/s]
Map (num_proc=8): 100%|##########| 121/121 [00:00<00:00, 322.25 examples/s]

Filter (num_proc=8):   0%|          | 0/121 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 61/121 [00:00<00:00, 558.49 examples/s]
Filter (num_proc=8): 100%|##########| 121/121 [00:00<00:00, 556.51 examples/s]INFO 2025-05-20 04:26:14,902 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 04:26:21,340 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/111 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 42/111 [00:00<00:00, 270.40 examples/s]
Map (num_proc=8):  88%|########8 | 98/111 [00:00<00:00, 407.19 examples/s]
Map (num_proc=8): 100%|##########| 111/111 [00:00<00:00, 297.83 examples/s]

Filter (num_proc=8):   0%|          | 0/111 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 70/111 [00:00<00:00, 611.30 examples/s]
Filter (num_proc=8): 100%|##########| 111/111 [00:00<00:00, 528.68 examples/s]INFO 2025-05-20 04:26:22,484 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_14_2_1__Additional_resources/data_checkpoint_a2c1313756244857a388e94c24dfca8c.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 146.93ba/s]INFO 2025-05-20 04:26:22,526 instructlab.sdg.generate_data:478: Generated 98 samples
INFO 2025-05-20 04:26:22,618 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_10__Running_cluster_checkups, generating from scratch
INFO 2025-05-20 04:26:22,618 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 04:26:22,621 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 04:26:23,241 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 04:26:23,254 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 04:26:23,262 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 04:26:34,684 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 04:26:50,005 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 04:26:56,507 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 04:26:56,529 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 04:26:56,544 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 04:28:30,342 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 04:32:03,344 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 441.30 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 671.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1185.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 473.76 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 730.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1266.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1184.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 327.55 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 583.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1185.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.20 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 879.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1194.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 599.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1274.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.57 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 765.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1184.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.44 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 623.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1230.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.30 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 771.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1451.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.13 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 643.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1134.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 525.85 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 808.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1278.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.65 examples/s]

Map (num_proc=8):   0%|          | 0/248 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 62/248 [00:00<00:00, 603.50 examples/s]
Map (num_proc=8):  50%|#####     | 124/248 [00:00<00:00, 538.29 examples/s]
Map (num_proc=8): 100%|##########| 248/248 [00:00<00:00, 596.63 examples/s]

Filter (num_proc=8):   0%|          | 0/248 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 124/248 [00:00<00:00, 1167.88 examples/s]
Filter (num_proc=8): 100%|##########| 248/248 [00:00<00:00, 1141.18 examples/s]INFO 2025-05-20 04:32:16,237 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 04:34:21,458 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 329.58 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 694.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1352.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 462.89 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 523.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1163.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.90 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 777.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1185.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.57 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1241.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.20 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 671.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1272.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.94 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 608.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1279.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 459.08 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 420.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1199.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.57 examples/s]

Map (num_proc=8):   0%|          | 0/219 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 56/219 [00:00<00:00, 382.42 examples/s]
Map (num_proc=8):  63%|######3   | 138/219 [00:00<00:00, 465.88 examples/s]
Map (num_proc=8): 100%|##########| 219/219 [00:00<00:00, 521.13 examples/s]

Filter (num_proc=8):   0%|          | 0/219 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 111/219 [00:00<00:00, 1073.57 examples/s]
Filter (num_proc=8): 100%|##########| 219/219 [00:00<00:00, 1002.97 examples/s]INFO 2025-05-20 04:34:31,044 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 04:35:59,735 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 325.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 791.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1287.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.32 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 636.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 933.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.97 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 536.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1212.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.95 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 612.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1161.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 535.17 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 479.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1163.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.12 examples/s]
Map (num_proc=8):  60%|######    | 154/256 [00:00<00:00, 599.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1161.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 657.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1150.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.07 examples/s]

Map (num_proc=8):   0%|          | 0/103 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 26/103 [00:00<00:00, 222.09 examples/s]
Map (num_proc=8):  63%|######3   | 65/103 [00:00<00:00, 272.52 examples/s]
Map (num_proc=8): 100%|##########| 103/103 [00:00<00:00, 269.94 examples/s]

Filter (num_proc=8):   0%|          | 0/103 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 52/103 [00:00<00:00, 472.51 examples/s]
Filter (num_proc=8): 100%|##########| 103/103 [00:00<00:00, 489.66 examples/s]INFO 2025-05-20 04:36:09,333 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_10__Running_cluster_checkups/data_checkpoint_a0687f33d5a04f778060ce45b9939ed8.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 28.47ba/s]INFO 2025-05-20 04:36:09,726 instructlab.sdg.generate_data:478: Generated 1297 samples
INFO 2025-05-20 04:36:09,789 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_12_2__Configuring_a_virtual_machine_with_the_node_exporter_service, generating from scratch
INFO 2025-05-20 04:36:09,790 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 04:36:09,793 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 04:36:10,752 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 04:36:10,766 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 04:36:10,773 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 04:36:25,181 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 04:36:56,899 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 04:37:08,876 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 04:37:08,899 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 04:37:08,914 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 04:38:10,114 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 04:39:49,629 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.00 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 580.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1256.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.13 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 646.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1237.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.34 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 494.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1200.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1186.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.46 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 726.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1232.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.85 examples/s]

Map (num_proc=8):   0%|          | 0/227 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 58/227 [00:00<00:00, 363.79 examples/s]
Map (num_proc=8):  75%|#######5  | 171/227 [00:00<00:00, 701.48 examples/s]
Map (num_proc=8): 100%|##########| 227/227 [00:00<00:00, 544.06 examples/s]

Filter (num_proc=8):   0%|          | 0/227 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 115/227 [00:00<00:00, 1063.82 examples/s]
Filter (num_proc=8): 100%|##########| 227/227 [00:00<00:00, 1037.25 examples/s]INFO 2025-05-20 04:39:55,607 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 04:40:59,322 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.32 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 649.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1231.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 476.95 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 513.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.02 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 580.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1181.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 436.31 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 632.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1257.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.40 examples/s]

Map (num_proc=8):   0%|          | 0/29 [00:00<?, ? examples/s]
Map (num_proc=8):  28%|##7       | 8/29 [00:00<00:00, 72.20 examples/s]
Map (num_proc=8):  90%|########9 | 26/29 [00:00<00:00, 122.68 examples/s]
Map (num_proc=8): 100%|##########| 29/29 [00:00<00:00, 85.17 examples/s] 

Filter (num_proc=8):   0%|          | 0/29 [00:00<?, ? examples/s]
Filter (num_proc=8):  55%|#####5    | 16/29 [00:00<00:00, 147.23 examples/s]
Filter (num_proc=8): 100%|##########| 29/29 [00:00<00:00, 139.21 examples/s]INFO 2025-05-20 04:41:05,225 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 04:41:53,796 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 825.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 956.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 477.71 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 547.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 847.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1156.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.93 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 460.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1256.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.38 examples/s]

Map (num_proc=8):   0%|          | 0/225 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 57/225 [00:00<00:00, 431.78 examples/s]
Map (num_proc=8):  63%|######2   | 141/225 [00:00<00:00, 606.23 examples/s]
Map (num_proc=8): 100%|##########| 225/225 [00:00<00:00, 539.73 examples/s]

Filter (num_proc=8):   0%|          | 0/225 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 113/225 [00:00<00:00, 1028.57 examples/s]
Filter (num_proc=8): 100%|##########| 225/225 [00:00<00:00, 1025.86 examples/s]INFO 2025-05-20 04:41:58,588 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_12_2__Configuring_a_virtual_machine_with_the_node_exporter_service/data_checkpoint_20d8ab68b128428db7935f34b301f5ba.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 22.94ba/s]INFO 2025-05-20 04:41:58,826 instructlab.sdg.generate_data:478: Generated 763 samples
INFO 2025-05-20 04:41:58,894 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_7_3__Additional_OpenShift_Container_Platform_security_context_constraints_and_Linux_capabilities_for_the_kubevirt_controller_service_account, generating from scratch
INFO 2025-05-20 04:41:58,894 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 04:41:58,898 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 04:41:59,113 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 04:41:59,126 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 04:41:59,134 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 04:42:03,556 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 04:42:12,167 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 04:42:17,724 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 04:42:17,743 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 04:42:17,758 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 04:43:09,927 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 04:43:59,227 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.04 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 556.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1219.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 454.14 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 513.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1279.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.56 examples/s]

Map (num_proc=8):   0%|          | 0/65 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 17/65 [00:00<00:00, 127.25 examples/s]
Map (num_proc=8):  88%|########7 | 57/65 [00:00<00:00, 229.87 examples/s]
Map (num_proc=8): 100%|##########| 65/65 [00:00<00:00, 182.71 examples/s]

Filter (num_proc=8):   0%|          | 0/65 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 41/65 [00:00<00:00, 382.96 examples/s]
Filter (num_proc=8): 100%|##########| 65/65 [00:00<00:00, 313.21 examples/s]INFO 2025-05-20 04:44:02,696 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 04:44:27,795 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.29 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 649.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1194.12 examples/s]

Map (num_proc=8):   0%|          | 0/123 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 32/123 [00:00<00:00, 237.59 examples/s]
Map (num_proc=8):  63%|######3   | 78/123 [00:00<00:00, 340.50 examples/s]
Map (num_proc=8): 100%|##########| 123/123 [00:00<00:00, 326.52 examples/s]

Filter (num_proc=8):   0%|          | 0/123 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 63/123 [00:00<00:00, 622.67 examples/s]
Filter (num_proc=8): 100%|##########| 123/123 [00:00<00:00, 576.84 examples/s]INFO 2025-05-20 04:44:30,110 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 04:44:48,984 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.91 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 695.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1187.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.85 examples/s]

Map (num_proc=8):   0%|          | 0/114 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 30/114 [00:00<00:00, 214.95 examples/s]
Map (num_proc=8):  75%|#######5  | 86/114 [00:00<00:00, 385.64 examples/s]
Map (num_proc=8): 100%|##########| 114/114 [00:00<00:00, 307.58 examples/s]

Filter (num_proc=8):   0%|          | 0/114 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 72/114 [00:00<00:00, 606.77 examples/s]
Filter (num_proc=8): 100%|##########| 114/114 [00:00<00:00, 532.29 examples/s]INFO 2025-05-20 04:44:51,291 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_7_3__Additional_OpenShift_Container_Platform_security_context_constraints_and_Linux_capabilities_for_the_kubevirt_controller_service_account/data_checkpoint_baa90ad9ab1e4765bc384ab931ac6d3f.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 45.22ba/s]INFO 2025-05-20 04:44:51,412 instructlab.sdg.generate_data:478: Generated 362 samples
INFO 2025-05-20 04:44:51,483 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_7__Known_issues, generating from scratch
INFO 2025-05-20 04:44:51,483 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 04:44:51,487 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 04:44:51,895 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 04:44:51,909 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 04:44:51,916 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 04:45:00,704 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 04:45:13,878 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 04:45:21,522 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 04:45:21,542 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 04:45:21,557 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 04:46:14,979 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 04:47:04,056 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 468.14 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 605.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1186.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.91 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 736.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1122.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.56 examples/s]

Map (num_proc=8):   0%|          | 0/41 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 11/41 [00:00<00:00, 89.17 examples/s]
Map (num_proc=8): 100%|##########| 41/41 [00:00<00:00, 167.92 examples/s]
Map (num_proc=8): 100%|##########| 41/41 [00:00<00:00, 115.84 examples/s]

Filter (num_proc=8):   0%|          | 0/41 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 26/41 [00:00<00:00, 242.78 examples/s]
Filter (num_proc=8): 100%|##########| 41/41 [00:00<00:00, 198.82 examples/s]INFO 2025-05-20 04:47:07,482 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 04:47:33,179 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.61 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 499.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1192.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1191.75 examples/s]

Map (num_proc=8):   0%|          | 0/139 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 36/139 [00:00<00:00, 227.18 examples/s]
Map (num_proc=8): 100%|##########| 139/139 [00:00<00:00, 355.78 examples/s]

Filter (num_proc=8):   0%|          | 0/139 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 88/139 [00:00<00:00, 775.87 examples/s]
Filter (num_proc=8): 100%|##########| 139/139 [00:00<00:00, 665.59 examples/s]INFO 2025-05-20 04:47:35,497 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 04:47:54,500 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 317.37 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 627.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1255.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.67 examples/s]

Map (num_proc=8):   0%|          | 0/83 [00:00<?, ? examples/s]
Map (num_proc=8):  40%|###9      | 33/83 [00:00<00:00, 180.89 examples/s]
Map (num_proc=8): 100%|##########| 83/83 [00:00<00:00, 319.54 examples/s]
Map (num_proc=8): 100%|##########| 83/83 [00:00<00:00, 226.20 examples/s]

Filter (num_proc=8):   0%|          | 0/83 [00:00<?, ? examples/s]
Filter (num_proc=8):  64%|######3   | 53/83 [00:00<00:00, 485.82 examples/s]
Filter (num_proc=8): 100%|##########| 83/83 [00:00<00:00, 395.10 examples/s]INFO 2025-05-20 04:47:56,801 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_4_7__Known_issues/data_checkpoint_add5246c68514550b71ec60ec9c3a2a3.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 51.42ba/s]INFO 2025-05-20 04:47:56,899 instructlab.sdg.generate_data:478: Generated 270 samples
INFO 2025-05-20 04:47:56,960 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_11_4__Migrating_a_virtual_machine_over_a_dedicated_additional_network, generating from scratch
INFO 2025-05-20 04:47:56,960 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 04:47:56,964 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 04:47:57,322 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 04:47:57,336 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 04:47:57,343 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 04:48:04,740 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 04:48:15,047 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 04:48:21,979 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 04:48:21,999 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 04:48:22,014 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 04:49:09,898 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 04:49:39,528 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 533.84 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 464.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.34 examples/s]

Map (num_proc=8):   0%|          | 0/84 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 22/84 [00:00<00:00, 179.33 examples/s]
Map (num_proc=8):  76%|#######6  | 64/84 [00:00<00:00, 294.67 examples/s]
Map (num_proc=8): 100%|##########| 84/84 [00:00<00:00, 237.12 examples/s]

Filter (num_proc=8):   0%|          | 0/84 [00:00<?, ? examples/s]
Filter (num_proc=8):  64%|######4   | 54/84 [00:00<00:00, 477.59 examples/s]
Filter (num_proc=8): 100%|##########| 84/84 [00:00<00:00, 404.01 examples/s]INFO 2025-05-20 04:49:41,808 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 04:49:55,960 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/224 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 56/224 [00:00<00:00, 318.09 examples/s]
Map (num_proc=8):  75%|#######5  | 168/224 [00:00<00:00, 612.04 examples/s]
Map (num_proc=8): 100%|##########| 224/224 [00:00<00:00, 538.54 examples/s]

Filter (num_proc=8):   0%|          | 0/224 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 112/224 [00:00<00:00, 1041.45 examples/s]
Filter (num_proc=8): 100%|##########| 224/224 [00:00<00:00, 1034.42 examples/s]INFO 2025-05-20 04:49:57,131 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 04:50:06,791 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/203 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 52/203 [00:00<00:00, 468.17 examples/s]
Map (num_proc=8):  51%|#####     | 103/203 [00:00<00:00, 447.81 examples/s]
Map (num_proc=8): 100%|##########| 203/203 [00:00<00:00, 495.51 examples/s]

Filter (num_proc=8):   0%|          | 0/203 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 103/203 [00:00<00:00, 950.69 examples/s]
Filter (num_proc=8): 100%|##########| 203/203 [00:00<00:00, 957.44 examples/s]INFO 2025-05-20 04:50:07,954 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_11_4__Migrating_a_virtual_machine_over_a_dedicated_additional_network/data_checkpoint_36daa227c1ee4e3cb816727bb2c9bd73.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 96.13ba/s]INFO 2025-05-20 04:50:08,013 instructlab.sdg.generate_data:478: Generated 164 samples
INFO 2025-05-20 04:50:08,083 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_10_1_3__Creating_a_virtual_machine_template_in_the_web_console, generating from scratch
INFO 2025-05-20 04:50:08,084 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 04:50:08,087 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 04:50:08,544 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 04:50:08,558 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 04:50:08,566 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 04:50:17,039 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 04:50:28,278 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 04:50:36,247 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 04:50:36,268 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 04:50:36,283 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 04:51:29,167 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 04:52:27,253 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 479.78 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 614.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 959.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.43 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 724.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1096.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.65 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 803.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 922.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.48 examples/s]
num_proc must be <= 7. Reducing num_proc to 7 for dataset of size 7.

Map (num_proc=7):   0%|          | 0/7 [00:00<?, ? examples/s]
Map (num_proc=7):  43%|####2     | 3/7 [00:00<00:00, 22.53 examples/s]
Map (num_proc=7): 100%|##########| 7/7 [00:00<00:00, 23.39 examples/s]
num_proc must be <= 7. Reducing num_proc to 7 for dataset of size 7.

Filter (num_proc=7):   0%|          | 0/7 [00:00<?, ? examples/s]
Filter (num_proc=7):  71%|#######1  | 5/7 [00:00<00:00, 49.67 examples/s]
Filter (num_proc=7): 100%|##########| 7/7 [00:00<00:00, 37.19 examples/s]INFO 2025-05-20 04:52:31,741 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 04:53:08,543 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.41 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 738.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1246.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.80 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 591.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1266.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.84 examples/s]

Map (num_proc=8):   0%|          | 0/71 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###8      | 27/71 [00:00<00:00, 183.36 examples/s]
Map (num_proc=8):  89%|########8 | 63/71 [00:00<00:00, 266.14 examples/s]
Map (num_proc=8): 100%|##########| 71/71 [00:00<00:00, 199.92 examples/s]

Filter (num_proc=8):   0%|          | 0/71 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 36/71 [00:00<00:00, 350.70 examples/s]
Filter (num_proc=8): 100%|##########| 71/71 [00:00<00:00, 341.33 examples/s]INFO 2025-05-20 04:53:11,993 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 04:53:37,464 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.44 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 481.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1193.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1182.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.05 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 636.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 858.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.38 examples/s]

Map (num_proc=8):   0%|          | 0/30 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 8/30 [00:00<00:00, 68.62 examples/s]
Map (num_proc=8): 100%|##########| 30/30 [00:00<00:00, 89.06 examples/s]

Filter (num_proc=8):   0%|          | 0/30 [00:00<?, ? examples/s]
Filter (num_proc=8):  53%|#####3    | 16/30 [00:00<00:00, 157.30 examples/s]
Filter (num_proc=8): 100%|##########| 30/30 [00:00<00:00, 146.28 examples/s]INFO 2025-05-20 04:53:40,892 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_10_1_3__Creating_a_virtual_machine_template_in_the_web_console/data_checkpoint_4c8351f746814df3a116a820a6d11215.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 32.58ba/s]INFO 2025-05-20 04:53:41,055 instructlab.sdg.generate_data:478: Generated 483 samples
INFO 2025-05-20 04:53:41,114 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_1_1__Hardware_and_operating_system_requirements, generating from scratch
INFO 2025-05-20 04:53:41,114 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 04:53:41,117 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 04:53:41,783 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 04:53:41,797 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 04:53:41,805 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 04:53:50,671 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 04:54:03,420 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 04:54:11,665 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 04:54:11,686 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 04:54:11,701 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 04:55:56,140 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 05:00:17,001 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 333.80 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 554.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1139.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 323.20 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 720.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1251.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 348.85 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 874.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1370.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.03 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 862.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1360.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1185.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 328.88 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 849.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1203.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.85 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 633.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1390.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1203.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.91 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 761.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1070.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.04 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 629.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1213.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1196.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 458.27 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 684.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1170.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1184.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 453.65 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 643.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 495.44 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 699.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1263.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.50 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 699.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1132.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.88 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 721.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 731.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1108.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1182.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 638.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.42 examples/s]
num_proc must be <= 3. Reducing num_proc to 3 for dataset of size 3.

Map (num_proc=3):   0%|          | 0/3 [00:00<?, ? examples/s]
Map (num_proc=3): 100%|##########| 3/3 [00:00<00:00, 14.43 examples/s]
num_proc must be <= 3. Reducing num_proc to 3 for dataset of size 3.

Filter (num_proc=3):   0%|          | 0/3 [00:00<?, ? examples/s]
Filter (num_proc=3): 100%|##########| 3/3 [00:00<00:00, 19.43 examples/s]INFO 2025-05-20 05:00:35,183 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 05:02:50,285 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 506.58 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 495.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 931.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.30 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 652.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 931.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.72 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 591.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1129.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.30 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 700.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 883.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1074.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.46 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 555.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1188.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.39 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 512.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1167.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 579.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1131.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 494.19 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 524.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 801.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1149.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.64 examples/s]

Map (num_proc=8):   0%|          | 0/223 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 56/223 [00:00<00:00, 341.33 examples/s]
Map (num_proc=8):  63%|######2   | 140/223 [00:00<00:00, 529.29 examples/s]
Map (num_proc=8): 100%|##########| 223/223 [00:00<00:00, 530.84 examples/s]

Filter (num_proc=8):   0%|          | 0/223 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######2   | 140/223 [00:00<00:00, 1125.80 examples/s]
Filter (num_proc=8): 100%|##########| 223/223 [00:00<00:00, 1018.12 examples/s]INFO 2025-05-20 05:03:01,049 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 05:04:41,062 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 325.89 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 821.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1153.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 337.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 847.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1401.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.86 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 634.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 604.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1100.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 544.93 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 565.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1126.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.78 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 703.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1444.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.16 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 556.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 313.61 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 495.06 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 739.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 955.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.55 examples/s]

Map (num_proc=8):   0%|          | 0/122 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 32/122 [00:00<00:00, 250.84 examples/s]
Map (num_proc=8):  63%|######3   | 77/122 [00:00<00:00, 334.87 examples/s]
Map (num_proc=8): 100%|##########| 122/122 [00:00<00:00, 319.38 examples/s]

Filter (num_proc=8):   0%|          | 0/122 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 77/122 [00:00<00:00, 666.34 examples/s]
Filter (num_proc=8): 100%|##########| 122/122 [00:00<00:00, 577.89 examples/s]INFO 2025-05-20 05:04:51,694 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_1_1__Hardware_and_operating_system_requirements/data_checkpoint_2dbf6071fdbb4e9fbcea9f6b4d3c7882.jsonl


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format:  67%|######6   | 2/3 [00:00<00:00, 19.39ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 28.49ba/s]INFO 2025-05-20 05:04:52,288 instructlab.sdg.generate_data:478: Generated 2008 samples
INFO 2025-05-20 05:04:52,354 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_13_1__About_VirtIO_drivers, generating from scratch
INFO 2025-05-20 05:04:52,354 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 05:04:52,358 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 05:04:52,687 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 05:04:52,700 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 05:04:52,708 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 05:04:59,908 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 05:05:09,376 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 05:05:16,321 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 05:05:16,341 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 05:05:16,356 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 05:06:19,720 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 05:08:16,605 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 479.72 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 611.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1248.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.09 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 675.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 898.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.72 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 553.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 887.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.11 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 746.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1444.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 625.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1189.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 436.01 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 531.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1182.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.67 examples/s]

Map (num_proc=8):   0%|          | 0/155 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 40/155 [00:00<00:00, 246.46 examples/s]
Map (num_proc=8): 100%|##########| 155/155 [00:00<00:00, 656.82 examples/s]
Map (num_proc=8): 100%|##########| 155/155 [00:00<00:00, 404.43 examples/s]

Filter (num_proc=8):   0%|          | 0/155 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 79/155 [00:00<00:00, 752.46 examples/s]
Filter (num_proc=8): 100%|##########| 155/155 [00:00<00:00, 737.50 examples/s]INFO 2025-05-20 05:08:24,811 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 05:09:50,477 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.26 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 655.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1193.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 348.61 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 708.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1165.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 351.41 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 719.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 420.21 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 634.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1184.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 546.55 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 558.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1222.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.18 examples/s]

Map (num_proc=8):   0%|          | 0/104 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 26/104 [00:00<00:00, 218.73 examples/s]
Map (num_proc=8):  88%|########7 | 91/104 [00:00<00:00, 377.10 examples/s]
Map (num_proc=8): 100%|##########| 104/104 [00:00<00:00, 282.76 examples/s]

Filter (num_proc=8):   0%|          | 0/104 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 39/104 [00:00<00:00, 376.98 examples/s]
Filter (num_proc=8): 100%|##########| 104/104 [00:00<00:00, 461.06 examples/s]INFO 2025-05-20 05:09:57,548 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 05:11:00,030 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.88 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 717.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1243.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.00 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 696.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.49 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 768.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1244.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.46 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 664.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1303.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.96 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 788.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1157.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.06 examples/s]

Map (num_proc=8):   0%|          | 0/72 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 18/72 [00:00<00:00, 137.55 examples/s]
Map (num_proc=8): 100%|##########| 72/72 [00:00<00:00, 301.42 examples/s]
Map (num_proc=8): 100%|##########| 72/72 [00:00<00:00, 203.64 examples/s]

Filter (num_proc=8):   0%|          | 0/72 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 36/72 [00:00<00:00, 346.89 examples/s]
Filter (num_proc=8): 100%|##########| 72/72 [00:00<00:00, 342.49 examples/s]INFO 2025-05-20 05:11:07,044 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_13_1__About_VirtIO_drivers/data_checkpoint_3663d638b59c46f3bcf86cae89b95820.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 33.72ba/s]INFO 2025-05-20 05:11:07,384 instructlab.sdg.generate_data:478: Generated 1131 samples
INFO 2025-05-20 05:11:07,441 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_10_5_1__Deleting_a_virtual_machine_template_in_the_web_console, generating from scratch
INFO 2025-05-20 05:11:07,441 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 05:11:07,445 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 05:11:07,565 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 05:11:07,578 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 05:11:07,586 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 05:11:10,290 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 05:11:13,234 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 05:11:16,407 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 05:11:16,425 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 05:11:16,440 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 05:12:08,781 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 05:13:15,773 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 533.28 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 602.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 640.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1260.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1213.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.16 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 789.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 637.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1206.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1197.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 485.75 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 658.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1236.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1195.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 601.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 644.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 922.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1216.48 examples/s]

Map (num_proc=8):   0%|          | 0/90 [00:00<?, ? examples/s]
Map (num_proc=8):  39%|###8      | 35/90 [00:00<00:00, 214.53 examples/s]
Map (num_proc=8): 100%|##########| 90/90 [00:00<00:00, 359.17 examples/s]
Map (num_proc=8): 100%|##########| 90/90 [00:00<00:00, 252.31 examples/s]

Filter (num_proc=8):   0%|          | 0/90 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 57/90 [00:00<00:00, 521.22 examples/s]
Filter (num_proc=8): 100%|##########| 90/90 [00:00<00:00, 431.43 examples/s]INFO 2025-05-20 05:13:21,464 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 05:14:01,507 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.59 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 731.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.67 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 800.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1226.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1195.20 examples/s]

Map (num_proc=8):   0%|          | 0/132 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 34/132 [00:00<00:00, 292.55 examples/s]
Map (num_proc=8):  63%|######2   | 83/132 [00:00<00:00, 400.24 examples/s]
Map (num_proc=8): 100%|##########| 132/132 [00:00<00:00, 361.37 examples/s]

Filter (num_proc=8):   0%|          | 0/132 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 68/132 [00:00<00:00, 651.11 examples/s]
Filter (num_proc=8): 100%|##########| 132/132 [00:00<00:00, 626.76 examples/s]INFO 2025-05-20 05:14:04,949 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 05:14:33,047 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.07 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 756.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1412.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1191.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 691.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1116.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.26 examples/s]

Map (num_proc=8):   0%|          | 0/107 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 28/107 [00:00<00:00, 213.34 examples/s]
Map (num_proc=8):  88%|########7 | 94/107 [00:00<00:00, 399.44 examples/s]
Map (num_proc=8): 100%|##########| 107/107 [00:00<00:00, 297.37 examples/s]

Filter (num_proc=8):   0%|          | 0/107 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 55/107 [00:00<00:00, 544.00 examples/s]
Filter (num_proc=8): 100%|##########| 107/107 [00:00<00:00, 511.60 examples/s]INFO 2025-05-20 05:14:36,484 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_10_5_1__Deleting_a_virtual_machine_template_in_the_web_console/data_checkpoint_5e432e9e622745e6b1ba3aa686d5c8d8.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 44.36ba/s]INFO 2025-05-20 05:14:36,643 instructlab.sdg.generate_data:478: Generated 565 samples
INFO 2025-05-20 05:14:36,712 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_7_1__Accessing_virtual_machine_consoles_in_the_OpenShift_Container_Platform_web_console, generating from scratch
INFO 2025-05-20 05:14:36,713 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 05:14:36,716 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 05:14:37,501 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 05:14:37,516 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 05:14:37,524 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 05:14:47,721 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 05:15:01,486 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 05:15:09,233 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 05:15:09,255 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 05:15:09,274 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 05:17:48,482 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 05:23:54,168 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 298.64 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 443.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 884.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1363.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.51 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 717.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1389.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 648.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1220.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.38 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 665.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 934.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.23 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 676.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1121.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.63 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 623.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 485.87 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 540.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1142.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.55 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 628.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 444.25 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 447.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 613.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 941.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 449.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 676.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 923.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.10 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 573.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1244.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.44 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 691.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 953.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.36 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 704.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1271.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.93 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 605.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1192.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.95 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 745.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 904.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.15 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 657.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1238.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.95 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 670.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1113.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 342.66 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 687.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.93 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 608.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1099.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 445.33 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 627.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 860.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.16 examples/s]

Map (num_proc=8):   0%|          | 0/151 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 38/151 [00:00<00:00, 292.23 examples/s]
Map (num_proc=8):  75%|#######4  | 113/151 [00:00<00:00, 462.99 examples/s]
Map (num_proc=8): 100%|##########| 151/151 [00:00<00:00, 379.04 examples/s]

Filter (num_proc=8):   0%|          | 0/151 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 76/151 [00:00<00:00, 710.70 examples/s]
Filter (num_proc=8): 100%|##########| 151/151 [00:00<00:00, 701.54 examples/s]INFO 2025-05-20 05:24:19,298 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 05:28:03,820 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.14 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 671.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1217.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 341.92 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 734.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1200.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 576.72 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 562.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 762.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.18 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 437.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 955.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.64 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 622.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1157.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 483.99 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 671.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 783.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 956.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.69 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 779.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 878.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.08 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 475.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1169.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.38 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 622.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1255.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.82 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 574.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.91 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 578.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 448.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 679.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 933.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 319.62 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 672.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 455.11 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 593.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 911.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.36 examples/s]

Map (num_proc=8):   0%|          | 0/15 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 4/15 [00:00<00:00, 36.52 examples/s]
Map (num_proc=8):  80%|########  | 12/15 [00:00<00:00, 60.54 examples/s]
Map (num_proc=8): 100%|##########| 15/15 [00:00<00:00, 44.20 examples/s]

Filter (num_proc=8):   0%|          | 0/15 [00:00<?, ? examples/s]
Filter (num_proc=8):  67%|######6   | 10/15 [00:00<00:00, 88.59 examples/s]
Filter (num_proc=8): 100%|##########| 15/15 [00:00<00:00, 71.10 examples/s]INFO 2025-05-20 05:28:21,948 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 05:31:02,498 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 349.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 585.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.16 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 464.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 571.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1182.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.98 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 535.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1165.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.03 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 644.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1103.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.61 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 510.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 918.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.87 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 556.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 947.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 589.46 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 435.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1238.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.30 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 578.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1167.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.58 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 581.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1215.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.16 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 471.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1066.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.83 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 635.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 586.36 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 541.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 809.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1255.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 294.75 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 478.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1129.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.85 examples/s]

Map (num_proc=8):   0%|          | 0/47 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 12/47 [00:00<00:00, 94.84 examples/s]
Map (num_proc=8):  87%|########7 | 41/47 [00:00<00:00, 184.95 examples/s]
Map (num_proc=8): 100%|##########| 47/47 [00:00<00:00, 130.94 examples/s]

Filter (num_proc=8):   0%|          | 0/47 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 24/47 [00:00<00:00, 226.06 examples/s]
Filter (num_proc=8): 100%|##########| 47/47 [00:00<00:00, 220.91 examples/s]INFO 2025-05-20 05:31:19,424 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_7_1__Accessing_virtual_machine_consoles_in_the_OpenShift_Container_Platform_web_console/data_checkpoint_b22fcae5d2ec4ceba53f11cf62e1afeb.jsonl


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format:  67%|######6   | 2/3 [00:00<00:00, 19.45ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 20.11ba/s]INFO 2025-05-20 05:31:20,321 instructlab.sdg.generate_data:478: Generated 2995 samples
INFO 2025-05-20 05:31:20,381 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_15__Using_virtual_Trusted_Platform_Module_devices, generating from scratch
INFO 2025-05-20 05:31:20,381 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 05:31:20,384 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 05:31:20,575 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 05:31:20,588 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 05:31:20,596 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 05:31:24,902 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 05:31:26,709 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 05:31:29,647 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 05:31:29,666 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 05:31:29,681 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 05:32:24,091 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 05:33:25,300 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.12 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 643.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1193.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 463.56 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 656.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 889.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.60 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 633.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1182.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.53 examples/s]

Map (num_proc=8):   0%|          | 0/152 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 38/152 [00:00<00:00, 231.14 examples/s]
Map (num_proc=8):  88%|########7 | 133/152 [00:00<00:00, 557.81 examples/s]
Map (num_proc=8): 100%|##########| 152/152 [00:00<00:00, 401.70 examples/s]

Filter (num_proc=8):   0%|          | 0/152 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 76/152 [00:00<00:00, 724.68 examples/s]
Filter (num_proc=8): 100%|##########| 152/152 [00:00<00:00, 714.80 examples/s]INFO 2025-05-20 05:33:29,989 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 05:33:56,760 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 520.88 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 566.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1217.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.49 examples/s]

Map (num_proc=8):   0%|          | 0/179 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 46/179 [00:00<00:00, 329.80 examples/s]
Map (num_proc=8):  51%|#####     | 91/179 [00:00<00:00, 383.42 examples/s]
Map (num_proc=8): 100%|##########| 179/179 [00:00<00:00, 466.20 examples/s]

Filter (num_proc=8):   0%|          | 0/179 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 91/179 [00:00<00:00, 860.65 examples/s]
Filter (num_proc=8): 100%|##########| 179/179 [00:00<00:00, 831.46 examples/s]INFO 2025-05-20 05:33:59,100 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 05:34:19,436 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.42 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 713.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.69 examples/s]

Map (num_proc=8):   0%|          | 0/166 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 42/166 [00:00<00:00, 311.91 examples/s]
Map (num_proc=8):  75%|#######5  | 125/166 [00:00<00:00, 535.82 examples/s]
Map (num_proc=8): 100%|##########| 166/166 [00:00<00:00, 437.74 examples/s]

Filter (num_proc=8):   0%|          | 0/166 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 84/166 [00:00<00:00, 768.95 examples/s]
Filter (num_proc=8): 100%|##########| 166/166 [00:00<00:00, 766.62 examples/s]INFO 2025-05-20 05:34:21,772 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_15__Using_virtual_Trusted_Platform_Module_devices/data_checkpoint_d5b5b168fc804a9a883fa33415128ca5.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 53.72ba/s]INFO 2025-05-20 05:34:21,892 instructlab.sdg.generate_data:478: Generated 394 samples
INFO 2025-05-20 05:34:21,952 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_11_2_2__Cluster_wide_live_migration_limits_and_timeouts, generating from scratch
INFO 2025-05-20 05:34:21,952 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 05:34:21,956 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 05:34:22,168 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 05:34:22,181 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 05:34:22,188 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 05:34:27,368 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 05:34:35,490 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 05:34:40,425 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 05:34:40,444 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 05:34:40,459 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 05:35:37,777 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 05:36:51,132 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 477.29 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 625.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1271.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 425.56 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 551.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.70 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 639.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1276.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.75 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 872.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1130.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.55 examples/s]
num_proc must be <= 1. Reducing num_proc to 1 for dataset of size 1.

Map:   0%|          | 0/1 [00:00<?, ? examples/s]
Map: 100%|##########| 1/1 [00:00<00:00, 190.62 examples/s]
num_proc must be <= 1. Reducing num_proc to 1 for dataset of size 1.

Filter:   0%|          | 0/1 [00:00<?, ? examples/s]
Filter: 100%|##########| 1/1 [00:00<00:00, 466.97 examples/s]INFO 2025-05-20 05:36:55,874 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 05:37:39,154 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.05 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 807.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.47 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 628.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1150.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.04 examples/s]

Map (num_proc=8):   0%|          | 0/198 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 50/198 [00:00<00:00, 313.75 examples/s]
Map (num_proc=8):  88%|########7 | 174/198 [00:00<00:00, 630.13 examples/s]
Map (num_proc=8): 100%|##########| 198/198 [00:00<00:00, 498.83 examples/s]

Filter (num_proc=8):   0%|          | 0/198 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 125/198 [00:00<00:00, 1067.46 examples/s]
Filter (num_proc=8): 100%|##########| 198/198 [00:00<00:00, 913.20 examples/s] INFO 2025-05-20 05:37:42,685 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 05:38:15,742 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 533.94 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 561.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1454.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1192.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 240.61 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 576.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.73 examples/s]

Map (num_proc=8):   0%|          | 0/192 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 48/192 [00:00<00:00, 332.21 examples/s]
Map (num_proc=8):  62%|######2   | 120/192 [00:00<00:00, 468.29 examples/s]
Map (num_proc=8): 100%|##########| 192/192 [00:00<00:00, 488.12 examples/s]

Filter (num_proc=8):   0%|          | 0/192 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 96/192 [00:00<00:00, 884.77 examples/s]
Filter (num_proc=8): 100%|##########| 192/192 [00:00<00:00, 884.50 examples/s]INFO 2025-05-20 05:38:19,288 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_11_2_2__Cluster_wide_live_migration_limits_and_timeouts/data_checkpoint_ee6ab5d639de48d1badff9cb3a5fea40.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 46.38ba/s]INFO 2025-05-20 05:38:19,419 instructlab.sdg.generate_data:478: Generated 431 samples
INFO 2025-05-20 05:38:19,483 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_8_3__Additional_resources, generating from scratch
INFO 2025-05-20 05:38:19,483 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 05:38:19,487 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 05:38:19,688 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 05:38:19,702 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 05:38:19,709 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 05:38:24,791 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 05:38:30,364 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 05:38:34,222 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 05:38:34,240 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 05:38:34,256 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 05:39:33,276 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 05:40:48,689 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 649.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1463.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 258.76 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 526.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1277.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 233.95 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 597.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 259.29 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 569.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1431.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1193.89 examples/s]

Map (num_proc=8):   0%|          | 0/172 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 44/172 [00:00<00:00, 314.41 examples/s]
Map (num_proc=8): 100%|##########| 172/172 [00:00<00:00, 741.39 examples/s]
Map (num_proc=8): 100%|##########| 172/172 [00:00<00:00, 456.18 examples/s]

Filter (num_proc=8):   0%|          | 0/172 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 88/172 [00:00<00:00, 840.39 examples/s]
Filter (num_proc=8): 100%|##########| 172/172 [00:00<00:00, 809.62 examples/s]INFO 2025-05-20 05:40:54,514 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 05:41:44,626 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 842.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.45 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 803.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1266.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 451.45 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 659.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1191.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.15 examples/s]

Map (num_proc=8):   0%|          | 0/55 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###8      | 21/55 [00:00<00:00, 151.27 examples/s]
Map (num_proc=8): 100%|##########| 55/55 [00:00<00:00, 228.72 examples/s]
Map (num_proc=8): 100%|##########| 55/55 [00:00<00:00, 158.80 examples/s]

Filter (num_proc=8):   0%|          | 0/55 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 28/55 [00:00<00:00, 277.50 examples/s]
Filter (num_proc=8): 100%|##########| 55/55 [00:00<00:00, 263.84 examples/s]INFO 2025-05-20 05:41:49,229 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 05:42:23,596 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.50 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 674.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1256.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.58 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 637.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1265.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.06 examples/s]

Map (num_proc=8):   0%|          | 0/235 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 60/235 [00:00<00:00, 427.13 examples/s]
Map (num_proc=8):  63%|######2   | 148/235 [00:00<00:00, 586.92 examples/s]
Map (num_proc=8): 100%|##########| 235/235 [00:00<00:00, 582.76 examples/s]

Filter (num_proc=8):   0%|          | 0/235 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 119/235 [00:00<00:00, 1130.79 examples/s]
Filter (num_proc=8): 100%|##########| 235/235 [00:00<00:00, 1071.56 examples/s]INFO 2025-05-20 05:42:27,108 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_8_3__Additional_resources/data_checkpoint_b98b7c7f745c47ec8946a5bfe57b1031.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 33.56ba/s]INFO 2025-05-20 05:42:27,292 instructlab.sdg.generate_data:478: Generated 639 samples
INFO 2025-05-20 05:42:27,347 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_14_2__About_VirtIO_drivers, generating from scratch
INFO 2025-05-20 05:42:27,347 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 05:42:27,351 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 05:42:27,709 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 05:42:27,722 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 05:42:27,730 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 05:42:36,933 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 05:42:44,816 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 05:42:53,232 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 05:42:53,252 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 05:42:53,268 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 05:44:03,937 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 05:46:11,725 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 442.11 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 491.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1226.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1191.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.19 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 772.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1166.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 490.94 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 728.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1168.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 420.99 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 515.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1256.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.33 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 630.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1276.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 481.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 706.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.28 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 601.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1232.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.48 examples/s]

Map (num_proc=8):   0%|          | 0/97 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 25/97 [00:00<00:00, 204.43 examples/s]
Map (num_proc=8):  75%|#######5  | 73/97 [00:00<00:00, 328.70 examples/s]
Map (num_proc=8): 100%|##########| 97/97 [00:00<00:00, 268.33 examples/s]

Filter (num_proc=8):   0%|          | 0/97 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######2   | 61/97 [00:00<00:00, 557.39 examples/s]
Filter (num_proc=8): 100%|##########| 97/97 [00:00<00:00, 453.61 examples/s]INFO 2025-05-20 05:46:21,099 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 05:47:46,812 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 607.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 971.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.07 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 702.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1157.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.21 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 747.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 928.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 541.64 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 510.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 941.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 332.71 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 683.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1266.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.64 examples/s]

Map (num_proc=8):   0%|          | 0/101 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 26/101 [00:00<00:00, 193.69 examples/s]
Map (num_proc=8):  88%|########8 | 89/101 [00:00<00:00, 384.73 examples/s]
Map (num_proc=8): 100%|##########| 101/101 [00:00<00:00, 271.87 examples/s]

Filter (num_proc=8):   0%|          | 0/101 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 52/101 [00:00<00:00, 489.25 examples/s]
Filter (num_proc=8): 100%|##########| 101/101 [00:00<00:00, 476.18 examples/s]INFO 2025-05-20 05:47:53,901 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 05:48:56,375 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.14 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 752.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1144.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.14 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 683.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.83 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 746.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1312.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.87 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 728.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1169.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 336.32 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 869.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1359.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.83 examples/s]

Map (num_proc=8):   0%|          | 0/48 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 12/48 [00:00<00:00, 96.02 examples/s]
Map (num_proc=8): 100%|##########| 48/48 [00:00<00:00, 203.16 examples/s]
Map (num_proc=8): 100%|##########| 48/48 [00:00<00:00, 139.20 examples/s]

Filter (num_proc=8):   0%|          | 0/48 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 24/48 [00:00<00:00, 222.92 examples/s]
Filter (num_proc=8): 100%|##########| 48/48 [00:00<00:00, 234.71 examples/s]INFO 2025-05-20 05:49:03,416 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_14_2__About_VirtIO_drivers/data_checkpoint_46c2f23ad3424959bfd8cca91601e19a.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 33.58ba/s]INFO 2025-05-20 05:49:03,756 instructlab.sdg.generate_data:478: Generated 1148 samples
INFO 2025-05-20 05:49:03,852 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_5_4__Editing_a_standalone_virtual_machine_instance_using_the_web_console, generating from scratch
INFO 2025-05-20 05:49:03,852 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 05:49:03,855 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 05:49:04,178 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 05:49:04,192 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 05:49:04,199 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 05:49:09,838 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 05:49:17,441 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 05:49:21,919 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 05:49:21,939 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 05:49:21,954 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 05:50:43,087 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 05:53:52,701 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.56 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 500.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1357.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.34 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 808.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 880.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.55 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 595.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1168.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.00 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 609.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1215.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1183.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 648.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 881.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.50 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 768.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1248.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 457.08 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 546.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1195.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.21 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 717.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1201.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.81 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 697.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1186.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 494.60 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 639.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1156.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.75 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 793.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1099.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.71 examples/s]

Map (num_proc=8):   0%|          | 0/170 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 44/170 [00:00<00:00, 338.97 examples/s]
Map (num_proc=8):  75%|#######5  | 128/170 [00:00<00:00, 563.29 examples/s]
Map (num_proc=8): 100%|##########| 170/170 [00:00<00:00, 447.46 examples/s]

Filter (num_proc=8):   0%|          | 0/170 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 86/170 [00:00<00:00, 772.80 examples/s]
Filter (num_proc=8): 100%|##########| 170/170 [00:00<00:00, 782.33 examples/s]INFO 2025-05-20 05:54:06,849 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 05:55:51,387 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.68 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 494.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.87 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 771.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1219.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.00 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 666.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1250.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.54 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 801.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1166.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 462.07 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 495.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1165.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.38 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 770.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1155.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.11 examples/s]

Map (num_proc=8):   0%|          | 0/194 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 50/194 [00:00<00:00, 419.97 examples/s]
Map (num_proc=8):  75%|#######5  | 146/194 [00:00<00:00, 613.08 examples/s]
Map (num_proc=8): 100%|##########| 194/194 [00:00<00:00, 498.95 examples/s]

Filter (num_proc=8):   0%|          | 0/194 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 98/194 [00:00<00:00, 829.04 examples/s]
Filter (num_proc=8): 100%|##########| 194/194 [00:00<00:00, 899.78 examples/s]INFO 2025-05-20 05:55:59,723 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 05:57:20,045 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 329.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 538.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 479.00 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 683.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.85 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 591.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1056.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.81 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 620.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1227.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.27 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 455.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1167.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.33 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 755.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1244.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.29 examples/s]

Map (num_proc=8):   0%|          | 0/148 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 38/148 [00:00<00:00, 313.35 examples/s]
Map (num_proc=8):  76%|#######5  | 112/148 [00:00<00:00, 465.86 examples/s]
Map (num_proc=8): 100%|##########| 148/148 [00:00<00:00, 393.76 examples/s]

Filter (num_proc=8):   0%|          | 0/148 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 76/148 [00:00<00:00, 698.11 examples/s]
Filter (num_proc=8): 100%|##########| 148/148 [00:00<00:00, 695.06 examples/s]INFO 2025-05-20 05:57:28,387 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_5_4__Editing_a_standalone_virtual_machine_instance_using_the_web_console/data_checkpoint_17faf3fbd2674116b9fc9574f2a7f466.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 33.48ba/s]INFO 2025-05-20 05:57:28,784 instructlab.sdg.generate_data:478: Generated 1398 samples
INFO 2025-05-20 05:57:28,837 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_6_1__Starting_a_virtual_machine, generating from scratch
INFO 2025-05-20 05:57:28,837 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 05:57:28,841 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 05:57:29,103 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 05:57:29,117 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 05:57:29,125 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 05:57:34,215 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 05:57:42,118 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 05:57:47,106 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 05:57:47,126 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 05:57:47,141 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 05:58:31,982 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 05:58:57,195 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 575.27 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 594.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1214.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1205.31 examples/s]

Map (num_proc=8):   0%|          | 0/126 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 32/126 [00:00<00:00, 247.20 examples/s]
Map (num_proc=8):  51%|#####     | 64/126 [00:00<00:00, 104.47 examples/s]
Map (num_proc=8): 100%|##########| 126/126 [00:01<00:00, 116.74 examples/s]

Filter (num_proc=8):   0%|          | 0/126 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 64/126 [00:00<00:00, 613.10 examples/s]
Filter (num_proc=8): 100%|##########| 126/126 [00:00<00:00, 605.31 examples/s]INFO 2025-05-20 05:59:00,216 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 05:59:14,991 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/231 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 58/231 [00:00<00:00, 369.52 examples/s]
Map (num_proc=8):  63%|######2   | 145/231 [00:00<00:00, 567.53 examples/s]
Map (num_proc=8): 100%|##########| 231/231 [00:00<00:00, 567.72 examples/s]

Filter (num_proc=8):   0%|          | 0/231 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 116/231 [00:00<00:00, 1083.48 examples/s]
Filter (num_proc=8): 100%|##########| 231/231 [00:00<00:00, 1083.60 examples/s]INFO 2025-05-20 05:59:16,171 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 05:59:25,736 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/212 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 54/212 [00:00<00:00, 432.65 examples/s]
Map (num_proc=8):  51%|#####     | 108/212 [00:00<00:00, 440.83 examples/s]
Map (num_proc=8): 100%|##########| 212/212 [00:00<00:00, 532.31 examples/s]

Filter (num_proc=8):   0%|          | 0/212 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 134/212 [00:00<00:00, 1245.19 examples/s]
Filter (num_proc=8): 100%|##########| 212/212 [00:00<00:00, 996.65 examples/s] INFO 2025-05-20 05:59:26,907 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_6_1__Starting_a_virtual_machine/data_checkpoint_709ba3a0eed849a29af68fd77615f086.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 93.91ba/s]INFO 2025-05-20 05:59:26,974 instructlab.sdg.generate_data:478: Generated 192 samples
INFO 2025-05-20 05:59:27,036 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_13_5__Removing_the_VirtIO_container_disk_from_a_virtual_machine, generating from scratch
INFO 2025-05-20 05:59:27,036 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 05:59:27,040 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 05:59:27,352 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 05:59:27,365 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 05:59:27,373 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 05:59:34,506 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 05:59:42,755 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 05:59:49,686 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 05:59:49,706 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 05:59:49,721 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 06:00:58,448 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 06:03:44,415 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.23 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 777.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1182.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1203.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.72 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1465.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1191.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 451.94 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 601.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1472.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1199.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 473.61 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 570.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1189.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1204.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.66 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 824.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1254.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.92 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 682.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1226.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1201.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.08 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 771.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1236.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.63 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 721.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1148.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1206.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 449.92 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 815.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 630.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1187.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1195.45 examples/s]

Map (num_proc=8):   0%|          | 0/165 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 42/165 [00:00<00:00, 320.57 examples/s]
Map (num_proc=8):  63%|######3   | 104/165 [00:00<00:00, 469.35 examples/s]
Map (num_proc=8): 100%|##########| 165/165 [00:00<00:00, 439.64 examples/s]

Filter (num_proc=8):   0%|          | 0/165 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 84/165 [00:00<00:00, 807.95 examples/s]
Filter (num_proc=8): 100%|##########| 165/165 [00:00<00:00, 781.27 examples/s]INFO 2025-05-20 06:03:56,016 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 06:05:35,614 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 453.64 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 535.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1475.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1195.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.31 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 696.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1185.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.40 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 693.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 624.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1081.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 596.09 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 562.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1225.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1190.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 350.60 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 850.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1185.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1194.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.85 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 773.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1483.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1200.92 examples/s]

Map (num_proc=8):   0%|          | 0/74 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##7       | 20/74 [00:00<00:00, 162.03 examples/s]
Map (num_proc=8):  76%|#######5  | 56/74 [00:00<00:00, 227.89 examples/s]
Map (num_proc=8): 100%|##########| 74/74 [00:00<00:00, 209.44 examples/s]

Filter (num_proc=8):   0%|          | 0/74 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 38/74 [00:00<00:00, 379.21 examples/s]
Filter (num_proc=8): 100%|##########| 74/74 [00:00<00:00, 361.39 examples/s]INFO 2025-05-20 06:05:43,789 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 06:06:52,221 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.77 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 620.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1196.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 614.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1186.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1201.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.38 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 668.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1129.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1196.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.31 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 609.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1120.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.40 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 675.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1168.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.30 examples/s]

Map (num_proc=8):   0%|          | 0/236 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 60/236 [00:00<00:00, 376.51 examples/s]
Map (num_proc=8):  63%|######3   | 149/236 [00:00<00:00, 602.91 examples/s]
Map (num_proc=8): 100%|##########| 236/236 [00:00<00:00, 583.48 examples/s]

Filter (num_proc=8):   0%|          | 0/236 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 120/236 [00:00<00:00, 1119.97 examples/s]
Filter (num_proc=8): 100%|##########| 236/236 [00:00<00:00, 1092.49 examples/s]INFO 2025-05-20 06:06:59,240 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_13_5__Removing_the_VirtIO_container_disk_from_a_virtual_machine/data_checkpoint_4fb3f85b490246a0bec461632ad532a1.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 35.44ba/s]INFO 2025-05-20 06:06:59,561 instructlab.sdg.generate_data:478: Generated 1092 samples
INFO 2025-05-20 06:06:59,654 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_13_4__Installing_VirtIO_drivers_on_an_existing_Windows_virtual_machine, generating from scratch
INFO 2025-05-20 06:06:59,654 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 06:06:59,658 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 06:07:00,007 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 06:07:00,021 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 06:07:00,028 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 06:07:07,193 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 06:07:15,958 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 06:07:22,867 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 06:07:22,887 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 06:07:22,902 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 06:08:30,752 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 06:11:02,617 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 451.74 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 604.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 908.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 490.63 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 550.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1192.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1197.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 455.99 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 696.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 640.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1241.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1196.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.35 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 730.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1355.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1193.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 452.16 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 521.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 637.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1211.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 441.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 818.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1439.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1195.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.50 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 607.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1187.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1182.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.07 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 805.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1225.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1199.03 examples/s]

Map (num_proc=8):   0%|          | 0/210 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 54/210 [00:00<00:00, 387.28 examples/s]
Map (num_proc=8):  63%|######2   | 132/210 [00:00<00:00, 538.91 examples/s]
Map (num_proc=8): 100%|##########| 210/210 [00:00<00:00, 541.81 examples/s]

Filter (num_proc=8):   0%|          | 0/210 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######2   | 132/210 [00:00<00:00, 1127.30 examples/s]
Filter (num_proc=8): 100%|##########| 210/210 [00:00<00:00, 997.09 examples/s] INFO 2025-05-20 06:11:13,064 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 06:12:56,462 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 341.64 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 789.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1167.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 590.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.18 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 616.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.45 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 808.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1172.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 458.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 638.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1344.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 442.24 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 586.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1258.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.35 examples/s]

Map (num_proc=8):   0%|          | 0/180 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 46/180 [00:00<00:00, 349.13 examples/s]
Map (num_proc=8):  63%|######3   | 114/180 [00:00<00:00, 473.88 examples/s]
Map (num_proc=8): 100%|##########| 180/180 [00:00<00:00, 470.63 examples/s]

Filter (num_proc=8):   0%|          | 0/180 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 91/180 [00:00<00:00, 881.77 examples/s]
Filter (num_proc=8): 100%|##########| 180/180 [00:00<00:00, 853.61 examples/s]INFO 2025-05-20 06:13:04,675 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 06:14:19,606 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.59 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 596.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 850.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1099.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.50 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 630.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1421.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.59 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 452.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1185.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1191.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.87 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 660.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1261.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.08 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 849.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1186.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 548.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1206.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1184.34 examples/s]

Map (num_proc=8):   0%|          | 0/78 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 20/78 [00:00<00:00, 167.89 examples/s]
Map (num_proc=8):  77%|#######6  | 60/78 [00:00<00:00, 282.13 examples/s]
Map (num_proc=8): 100%|##########| 78/78 [00:00<00:00, 220.73 examples/s]

Filter (num_proc=8):   0%|          | 0/78 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 40/78 [00:00<00:00, 382.84 examples/s]
Filter (num_proc=8): 100%|##########| 78/78 [00:00<00:00, 368.13 examples/s]INFO 2025-05-20 06:14:27,786 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_13_4__Installing_VirtIO_drivers_on_an_existing_Windows_virtual_machine/data_checkpoint_f986a7e16abb4f5ab189bcead86404fe.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 33.32ba/s]INFO 2025-05-20 06:14:28,151 instructlab.sdg.generate_data:478: Generated 1246 samples
INFO 2025-05-20 06:14:28,216 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_2_1__How_OpenShift_Virtualization_architecture_works, generating from scratch
INFO 2025-05-20 06:14:28,216 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 06:14:28,219 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 06:14:28,415 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 06:14:28,428 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 06:14:28,436 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 06:14:34,122 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 06:14:39,765 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 06:14:45,861 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 06:14:45,879 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 06:14:45,894 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 06:15:31,864 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 06:15:58,193 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 614.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1085.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.90 examples/s]

Map (num_proc=8):   0%|          | 0/83 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 22/83 [00:00<00:00, 185.95 examples/s]
Map (num_proc=8):  76%|#######5  | 63/83 [00:00<00:00, 272.69 examples/s]
Map (num_proc=8): 100%|##########| 83/83 [00:00<00:00, 229.01 examples/s]

Filter (num_proc=8):   0%|          | 0/83 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 43/83 [00:00<00:00, 413.36 examples/s]
Filter (num_proc=8): 100%|##########| 83/83 [00:00<00:00, 392.04 examples/s]INFO 2025-05-20 06:16:00,502 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 06:16:22,467 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 462.45 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 491.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 788.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.35 examples/s]

Map (num_proc=8):   0%|          | 0/71 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###8      | 27/71 [00:00<00:00, 167.04 examples/s]
Map (num_proc=8): 100%|##########| 71/71 [00:00<00:00, 197.25 examples/s]

Filter (num_proc=8):   0%|          | 0/71 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 45/71 [00:00<00:00, 404.20 examples/s]
Filter (num_proc=8): 100%|##########| 71/71 [00:00<00:00, 344.28 examples/s]INFO 2025-05-20 06:16:24,787 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 06:16:41,316 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.92 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 472.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1134.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.96 examples/s]

Map (num_proc=8):   0%|          | 0/46 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 12/46 [00:00<00:00, 107.21 examples/s]
Map (num_proc=8):  78%|#######8  | 36/46 [00:00<00:00, 175.39 examples/s]
Map (num_proc=8): 100%|##########| 46/46 [00:00<00:00, 132.24 examples/s]

Filter (num_proc=8):   0%|          | 0/46 [00:00<?, ? examples/s]
Filter (num_proc=8):  65%|######5   | 30/46 [00:00<00:00, 268.40 examples/s]
Filter (num_proc=8): 100%|##########| 46/46 [00:00<00:00, 223.24 examples/s]INFO 2025-05-20 06:16:43,610 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_2_1__How_OpenShift_Virtualization_architecture_works/data_checkpoint_6c0aca484b72409d864c780fab455e32.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 52.09ba/s]INFO 2025-05-20 06:16:43,711 instructlab.sdg.generate_data:478: Generated 288 samples
INFO 2025-05-20 06:16:43,774 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_7_1__About_the_OpenShift_Container_Platform_dashboards_page, generating from scratch
INFO 2025-05-20 06:16:43,774 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 06:16:43,777 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 06:16:43,930 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 06:16:43,943 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 06:16:43,951 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 06:16:49,063 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 06:16:58,235 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 06:17:02,518 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 06:17:02,537 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 06:17:02,553 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 06:17:42,898 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 06:17:57,447 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/184 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 46/184 [00:00<00:00, 310.07 examples/s]
Map (num_proc=8):  75%|#######5  | 138/184 [00:00<00:00, 590.42 examples/s]
Map (num_proc=8): 100%|##########| 184/184 [00:00<00:00, 472.75 examples/s]

Filter (num_proc=8):   0%|          | 0/184 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 92/184 [00:00<00:00, 783.39 examples/s]
Filter (num_proc=8): 100%|##########| 184/184 [00:00<00:00, 833.86 examples/s]INFO 2025-05-20 06:17:58,608 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 06:18:09,584 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/164 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 42/164 [00:00<00:00, 301.56 examples/s]
Map (num_proc=8):  76%|#######5  | 124/164 [00:00<00:00, 547.98 examples/s]
Map (num_proc=8): 100%|##########| 164/164 [00:00<00:00, 427.95 examples/s]

Filter (num_proc=8):   0%|          | 0/164 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 84/164 [00:00<00:00, 796.24 examples/s]
Filter (num_proc=8): 100%|##########| 164/164 [00:00<00:00, 775.09 examples/s]INFO 2025-05-20 06:18:10,727 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 06:18:19,977 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/163 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 42/163 [00:00<00:00, 302.78 examples/s]
Map (num_proc=8):  88%|########7 | 143/163 [00:00<00:00, 544.55 examples/s]
Map (num_proc=8): 100%|##########| 163/163 [00:00<00:00, 427.98 examples/s]

Filter (num_proc=8):   0%|          | 0/163 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 103/163 [00:00<00:00, 888.48 examples/s]
Filter (num_proc=8): 100%|##########| 163/163 [00:00<00:00, 761.57 examples/s]INFO 2025-05-20 06:18:21,121 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_7_1__About_the_OpenShift_Container_Platform_dashboards_page/data_checkpoint_71018e4d198b49799b91337611226a1e.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 95.64ba/s]INFO 2025-05-20 06:18:21,179 instructlab.sdg.generate_data:478: Generated 156 samples
INFO 2025-05-20 06:18:21,250 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_10_4__Deploying_a_virtual_machine_template_to_a_custom_namespace, generating from scratch
INFO 2025-05-20 06:18:21,250 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 06:18:21,254 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 06:18:21,571 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 06:18:21,585 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 06:18:21,592 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 06:18:28,518 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 06:18:36,068 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 06:18:42,663 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 06:18:42,683 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 06:18:42,699 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 06:19:27,612 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 06:19:47,832 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.02 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 603.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1219.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1199.21 examples/s]

Map (num_proc=8):   0%|          | 0/33 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##7       | 9/33 [00:00<00:00, 80.14 examples/s]
Map (num_proc=8):  88%|########7 | 29/33 [00:00<00:00, 140.28 examples/s]
Map (num_proc=8): 100%|##########| 33/33 [00:00<00:00, 99.84 examples/s] 

Filter (num_proc=8):   0%|          | 0/33 [00:00<?, ? examples/s]
Filter (num_proc=8):  64%|######3   | 21/33 [00:00<00:00, 206.10 examples/s]
Filter (num_proc=8): 100%|##########| 33/33 [00:00<00:00, 167.74 examples/s]INFO 2025-05-20 06:19:50,061 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 06:20:03,127 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/204 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 52/204 [00:00<00:00, 345.01 examples/s]
Map (num_proc=8):  88%|########7 | 179/204 [00:00<00:00, 711.93 examples/s]
Map (num_proc=8): 100%|##########| 204/204 [00:00<00:00, 527.23 examples/s]

Filter (num_proc=8):   0%|          | 0/204 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 129/204 [00:00<00:00, 1175.47 examples/s]
Filter (num_proc=8): 100%|##########| 204/204 [00:00<00:00, 962.23 examples/s] INFO 2025-05-20 06:20:04,271 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 06:20:13,682 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/190 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 48/190 [00:00<00:00, 407.49 examples/s]
Map (num_proc=8):  63%|######3   | 120/190 [00:00<00:00, 530.05 examples/s]
Map (num_proc=8): 100%|##########| 190/190 [00:00<00:00, 495.85 examples/s]

Filter (num_proc=8):   0%|          | 0/190 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 96/190 [00:00<00:00, 948.01 examples/s]
Filter (num_proc=8): 100%|##########| 190/190 [00:00<00:00, 867.61 examples/s]INFO 2025-05-20 06:20:14,828 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_10_4__Deploying_a_virtual_machine_template_to_a_custom_namespace/data_checkpoint_15aa5e36f845401cb42f5f16a42b9eaf.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 100.83ba/s]INFO 2025-05-20 06:20:14,884 instructlab.sdg.generate_data:478: Generated 160 samples
INFO 2025-05-20 06:20:14,953 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_13_4__Additional_resources, generating from scratch
INFO 2025-05-20 06:20:14,954 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 06:20:14,957 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 06:20:16,577 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 06:20:16,592 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 06:20:16,601 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 06:20:29,814 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 06:20:47,654 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 06:20:59,709 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 06:20:59,736 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 06:20:59,755 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 06:23:58,077 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 06:30:04,377 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 425.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 611.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.30 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 622.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1159.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.89 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 576.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 906.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.27 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 615.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.15 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1176.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.76 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 593.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1166.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.37 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 600.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 614.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1047.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 441.36 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 539.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1137.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 354.85 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 734.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1280.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1182.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.26 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 667.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 872.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.05 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 753.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1172.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.15 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 656.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 938.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.35 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 633.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 594.14 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 509.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1264.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.08 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 592.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1159.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 452.48 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 691.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 638.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1231.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1190.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.25 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 615.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1169.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 453.12 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 537.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1265.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.99 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.87 examples/s]

Map (num_proc=8):   0%|          | 0/229 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 58/229 [00:00<00:00, 337.15 examples/s]
Map (num_proc=8):  88%|########7 | 201/229 [00:00<00:00, 780.54 examples/s]
Map (num_proc=8): 100%|##########| 229/229 [00:00<00:00, 564.28 examples/s]

Filter (num_proc=8):   0%|          | 0/229 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 116/229 [00:00<00:00, 1082.97 examples/s]
Filter (num_proc=8): 100%|##########| 229/229 [00:00<00:00, 1055.91 examples/s]INFO 2025-05-20 06:30:29,443 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 06:34:25,948 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.59 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 748.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1239.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 621.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 639.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.89 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 594.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1043.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 575.74 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 572.51 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 679.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1265.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 447.01 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 489.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1224.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.26 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 519.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1094.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.72 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 521.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 883.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 454.44 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 505.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.85 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 742.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 896.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.62 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 758.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1272.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 345.97 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 515.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1135.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.64 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 568.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1145.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.44 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 664.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1279.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 261.75 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 543.95 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 653.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 533.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1265.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 443.77 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 521.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1090.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.18 examples/s]

Map (num_proc=8):   0%|          | 0/66 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##7       | 18/66 [00:00<00:00, 143.28 examples/s]
Map (num_proc=8):  76%|#######5  | 50/66 [00:00<00:00, 214.64 examples/s]
Map (num_proc=8): 100%|##########| 66/66 [00:00<00:00, 182.55 examples/s]

Filter (num_proc=8):   0%|          | 0/66 [00:00<?, ? examples/s]
Filter (num_proc=8):  64%|######3   | 42/66 [00:00<00:00, 391.03 examples/s]
Filter (num_proc=8): 100%|##########| 66/66 [00:00<00:00, 311.64 examples/s]INFO 2025-05-20 06:34:45,335 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 06:37:36,360 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 330.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 688.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 351.23 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 472.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1141.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.28 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 571.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.01 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 492.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1156.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 527.37 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 509.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.71 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 529.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 939.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.26 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 506.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1181.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 354.18 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 870.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1186.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.17 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 489.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1257.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 577.80 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 452.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 489.07 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 460.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1252.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.02 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 597.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1148.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 323.65 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 695.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.00 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 909.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 582.68 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 496.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 870.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.78 examples/s]

Map (num_proc=8):   0%|          | 0/145 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 37/145 [00:00<00:00, 342.69 examples/s]
Map (num_proc=8):  63%|######2   | 91/145 [00:00<00:00, 387.31 examples/s]
Map (num_proc=8): 100%|##########| 145/145 [00:00<00:00, 373.15 examples/s]

Filter (num_proc=8):   0%|          | 0/145 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 73/145 [00:00<00:00, 695.53 examples/s]
Filter (num_proc=8): 100%|##########| 145/145 [00:00<00:00, 669.93 examples/s]INFO 2025-05-20 06:37:54,602 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_13_4__Additional_resources/data_checkpoint_85a6d80b258e42a9b303fa5a31b6e06b.jsonl


Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format:  50%|#####     | 2/4 [00:00<00:00, 18.62ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 24.62ba/s]INFO 2025-05-20 06:37:55,525 instructlab.sdg.generate_data:478: Generated 3081 samples
INFO 2025-05-20 06:37:55,596 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_1_4__Reviewing_top_consumers_for_projects__virtual_machines__and_nodes, generating from scratch
INFO 2025-05-20 06:37:55,596 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 06:37:55,600 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 06:37:55,893 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 06:37:55,907 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 06:37:55,915 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 06:38:01,285 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 06:38:06,565 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 06:38:12,236 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 06:38:12,257 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 06:38:12,272 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 06:39:26,816 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 06:41:39,049 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.24 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 779.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 919.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 454.54 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 539.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1139.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.88 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 643.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1195.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.94 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 619.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1195.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 491.91 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 567.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1240.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.65 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 670.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 904.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.67 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 750.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 931.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 582.93 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 564.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1186.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.50 examples/s]

Map (num_proc=8):   0%|          | 0/9 [00:00<?, ? examples/s]
Map (num_proc=8):  33%|###3      | 3/9 [00:00<00:00, 28.67 examples/s]
Map (num_proc=8):  89%|########8 | 8/9 [00:00<00:00, 36.18 examples/s]
Map (num_proc=8): 100%|##########| 9/9 [00:00<00:00, 26.88 examples/s]

Filter (num_proc=8):   0%|          | 0/9 [00:00<?, ? examples/s]
Filter (num_proc=8):  56%|#####5    | 5/9 [00:00<00:00, 49.18 examples/s]
Filter (num_proc=8): 100%|##########| 9/9 [00:00<00:00, 45.24 examples/s]INFO 2025-05-20 06:41:49,614 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 06:43:19,238 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.53 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 651.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1266.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 442.51 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 600.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1154.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 349.77 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 728.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1229.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.88 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 702.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 878.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 449.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 583.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1128.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.72 examples/s]

Map (num_proc=8):   0%|          | 0/176 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 44/176 [00:00<00:00, 321.30 examples/s]
Map (num_proc=8):  88%|########7 | 154/176 [00:00<00:00, 641.25 examples/s]
Map (num_proc=8): 100%|##########| 176/176 [00:00<00:00, 457.39 examples/s]

Filter (num_proc=8):   0%|          | 0/176 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 88/176 [00:00<00:00, 813.20 examples/s]
Filter (num_proc=8): 100%|##########| 176/176 [00:00<00:00, 812.86 examples/s]INFO 2025-05-20 06:43:26,348 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 06:44:31,649 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 349.73 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 879.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 859.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1187.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 335.55 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 863.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1200.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 348.68 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 664.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 893.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 572.82 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 649.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1238.10 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.84 examples/s]

Map (num_proc=8):   0%|          | 0/116 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 30/116 [00:00<00:00, 239.47 examples/s]
Map (num_proc=8):  76%|#######5  | 88/116 [00:00<00:00, 407.87 examples/s]
Map (num_proc=8): 100%|##########| 116/116 [00:00<00:00, 315.18 examples/s]

Filter (num_proc=8):   0%|          | 0/116 [00:00<?, ? examples/s]
Filter (num_proc=8):  39%|###8      | 45/116 [00:00<00:00, 422.27 examples/s]
Filter (num_proc=8): 100%|##########| 116/116 [00:00<00:00, 539.61 examples/s]INFO 2025-05-20 06:44:38,759 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_1_4__Reviewing_top_consumers_for_projects__virtual_machines__and_nodes/data_checkpoint_bbc8697294c54cfc95015cdebf05c090.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 39.51ba/s]INFO 2025-05-20 06:44:39,066 instructlab.sdg.generate_data:478: Generated 1044 samples
INFO 2025-05-20 06:44:39,132 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_7__Accessing_virtual_machine_consoles, generating from scratch
INFO 2025-05-20 06:44:39,132 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 06:44:39,136 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 06:44:39,921 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 06:44:39,935 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 06:44:39,943 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 06:44:50,128 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 06:45:05,959 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 06:45:13,697 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 06:45:13,720 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 06:45:13,738 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 06:47:47,248 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 06:53:37,733 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 452.58 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 542.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1215.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1191.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 626.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1234.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.55 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 695.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1215.83 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.58 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 609.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1433.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 444.03 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 632.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1253.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.03 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 639.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.28 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 582.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 896.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.75 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 985.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1206.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1198.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.59 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 694.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 893.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.81 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 720.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 934.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 446.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 702.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 638.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 894.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 446.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 679.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.25 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 825.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 905.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.75 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 904.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 901.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 318.29 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 714.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.37 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1247.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 521.17 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 630.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1178.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 397.88 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 747.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 639.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 855.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1159.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1185.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.07 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 728.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1193.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.54 examples/s]

Map (num_proc=8):   0%|          | 0/251 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/251 [00:00<00:00, 476.34 examples/s]
Map (num_proc=8):  63%|######2   | 158/251 [00:00<00:00, 660.21 examples/s]
Map (num_proc=8): 100%|##########| 251/251 [00:00<00:00, 622.45 examples/s]

Filter (num_proc=8):   0%|          | 0/251 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 127/251 [00:00<00:00, 1120.96 examples/s]
Filter (num_proc=8): 100%|##########| 251/251 [00:00<00:00, 1135.47 examples/s]INFO 2025-05-20 06:54:01,467 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 06:57:39,231 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.98 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 618.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1352.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.87 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 633.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.77 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 470.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1086.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.27 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 663.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 688.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1263.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.50 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 700.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1253.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.74 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 519.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1151.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.70 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 614.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1140.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 350.24 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 679.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 932.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.05 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 754.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1197.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 354.63 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 765.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.59 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 730.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1113.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.93 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 587.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.04 examples/s]

Map (num_proc=8):   0%|          | 0/226 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 58/226 [00:00<00:00, 506.65 examples/s]
Map (num_proc=8):  50%|#####     | 114/226 [00:00<00:00, 495.39 examples/s]
Map (num_proc=8): 100%|##########| 226/226 [00:00<00:00, 553.18 examples/s]

Filter (num_proc=8):   0%|          | 0/226 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 114/226 [00:00<00:00, 1050.27 examples/s]
Filter (num_proc=8): 100%|##########| 226/226 [00:00<00:00, 1015.82 examples/s]INFO 2025-05-20 06:57:55,951 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 07:00:32,165 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.21 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 771.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1211.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.41 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 663.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 918.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.21 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 598.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1200.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.74 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 852.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.64 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 694.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.79 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 699.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.15 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 916.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 426.30 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 744.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1194.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 613.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1218.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.02 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 684.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 895.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 487.80 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 577.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1224.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 762.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1397.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 442.97 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 776.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1206.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.95 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 598.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1271.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.15 examples/s]

Map (num_proc=8):   0%|          | 0/13 [00:00<?, ? examples/s]
Map (num_proc=8):  46%|####6     | 6/13 [00:00<00:00, 45.42 examples/s]
Map (num_proc=8):  92%|#########2| 12/13 [00:00<00:00, 49.45 examples/s]
Map (num_proc=8): 100%|##########| 13/13 [00:00<00:00, 38.82 examples/s]

Filter (num_proc=8):   0%|          | 0/13 [00:00<?, ? examples/s]
Filter (num_proc=8):  77%|#######6  | 10/13 [00:00<00:00, 91.36 examples/s]
Filter (num_proc=8): 100%|##########| 13/13 [00:00<00:00, 64.76 examples/s]INFO 2025-05-20 07:00:48,805 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_7__Accessing_virtual_machine_consoles/data_checkpoint_8b28f1406b694573b8235fb1402dc00b.jsonl


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 26.01ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 25.97ba/s]INFO 2025-05-20 07:00:49,595 instructlab.sdg.generate_data:478: Generated 2879 samples
INFO 2025-05-20 07:00:49,654 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_2_5__About_the_hostpath_provisioner_operator, generating from scratch
INFO 2025-05-20 07:00:49,655 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 07:00:49,658 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 07:00:49,795 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 07:00:49,808 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 07:00:49,816 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 07:00:55,611 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 07:01:00,891 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 07:01:04,995 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 07:01:05,014 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 07:01:05,029 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 07:01:50,475 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 07:02:17,216 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.38 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 770.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1152.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.55 examples/s]

Map (num_proc=8):   0%|          | 0/11 [00:00<?, ? examples/s]
Map (num_proc=8):  36%|###6      | 4/11 [00:00<00:00, 36.65 examples/s]
Map (num_proc=8):  91%|######### | 10/11 [00:00<00:00, 48.14 examples/s]
Map (num_proc=8): 100%|##########| 11/11 [00:00<00:00, 33.53 examples/s]

Filter (num_proc=8):   0%|          | 0/11 [00:00<?, ? examples/s]
Filter (num_proc=8):  55%|#####4    | 6/11 [00:00<00:00, 59.57 examples/s]
Filter (num_proc=8): 100%|##########| 11/11 [00:00<00:00, 53.86 examples/s]INFO 2025-05-20 07:02:19,486 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 07:02:35,228 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/242 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 62/242 [00:00<00:00, 466.48 examples/s]
Map (num_proc=8):  50%|#####     | 122/242 [00:00<00:00, 514.00 examples/s]
Map (num_proc=8): 100%|##########| 242/242 [00:00<00:00, 593.95 examples/s]

Filter (num_proc=8):   0%|          | 0/242 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 122/242 [00:00<00:00, 1093.22 examples/s]
Filter (num_proc=8): 100%|##########| 242/242 [00:00<00:00, 1113.78 examples/s]INFO 2025-05-20 07:02:36,412 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 07:02:47,482 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/231 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 29/231 [00:00<00:00, 280.39 examples/s]
Map (num_proc=8):  50%|#####     | 116/231 [00:00<00:00, 550.03 examples/s]
Map (num_proc=8): 100%|##########| 231/231 [00:00<00:00, 572.42 examples/s]

Filter (num_proc=8):   0%|          | 0/231 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 116/231 [00:00<00:00, 1004.56 examples/s]
Filter (num_proc=8): 100%|##########| 231/231 [00:00<00:00, 1061.66 examples/s]INFO 2025-05-20 07:02:48,658 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_2_5__About_the_hostpath_provisioner_operator/data_checkpoint_5723aa8ca718461c995f4bd9118c95e6.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 97.90ba/s]INFO 2025-05-20 07:02:48,724 instructlab.sdg.generate_data:478: Generated 193 samples
INFO 2025-05-20 07:02:48,793 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_1_4__Using_the_CLI_to_create_a_virtual_machine, generating from scratch
INFO 2025-05-20 07:02:48,793 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 07:02:48,796 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 07:02:49,739 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 07:02:49,753 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 07:02:49,761 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 07:03:05,545 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 07:03:28,588 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 07:03:45,091 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 07:03:45,114 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 07:03:45,132 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 07:06:39,572 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 07:13:48,461 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.96 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 617.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1266.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.00 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 881.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1115.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.43 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 738.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1259.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 512.18 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 467.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1242.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 475.01 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 642.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1161.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.29 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 528.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1155.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 338.62 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 824.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 883.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.18 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 704.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1040.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.36 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 609.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1129.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.48 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 543.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1176.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 478.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 863.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.91 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 675.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1260.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 450.75 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 600.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1140.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.87 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 651.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1191.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 529.08 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 529.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1088.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 410.64 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 692.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 918.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 424.79 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 508.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.16 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 505.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1277.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.64 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 789.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 883.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.35 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 648.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1105.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.88 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 678.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1201.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 394.29 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 757.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1306.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.65 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 484.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1133.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.99 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 493.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.11 examples/s]

Map (num_proc=8):   0%|          | 0/64 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 16/64 [00:00<00:00, 138.11 examples/s]
Map (num_proc=8):  88%|########7 | 56/64 [00:00<00:00, 243.97 examples/s]
Map (num_proc=8): 100%|##########| 64/64 [00:00<00:00, 183.37 examples/s]

Filter (num_proc=8):   0%|          | 0/64 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 32/64 [00:00<00:00, 307.74 examples/s]
Filter (num_proc=8): 100%|##########| 64/64 [00:00<00:00, 300.43 examples/s]INFO 2025-05-20 07:14:18,407 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 07:18:20,679 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.30 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 593.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 923.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.48 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 769.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1087.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.69 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 588.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1407.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.46 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.24 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 538.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 956.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 352.74 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 758.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1217.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.96 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 513.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 341.63 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 776.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.51 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 598.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 455.54 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 591.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.37 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 618.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1191.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.65 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 713.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.09 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 588.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1210.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 851.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1200.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 495.10 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 597.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1193.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.30 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 781.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1209.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.00 examples/s]

Map (num_proc=8):   0%|          | 0/157 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 40/157 [00:00<00:00, 309.14 examples/s]
Map (num_proc=8):  76%|#######5  | 119/157 [00:00<00:00, 517.67 examples/s]
Map (num_proc=8): 100%|##########| 157/157 [00:00<00:00, 405.09 examples/s]

Filter (num_proc=8):   0%|          | 0/157 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 80/157 [00:00<00:00, 749.05 examples/s]
Filter (num_proc=8): 100%|##########| 157/157 [00:00<00:00, 722.70 examples/s]INFO 2025-05-20 07:18:40,164 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 07:21:30,768 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 344.11 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 673.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 894.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.02 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 634.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.15 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 825.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1007.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1148.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.03 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 589.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 574.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1154.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 342.64 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 577.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1374.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.72 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 597.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1101.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 287.15 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 475.27 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 819.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1079.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.26 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 645.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 948.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 527.43 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 564.85 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 712.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1161.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 339.37 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 827.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1213.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.29 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 723.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1084.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.12 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 490.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1142.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 442.89 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 535.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1148.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.03 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 573.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1153.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.16 examples/s]

Map (num_proc=8):   0%|          | 0/122 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 32/122 [00:00<00:00, 245.13 examples/s]
Map (num_proc=8):  63%|######3   | 77/122 [00:00<00:00, 312.96 examples/s]
Map (num_proc=8): 100%|##########| 122/122 [00:00<00:00, 315.50 examples/s]

Filter (num_proc=8):   0%|          | 0/122 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 62/122 [00:00<00:00, 613.81 examples/s]
Filter (num_proc=8): 100%|##########| 122/122 [00:00<00:00, 562.35 examples/s]INFO 2025-05-20 07:21:49,135 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_1_4__Using_the_CLI_to_create_a_virtual_machine/data_checkpoint_010ef33641d740ea9e3749f28e02d7f6.jsonl


Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format:  75%|#######5  | 3/4 [00:00<00:00, 21.63ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 27.05ba/s]INFO 2025-05-20 07:21:50,048 instructlab.sdg.generate_data:478: Generated 3191 samples
INFO 2025-05-20 07:21:50,114 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_14_2__Collecting_data_about_virtual_machines, generating from scratch
INFO 2025-05-20 07:21:50,114 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 07:21:50,117 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 07:21:50,639 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 07:21:50,653 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 07:21:50,661 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 07:22:00,143 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 07:22:10,286 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 07:22:18,085 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 07:22:18,106 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 07:22:18,121 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 07:23:16,846 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 07:24:19,169 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.47 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 671.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1177.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.37 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 680.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1120.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.18 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 724.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.47 examples/s]

Map (num_proc=8):   0%|          | 0/105 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 27/105 [00:00<00:00, 198.91 examples/s]
Map (num_proc=8): 100%|##########| 105/105 [00:00<00:00, 433.58 examples/s]
Map (num_proc=8): 100%|##########| 105/105 [00:00<00:00, 287.08 examples/s]

Filter (num_proc=8):   0%|          | 0/105 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 53/105 [00:00<00:00, 527.03 examples/s]
Filter (num_proc=8): 100%|##########| 105/105 [00:00<00:00, 495.37 examples/s]INFO 2025-05-20 07:24:23,895 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 07:25:01,667 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 566.47 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 563.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 823.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.45 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1158.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.93 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 595.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.89 examples/s]

Map (num_proc=8):   0%|          | 0/75 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 20/75 [00:00<00:00, 176.50 examples/s]
Map (num_proc=8):  76%|#######6  | 57/75 [00:00<00:00, 249.81 examples/s]
Map (num_proc=8): 100%|##########| 75/75 [00:00<00:00, 211.34 examples/s]

Filter (num_proc=8):   0%|          | 0/75 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####2    | 39/75 [00:00<00:00, 362.72 examples/s]
Filter (num_proc=8): 100%|##########| 75/75 [00:00<00:00, 355.23 examples/s]INFO 2025-05-20 07:25:05,226 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 07:25:30,536 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.75 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 523.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1132.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.38 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 570.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 602.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1110.43 examples/s]
num_proc must be <= 1. Reducing num_proc to 1 for dataset of size 1.

Map:   0%|          | 0/1 [00:00<?, ? examples/s]
Map: 100%|##########| 1/1 [00:00<00:00, 186.44 examples/s]
num_proc must be <= 1. Reducing num_proc to 1 for dataset of size 1.

Filter:   0%|          | 0/1 [00:00<?, ? examples/s]
Filter: 100%|##########| 1/1 [00:00<00:00, 564.43 examples/s]INFO 2025-05-20 07:25:32,970 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_14_2__Collecting_data_about_virtual_machines/data_checkpoint_75f78ffc821f47ecaf662cf45383ed41.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 42.92ba/s]INFO 2025-05-20 07:25:33,108 instructlab.sdg.generate_data:478: Generated 427 samples
INFO 2025-05-20 07:25:33,200 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_14_6__Removing_the_VirtIO_container_disk_from_a_virtual_machine, generating from scratch
INFO 2025-05-20 07:25:33,200 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 07:25:33,204 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 07:25:33,533 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 07:25:33,546 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 07:25:33,554 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 07:25:42,764 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 07:25:51,186 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 07:25:59,627 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 07:25:59,648 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 07:25:59,663 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 07:27:16,143 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 07:30:28,126 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.80 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 737.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 913.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.81 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 593.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 620.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1227.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.73 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 504.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1087.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.30 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 604.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1138.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 485.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 675.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 640.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1200.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 440.22 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 672.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1213.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 352.23 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 900.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1183.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 458.71 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 578.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1266.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 541.56 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 543.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.29 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 748.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1252.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.48 examples/s]

Map (num_proc=8):   0%|          | 0/54 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 14/54 [00:00<00:00, 110.56 examples/s]
Map (num_proc=8):  76%|#######5  | 41/54 [00:00<00:00, 194.56 examples/s]
Map (num_proc=8): 100%|##########| 54/54 [00:00<00:00, 154.09 examples/s]

Filter (num_proc=8):   0%|          | 0/54 [00:00<?, ? examples/s]
Filter (num_proc=8):  65%|######4   | 35/54 [00:00<00:00, 306.95 examples/s]
Filter (num_proc=8): 100%|##########| 54/54 [00:00<00:00, 258.03 examples/s]INFO 2025-05-20 07:30:42,324 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 07:32:25,383 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.56 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 731.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1142.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 349.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 526.75 examples/s]
Map (num_proc=8):  99%|#########9| 254/256 [00:00<00:00, 365.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 335.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1277.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.38 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 483.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1199.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 908.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1201.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.31 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 177.98 examples/s]
Map (num_proc=8):  99%|#########8| 253/256 [00:01<00:00, 247.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:01<00:00, 222.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1229.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1192.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 463.97 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 468.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1195.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.48 examples/s]

Map (num_proc=8):   0%|          | 0/135 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 34/135 [00:00<00:00, 215.60 examples/s]
Map (num_proc=8): 100%|##########| 135/135 [00:00<00:00, 556.12 examples/s]
Map (num_proc=8): 100%|##########| 135/135 [00:00<00:00, 356.39 examples/s]

Filter (num_proc=8):   0%|          | 0/135 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######2   | 85/135 [00:00<00:00, 730.40 examples/s]
Filter (num_proc=8): 100%|##########| 135/135 [00:00<00:00, 631.82 examples/s]INFO 2025-05-20 07:32:34,819 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 07:33:48,451 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 343.16 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 683.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1260.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.96 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 456.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 944.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 277.97 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 480.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1178.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.68 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 665.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1184.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.28 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 331.19 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 657.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1208.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1190.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 449.79 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 488.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.85 examples/s]

Map (num_proc=8):   0%|          | 0/37 [00:00<?, ? examples/s]
Map (num_proc=8):  41%|####      | 15/37 [00:00<00:00, 113.27 examples/s]
Map (num_proc=8):  89%|########9 | 33/37 [00:00<00:00, 138.65 examples/s]
Map (num_proc=8): 100%|##########| 37/37 [00:00<00:00, 107.93 examples/s]

Filter (num_proc=8):   0%|          | 0/37 [00:00<?, ? examples/s]
Filter (num_proc=8):  54%|#####4    | 20/37 [00:00<00:00, 197.91 examples/s]
Filter (num_proc=8): 100%|##########| 37/37 [00:00<00:00, 180.12 examples/s]INFO 2025-05-20 07:33:56,826 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_14_6__Removing_the_VirtIO_container_disk_from_a_virtual_machine/data_checkpoint_83ede40985bd40d1b43c7105150e14fa.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 33.96ba/s]INFO 2025-05-20 07:33:57,185 instructlab.sdg.generate_data:478: Generated 1216 samples
INFO 2025-05-20 07:33:57,247 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_14_1__Collecting_data_about_your_environment, generating from scratch
INFO 2025-05-20 07:33:57,247 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 07:33:57,250 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 07:33:57,773 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 07:33:57,787 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 07:33:57,795 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 07:34:07,273 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 07:34:18,723 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 07:34:26,496 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 07:34:26,517 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 07:34:26,533 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 07:35:22,777 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 07:36:12,179 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 436.43 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 633.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1169.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 344.45 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 765.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1204.57 examples/s]

Map (num_proc=8):   0%|          | 0/215 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 54/215 [00:00<00:00, 403.82 examples/s]
Map (num_proc=8):  88%|########7 | 189/215 [00:00<00:00, 677.19 examples/s]
Map (num_proc=8): 100%|##########| 215/215 [00:00<00:00, 549.48 examples/s]

Filter (num_proc=8):   0%|          | 0/215 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 108/215 [00:00<00:00, 1069.97 examples/s]
Filter (num_proc=8): 100%|##########| 215/215 [00:00<00:00, 988.75 examples/s] INFO 2025-05-20 07:36:15,687 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 07:36:51,303 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.96 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 635.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1180.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.30 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 651.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1126.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.06 examples/s]

Map (num_proc=8):   0%|          | 0/47 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###8      | 18/47 [00:00<00:00, 126.72 examples/s]
Map (num_proc=8): 100%|##########| 47/47 [00:00<00:00, 137.92 examples/s]

Filter (num_proc=8):   0%|          | 0/47 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 18/47 [00:00<00:00, 168.13 examples/s]
Filter (num_proc=8): 100%|##########| 47/47 [00:00<00:00, 214.36 examples/s]INFO 2025-05-20 07:36:54,758 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 07:37:16,504 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 352.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 915.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1181.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.70 examples/s]

Map (num_proc=8):   0%|          | 0/232 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 58/232 [00:00<00:00, 383.33 examples/s]
Map (num_proc=8):  75%|#######5  | 174/232 [00:00<00:00, 648.64 examples/s]
Map (num_proc=8): 100%|##########| 232/232 [00:00<00:00, 577.92 examples/s]

Filter (num_proc=8):   0%|          | 0/232 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 116/232 [00:00<00:00, 1072.75 examples/s]
Filter (num_proc=8): 100%|##########| 232/232 [00:00<00:00, 1081.52 examples/s]INFO 2025-05-20 07:37:18,851 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_14_1__Collecting_data_about_your_environment/data_checkpoint_7c3031c9fec44309b2c8aa665c40f3cf.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 44.70ba/s]INFO 2025-05-20 07:37:18,980 instructlab.sdg.generate_data:478: Generated 411 samples
INFO 2025-05-20 07:37:19,041 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_3_1__Before_you_begin, generating from scratch
INFO 2025-05-20 07:37:19,041 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 07:37:19,044 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 07:37:19,172 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 07:37:19,185 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 07:37:19,193 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 07:37:22,905 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 07:37:25,632 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 07:37:29,708 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 07:37:29,726 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 07:37:29,742 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 07:38:11,503 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 07:38:24,283 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/191 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 48/191 [00:00<00:00, 358.79 examples/s]
Map (num_proc=8):  75%|#######5  | 144/191 [00:00<00:00, 560.73 examples/s]
Map (num_proc=8): 100%|##########| 191/191 [00:00<00:00, 503.06 examples/s]

Filter (num_proc=8):   0%|          | 0/191 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 72/191 [00:00<00:00, 688.71 examples/s]
Filter (num_proc=8): 100%|##########| 191/191 [00:00<00:00, 909.11 examples/s]INFO 2025-05-20 07:38:25,419 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 07:38:33,801 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/95 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 24/95 [00:00<00:00, 193.90 examples/s]
Map (num_proc=8):  63%|######3   | 60/95 [00:00<00:00, 277.65 examples/s]
Map (num_proc=8): 100%|##########| 95/95 [00:00<00:00, 270.51 examples/s]

Filter (num_proc=8):   0%|          | 0/95 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 60/95 [00:00<00:00, 564.10 examples/s]
Filter (num_proc=8): 100%|##########| 95/95 [00:00<00:00, 460.38 examples/s]INFO 2025-05-20 07:38:34,905 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 07:38:41,560 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/87 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 22/87 [00:00<00:00, 189.46 examples/s]
Map (num_proc=8): 100%|##########| 87/87 [00:00<00:00, 334.60 examples/s]
Map (num_proc=8): 100%|##########| 87/87 [00:00<00:00, 245.05 examples/s]

Filter (num_proc=8):   0%|          | 0/87 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 44/87 [00:00<00:00, 418.56 examples/s]
Filter (num_proc=8): 100%|##########| 87/87 [00:00<00:00, 419.60 examples/s]INFO 2025-05-20 07:38:42,665 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_3_1__Before_you_begin/data_checkpoint_f96a8f8d191b4676ac34d08ddc084c0a.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 216.10ba/s]INFO 2025-05-20 07:38:42,694 instructlab.sdg.generate_data:478: Generated 68 samples
INFO 2025-05-20 07:38:42,783 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_4_3__Deploying_the_OpenShift_Virtualization_Operator_by_using_the_CLI, generating from scratch
INFO 2025-05-20 07:38:42,783 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 07:38:42,787 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 07:38:43,067 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 07:38:43,081 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 07:38:43,088 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 07:38:53,732 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 07:39:02,373 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 07:39:11,953 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 07:39:11,973 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 07:39:11,988 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 07:40:19,832 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 07:43:01,265 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.16 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 806.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1192.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.27 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 709.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1166.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1184.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.17 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 608.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 640.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1151.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1186.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 441.65 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 456.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1443.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1195.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.51 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 616.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1269.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1217.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.28 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 809.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1501.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1200.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 447.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 675.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1432.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1193.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 607.86 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 539.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 639.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1216.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1196.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 489.58 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 726.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.50 examples/s]

Map (num_proc=8):   0%|          | 0/13 [00:00<?, ? examples/s]
Map (num_proc=8):  31%|###       | 4/13 [00:00<00:00, 39.56 examples/s]
Map (num_proc=8):  92%|#########2| 12/13 [00:00<00:00, 53.85 examples/s]
Map (num_proc=8): 100%|##########| 13/13 [00:00<00:00, 39.50 examples/s]

Filter (num_proc=8):   0%|          | 0/13 [00:00<?, ? examples/s]
Filter (num_proc=8):  77%|#######6  | 10/13 [00:00<00:00, 98.09 examples/s]
Filter (num_proc=8): 100%|##########| 13/13 [00:00<00:00, 65.86 examples/s]INFO 2025-05-20 07:43:12,746 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 07:44:52,302 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.24 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 638.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1105.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1183.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 457.79 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 549.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.15 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1167.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.23 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 494.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.91 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 636.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1250.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 449.63 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 830.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1104.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1194.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.56 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 652.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.82 examples/s]

Map (num_proc=8):   0%|          | 0/75 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##6       | 20/75 [00:00<00:00, 176.03 examples/s]
Map (num_proc=8):  76%|#######6  | 57/75 [00:00<00:00, 262.57 examples/s]
Map (num_proc=8): 100%|##########| 75/75 [00:00<00:00, 211.11 examples/s]

Filter (num_proc=8):   0%|          | 0/75 [00:00<?, ? examples/s]
Filter (num_proc=8):  64%|######4   | 48/75 [00:00<00:00, 455.10 examples/s]
Filter (num_proc=8): 100%|##########| 75/75 [00:00<00:00, 360.42 examples/s]INFO 2025-05-20 07:45:00,459 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 07:46:15,792 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 337.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 572.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1230.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 673.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 426.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 781.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1163.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.77 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 640.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1209.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1198.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 476.08 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 642.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1245.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.86 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 737.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1182.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.65 examples/s]

Map (num_proc=8):   0%|          | 0/35 [00:00<?, ? examples/s]
Map (num_proc=8):  29%|##8       | 10/35 [00:00<00:00, 91.79 examples/s]
Map (num_proc=8):  77%|#######7  | 27/35 [00:00<00:00, 131.68 examples/s]
Map (num_proc=8): 100%|##########| 35/35 [00:00<00:00, 103.09 examples/s]

Filter (num_proc=8):   0%|          | 0/35 [00:00<?, ? examples/s]
Filter (num_proc=8):  54%|#####4    | 19/35 [00:00<00:00, 188.15 examples/s]
Filter (num_proc=8): 100%|##########| 35/35 [00:00<00:00, 170.28 examples/s]INFO 2025-05-20 07:46:23,931 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_4_3__Deploying_the_OpenShift_Virtualization_Operator_by_using_the_CLI/data_checkpoint_6970f7f856fa4239bfd3ed3acd89197a.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 30.59ba/s]INFO 2025-05-20 07:46:24,336 instructlab.sdg.generate_data:478: Generated 1415 samples
INFO 2025-05-20 07:46:24,392 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_8_4__Creating_a_container_using_virtctl_guestfs, generating from scratch
INFO 2025-05-20 07:46:24,392 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 07:46:24,396 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 07:46:24,508 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 07:46:24,521 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 07:46:24,528 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 07:46:28,321 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 07:46:31,608 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 07:46:33,637 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 07:46:33,655 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 07:46:33,670 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 07:47:19,335 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 07:47:45,822 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 601.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1191.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.49 examples/s]

Map (num_proc=8):   0%|          | 0/72 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 18/72 [00:00<00:00, 143.46 examples/s]
Map (num_proc=8): 100%|##########| 72/72 [00:00<00:00, 303.07 examples/s]
Map (num_proc=8): 100%|##########| 72/72 [00:00<00:00, 208.57 examples/s]

Filter (num_proc=8):   0%|          | 0/72 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 36/72 [00:00<00:00, 351.85 examples/s]
Filter (num_proc=8): 100%|##########| 72/72 [00:00<00:00, 348.44 examples/s]INFO 2025-05-20 07:47:48,089 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 07:48:07,025 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 628.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1142.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.87 examples/s]

Map (num_proc=8):   0%|          | 0/14 [00:00<?, ? examples/s]
Map (num_proc=8):  43%|####2     | 6/14 [00:00<00:00, 40.80 examples/s]
Map (num_proc=8): 100%|##########| 14/14 [00:00<00:00, 42.53 examples/s]

Filter (num_proc=8):   0%|          | 0/14 [00:00<?, ? examples/s]
Filter (num_proc=8):  57%|#####7    | 8/14 [00:00<00:00, 74.38 examples/s]
Filter (num_proc=8): 100%|##########| 14/14 [00:00<00:00, 69.03 examples/s]INFO 2025-05-20 07:48:09,260 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 07:48:23,651 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 458.96 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 659.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1201.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1201.16 examples/s]
num_proc must be <= 2. Reducing num_proc to 2 for dataset of size 2.

Map (num_proc=2):   0%|          | 0/2 [00:00<?, ? examples/s]
Map (num_proc=2): 100%|##########| 2/2 [00:00<00:00, 11.38 examples/s]
num_proc must be <= 2. Reducing num_proc to 2 for dataset of size 2.

Filter (num_proc=2):   0%|          | 0/2 [00:00<?, ? examples/s]
Filter (num_proc=2): 100%|##########| 2/2 [00:00<00:00, 13.66 examples/s]INFO 2025-05-20 07:48:25,296 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_8_4__Creating_a_container_using_virtctl_guestfs/data_checkpoint_8c7a4d5ac30142f3abf6072f31f85be0.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 92.51ba/s]INFO 2025-05-20 07:48:25,367 instructlab.sdg.generate_data:478: Generated 220 samples
INFO 2025-05-20 07:48:25,433 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_1_6__Snapshots_and_cloning, generating from scratch
INFO 2025-05-20 07:48:25,433 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 07:48:25,437 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 07:48:26,102 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 07:48:26,116 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 07:48:26,124 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 07:48:34,999 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 07:48:46,185 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 07:48:54,428 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 07:48:54,450 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 07:48:54,465 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 07:50:39,998 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 07:54:19,015 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 452.79 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 692.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 787.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1140.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 461.43 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 519.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 821.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1340.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1191.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.69 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 946.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1391.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1204.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.61 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 764.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 635.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1208.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1196.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 436.43 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 596.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1226.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1203.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 351.43 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 831.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1112.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1201.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.21 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1188.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1184.54 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.31 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 736.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 636.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1111.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1198.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 425.63 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 824.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1186.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1195.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 469.16 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 624.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 636.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1350.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1211.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 462.49 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 622.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 635.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1220.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1204.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 425.58 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 522.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 884.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1184.53 examples/s]

Map (num_proc=8):   0%|          | 0/244 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 62/244 [00:00<00:00, 518.80 examples/s]
Map (num_proc=8):  51%|#####     | 124/244 [00:00<00:00, 565.93 examples/s]
Map (num_proc=8): 100%|##########| 244/244 [00:00<00:00, 616.75 examples/s]

Filter (num_proc=8):   0%|          | 0/244 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 154/244 [00:00<00:00, 1307.62 examples/s]
Filter (num_proc=8): 100%|##########| 244/244 [00:00<00:00, 1134.47 examples/s]INFO 2025-05-20 07:54:33,835 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 07:56:14,221 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.97 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 572.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1318.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 565.35 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 541.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1183.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 328.80 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 862.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 488.62 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 548.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 910.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.44 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 848.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1263.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.43 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 583.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 925.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.99 examples/s]

Map (num_proc=8):   0%|          | 0/137 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 35/137 [00:00<00:00, 247.28 examples/s]
Map (num_proc=8):  75%|#######5  | 103/137 [00:00<00:00, 420.03 examples/s]
Map (num_proc=8): 100%|##########| 137/137 [00:00<00:00, 365.64 examples/s]

Filter (num_proc=8):   0%|          | 0/137 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######2   | 86/137 [00:00<00:00, 790.98 examples/s]
Filter (num_proc=8): 100%|##########| 137/137 [00:00<00:00, 645.22 examples/s]INFO 2025-05-20 07:56:22,327 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 07:57:35,334 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.86 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 709.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1145.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.22 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 642.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1221.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 414.23 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 550.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1186.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.58 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 771.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 937.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 330.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 840.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1184.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1191.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 436.24 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 531.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1258.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1182.51 examples/s]

Map (num_proc=8):   0%|          | 0/56 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 14/56 [00:00<00:00, 116.01 examples/s]
Map (num_proc=8):  88%|########7 | 49/56 [00:00<00:00, 237.28 examples/s]
Map (num_proc=8): 100%|##########| 56/56 [00:00<00:00, 160.89 examples/s]

Filter (num_proc=8):   0%|          | 0/56 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 35/56 [00:00<00:00, 316.20 examples/s]
Filter (num_proc=8): 100%|##########| 56/56 [00:00<00:00, 276.54 examples/s]INFO 2025-05-20 07:57:43,367 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_1_6__Snapshots_and_cloning/data_checkpoint_7f2a93ec46b64fa4b7acc5a8ed3dc13c.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 29.89ba/s]INFO 2025-05-20 07:57:43,779 instructlab.sdg.generate_data:478: Generated 1445 samples
INFO 2025-05-20 07:57:43,849 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_2_1__Editing_a_virtual_machine_in_the_web_console, generating from scratch
INFO 2025-05-20 07:57:43,849 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 07:57:43,852 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 07:57:44,400 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 07:57:44,414 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 07:57:44,422 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 07:57:53,765 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 07:58:06,445 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 07:58:15,482 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 07:58:15,505 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 07:58:15,520 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 08:00:01,520 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 08:04:51,643 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.37 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 765.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 939.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.65 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 666.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1259.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.80 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 473.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 647.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1194.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.28 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 678.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1220.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1193.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.37 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 831.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1235.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1198.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 451.22 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 671.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1411.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1190.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.49 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 860.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1102.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1197.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 456.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 630.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1216.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.29 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 898.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1187.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 447.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 675.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1193.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1197.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 442.40 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 509.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1225.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1214.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.00 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 692.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1191.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.37 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 808.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1167.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.36 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 670.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 630.92 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1366.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1191.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 456.62 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 521.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1262.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 446.56 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 585.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.15 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1216.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.48 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 566.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 638.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1103.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.97 examples/s]

Map (num_proc=8):   0%|          | 0/54 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 14/54 [00:00<00:00, 113.73 examples/s]
Map (num_proc=8):  65%|######4   | 35/54 [00:00<00:00, 164.72 examples/s]
Map (num_proc=8): 100%|##########| 54/54 [00:00<00:00, 156.11 examples/s]

Filter (num_proc=8):   0%|          | 0/54 [00:00<?, ? examples/s]
Filter (num_proc=8):  65%|######4   | 35/54 [00:00<00:00, 343.04 examples/s]
Filter (num_proc=8): 100%|##########| 54/54 [00:00<00:00, 270.72 examples/s]INFO 2025-05-20 08:05:12,258 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 08:08:09,445 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 499.15 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 478.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1226.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 434.20 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 494.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1136.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.91 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 748.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1155.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.90 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 759.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1203.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 330.13 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 768.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1094.98 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.84 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 749.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1193.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 484.13 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 517.44 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.97 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 699.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1256.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 539.34 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 514.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1213.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1183.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 500.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 526.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1127.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.37 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 744.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1153.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.02 examples/s]

Map (num_proc=8):   0%|          | 0/124 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 32/124 [00:00<00:00, 218.82 examples/s]
Map (num_proc=8):  88%|########7 | 109/124 [00:00<00:00, 428.75 examples/s]
Map (num_proc=8): 100%|##########| 124/124 [00:00<00:00, 339.48 examples/s]

Filter (num_proc=8):   0%|          | 0/124 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 64/124 [00:00<00:00, 617.01 examples/s]
Filter (num_proc=8): 100%|##########| 124/124 [00:00<00:00, 600.00 examples/s]INFO 2025-05-20 08:08:23,393 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 08:10:22,945 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 630.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1149.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.31 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 595.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1219.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 327.62 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 680.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1077.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1190.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 425.47 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 480.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1278.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 663.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1182.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.10 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 712.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1214.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.87 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 776.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1168.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.40 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 787.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1108.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1195.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1184.22 examples/s]

Map (num_proc=8):   0%|          | 0/111 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 28/111 [00:00<00:00, 240.35 examples/s]
Map (num_proc=8):  76%|#######5  | 84/111 [00:00<00:00, 395.54 examples/s]
Map (num_proc=8): 100%|##########| 111/111 [00:00<00:00, 304.83 examples/s]

Filter (num_proc=8):   0%|          | 0/111 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 56/111 [00:00<00:00, 529.55 examples/s]
Filter (num_proc=8): 100%|##########| 111/111 [00:00<00:00, 532.13 examples/s]INFO 2025-05-20 08:10:35,701 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_2_1__Editing_a_virtual_machine_in_the_web_console/data_checkpoint_de0963eeeadb4afcacd945169b38d369.jsonl


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 30.96ba/s]INFO 2025-05-20 08:10:36,320 instructlab.sdg.generate_data:478: Generated 2147 samples
INFO 2025-05-20 08:10:36,388 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_9_1__About_OpenShift_Container_Platform_monitoring, generating from scratch
INFO 2025-05-20 08:10:36,388 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 08:10:36,391 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 08:10:36,806 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 08:10:36,819 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 08:10:36,827 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 08:10:46,216 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 08:10:56,366 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 08:11:01,454 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 08:11:01,475 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 08:11:01,490 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 08:12:31,754 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 08:15:13,145 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.44 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 640.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1197.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1195.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 447.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 822.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1457.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1196.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 440.55 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 488.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1197.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 450.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 625.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1035.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1136.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 530.62 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 539.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1346.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1218.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 484.12 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 559.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 844.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1166.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1197.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.98 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1246.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1195.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.42 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 751.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 455.63 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 492.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1120.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.83 examples/s]

Map (num_proc=8):   0%|          | 0/198 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 50/198 [00:00<00:00, 454.30 examples/s]
Map (num_proc=8):  63%|######3   | 125/198 [00:00<00:00, 517.37 examples/s]
Map (num_proc=8): 100%|##########| 198/198 [00:00<00:00, 504.90 examples/s]

Filter (num_proc=8):   0%|          | 0/198 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 125/198 [00:00<00:00, 1086.64 examples/s]
Filter (num_proc=8): 100%|##########| 198/198 [00:00<00:00, 940.01 examples/s] INFO 2025-05-20 08:15:24,687 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 08:17:13,080 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 456.46 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 549.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1117.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.31 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 507.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1411.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.15 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 647.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1158.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 380.08 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 887.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 946.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.98 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 599.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.67 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.48 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 790.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1112.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.72 examples/s]

Map (num_proc=8):   0%|          | 0/206 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 52/206 [00:00<00:00, 355.27 examples/s]
Map (num_proc=8):  63%|######3   | 130/206 [00:00<00:00, 556.04 examples/s]
Map (num_proc=8): 100%|##########| 206/206 [00:00<00:00, 509.23 examples/s]

Filter (num_proc=8):   0%|          | 0/206 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 130/206 [00:00<00:00, 1161.75 examples/s]
Filter (num_proc=8): 100%|##########| 206/206 [00:00<00:00, 955.80 examples/s] INFO 2025-05-20 08:17:21,258 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 08:18:35,636 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.63 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 635.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 897.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 449.85 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 850.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1086.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 581.67 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 561.27 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 724.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1194.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1184.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.93 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 690.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1203.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1191.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.30 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 738.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1265.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1186.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.61 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 505.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1473.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1186.06 examples/s]

Map (num_proc=8):   0%|          | 0/124 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 32/124 [00:00<00:00, 252.75 examples/s]
Map (num_proc=8):  64%|######3   | 79/124 [00:00<00:00, 362.32 examples/s]
Map (num_proc=8): 100%|##########| 124/124 [00:00<00:00, 331.84 examples/s]

Filter (num_proc=8):   0%|          | 0/124 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 64/124 [00:00<00:00, 603.33 examples/s]
Filter (num_proc=8): 100%|##########| 124/124 [00:00<00:00, 585.78 examples/s]INFO 2025-05-20 08:18:43,739 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_9_1__About_OpenShift_Container_Platform_monitoring/data_checkpoint_e262c43dfe364252a20c730c0ab33aec.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 28.64ba/s]INFO 2025-05-20 08:18:44,182 instructlab.sdg.generate_data:478: Generated 1585 samples
INFO 2025-05-20 08:18:44,242 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_7_1__About_workload_security, generating from scratch
INFO 2025-05-20 08:18:44,242 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 08:18:44,246 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 08:18:44,386 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 08:18:44,399 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 08:18:44,407 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 08:18:46,761 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 08:18:50,186 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 08:18:52,381 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 08:18:52,399 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 08:18:52,414 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 08:19:34,784 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 08:19:57,162 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.50 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 801.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1145.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1100.27 examples/s]

Map (num_proc=8):   0%|          | 0/49 [00:00<?, ? examples/s]
Map (num_proc=8):  39%|###8      | 19/49 [00:00<00:00, 128.33 examples/s]
Map (num_proc=8): 100%|##########| 49/49 [00:00<00:00, 138.10 examples/s]

Filter (num_proc=8):   0%|          | 0/49 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 25/49 [00:00<00:00, 234.65 examples/s]
Filter (num_proc=8): 100%|##########| 49/49 [00:00<00:00, 226.49 examples/s]INFO 2025-05-20 08:19:59,590 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 08:20:19,344 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 312.09 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 499.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 870.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 897.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.90 examples/s]

Map (num_proc=8):   0%|          | 0/43 [00:00<?, ? examples/s]
Map (num_proc=8):  28%|##7       | 12/43 [00:00<00:00, 99.24 examples/s]
Map (num_proc=8): 100%|##########| 43/43 [00:00<00:00, 176.72 examples/s]
Map (num_proc=8): 100%|##########| 43/43 [00:00<00:00, 126.67 examples/s]

Filter (num_proc=8):   0%|          | 0/43 [00:00<?, ? examples/s]
Filter (num_proc=8):  53%|#####3    | 23/43 [00:00<00:00, 208.95 examples/s]
Filter (num_proc=8): 100%|##########| 43/43 [00:00<00:00, 207.58 examples/s]INFO 2025-05-20 08:20:21,599 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 08:20:39,016 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.53 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 728.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1211.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1203.83 examples/s]

Map (num_proc=8):   0%|          | 0/39 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 10/39 [00:00<00:00, 90.27 examples/s]
Map (num_proc=8):  90%|########9 | 35/39 [00:00<00:00, 158.15 examples/s]
Map (num_proc=8): 100%|##########| 39/39 [00:00<00:00, 114.50 examples/s]

Filter (num_proc=8):   0%|          | 0/39 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 20/39 [00:00<00:00, 192.91 examples/s]
Filter (num_proc=8): 100%|##########| 39/39 [00:00<00:00, 188.68 examples/s]INFO 2025-05-20 08:20:41,241 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_7_1__About_workload_security/data_checkpoint_1ec961b6688743e3b52d2172397442b1.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 71.46ba/s]INFO 2025-05-20 08:20:41,332 instructlab.sdg.generate_data:478: Generated 287 samples
INFO 2025-05-20 08:20:41,395 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_10_1_5__Additional_resources, generating from scratch
INFO 2025-05-20 08:20:41,395 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 08:20:41,398 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 08:20:41,854 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 08:20:41,867 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 08:20:41,875 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 08:20:50,360 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 08:21:01,604 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 08:21:09,565 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 08:21:09,586 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 08:21:09,602 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 08:22:02,350 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 08:22:49,103 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 346.27 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 674.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 902.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1104.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.73 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 774.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1436.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.14 examples/s]

Map (num_proc=8):   0%|          | 0/163 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 42/163 [00:00<00:00, 256.36 examples/s]
Map (num_proc=8):  75%|#######5  | 123/163 [00:00<00:00, 501.58 examples/s]
Map (num_proc=8): 100%|##########| 163/163 [00:00<00:00, 417.10 examples/s]

Filter (num_proc=8):   0%|          | 0/163 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 83/163 [00:00<00:00, 827.70 examples/s]
Filter (num_proc=8): 100%|##########| 163/163 [00:00<00:00, 771.01 examples/s]INFO 2025-05-20 08:22:52,602 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 08:23:23,605 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.24 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 468.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1229.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.01 examples/s]

Map (num_proc=8):   0%|          | 0/245 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 62/245 [00:00<00:00, 344.66 examples/s]
Map (num_proc=8):  76%|#######5  | 185/245 [00:00<00:00, 615.90 examples/s]
Map (num_proc=8): 100%|##########| 245/245 [00:00<00:00, 561.07 examples/s]

Filter (num_proc=8):   0%|          | 0/245 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 124/245 [00:00<00:00, 1065.03 examples/s]
Filter (num_proc=8): 100%|##########| 245/245 [00:00<00:00, 1147.79 examples/s]INFO 2025-05-20 08:23:25,977 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 08:23:46,185 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.78 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 495.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1104.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1186.72 examples/s]

Map (num_proc=8):   0%|          | 0/209 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 53/209 [00:00<00:00, 307.90 examples/s]
Map (num_proc=8):  88%|########7 | 183/209 [00:00<00:00, 685.03 examples/s]
Map (num_proc=8): 100%|##########| 209/209 [00:00<00:00, 508.63 examples/s]

Filter (num_proc=8):   0%|          | 0/209 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 79/209 [00:00<00:00, 770.28 examples/s]
Filter (num_proc=8): 100%|##########| 209/209 [00:00<00:00, 952.68 examples/s]INFO 2025-05-20 08:23:48,531 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_10_1_5__Additional_resources/data_checkpoint_8546edbabb2c47289a49834ead36c7c2.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 36.23ba/s]INFO 2025-05-20 08:23:48,670 instructlab.sdg.generate_data:478: Generated 407 samples
INFO 2025-05-20 08:23:48,729 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_2_3__About_the_cdi_operator, generating from scratch
INFO 2025-05-20 08:23:48,730 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 08:23:48,733 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 08:23:48,902 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 08:23:48,915 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 08:23:48,922 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 08:23:51,213 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 08:23:53,583 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 08:23:56,139 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 08:23:56,157 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 08:23:56,172 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 08:24:41,487 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 08:25:11,054 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 410.73 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 722.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 636.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1381.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1186.49 examples/s]

Map (num_proc=8):   0%|          | 0/130 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 34/130 [00:00<00:00, 278.83 examples/s]
Map (num_proc=8):  88%|########7 | 114/130 [00:00<00:00, 555.10 examples/s]
Map (num_proc=8): 100%|##########| 130/130 [00:00<00:00, 358.20 examples/s]

Filter (num_proc=8):   0%|          | 0/130 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 82/130 [00:00<00:00, 696.38 examples/s]
Filter (num_proc=8): 100%|##########| 130/130 [00:00<00:00, 618.48 examples/s]INFO 2025-05-20 08:25:13,293 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 08:25:37,650 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 426.79 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 764.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 638.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1263.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.28 examples/s]

Map (num_proc=8):   0%|          | 0/99 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 26/99 [00:00<00:00, 203.00 examples/s]
Map (num_proc=8):  76%|#######5  | 75/99 [00:00<00:00, 349.38 examples/s]
Map (num_proc=8): 100%|##########| 99/99 [00:00<00:00, 277.22 examples/s]

Filter (num_proc=8):   0%|          | 0/99 [00:00<?, ? examples/s]
Filter (num_proc=8):  64%|######3   | 63/99 [00:00<00:00, 546.55 examples/s]
Filter (num_proc=8): 100%|##########| 99/99 [00:00<00:00, 474.89 examples/s]INFO 2025-05-20 08:25:39,876 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 08:25:58,459 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 444.88 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 592.38 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 633.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1456.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1191.75 examples/s]

Map (num_proc=8):   0%|          | 0/96 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 24/96 [00:00<00:00, 215.31 examples/s]
Map (num_proc=8):  88%|########7 | 84/96 [00:00<00:00, 346.71 examples/s]
Map (num_proc=8): 100%|##########| 96/96 [00:00<00:00, 274.74 examples/s]

Filter (num_proc=8):   0%|          | 0/96 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 48/96 [00:00<00:00, 454.04 examples/s]
Filter (num_proc=8): 100%|##########| 96/96 [00:00<00:00, 462.13 examples/s]INFO 2025-05-20 08:26:00,675 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_2_3__About_the_cdi_operator/data_checkpoint_1b93ec456f8648648a23567da090562c.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 82.21ba/s]INFO 2025-05-20 08:26:00,759 instructlab.sdg.generate_data:478: Generated 283 samples
INFO 2025-05-20 08:26:00,823 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_1_5__Virtual_machine_storage_volume_types, generating from scratch
INFO 2025-05-20 08:26:00,824 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 08:26:00,827 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 08:26:01,777 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 08:26:01,791 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 08:26:01,799 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 08:26:17,603 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 08:26:43,941 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 08:27:00,440 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 08:27:00,464 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 08:27:00,482 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 08:29:55,837 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 08:35:47,160 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.92 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 549.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1158.13 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.70 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 819.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1151.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.67 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 707.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.53 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 579.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.84 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 672.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 337.47 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 571.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1250.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1185.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.43 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 605.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1254.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 445.59 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 615.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1244.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.87 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 633.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 931.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.40 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 600.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1256.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.64 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 523.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1184.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 515.21 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 539.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1197.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 420.08 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 653.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1208.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.33 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 672.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1195.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1093.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 448.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 635.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 914.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 442.60 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 694.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1160.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 766.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1270.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 643.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 879.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.11 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 736.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1163.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.11 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 648.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1135.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.56 examples/s]

Map (num_proc=8):   0%|          | 0/23 [00:00<?, ? examples/s]
Map (num_proc=8):  39%|###9      | 9/23 [00:00<00:00, 54.76 examples/s]
Map (num_proc=8): 100%|##########| 23/23 [00:00<00:00, 67.67 examples/s]

Filter (num_proc=8):   0%|          | 0/23 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####2    | 12/23 [00:00<00:00, 117.27 examples/s]
Filter (num_proc=8): 100%|##########| 23/23 [00:00<00:00, 115.16 examples/s]INFO 2025-05-20 08:36:11,698 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 08:39:21,134 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 350.64 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 731.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.93 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.43 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 593.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1194.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.26 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 565.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1149.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 334.20 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 731.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 927.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.89 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 504.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.28 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 410.39 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 590.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1209.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 322.04 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 632.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1124.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 341.02 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 664.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1091.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 354.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 682.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1178.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.34 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 664.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1236.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 350.81 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 466.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1146.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.04 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 587.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1228.88 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.60 examples/s]

Map (num_proc=8):   0%|          | 0/46 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 12/46 [00:00<00:00, 99.22 examples/s]
Map (num_proc=8):  89%|########9 | 41/46 [00:00<00:00, 189.38 examples/s]
Map (num_proc=8): 100%|##########| 46/46 [00:00<00:00, 133.16 examples/s]

Filter (num_proc=8):   0%|          | 0/46 [00:00<?, ? examples/s]
Filter (num_proc=8):  39%|###9      | 18/46 [00:00<00:00, 176.43 examples/s]
Filter (num_proc=8): 100%|##########| 46/46 [00:00<00:00, 219.51 examples/s]INFO 2025-05-20 08:39:36,514 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 08:41:52,764 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.75 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 568.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1265.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 331.97 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 869.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.78 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 786.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1121.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 349.53 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 877.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1259.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.68 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 778.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1248.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 357.55 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 641.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1270.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 410.60 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 647.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1238.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 295.60 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 670.99 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 763.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 566.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1214.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.34 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 633.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1272.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.01 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 647.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1201.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1182.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.03 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 810.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1152.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.41 examples/s]

Map (num_proc=8):   0%|          | 0/165 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 42/165 [00:00<00:00, 260.51 examples/s]
Map (num_proc=8):  88%|########7 | 145/165 [00:00<00:00, 552.76 examples/s]
Map (num_proc=8): 100%|##########| 165/165 [00:00<00:00, 418.01 examples/s]

Filter (num_proc=8):   0%|          | 0/165 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 84/165 [00:00<00:00, 809.54 examples/s]
Filter (num_proc=8): 100%|##########| 165/165 [00:00<00:00, 773.37 examples/s]INFO 2025-05-20 08:42:06,995 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_1_5__Virtual_machine_storage_volume_types/data_checkpoint_3a355964652e432f967ee25d4a3f56bc.jsonl


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 25.85ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 25.81ba/s]INFO 2025-05-20 08:42:07,711 instructlab.sdg.generate_data:478: Generated 2488 samples
INFO 2025-05-20 08:42:07,772 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_1__Creating_virtual_machines, generating from scratch
INFO 2025-05-20 08:42:07,772 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 08:42:07,775 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 08:42:08,715 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 08:42:08,729 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 08:42:08,737 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 08:42:24,529 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 08:42:57,398 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 08:43:13,893 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 08:43:13,917 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 08:43:13,935 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 08:46:07,137 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 08:52:43,875 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.47 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 749.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.77 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 894.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 945.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.28 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 718.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1218.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.02 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 600.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.73 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 808.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1148.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 493.86 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 693.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.93 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1248.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 369.52 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 484.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 959.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 678.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.74 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.48 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 701.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.90 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 778.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 911.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.07 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 704.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1161.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 575.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 883.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.30 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.34 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 844.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 865.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.30 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 679.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1271.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.99 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 780.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.38 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 602.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1440.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 508.79 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 640.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1261.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 336.31 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 813.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 953.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.91 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 484.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 434.52 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 682.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.90 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 589.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 436.54 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 817.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1272.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1097.09 examples/s]

Map (num_proc=8):   0%|          | 0/211 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 54/211 [00:00<00:00, 336.80 examples/s]
Map (num_proc=8):  63%|######3   | 133/211 [00:00<00:00, 522.48 examples/s]
Map (num_proc=8): 100%|##########| 211/211 [00:00<00:00, 530.90 examples/s]

Filter (num_proc=8):   0%|          | 0/211 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 107/211 [00:00<00:00, 977.70 examples/s]
Filter (num_proc=8): 100%|##########| 211/211 [00:00<00:00, 975.39 examples/s]INFO 2025-05-20 08:53:10,855 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 08:57:36,192 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.80 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 616.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1145.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 477.56 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 444.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1142.52 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 427.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 633.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1189.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.12 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 671.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 956.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.22 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 653.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1121.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.46 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 717.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.29 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.52 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 679.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1180.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 408.39 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 668.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.57 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1250.99 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 458.89 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 481.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1067.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 471.36 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 647.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 383.17 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 608.80 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.15 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 941.97 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.45 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 757.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 351.59 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 561.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 911.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.97 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 698.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1153.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 388.68 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 712.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 873.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.96 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.96 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 693.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1158.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 374.06 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 734.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 958.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.73 examples/s]

Map (num_proc=8):   0%|          | 0/29 [00:00<?, ? examples/s]
Map (num_proc=8):  28%|##7       | 8/29 [00:00<00:00, 72.17 examples/s]
Map (num_proc=8): 100%|##########| 29/29 [00:00<00:00, 148.19 examples/s]
Map (num_proc=8): 100%|##########| 29/29 [00:00<00:00, 85.35 examples/s] 

Filter (num_proc=8):   0%|          | 0/29 [00:00<?, ? examples/s]
Filter (num_proc=8):  55%|#####5    | 16/29 [00:00<00:00, 146.60 examples/s]
Filter (num_proc=8): 100%|##########| 29/29 [00:00<00:00, 140.07 examples/s]INFO 2025-05-20 08:57:57,517 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 09:00:56,427 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 410.98 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 630.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1266.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.20 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 716.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1250.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.71 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 462.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1235.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1191.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 332.72 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 687.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1438.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1213.87 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 340.97 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 692.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 903.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.92 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 606.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1091.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.59 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 730.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1107.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 351.09 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 667.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1115.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.88 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 483.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1155.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 366.24 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 603.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 953.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1161.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.14 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 785.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1343.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.76 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 552.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1264.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 439.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 600.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1269.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.69 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 691.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1181.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1184.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 349.56 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 728.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1246.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.12 examples/s]

Map (num_proc=8):   0%|          | 0/184 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 46/184 [00:00<00:00, 349.76 examples/s]
Map (num_proc=8):  50%|#####     | 92/184 [00:00<00:00, 399.67 examples/s]
Map (num_proc=8): 100%|##########| 184/184 [00:00<00:00, 458.79 examples/s]

Filter (num_proc=8):   0%|          | 0/184 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 69/184 [00:00<00:00, 688.88 examples/s]
Filter (num_proc=8): 100%|##########| 184/184 [00:00<00:00, 841.86 examples/s]INFO 2025-05-20 09:01:15,529 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_1__Creating_virtual_machines/data_checkpoint_89d9346d8f3f4f5eb800a89a0d3cafdd.jsonl


Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format:  75%|#######5  | 3/4 [00:00<00:00, 22.71ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 27.31ba/s]INFO 2025-05-20 09:01:16,483 instructlab.sdg.generate_data:478: Generated 3348 samples
INFO 2025-05-20 09:01:16,563 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_4_1__Deleting_a_virtual_machine_using_the_web_console, generating from scratch
INFO 2025-05-20 09:01:16,563 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 09:01:16,567 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 09:01:16,733 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 09:01:16,746 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 09:01:16,754 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 09:01:20,427 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 09:01:24,608 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 09:01:27,797 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 09:01:27,816 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 09:01:27,831 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 09:02:24,394 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 09:03:38,819 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 922.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1166.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1194.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 461.97 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 689.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 638.03 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.34 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1210.73 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.44 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 798.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 638.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1189.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.51 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 753.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1199.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1202.88 examples/s]

Map (num_proc=8):   0%|          | 0/242 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 62/242 [00:00<00:00, 400.53 examples/s]
Map (num_proc=8):  63%|######2   | 152/242 [00:00<00:00, 632.19 examples/s]
Map (num_proc=8): 100%|##########| 242/242 [00:00<00:00, 606.50 examples/s]

Filter (num_proc=8):   0%|          | 0/242 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 122/242 [00:00<00:00, 1176.34 examples/s]
Filter (num_proc=8): 100%|##########| 242/242 [00:00<00:00, 1144.63 examples/s]INFO 2025-05-20 09:03:44,550 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 09:04:36,585 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 395.32 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 941.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1125.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1183.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 445.92 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 627.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.47 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1138.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 445.00 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 710.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 632.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1228.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.06 examples/s]

Map (num_proc=8):   0%|          | 0/58 [00:00<?, ? examples/s]
Map (num_proc=8):  28%|##7       | 16/58 [00:00<00:00, 112.63 examples/s]
Map (num_proc=8): 100%|##########| 58/58 [00:00<00:00, 167.81 examples/s]

Filter (num_proc=8):   0%|          | 0/58 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 30/58 [00:00<00:00, 291.31 examples/s]
Filter (num_proc=8): 100%|##########| 58/58 [00:00<00:00, 276.86 examples/s]INFO 2025-05-20 09:04:41,138 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 09:05:20,732 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 647.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1187.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 420.24 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 671.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.02 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.31 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 631.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 639.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1203.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1195.26 examples/s]

Map (num_proc=8):   0%|          | 0/31 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 8/31 [00:00<00:00, 68.77 examples/s]
Map (num_proc=8):  90%|######### | 28/31 [00:00<00:00, 124.04 examples/s]
Map (num_proc=8): 100%|##########| 31/31 [00:00<00:00, 92.14 examples/s] 

Filter (num_proc=8):   0%|          | 0/31 [00:00<?, ? examples/s]
Filter (num_proc=8):  52%|#####1    | 16/31 [00:00<00:00, 144.83 examples/s]
Filter (num_proc=8): 100%|##########| 31/31 [00:00<00:00, 150.21 examples/s]INFO 2025-05-20 09:05:25,266 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_4_1__Deleting_a_virtual_machine_using_the_web_console/data_checkpoint_0299ca1d90304ce389dd50a668a3ae0b.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 28.45ba/s]INFO 2025-05-20 09:05:25,489 instructlab.sdg.generate_data:478: Generated 757 samples
INFO 2025-05-20 09:05:25,550 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_12_1_1__About_node_maintenance_mode, generating from scratch
INFO 2025-05-20 09:05:25,550 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 09:05:25,554 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 09:05:25,855 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 09:05:25,868 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 09:05:25,876 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 09:05:31,684 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 09:05:39,462 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 09:05:44,366 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 09:05:44,385 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 09:05:44,400 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 09:06:26,220 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 09:06:51,080 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.28 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 682.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1201.03 examples/s]

Map (num_proc=8):   0%|          | 0/65 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###8      | 25/65 [00:00<00:00, 178.88 examples/s]
Map (num_proc=8):  88%|########7 | 57/65 [00:00<00:00, 227.27 examples/s]
Map (num_proc=8): 100%|##########| 65/65 [00:00<00:00, 185.93 examples/s]

Filter (num_proc=8):   0%|          | 0/65 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 33/65 [00:00<00:00, 322.11 examples/s]
Filter (num_proc=8): 100%|##########| 65/65 [00:00<00:00, 314.63 examples/s]INFO 2025-05-20 09:06:53,322 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 09:07:11,871 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.36 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 620.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1276.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.21 examples/s]

Map (num_proc=8):   0%|          | 0/9 [00:00<?, ? examples/s]
Map (num_proc=8):  44%|####4     | 4/9 [00:00<00:00, 29.50 examples/s]
Map (num_proc=8): 100%|##########| 9/9 [00:00<00:00, 27.52 examples/s]

Filter (num_proc=8):   0%|          | 0/9 [00:00<?, ? examples/s]
Filter (num_proc=8):  67%|######6   | 6/9 [00:00<00:00, 55.62 examples/s]
Filter (num_proc=8): 100%|##########| 9/9 [00:00<00:00, 44.61 examples/s]INFO 2025-05-20 09:07:14,092 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 09:07:24,940 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/242 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 62/242 [00:00<00:00, 480.58 examples/s]
Map (num_proc=8):  63%|######2   | 152/242 [00:00<00:00, 633.47 examples/s]
Map (num_proc=8): 100%|##########| 242/242 [00:00<00:00, 598.68 examples/s]

Filter (num_proc=8):   0%|          | 0/242 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 122/242 [00:00<00:00, 1183.61 examples/s]
Filter (num_proc=8): 100%|##########| 242/242 [00:00<00:00, 1101.16 examples/s]INFO 2025-05-20 09:07:26,099 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_12_1_1__About_node_maintenance_mode/data_checkpoint_cb5c9bbca2164fc283215b320c821d2c.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 76.30ba/s]INFO 2025-05-20 09:07:26,175 instructlab.sdg.generate_data:478: Generated 220 samples
INFO 2025-05-20 09:07:26,238 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_14_3__Using_the_must_gather_tool_for_OpenShift_Virtualization, generating from scratch
INFO 2025-05-20 09:07:26,239 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 09:07:26,242 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 09:07:26,711 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 09:07:26,725 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 09:07:26,733 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 09:07:36,217 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 09:07:45,578 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 09:07:53,356 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 09:07:53,378 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 09:07:53,393 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 09:08:56,227 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 09:10:02,493 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.89 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 653.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 494.98 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 656.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1172.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.19 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 765.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1170.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1182.65 examples/s]

Map (num_proc=8):   0%|          | 0/219 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 56/219 [00:00<00:00, 379.36 examples/s]
Map (num_proc=8):  63%|######3   | 138/219 [00:00<00:00, 562.34 examples/s]
Map (num_proc=8): 100%|##########| 219/219 [00:00<00:00, 557.25 examples/s]

Filter (num_proc=8):   0%|          | 0/219 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 111/219 [00:00<00:00, 1035.71 examples/s]
Filter (num_proc=8): 100%|##########| 219/219 [00:00<00:00, 1023.50 examples/s]INFO 2025-05-20 09:10:07,121 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 09:10:47,411 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.72 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 878.42 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1427.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1192.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 425.86 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 794.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1329.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1191.24 examples/s]

Map (num_proc=8):   0%|          | 0/117 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 30/117 [00:00<00:00, 232.56 examples/s]
Map (num_proc=8):  88%|########8 | 103/117 [00:00<00:00, 482.02 examples/s]
Map (num_proc=8): 100%|##########| 117/117 [00:00<00:00, 315.85 examples/s]

Filter (num_proc=8):   0%|          | 0/117 [00:00<?, ? examples/s]
Filter (num_proc=8):  64%|######4   | 75/117 [00:00<00:00, 669.79 examples/s]
Filter (num_proc=8): 100%|##########| 117/117 [00:00<00:00, 563.84 examples/s]INFO 2025-05-20 09:10:50,839 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 09:11:20,411 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.24 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 679.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1129.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.15 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 602.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1381.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1196.45 examples/s]

Map (num_proc=8):   0%|          | 0/63 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 16/63 [00:00<00:00, 142.31 examples/s]
Map (num_proc=8):  76%|#######6  | 48/63 [00:00<00:00, 229.76 examples/s]
Map (num_proc=8): 100%|##########| 63/63 [00:00<00:00, 180.54 examples/s]

Filter (num_proc=8):   0%|          | 0/63 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 40/63 [00:00<00:00, 383.45 examples/s]
Filter (num_proc=8): 100%|##########| 63/63 [00:00<00:00, 306.79 examples/s]INFO 2025-05-20 09:11:23,820 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_13_14_3__Using_the_must_gather_tool_for_OpenShift_Virtualization/data_checkpoint_11838b24b90349e0b6d4e133332673b5.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 39.94ba/s]INFO 2025-05-20 09:11:23,964 instructlab.sdg.generate_data:478: Generated 475 samples
INFO 2025-05-20 09:11:24,021 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_6__Uninstalling_OpenShift_Virtualization, generating from scratch
INFO 2025-05-20 09:11:24,021 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 09:11:24,025 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 09:11:24,522 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 09:11:24,536 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 09:11:24,544 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 09:11:33,596 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 09:11:43,599 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 09:11:50,875 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 09:11:50,895 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 09:11:50,911 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 09:13:22,970 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 09:17:19,292 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.88 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 743.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1220.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1197.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 567.90 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1182.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.85 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 722.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1181.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.00 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 654.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.17 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1125.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1199.42 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 425.95 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 632.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1173.39 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1194.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 470.58 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 663.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1354.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1202.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.80 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 588.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1188.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.43 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 752.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 628.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.62 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 519.34 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 638.80 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1454.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1190.66 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 579.77 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 583.87 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.65 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1197.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1184.52 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.72 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 625.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1241.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 550.49 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 595.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 634.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1220.32 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1198.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.34 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 789.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 638.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1421.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.02 examples/s]

Map (num_proc=8):   0%|          | 0/166 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 42/166 [00:00<00:00, 404.67 examples/s]
Map (num_proc=8):  51%|#####     | 84/166 [00:00<00:00, 408.85 examples/s]
Map (num_proc=8): 100%|##########| 166/166 [00:00<00:00, 433.97 examples/s]

Filter (num_proc=8):   0%|          | 0/166 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 84/166 [00:00<00:00, 749.11 examples/s]
Filter (num_proc=8): 100%|##########| 166/166 [00:00<00:00, 774.10 examples/s]INFO 2025-05-20 09:17:35,471 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 09:19:43,436 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.91 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 707.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1184.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 400.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 631.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1088.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.04 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.04 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 814.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1138.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.74 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 690.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 631.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1124.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1186.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.41 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 571.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 622.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 877.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 629.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.96 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 591.94 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1099.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1191.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 665.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.11 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1184.07 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.24 examples/s]

Map (num_proc=8):   0%|          | 0/26 [00:00<?, ? examples/s]
Map (num_proc=8):  42%|####2     | 11/26 [00:00<00:00, 75.87 examples/s]
Map (num_proc=8): 100%|##########| 26/26 [00:00<00:00, 111.25 examples/s]
Map (num_proc=8): 100%|##########| 26/26 [00:00<00:00, 76.67 examples/s] 

Filter (num_proc=8):   0%|          | 0/26 [00:00<?, ? examples/s]
Filter (num_proc=8):  42%|####2     | 11/26 [00:00<00:00, 108.35 examples/s]
Filter (num_proc=8): 100%|##########| 26/26 [00:00<00:00, 124.16 examples/s]INFO 2025-05-20 09:19:53,819 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 09:21:26,418 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.30 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 736.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1188.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.26 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 790.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1332.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.55 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 601.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1176.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 473.35 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 673.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1204.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.19 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 591.02 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.46 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 780.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1434.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 433.67 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 447.65 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1203.53 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1191.66 examples/s]

Map (num_proc=8):   0%|          | 0/219 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 56/219 [00:00<00:00, 354.49 examples/s]
Map (num_proc=8):  51%|#####     | 111/219 [00:00<00:00, 419.95 examples/s]
Map (num_proc=8): 100%|##########| 219/219 [00:00<00:00, 546.37 examples/s]

Filter (num_proc=8):   0%|          | 0/219 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####     | 111/219 [00:00<00:00, 1100.51 examples/s]
Filter (num_proc=8): 100%|##########| 219/219 [00:00<00:00, 1019.33 examples/s]INFO 2025-05-20 09:21:35,754 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_5_6__Uninstalling_OpenShift_Virtualization/data_checkpoint_7e8e293cf5734c14b3eb21b7b546977f.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 24.49ba/s]INFO 2025-05-20 09:21:36,279 instructlab.sdg.generate_data:478: Generated 1886 samples
INFO 2025-05-20 09:21:36,336 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_14_1__Prerequisites, generating from scratch
INFO 2025-05-20 09:21:36,336 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 09:21:36,340 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 09:21:36,690 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 09:21:36,704 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 09:21:36,712 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 09:21:46,791 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 09:21:55,700 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 09:22:04,131 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 09:22:04,151 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 09:22:04,167 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 09:23:10,143 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 09:25:09,175 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 449.88 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 676.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1188.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1180.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 342.32 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 652.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1227.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.44 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 721.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 930.04 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.13 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 632.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1247.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 412.21 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 591.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.75 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1276.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 870.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1252.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.24 examples/s]

Map (num_proc=8):   0%|          | 0/184 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 46/184 [00:00<00:00, 279.97 examples/s]
Map (num_proc=8):  88%|########7 | 161/184 [00:00<00:00, 626.81 examples/s]
Map (num_proc=8): 100%|##########| 184/184 [00:00<00:00, 471.94 examples/s]

Filter (num_proc=8):   0%|          | 0/184 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 92/184 [00:00<00:00, 895.51 examples/s]
Filter (num_proc=8): 100%|##########| 184/184 [00:00<00:00, 866.36 examples/s]INFO 2025-05-20 09:25:17,314 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 09:26:41,318 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 476.76 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 533.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.36 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1109.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.89 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 648.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 596.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1205.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 432.47 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 646.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1214.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1187.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.61 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 667.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 594.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1259.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 316.12 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 682.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1312.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.44 examples/s]

Map (num_proc=8):   0%|          | 0/29 [00:00<?, ? examples/s]
Map (num_proc=8):  41%|####1     | 12/29 [00:00<00:00, 82.14 examples/s]
Map (num_proc=8): 100%|##########| 29/29 [00:00<00:00, 86.56 examples/s]

Filter (num_proc=8):   0%|          | 0/29 [00:00<?, ? examples/s]
Filter (num_proc=8):  55%|#####5    | 16/29 [00:00<00:00, 145.82 examples/s]
Filter (num_proc=8): 100%|##########| 29/29 [00:00<00:00, 140.79 examples/s]INFO 2025-05-20 09:26:48,271 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 09:27:43,185 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 304.34 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 562.65 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 583.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1207.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 454.64 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 576.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1259.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 351.51 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 749.99 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1388.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 342.93 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 604.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1233.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1155.52 examples/s]

Map (num_proc=8):   0%|          | 0/230 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 58/230 [00:00<00:00, 330.47 examples/s]
Map (num_proc=8):  75%|#######5  | 173/230 [00:00<00:00, 619.74 examples/s]
Map (num_proc=8): 100%|##########| 230/230 [00:00<00:00, 560.96 examples/s]

Filter (num_proc=8):   0%|          | 0/230 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 116/230 [00:00<00:00, 1055.75 examples/s]
Filter (num_proc=8): 100%|##########| 230/230 [00:00<00:00, 1052.17 examples/s]INFO 2025-05-20 09:27:49,111 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_14_1__Prerequisites/data_checkpoint_d79f74d8bfe54c879ce5c8881b8af30e.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 31.68ba/s]INFO 2025-05-20 09:27:49,438 instructlab.sdg.generate_data:478: Generated 1064 samples
INFO 2025-05-20 09:27:49,509 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_3_2__Editing_a_boot_order_list_in_the_web_console, generating from scratch
INFO 2025-05-20 09:27:49,509 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 09:27:49,512 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 09:27:49,838 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 09:27:49,851 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 09:27:49,859 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 09:27:55,311 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 09:28:02,769 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 09:28:08,395 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 09:28:08,415 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 09:28:08,430 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 09:29:13,206 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 09:31:25,101 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.17 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 568.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.25 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1196.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.07 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 432.35 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 524.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1289.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.14 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 402.84 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 511.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1216.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1176.95 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 582.54 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 505.18 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1276.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 609.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1130.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1158.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 434.40 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 679.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.97 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1268.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1174.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.11 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 768.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.87 examples/s]

Map (num_proc=8):   0%|          | 0/174 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 44/174 [00:00<00:00, 317.48 examples/s]
Map (num_proc=8):  76%|#######5  | 132/174 [00:00<00:00, 581.14 examples/s]
Map (num_proc=8): 100%|##########| 174/174 [00:00<00:00, 454.85 examples/s]

Filter (num_proc=8):   0%|          | 0/174 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 110/174 [00:00<00:00, 985.01 examples/s]
Filter (num_proc=8): 100%|##########| 174/174 [00:00<00:00, 810.36 examples/s]INFO 2025-05-20 09:31:34,483 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 09:33:01,146 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.27 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 930.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1166.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1181.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 435.22 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 518.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1186.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1159.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.39 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 502.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.27 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 416.01 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 717.67 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1144.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1206.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 636.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.59 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1372.92 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.28 examples/s]

Map (num_proc=8):   0%|          | 0/166 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 42/166 [00:00<00:00, 315.14 examples/s]
Map (num_proc=8):  88%|########7 | 146/166 [00:00<00:00, 522.94 examples/s]
Map (num_proc=8): 100%|##########| 166/166 [00:00<00:00, 427.53 examples/s]

Filter (num_proc=8):   0%|          | 0/166 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 105/166 [00:00<00:00, 899.13 examples/s]
Filter (num_proc=8): 100%|##########| 166/166 [00:00<00:00, 784.26 examples/s]INFO 2025-05-20 09:33:08,120 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 09:34:13,251 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.93 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 667.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1156.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1188.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.90 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 809.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.08 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1270.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.06 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 518.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1188.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.46 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 806.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1247.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1164.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.40 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 593.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1074.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1178.69 examples/s]

Map (num_proc=8):   0%|          | 0/112 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 28/112 [00:00<00:00, 235.78 examples/s]
Map (num_proc=8):  62%|######2   | 70/112 [00:00<00:00, 314.06 examples/s]
Map (num_proc=8): 100%|##########| 112/112 [00:00<00:00, 300.67 examples/s]

Filter (num_proc=8):   0%|          | 0/112 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 70/112 [00:00<00:00, 619.19 examples/s]
Filter (num_proc=8): 100%|##########| 112/112 [00:00<00:00, 537.00 examples/s]INFO 2025-05-20 09:34:20,204 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_3_2__Editing_a_boot_order_list_in_the_web_console/data_checkpoint_c9c4fe990beb4c3285bcc06385525c5a.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 36.60ba/s]INFO 2025-05-20 09:34:20,541 instructlab.sdg.generate_data:478: Generated 1148 samples
INFO 2025-05-20 09:34:20,621 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_16_5__Configuring_PXE_booting_for_virtual_machines, generating from scratch
INFO 2025-05-20 09:34:20,621 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 09:34:20,626 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 09:34:20,628 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 09:34:24,025 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 09:34:24,041 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 09:34:24,049 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 09:34:26,559 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 09:34:26,573 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 09:34:26,582 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 09:34:50,134 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 09:34:50,702 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 09:35:33,277 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 09:35:45,544 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 09:35:55,745 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 09:35:55,775 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 09:35:55,800 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 09:36:15,011 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 09:36:15,041 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 09:36:15,065 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 09:44:05,779 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 09:44:42,827 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 09:54:17,891 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 508.12 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 535.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 582.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1070.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 338.00 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 659.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.14 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1192.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 377.49 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 614.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 609.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1198.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.61 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 500.40 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 735.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 611.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1191.25 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.85 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 713.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1150.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 563.10 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 530.71 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 956.70 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 575.15 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 539.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1181.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 364.27 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 766.58 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.79 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1206.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1179.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 410.52 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 604.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1109.51 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.76 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 562.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1149.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 434.93 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 623.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1257.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 389.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 614.97 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1174.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.89 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 398.21 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 671.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1161.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 404.34 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 633.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1209.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1172.34 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.17 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 546.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 537.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 855.09 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1101.86 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.85 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 592.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1171.49 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 390.07 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 612.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1241.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1173.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 444.66 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 669.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1137.38 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1189.12 examples/s]

Map (num_proc=8):   0%|          | 0/149 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 38/149 [00:00<00:00, 227.93 examples/s]
Map (num_proc=8): 100%|##########| 149/149 [00:00<00:00, 555.21 examples/s]
Map (num_proc=8): 100%|##########| 149/149 [00:00<00:00, 380.11 examples/s]

Filter (num_proc=8):   0%|          | 0/149 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 76/149 [00:00<00:00, 715.24 examples/s]
Filter (num_proc=8): 100%|##########| 149/149 [00:00<00:00, 679.30 examples/s]INFO 2025-05-20 09:54:41,294 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 09:59:38,080 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.51 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 747.35 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.86 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1131.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.57 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.39 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 700.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1152.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.12 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 702.10 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1190.82 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.98 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.88 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 475.37 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.61 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.93 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.03 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 622.46 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.99 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.32 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 570.03 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.68 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1138.89 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 309.84 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 823.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1201.44 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1170.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 292.90 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 530.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 915.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1007.51 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 309.32 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 735.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 936.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 472.34 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 531.41 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1175.50 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1168.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.60 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 472.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1260.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.43 examples/s]

Map (num_proc=8):   0%|          | 0/37 [00:00<?, ? examples/s]
Map (num_proc=8):  27%|##7       | 10/37 [00:00<00:00, 83.87 examples/s]
Map (num_proc=8):  89%|########9 | 33/37 [00:00<00:00, 150.08 examples/s]
Map (num_proc=8): 100%|##########| 37/37 [00:00<00:00, 104.30 examples/s]

Filter (num_proc=8):   0%|          | 0/37 [00:00<?, ? examples/s]
Filter (num_proc=8):  54%|#####4    | 20/37 [00:00<00:00, 185.09 examples/s]
Filter (num_proc=8): 100%|##########| 37/37 [00:00<00:00, 180.93 examples/s]INFO 2025-05-20 09:59:53,120 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 10:04:12,167 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 375.61 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 631.70 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1131.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 378.36 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 602.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.53 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 634.36 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1117.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:01, 200.12 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 425.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 501.48 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1170.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.26 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.00 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 658.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1202.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1171.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.14 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 568.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1196.80 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 335.95 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 563.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1162.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 353.56 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 651.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 863.37 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 352.20 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 769.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.64 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.79 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 756.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 592.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 949.85 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.16 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 619.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.73 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1130.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.13 examples/s]

Map (num_proc=8):   0%|          | 0/80 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 20/80 [00:00<00:00, 155.38 examples/s]
Map (num_proc=8):  50%|#####     | 40/80 [00:00<00:00, 175.68 examples/s]
Map (num_proc=8): 100%|##########| 80/80 [00:00<00:00, 215.91 examples/s]

Filter (num_proc=8):   0%|          | 0/80 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 40/80 [00:00<00:00, 375.21 examples/s]
Filter (num_proc=8): 100%|##########| 80/80 [00:00<00:00, 370.73 examples/s]INFO 2025-05-20 10:04:26,054 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_16_5__Configuring_PXE_booting_for_virtual_machines/data_checkpoint_aa8c2c62b60b436b80c60ba20fd3014c.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 21.72ba/s]INFO 2025-05-20 10:07:33,139 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 453.82 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 701.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.16 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.47 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 367.23 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 709.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 614.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1054.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.93 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 691.59 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.72 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.63 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.26 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 735.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.83 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1087.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 361.44 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 751.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1242.95 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 488.39 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 531.85 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1195.31 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 368.94 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 599.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1169.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.91 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.86 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 701.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1236.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1163.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 392.26 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 955.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1219.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 458.91 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 661.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.02 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.58 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.16 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 830.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 605.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1197.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.17 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 467.87 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 699.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 591.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 862.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 436.52 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 527.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1277.06 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 344.41 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 679.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1151.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 447.23 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 539.12 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1179.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.18 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 440.21 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 854.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.96 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1267.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.27 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 431.52 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 688.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.33 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1178.28 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 381.40 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 778.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.70 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 415.87 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 735.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.54 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1252.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 372.13 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 608.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 623.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 922.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.97 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.63 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 933.26 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.01 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1165.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 379.92 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 617.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.09 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 862.54 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.94 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 450.28 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 670.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.21 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 956.78 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.42 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 639.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 612.62 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1159.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.10 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 371.90 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 718.19 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 618.41 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 918.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1156.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 429.21 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 610.60 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.43 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1102.90 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1169.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 484.05 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 514.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 617.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1192.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1150.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 432.07 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 815.64 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1149.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1137.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.66 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 500.01 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 940.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1146.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.24 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 728.96 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.04 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1135.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1153.25 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 441.67 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 531.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.65 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 385.37 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 776.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.56 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1154.66 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.03 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 588.51 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1097.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.96 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 638.07 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 624.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1151.83 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 407.52 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 658.74 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 619.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 421.76 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 487.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.77 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1147.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 401.76 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 713.45 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 608.29 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.41 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 438.00 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 528.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 599.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1151.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 405.28 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 620.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.95 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1188.46 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.81 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 499.05 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 521.93 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 607.51 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 584.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 991.09 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 362.33 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 772.50 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 917.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.08 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 551.30 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 546.86 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 832.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 601.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 925.87 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1094.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 422.53 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 725.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.50 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 868.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1160.48 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 469.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 905.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 627.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1194.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1166.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 631.11 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 538.72 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 613.85 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1239.60 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.90 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 470.57 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 874.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 615.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1221.12 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.49 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 589.55 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 480.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1211.91 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1144.78 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 581.92 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 559.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 620.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 925.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 501.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 625.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1149.68 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 471.90 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 554.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 626.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1145.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 484.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 888.83 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 616.30 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.68 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 624.48 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 545.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1186.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.24 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 519.47 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 584.48 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.18 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 935.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.75 examples/s]

Map (num_proc=8):   0%|          | 0/165 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 42/165 [00:00<00:00, 294.61 examples/s]
Map (num_proc=8):  88%|########7 | 145/165 [00:00<00:00, 594.16 examples/s]
Map (num_proc=8): 100%|##########| 165/165 [00:00<00:00, 428.31 examples/s]

Filter (num_proc=8):   0%|          | 0/165 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 63/165 [00:00<00:00, 626.50 examples/s]
Filter (num_proc=8): 100%|##########| 165/165 [00:00<00:00, 752.87 examples/s]INFO 2025-05-20 10:08:39,652 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 10:16:57,073 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 376.34 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 905.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.24 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1135.79 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.50 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 551.25 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 513.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 799.53 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 566.69 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1269.03 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 609.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 590.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 919.20 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.19 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 492.17 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.66 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1212.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 337.07 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 716.49 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1087.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1134.38 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.15 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 862.55 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1121.16 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.56 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 355.59 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 623.69 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 583.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 956.71 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1148.55 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.38 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 789.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.26 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1154.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 393.25 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 539.61 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 556.64 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1238.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.32 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.58 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 631.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.44 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 952.72 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1133.06 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 360.52 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 693.98 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.35 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1138.76 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.35 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 387.17 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 568.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.32 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 907.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.40 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 406.44 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 549.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 544.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 1271.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1135.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.81 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 694.00 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.40 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 923.22 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.11 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 611.64 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 422.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1090.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.22 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 373.23 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 672.36 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 593.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1237.57 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1138.76 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.01 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 615.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 902.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.33 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 363.79 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 604.28 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.87 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 950.05 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.02 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 339.44 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 689.05 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 569.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1183.63 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1103.53 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 440.66 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 582.39 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 589.58 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 884.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.29 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 321.05 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 669.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.46 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 942.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1111.85 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 350.66 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 597.47 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.24 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.21 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 343.43 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 713.21 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.61 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1227.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 358.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 587.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 566.31 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 845.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1104.69 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.55 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 759.88 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.39 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1112.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1123.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 386.27 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 470.40 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 552.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1242.19 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.45 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 558.53 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 510.06 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 751.68 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 558.23 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 914.21 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.03 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 428.10 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 468.11 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 562.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1091.26 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1130.92 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 335.47 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 579.78 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 558.60 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1166.40 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 384.42 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 606.82 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.63 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1192.14 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1140.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 399.06 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 638.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 570.13 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1123.30 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.11 examples/s]

Map (num_proc=8):   0%|          | 0/218 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 56/218 [00:00<00:00, 341.84 examples/s]
Map (num_proc=8): 100%|##########| 218/218 [00:00<00:00, 700.86 examples/s]
Map (num_proc=8): 100%|##########| 218/218 [00:00<00:00, 516.17 examples/s]

Filter (num_proc=8):   0%|          | 0/218 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 110/218 [00:00<00:00, 1005.80 examples/s]
Filter (num_proc=8): 100%|##########| 218/218 [00:00<00:00, 959.34 examples/s] INFO 2025-05-20 10:17:37,904 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 10:23:35,168 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 350.34 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 606.08 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 586.90 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1236.64 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.71 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.20 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 614.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 575.06 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 902.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1121.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 449.96 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 541.23 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 597.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1232.23 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.62 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 350.07 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 652.66 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 568.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 957.35 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.82 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.37 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 491.92 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1162.96 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.37 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 413.82 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 689.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.12 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1142.11 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.39 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 330.44 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 694.31 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 600.78 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1001.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1088.43 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 543.61 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 530.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 801.75 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.27 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 892.94 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1118.20 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 396.47 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 470.79 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.84 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1164.75 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1139.88 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 356.05 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 871.25 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 573.05 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1225.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.87 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 498.30 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.94 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 951.42 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1141.01 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 365.54 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 750.81 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.07 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1250.56 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1119.13 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 441.53 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 584.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1163.47 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1128.44 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 359.88 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 583.91 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 584.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1252.69 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1147.36 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 370.27 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 691.43 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 604.38 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1172.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.19 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 447.73 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 662.89 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 595.91 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 856.62 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.74 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.54 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 843.95 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 602.10 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1156.59 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1143.75 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 326.96 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 851.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 577.81 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 935.45 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.05 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 342.78 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 886.15 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 585.98 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1229.15 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1129.23 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 411.54 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 667.13 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 578.52 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 943.77 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1142.67 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 382.84 examples/s]
Map (num_proc=8):  88%|########7 | 224/256 [00:00<00:00, 820.33 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 606.34 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1112.17 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1112.84 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 308.81 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 646.52 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.67 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1159.33 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1126.79 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 344.24 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 627.56 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 572.76 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1231.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1131.59 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 32/256 [00:00<00:00, 312.73 examples/s]
Map (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 448.10 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 627.62 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.15 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 959.00 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1127.00 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 484.60 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 369.29 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 560.89 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1235.48 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1122.15 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 313.51 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 710.24 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 581.22 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 918.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1116.65 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 423.12 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 456.14 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 567.55 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1158.84 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1124.60 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 410.21 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 661.73 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 580.71 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1213.81 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1132.41 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 307.38 examples/s]
Map (num_proc=8):  75%|#######5  | 192/256 [00:00<00:00, 673.20 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 576.74 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 900.73 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1114.70 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 437.81 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 517.32 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 588.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1217.43 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1125.75 examples/s]

Map (num_proc=8):   0%|          | 0/11 [00:00<?, ? examples/s]
Map (num_proc=8):  36%|###6      | 4/11 [00:00<00:00, 33.18 examples/s]
Map (num_proc=8):  91%|######### | 10/11 [00:00<00:00, 42.46 examples/s]
Map (num_proc=8): 100%|##########| 11/11 [00:00<00:00, 31.33 examples/s]

Filter (num_proc=8):   0%|          | 0/11 [00:00<?, ? examples/s]
Filter (num_proc=8):  64%|######3   | 7/11 [00:00<00:00, 62.30 examples/s]
Filter (num_proc=8): 100%|##########| 11/11 [00:00<00:00, 52.47 examples/s]INFO 2025-05-20 10:24:14,445 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_9_16_5__Configuring_PXE_booting_for_virtual_machines/data_checkpoint_453ca033eff043c0a5c3b3cbec744d0f.jsonl


Creating json from Arrow format:   0%|          | 0/7 [00:00<?, ?ba/s]
Creating json from Arrow format:  43%|####2     | 3/7 [00:00<00:00, 23.66ba/s]
Creating json from Arrow format:  86%|########5 | 6/7 [00:00<00:00, 24.16ba/s]
Creating json from Arrow format: 100%|##########| 7/7 [00:00<00:00, 26.52ba/s]INFO 2025-05-20 10:24:16,985 instructlab.sdg.generate_data:478: Generated 8322 samples
INFO 2025-05-20 10:24:17,056 instructlab.sdg.checkpointing:59: No existing checkpoints found in /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_6_4_1__Monitoring_OpenShift_Virtualization_upgrade_status, generating from scratch
INFO 2025-05-20 10:24:17,056 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:24:17,059 instructlab.sdg.pipeline:199: Running block: router
INFO 2025-05-20 10:24:17,282 instructlab.sdg.pipeline:199: Running block: SetClassifierValue
INFO 2025-05-20 10:24:17,295 instructlab.sdg.pipeline:199: Running block: duplicate_document_col
INFO 2025-05-20 10:24:17,302 instructlab.sdg.pipeline:199: Running block: gen_detailed_summary
INFO 2025-05-20 10:24:22,586 instructlab.sdg.pipeline:199: Running block: gen_atomic_facts
INFO 2025-05-20 10:24:29,056 instructlab.sdg.pipeline:199: Running block: gen_extractive_summary
INFO 2025-05-20 10:24:35,464 instructlab.sdg.pipeline:199: Running block: flatten_summary_columns
INFO 2025-05-20 10:24:35,483 instructlab.sdg.pipeline:199: Running block: rename_to_document_column
INFO 2025-05-20 10:24:35,498 instructlab.sdg.pipeline:199: Running block: knowledge generation
INFO 2025-05-20 10:25:27,397 instructlab.sdg.pipeline:199: Running block: eval_faithfulness_qa_pair
INFO 2025-05-20 10:26:20,793 instructlab.sdg.pipeline:199: Running block: filter_faithfulness


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 409.68 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 644.22 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 621.82 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1168.18 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1157.99 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 391.18 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 609.09 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 610.49 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###7      | 96/256 [00:00<00:00, 954.86 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1167.60 examples/s]

Map (num_proc=8):   0%|          | 0/244 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 62/244 [00:00<00:00, 377.19 examples/s]
Map (num_proc=8):  63%|######3   | 154/244 [00:00<00:00, 568.85 examples/s]
Map (num_proc=8): 100%|##########| 244/244 [00:00<00:00, 590.47 examples/s]

Filter (num_proc=8):   0%|          | 0/244 [00:00<?, ? examples/s]
Filter (num_proc=8):  38%|###8      | 93/244 [00:00<00:00, 926.37 examples/s]
Filter (num_proc=8): 100%|##########| 244/244 [00:00<00:00, 1104.13 examples/s]INFO 2025-05-20 10:26:24,367 instructlab.sdg.pipeline:199: Running block: eval_relevancy_qa_pair
INFO 2025-05-20 10:27:02,284 instructlab.sdg.pipeline:199: Running block: filter_relevancy


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 403.68 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 640.04 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 587.20 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 586.01 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1095.16 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 418.51 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 514.76 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 598.19 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1274.08 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1177.30 examples/s]

Map (num_proc=8):   0%|          | 0/73 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##6       | 19/73 [00:00<00:00, 144.87 examples/s]
Map (num_proc=8):  88%|########7 | 64/73 [00:00<00:00, 269.35 examples/s]
Map (num_proc=8): 100%|##########| 73/73 [00:00<00:00, 205.87 examples/s]

Filter (num_proc=8):   0%|          | 0/73 [00:00<?, ? examples/s]
Filter (num_proc=8):  63%|######3   | 46/73 [00:00<00:00, 424.33 examples/s]
Filter (num_proc=8): 100%|##########| 73/73 [00:00<00:00, 345.47 examples/s]INFO 2025-05-20 10:27:05,827 instructlab.sdg.pipeline:199: Running block: eval_verify_question
INFO 2025-05-20 10:27:35,223 instructlab.sdg.pipeline:199: Running block: filter_verify_question


Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 417.46 examples/s]
Map (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 521.27 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 579.88 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1158.55 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1165.72 examples/s]

Map (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Map (num_proc=8):  25%|##5       | 64/256 [00:00<00:00, 430.98 examples/s]
Map (num_proc=8):  62%|######2   | 160/256 [00:00<00:00, 583.77 examples/s]
Map (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 603.42 examples/s]

Filter (num_proc=8):   0%|          | 0/256 [00:00<?, ? examples/s]
Filter (num_proc=8):  50%|#####     | 128/256 [00:00<00:00, 1273.58 examples/s]
Filter (num_proc=8): 100%|##########| 256/256 [00:00<00:00, 1175.65 examples/s]

Map (num_proc=8):   0%|          | 0/47 [00:00<?, ? examples/s]
Map (num_proc=8):  26%|##5       | 12/47 [00:00<00:00, 103.22 examples/s]
Map (num_proc=8):  89%|########9 | 42/47 [00:00<00:00, 201.32 examples/s]
Map (num_proc=8): 100%|##########| 47/47 [00:00<00:00, 137.72 examples/s]

Filter (num_proc=8):   0%|          | 0/47 [00:00<?, ? examples/s]
Filter (num_proc=8):  51%|#####1    | 24/47 [00:00<00:00, 232.33 examples/s]
Filter (num_proc=8): 100%|##########| 47/47 [00:00<00:00, 224.99 examples/s]INFO 2025-05-20 10:27:38,733 instructlab.sdg.checkpointing:44: Saving checkpoint to /var/home/vkadlec/.local/share/instructlab/datasets/checkpoints/knowledge_technology_cloud-computing_6_4_1__Monitoring_OpenShift_Virtualization_upgrade_status/data_checkpoint_3003c9406e6e461d9b6bcb4e5ae63e82.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 38.81ba/s]INFO 2025-05-20 10:27:38,893 instructlab.sdg.generate_data:478: Generated 463 samples
INFO 2025-05-20 10:27:38,908 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:27:38,914 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/57 [00:00<?, ? examples/s]
Filter: 100%|##########| 57/57 [00:00<00:00, 17711.91 examples/s]

Filter:   0%|          | 0/57 [00:00<?, ? examples/s]
Filter: 100%|##########| 57/57 [00:00<00:00, 12745.25 examples/s]

Flattening the indices:   0%|          | 0/57 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 57/57 [00:00<00:00, 13313.77 examples/s]

Map:   0%|          | 0/57 [00:00<?, ? examples/s]
Map: 100%|##########| 57/57 [00:00<00:00, 5757.11 examples/s]

Map:   0%|          | 0/57 [00:00<?, ? examples/s]
Map: 100%|##########| 57/57 [00:00<00:00, 5539.15 examples/s]

Map:   0%|          | 0/57 [00:00<?, ? examples/s]
Map: 100%|##########| 57/57 [00:00<00:00, 5427.49 examples/s]

Filter:   0%|          | 0/57 [00:00<?, ? examples/s]
Filter: 100%|##########| 57/57 [00:00<00:00, 16886.24 examples/s]

Filter:   0%|          | 0/57 [00:00<?, ? examples/s]
Filter: 100%|##########| 57/57 [00:00<00:00, 11581.42 examples/s]

Filter:   0%|          | 0/55 [00:00<?, ? examples/s]
Filter: 100%|##########| 55/55 [00:00<00:00, 11573.11 examples/s]

Flattening the indices:   0%|          | 0/55 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 55/55 [00:00<00:00, 11291.57 examples/s]

Casting to class labels:   0%|          | 0/55 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 55/55 [00:00<00:00, 5773.95 examples/s]INFO 2025-05-20 10:27:49,193 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_6_2_1__About_workload_updates.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 434.60ba/s]INFO 2025-05-20 10:27:49,196 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_6_2_1__About_workload_updates_task.yaml
INFO 2025-05-20 10:27:49,208 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:27:49,213 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/153 [00:00<?, ? examples/s]
Filter: 100%|##########| 153/153 [00:00<00:00, 32168.46 examples/s]

Filter:   0%|          | 0/153 [00:00<?, ? examples/s]
Filter: 100%|##########| 153/153 [00:00<00:00, 19132.09 examples/s]

Flattening the indices:   0%|          | 0/153 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 153/153 [00:00<00:00, 28924.93 examples/s]

Map:   0%|          | 0/153 [00:00<?, ? examples/s]
Map: 100%|##########| 153/153 [00:00<00:00, 8174.36 examples/s]

Map:   0%|          | 0/153 [00:00<?, ? examples/s]
Map: 100%|##########| 153/153 [00:00<00:00, 7648.82 examples/s]

Map:   0%|          | 0/153 [00:00<?, ? examples/s]
Map: 100%|##########| 153/153 [00:00<00:00, 7473.08 examples/s]

Filter:   0%|          | 0/153 [00:00<?, ? examples/s]
Filter: 100%|##########| 153/153 [00:00<00:00, 26755.41 examples/s]

Filter:   0%|          | 0/153 [00:00<?, ? examples/s]
Filter: 100%|##########| 153/153 [00:00<00:00, 14971.62 examples/s]

Filter:   0%|          | 0/151 [00:00<?, ? examples/s]
Filter: 100%|##########| 151/151 [00:00<00:00, 15261.20 examples/s]

Flattening the indices:   0%|          | 0/151 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 151/151 [00:00<00:00, 23538.98 examples/s]

Casting to class labels:   0%|          | 0/151 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 151/151 [00:00<00:00, 7992.07 examples/s]INFO 2025-05-20 10:28:05,941 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_1_6__About_RunStrategies_for_virtual_machines.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 214.24ba/s]INFO 2025-05-20 10:28:05,946 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1_6__About_RunStrategies_for_virtual_machines_task.yaml
INFO 2025-05-20 10:28:05,955 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:28:05,961 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/25 [00:00<?, ? examples/s]
Filter: 100%|##########| 25/25 [00:00<00:00, 7625.45 examples/s]

Filter:   0%|          | 0/25 [00:00<?, ? examples/s]
Filter: 100%|##########| 25/25 [00:00<00:00, 8510.48 examples/s]

Flattening the indices:   0%|          | 0/25 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 25/25 [00:00<00:00, 6952.50 examples/s]

Map:   0%|          | 0/25 [00:00<?, ? examples/s]
Map: 100%|##########| 25/25 [00:00<00:00, 4008.32 examples/s]

Map:   0%|          | 0/25 [00:00<?, ? examples/s]
Map: 100%|##########| 25/25 [00:00<00:00, 3810.37 examples/s]

Map:   0%|          | 0/25 [00:00<?, ? examples/s]
Map: 100%|##########| 25/25 [00:00<00:00, 3689.18 examples/s]

Filter:   0%|          | 0/25 [00:00<?, ? examples/s]
Filter: 100%|##########| 25/25 [00:00<00:00, 9865.24 examples/s]

Filter:   0%|          | 0/24 [00:00<?, ? examples/s]
Filter: 100%|##########| 24/24 [00:00<00:00, 7333.23 examples/s]

Filter:   0%|          | 0/24 [00:00<?, ? examples/s]
Filter: 100%|##########| 24/24 [00:00<00:00, 7290.22 examples/s]

Flattening the indices:   0%|          | 0/24 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 24/24 [00:00<00:00, 5503.43 examples/s]

Casting to class labels:   0%|          | 0/24 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 24/24 [00:00<00:00, 3691.90 examples/s]INFO 2025-05-20 10:28:12,599 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_3_1__About_virtual_machine_events.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 768.47ba/s]INFO 2025-05-20 10:28:12,601 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_3_1__About_virtual_machine_events_task.yaml
INFO 2025-05-20 10:28:12,610 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:28:12,615 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 16038.69 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 11546.42 examples/s]

Flattening the indices:   0%|          | 0/43 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 43/43 [00:00<00:00, 10997.26 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 5223.60 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 4955.35 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 4796.55 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 14372.07 examples/s]

Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 9838.46 examples/s]

Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 9782.49 examples/s]

Flattening the indices:   0%|          | 0/41 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 41/41 [00:00<00:00, 8920.81 examples/s]

Casting to class labels:   0%|          | 0/41 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 41/41 [00:00<00:00, 5052.34 examples/s]INFO 2025-05-20 10:28:25,248 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_8_5__Libguestfs_tools_and_virtctl_guestfs.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 485.79ba/s]INFO 2025-05-20 10:28:25,251 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_8_5__Libguestfs_tools_and_virtctl_guestfs_task.yaml
INFO 2025-05-20 10:28:25,260 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:28:25,265 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/63 [00:00<?, ? examples/s]
Filter: 100%|##########| 63/63 [00:00<00:00, 20531.56 examples/s]

Filter:   0%|          | 0/63 [00:00<?, ? examples/s]
Filter: 100%|##########| 63/63 [00:00<00:00, 13669.30 examples/s]

Flattening the indices:   0%|          | 0/63 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 63/63 [00:00<00:00, 15147.97 examples/s]

Map:   0%|          | 0/63 [00:00<?, ? examples/s]
Map: 100%|##########| 63/63 [00:00<00:00, 6183.83 examples/s]

Map:   0%|          | 0/63 [00:00<?, ? examples/s]
Map: 100%|##########| 63/63 [00:00<00:00, 5988.20 examples/s]

Map:   0%|          | 0/63 [00:00<?, ? examples/s]
Map: 100%|##########| 63/63 [00:00<00:00, 5838.68 examples/s]

Filter:   0%|          | 0/63 [00:00<?, ? examples/s]
Filter: 100%|##########| 63/63 [00:00<00:00, 18376.88 examples/s]

Filter:   0%|          | 0/63 [00:00<?, ? examples/s]
Filter: 100%|##########| 63/63 [00:00<00:00, 11651.36 examples/s]

Filter:   0%|          | 0/63 [00:00<?, ? examples/s]
Filter: 100%|##########| 63/63 [00:00<00:00, 11839.29 examples/s]

Casting to class labels:   0%|          | 0/63 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 63/63 [00:00<00:00, 6192.67 examples/s]INFO 2025-05-20 10:28:36,085 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_12_2__Removing_a_secret__config_map__or_service_account_from_a_virtual_machine.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 399.95ba/s]INFO 2025-05-20 10:28:36,088 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_12_2__Removing_a_secret__config_map__or_service_account_from_a_virtual_machine_task.yaml
INFO 2025-05-20 10:28:36,115 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:28:36,121 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge
INFO 2025-05-20 10:28:36,125 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/849 [00:00<?, ? examples/s]
Filter: 100%|##########| 849/849 [00:00<00:00, 48776.32 examples/s]

Filter:   0%|          | 0/849 [00:00<?, ? examples/s]
Filter: 100%|##########| 849/849 [00:00<00:00, 22710.52 examples/s]

Flattening the indices:   0%|          | 0/849 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 849/849 [00:00<00:00, 52476.70 examples/s]

Map:   0%|          | 0/849 [00:00<?, ? examples/s]
Map: 100%|##########| 849/849 [00:00<00:00, 9835.83 examples/s]

Map:   0%|          | 0/849 [00:00<?, ? examples/s]
Map: 100%|##########| 849/849 [00:00<00:00, 9121.42 examples/s]

Map:   0%|          | 0/849 [00:00<?, ? examples/s]
Map: 100%|##########| 849/849 [00:00<00:00, 8927.81 examples/s]

Filter:   0%|          | 0/849 [00:00<?, ? examples/s]
Filter: 100%|##########| 849/849 [00:00<00:00, 35475.90 examples/s]

Filter:   0%|          | 0/848 [00:00<?, ? examples/s]
Filter: 100%|##########| 848/848 [00:00<00:00, 17632.30 examples/s]

Filter:   0%|          | 0/830 [00:00<?, ? examples/s]
Filter: 100%|##########| 830/830 [00:00<00:00, 17693.98 examples/s]

Flattening the indices:   0%|          | 0/830 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 830/830 [00:00<00:00, 44223.48 examples/s]

Casting to class labels:   0%|          | 0/830 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 830/830 [00:00<00:00, 9535.22 examples/s]INFO 2025-05-20 10:29:24,957 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_16_10__Configuring_vGPU_passthrough.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 42.10ba/s]INFO 2025-05-20 10:29:24,982 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_16_10__Configuring_vGPU_passthrough_task.yaml
INFO 2025-05-20 10:29:24,992 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:29:24,997 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/40 [00:00<?, ? examples/s]
Filter: 100%|##########| 40/40 [00:00<00:00, 15512.91 examples/s]

Filter:   0%|          | 0/40 [00:00<?, ? examples/s]
Filter: 100%|##########| 40/40 [00:00<00:00, 10992.80 examples/s]

Flattening the indices:   0%|          | 0/40 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 40/40 [00:00<00:00, 10187.77 examples/s]

Map:   0%|          | 0/40 [00:00<?, ? examples/s]
Map: 100%|##########| 40/40 [00:00<00:00, 5119.38 examples/s]

Map:   0%|          | 0/40 [00:00<?, ? examples/s]
Map: 100%|##########| 40/40 [00:00<00:00, 4843.73 examples/s]

Map:   0%|          | 0/40 [00:00<?, ? examples/s]
Map: 100%|##########| 40/40 [00:00<00:00, 4825.75 examples/s]

Filter:   0%|          | 0/40 [00:00<?, ? examples/s]
Filter: 100%|##########| 40/40 [00:00<00:00, 13988.01 examples/s]

Filter:   0%|          | 0/40 [00:00<?, ? examples/s]
Filter: 100%|##########| 40/40 [00:00<00:00, 9811.24 examples/s]

Filter:   0%|          | 0/40 [00:00<?, ? examples/s]
Filter: 100%|##########| 40/40 [00:00<00:00, 9909.76 examples/s]

Casting to class labels:   0%|          | 0/40 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 40/40 [00:00<00:00, 5059.63 examples/s]INFO 2025-05-20 10:29:33,253 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_4_3_2__Storage.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 556.42ba/s]INFO 2025-05-20 10:29:33,255 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_3_2__Storage_task.yaml
INFO 2025-05-20 10:29:33,264 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:29:33,269 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 18589.91 examples/s]

Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 13227.31 examples/s]

Flattening the indices:   0%|          | 0/53 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 53/53 [00:00<00:00, 13277.08 examples/s]

Map:   0%|          | 0/53 [00:00<?, ? examples/s]
Map: 100%|##########| 53/53 [00:00<00:00, 5789.62 examples/s]

Map:   0%|          | 0/53 [00:00<?, ? examples/s]
Map: 100%|##########| 53/53 [00:00<00:00, 5529.12 examples/s]

Map:   0%|          | 0/53 [00:00<?, ? examples/s]
Map: 100%|##########| 53/53 [00:00<00:00, 5443.01 examples/s]

Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 15753.53 examples/s]

Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 10975.52 examples/s]

Filter:   0%|          | 0/51 [00:00<?, ? examples/s]
Filter: 100%|##########| 51/51 [00:00<00:00, 10851.74 examples/s]

Flattening the indices:   0%|          | 0/51 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 51/51 [00:00<00:00, 10361.82 examples/s]

Casting to class labels:   0%|          | 0/51 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 51/51 [00:00<00:00, 5526.81 examples/s]INFO 2025-05-20 10:29:43,604 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_5_4_1__Prerequisites.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 425.95ba/s]INFO 2025-05-20 10:29:43,607 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_4_1__Prerequisites_task.yaml
INFO 2025-05-20 10:29:43,617 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:29:43,621 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/56 [00:00<?, ? examples/s]
Filter: 100%|##########| 56/56 [00:00<00:00, 19141.15 examples/s]

Filter:   0%|          | 0/56 [00:00<?, ? examples/s]
Filter: 100%|##########| 56/56 [00:00<00:00, 13383.53 examples/s]

Flattening the indices:   0%|          | 0/56 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 56/56 [00:00<00:00, 14075.69 examples/s]

Map:   0%|          | 0/56 [00:00<?, ? examples/s]
Map: 100%|##########| 56/56 [00:00<00:00, 6033.11 examples/s]

Map:   0%|          | 0/56 [00:00<?, ? examples/s]
Map: 100%|##########| 56/56 [00:00<00:00, 5650.12 examples/s]

Map:   0%|          | 0/56 [00:00<?, ? examples/s]
Map: 100%|##########| 56/56 [00:00<00:00, 5730.62 examples/s]

Filter:   0%|          | 0/56 [00:00<?, ? examples/s]
Filter: 100%|##########| 56/56 [00:00<00:00, 17001.88 examples/s]

Filter:   0%|          | 0/56 [00:00<?, ? examples/s]
Filter: 100%|##########| 56/56 [00:00<00:00, 11327.21 examples/s]

Filter:   0%|          | 0/54 [00:00<?, ? examples/s]
Filter: 100%|##########| 54/54 [00:00<00:00, 11257.08 examples/s]

Flattening the indices:   0%|          | 0/54 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 54/54 [00:00<00:00, 11488.91 examples/s]

Casting to class labels:   0%|          | 0/54 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 54/54 [00:00<00:00, 5742.71 examples/s]INFO 2025-05-20 10:29:52,881 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_6_2__Restarting_a_virtual_machine.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 473.29ba/s]INFO 2025-05-20 10:29:52,884 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_6_2__Restarting_a_virtual_machine_task.yaml
INFO 2025-05-20 10:29:52,895 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:29:52,899 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/103 [00:00<?, ? examples/s]
Filter: 100%|##########| 103/103 [00:00<00:00, 27580.01 examples/s]

Filter:   0%|          | 0/103 [00:00<?, ? examples/s]
Filter: 100%|##########| 103/103 [00:00<00:00, 17525.18 examples/s]

Flattening the indices:   0%|          | 0/103 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 103/103 [00:00<00:00, 21347.70 examples/s]

Map:   0%|          | 0/103 [00:00<?, ? examples/s]
Map: 100%|##########| 103/103 [00:00<00:00, 7346.67 examples/s]

Map:   0%|          | 0/103 [00:00<?, ? examples/s]
Map: 100%|##########| 103/103 [00:00<00:00, 7037.54 examples/s]

Map:   0%|          | 0/103 [00:00<?, ? examples/s]
Map: 100%|##########| 103/103 [00:00<00:00, 6758.02 examples/s]

Filter:   0%|          | 0/103 [00:00<?, ? examples/s]
Filter: 100%|##########| 103/103 [00:00<00:00, 23123.34 examples/s]

Filter:   0%|          | 0/103 [00:00<?, ? examples/s]
Filter: 100%|##########| 103/103 [00:00<00:00, 13957.52 examples/s]

Filter:   0%|          | 0/102 [00:00<?, ? examples/s]
Filter: 100%|##########| 102/102 [00:00<00:00, 13886.62 examples/s]

Flattening the indices:   0%|          | 0/102 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 102/102 [00:00<00:00, 18556.45 examples/s]

Casting to class labels:   0%|          | 0/102 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 102/102 [00:00<00:00, 7228.02 examples/s]INFO 2025-05-20 10:30:05,475 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_5_2_2__Example_manifests.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 286.07ba/s]INFO 2025-05-20 10:30:05,479 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_2_2__Example_manifests_task.yaml
INFO 2025-05-20 10:30:05,489 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:30:05,494 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/62 [00:00<?, ? examples/s]
Filter: 100%|##########| 62/62 [00:00<00:00, 20640.28 examples/s]

Filter:   0%|          | 0/61 [00:00<?, ? examples/s]
Filter: 100%|##########| 61/61 [00:00<00:00, 14027.00 examples/s]

Flattening the indices:   0%|          | 0/61 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 61/61 [00:00<00:00, 15185.03 examples/s]

Map:   0%|          | 0/61 [00:00<?, ? examples/s]
Map: 100%|##########| 61/61 [00:00<00:00, 6287.54 examples/s]

Map:   0%|          | 0/61 [00:00<?, ? examples/s]
Map: 100%|##########| 61/61 [00:00<00:00, 5945.91 examples/s]

Map:   0%|          | 0/61 [00:00<?, ? examples/s]
Map: 100%|##########| 61/61 [00:00<00:00, 5799.67 examples/s]

Filter:   0%|          | 0/61 [00:00<?, ? examples/s]
Filter: 100%|##########| 61/61 [00:00<00:00, 18045.74 examples/s]

Filter:   0%|          | 0/59 [00:00<?, ? examples/s]
Filter: 100%|##########| 59/59 [00:00<00:00, 11726.48 examples/s]

Filter:   0%|          | 0/59 [00:00<?, ? examples/s]
Filter: 100%|##########| 59/59 [00:00<00:00, 11622.94 examples/s]

Flattening the indices:   0%|          | 0/59 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 59/59 [00:00<00:00, 11901.88 examples/s]

Casting to class labels:   0%|          | 0/59 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 59/59 [00:00<00:00, 6118.08 examples/s]INFO 2025-05-20 10:30:16,438 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_2_2__Viewing_virtual_machine_logs_in_the_web_console.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 443.98ba/s]INFO 2025-05-20 10:30:16,441 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_2_2__Viewing_virtual_machine_logs_in_the_web_console_task.yaml
INFO 2025-05-20 10:30:16,450 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:30:16,455 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/54 [00:00<?, ? examples/s]
Filter: 100%|##########| 54/54 [00:00<00:00, 19114.90 examples/s]

Filter:   0%|          | 0/54 [00:00<?, ? examples/s]
Filter: 100%|##########| 54/54 [00:00<00:00, 13205.78 examples/s]

Flattening the indices:   0%|          | 0/54 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 54/54 [00:00<00:00, 13221.97 examples/s]

Map:   0%|          | 0/54 [00:00<?, ? examples/s]
Map: 100%|##########| 54/54 [00:00<00:00, 5808.99 examples/s]

Map:   0%|          | 0/54 [00:00<?, ? examples/s]
Map: 100%|##########| 54/54 [00:00<00:00, 5598.21 examples/s]

Map:   0%|          | 0/54 [00:00<?, ? examples/s]
Map: 100%|##########| 54/54 [00:00<00:00, 5447.54 examples/s]

Filter:   0%|          | 0/54 [00:00<?, ? examples/s]
Filter: 100%|##########| 54/54 [00:00<00:00, 16705.44 examples/s]

Filter:   0%|          | 0/54 [00:00<?, ? examples/s]
Filter: 100%|##########| 54/54 [00:00<00:00, 11122.74 examples/s]

Filter:   0%|          | 0/54 [00:00<?, ? examples/s]
Filter: 100%|##########| 54/54 [00:00<00:00, 11081.38 examples/s]

Casting to class labels:   0%|          | 0/54 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 54/54 [00:00<00:00, 5735.73 examples/s]INFO 2025-05-20 10:30:28,378 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_12__Managing_config_maps__secrets__and_service_accounts_in_virtual_machines.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 466.19ba/s]INFO 2025-05-20 10:30:28,380 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_12__Managing_config_maps__secrets__and_service_accounts_in_virtual_machines_task.yaml
INFO 2025-05-20 10:30:28,390 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:30:28,394 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 15967.69 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 11353.80 examples/s]

Flattening the indices:   0%|          | 0/43 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 43/43 [00:00<00:00, 11381.02 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 5342.27 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 5026.06 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 4956.31 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 14358.34 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 9993.08 examples/s]

Filter:   0%|          | 0/42 [00:00<?, ? examples/s]
Filter: 100%|##########| 42/42 [00:00<00:00, 9754.74 examples/s]

Flattening the indices:   0%|          | 0/42 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 42/42 [00:00<00:00, 9229.36 examples/s]

Casting to class labels:   0%|          | 0/42 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 42/42 [00:00<00:00, 5221.90 examples/s]INFO 2025-05-20 10:30:38,798 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_5_6__Deleting_a_standalone_virtual_machine_instance_using_the_web_console.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 523.18ba/s]INFO 2025-05-20 10:30:38,801 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_5_6__Deleting_a_standalone_virtual_machine_instance_using_the_web_console_task.yaml
INFO 2025-05-20 10:30:38,810 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:30:38,814 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 9303.94 examples/s]

Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 7659.83 examples/s]

Flattening the indices:   0%|          | 0/21 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 21/21 [00:00<00:00, 6005.75 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 3568.17 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 3441.45 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 3279.61 examples/s]

Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 8879.97 examples/s]

Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 6879.13 examples/s]

Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 6839.60 examples/s]

Casting to class labels:   0%|          | 0/21 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 21/21 [00:00<00:00, 3433.40 examples/s]INFO 2025-05-20 10:30:45,060 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_OpenShift_Virtualization_installation__usage__and_release_notes.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 802.12ba/s]INFO 2025-05-20 10:30:45,061 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_OpenShift_Virtualization_installation__usage__and_release_notes_task.yaml
INFO 2025-05-20 10:30:45,070 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:30:45,075 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/9 [00:00<?, ? examples/s]
Filter: 100%|##########| 9/9 [00:00<00:00, 4390.41 examples/s]

Filter:   0%|          | 0/9 [00:00<?, ? examples/s]
Filter: 100%|##########| 9/9 [00:00<00:00, 3890.42 examples/s]

Flattening the indices:   0%|          | 0/9 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 9/9 [00:00<00:00, 2633.69 examples/s]

Map:   0%|          | 0/9 [00:00<?, ? examples/s]
Map: 100%|##########| 9/9 [00:00<00:00, 1912.78 examples/s]

Map:   0%|          | 0/9 [00:00<?, ? examples/s]
Map: 100%|##########| 9/9 [00:00<00:00, 1848.71 examples/s]

Map:   0%|          | 0/9 [00:00<?, ? examples/s]
Map: 100%|##########| 9/9 [00:00<00:00, 1725.42 examples/s]

Filter:   0%|          | 0/9 [00:00<?, ? examples/s]
Filter: 100%|##########| 9/9 [00:00<00:00, 4198.97 examples/s]

Filter:   0%|          | 0/9 [00:00<?, ? examples/s]
Filter: 100%|##########| 9/9 [00:00<00:00, 3596.49 examples/s]

Filter:   0%|          | 0/9 [00:00<?, ? examples/s]
Filter: 100%|##########| 9/9 [00:00<00:00, 3641.24 examples/s]

Casting to class labels:   0%|          | 0/9 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 9/9 [00:00<00:00, 1857.35 examples/s]INFO 2025-05-20 10:30:49,496 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_4_1__Making_open_source_more_inclusive.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 949.80ba/s]INFO 2025-05-20 10:30:49,498 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_1__Making_open_source_more_inclusive_task.yaml
INFO 2025-05-20 10:30:49,528 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:30:49,535 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge
INFO 2025-05-20 10:30:49,539 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge
INFO 2025-05-20 10:31:38,398 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/1317 [00:00<?, ? examples/s]
Filter: 100%|##########| 1317/1317 [00:00<00:00, 51983.27 examples/s]

Filter:   0%|          | 0/1315 [00:00<?, ? examples/s]
Filter: 100%|##########| 1315/1315 [00:00<00:00, 23210.69 examples/s]

Flattening the indices:   0%|          | 0/1315 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 1315/1315 [00:00<00:00, 30338.67 examples/s]

Map:   0%|          | 0/1315 [00:00<?, ? examples/s]
Map:  77%|#######6  | 1012/1315 [00:00<00:00, 10038.94 examples/s]
Map: 100%|##########| 1315/1315 [00:00<00:00, 9996.97 examples/s] 

Map:   0%|          | 0/1315 [00:00<?, ? examples/s]
Map:  76%|#######6  | 1000/1315 [00:00<00:00, 9200.92 examples/s]
Map: 100%|##########| 1315/1315 [00:00<00:00, 9177.01 examples/s]

Map:   0%|          | 0/1315 [00:00<?, ? examples/s]
Map:  76%|#######6  | 1000/1315 [00:00<00:00, 9084.64 examples/s]
Map: 100%|##########| 1315/1315 [00:00<00:00, 9083.81 examples/s]

Filter:   0%|          | 0/1315 [00:00<?, ? examples/s]
Filter: 100%|##########| 1315/1315 [00:00<00:00, 36923.91 examples/s]

Filter:   0%|          | 0/1307 [00:00<?, ? examples/s]
Filter: 100%|##########| 1307/1307 [00:00<00:00, 17845.20 examples/s]

Filter:   0%|          | 0/1291 [00:00<?, ? examples/s]
Filter: 100%|##########| 1291/1291 [00:00<00:00, 17834.35 examples/s]

Flattening the indices:   0%|          | 0/1287 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 1287/1287 [00:00<00:00, 24361.50 examples/s]

Casting to class labels:   0%|          | 0/1287 [00:00<?, ? examples/s]
Casting to class labels:  78%|#######7  | 1000/1287 [00:00<00:00, 9918.71 examples/s]
Casting to class labels: 100%|##########| 1287/1287 [00:00<00:00, 9591.42 examples/s]INFO 2025-05-20 10:32:04,042 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_20_20__Expanding_a_virtual_machine_disk.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 59.74ba/s]INFO 2025-05-20 10:32:04,076 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_20_20__Expanding_a_virtual_machine_disk_task.yaml
INFO 2025-05-20 10:32:04,087 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:32:04,092 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/61 [00:00<?, ? examples/s]
Filter: 100%|##########| 61/61 [00:00<00:00, 21057.82 examples/s]

Filter:   0%|          | 0/61 [00:00<?, ? examples/s]
Filter: 100%|##########| 61/61 [00:00<00:00, 13676.10 examples/s]

Flattening the indices:   0%|          | 0/61 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 61/61 [00:00<00:00, 14547.82 examples/s]

Map:   0%|          | 0/61 [00:00<?, ? examples/s]
Map: 100%|##########| 61/61 [00:00<00:00, 6141.74 examples/s]

Map:   0%|          | 0/61 [00:00<?, ? examples/s]
Map: 100%|##########| 61/61 [00:00<00:00, 5956.71 examples/s]

Map:   0%|          | 0/61 [00:00<?, ? examples/s]
Map: 100%|##########| 61/61 [00:00<00:00, 5845.12 examples/s]

Filter:   0%|          | 0/61 [00:00<?, ? examples/s]
Filter: 100%|##########| 61/61 [00:00<00:00, 18659.02 examples/s]

Filter:   0%|          | 0/61 [00:00<?, ? examples/s]
Filter: 100%|##########| 61/61 [00:00<00:00, 11759.55 examples/s]

Filter:   0%|          | 0/59 [00:00<?, ? examples/s]
Filter: 100%|##########| 59/59 [00:00<00:00, 11028.30 examples/s]

Flattening the indices:   0%|          | 0/59 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 59/59 [00:00<00:00, 11833.02 examples/s]

Casting to class labels:   0%|          | 0/59 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 59/59 [00:00<00:00, 6028.65 examples/s]INFO 2025-05-20 10:32:15,859 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_8_3__Generalizing_a_Windows_VM_using_sysprep.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 480.39ba/s]INFO 2025-05-20 10:32:15,861 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_8_3__Generalizing_a_Windows_VM_using_sysprep_task.yaml
INFO 2025-05-20 10:32:15,879 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:32:15,883 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/443 [00:00<?, ? examples/s]
Filter: 100%|##########| 443/443 [00:00<00:00, 45792.50 examples/s]

Filter:   0%|          | 0/441 [00:00<?, ? examples/s]
Filter: 100%|##########| 441/441 [00:00<00:00, 22090.05 examples/s]

Flattening the indices:   0%|          | 0/441 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 441/441 [00:00<00:00, 40307.87 examples/s]

Map:   0%|          | 0/441 [00:00<?, ? examples/s]
Map: 100%|##########| 441/441 [00:00<00:00, 9218.99 examples/s]

Map:   0%|          | 0/441 [00:00<?, ? examples/s]
Map: 100%|##########| 441/441 [00:00<00:00, 8736.65 examples/s]

Map:   0%|          | 0/441 [00:00<?, ? examples/s]
Map: 100%|##########| 441/441 [00:00<00:00, 8845.82 examples/s]

Filter:   0%|          | 0/441 [00:00<?, ? examples/s]
Filter: 100%|##########| 441/441 [00:00<00:00, 33975.39 examples/s]

Filter:   0%|          | 0/435 [00:00<?, ? examples/s]
Filter: 100%|##########| 435/435 [00:00<00:00, 17176.99 examples/s]

Filter:   0%|          | 0/424 [00:00<?, ? examples/s]
Filter: 100%|##########| 424/424 [00:00<00:00, 17088.02 examples/s]

Flattening the indices:   0%|          | 0/424 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 424/424 [00:00<00:00, 37507.59 examples/s]

Casting to class labels:   0%|          | 0/424 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 424/424 [00:00<00:00, 9302.73 examples/s]INFO 2025-05-20 10:32:45,918 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_19_7__Viewing_the_IP_address_of_NICs_on_a_virtual_machine.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 81.45ba/s]INFO 2025-05-20 10:32:45,931 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_19_7__Viewing_the_IP_address_of_NICs_on_a_virtual_machine_task.yaml
INFO 2025-05-20 10:32:45,942 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:32:45,946 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/118 [00:00<?, ? examples/s]
Filter: 100%|##########| 118/118 [00:00<00:00, 29593.87 examples/s]

Filter:   0%|          | 0/118 [00:00<?, ? examples/s]
Filter: 100%|##########| 118/118 [00:00<00:00, 17404.36 examples/s]

Flattening the indices:   0%|          | 0/118 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 118/118 [00:00<00:00, 24545.12 examples/s]

Map:   0%|          | 0/118 [00:00<?, ? examples/s]
Map: 100%|##########| 118/118 [00:00<00:00, 7719.98 examples/s]

Map:   0%|          | 0/118 [00:00<?, ? examples/s]
Map: 100%|##########| 118/118 [00:00<00:00, 7257.33 examples/s]

Map:   0%|          | 0/118 [00:00<?, ? examples/s]
Map: 100%|##########| 118/118 [00:00<00:00, 7209.44 examples/s]

Filter:   0%|          | 0/118 [00:00<?, ? examples/s]
Filter: 100%|##########| 118/118 [00:00<00:00, 24609.81 examples/s]

Filter:   0%|          | 0/118 [00:00<?, ? examples/s]
Filter: 100%|##########| 118/118 [00:00<00:00, 14201.66 examples/s]

Filter:   0%|          | 0/115 [00:00<?, ? examples/s]
Filter: 100%|##########| 115/115 [00:00<00:00, 14587.78 examples/s]

Flattening the indices:   0%|          | 0/115 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 115/115 [00:00<00:00, 19842.24 examples/s]

Casting to class labels:   0%|          | 0/115 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 115/115 [00:00<00:00, 7582.85 examples/s]INFO 2025-05-20 10:33:00,647 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_5_1_4__Restricted_network_environments.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 266.24ba/s]INFO 2025-05-20 10:33:00,652 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_1_4__Restricted_network_environments_task.yaml
INFO 2025-05-20 10:33:00,661 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:33:00,666 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/48 [00:00<?, ? examples/s]
Filter: 100%|##########| 48/48 [00:00<00:00, 17552.45 examples/s]

Filter:   0%|          | 0/47 [00:00<?, ? examples/s]
Filter: 100%|##########| 47/47 [00:00<00:00, 12470.41 examples/s]

Flattening the indices:   0%|          | 0/47 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 47/47 [00:00<00:00, 12417.78 examples/s]

Map:   0%|          | 0/47 [00:00<?, ? examples/s]
Map: 100%|##########| 47/47 [00:00<00:00, 5586.22 examples/s]

Map:   0%|          | 0/47 [00:00<?, ? examples/s]
Map: 100%|##########| 47/47 [00:00<00:00, 5311.25 examples/s]

Map:   0%|          | 0/47 [00:00<?, ? examples/s]
Map: 100%|##########| 47/47 [00:00<00:00, 5200.42 examples/s]

Filter:   0%|          | 0/47 [00:00<?, ? examples/s]
Filter: 100%|##########| 47/47 [00:00<00:00, 15255.56 examples/s]

Filter:   0%|          | 0/47 [00:00<?, ? examples/s]
Filter: 100%|##########| 47/47 [00:00<00:00, 10425.87 examples/s]

Filter:   0%|          | 0/46 [00:00<?, ? examples/s]
Filter: 100%|##########| 46/46 [00:00<00:00, 10736.07 examples/s]

Flattening the indices:   0%|          | 0/46 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 46/46 [00:00<00:00, 9872.99 examples/s]

Casting to class labels:   0%|          | 0/46 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 46/46 [00:00<00:00, 5438.86 examples/s]INFO 2025-05-20 10:33:08,908 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_10_2__Editing_virtual_machine_templates.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 539.95ba/s]INFO 2025-05-20 10:33:08,911 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_2__Editing_virtual_machine_templates_task.yaml
INFO 2025-05-20 10:33:08,920 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:33:08,925 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/32 [00:00<?, ? examples/s]
Filter: 100%|##########| 32/32 [00:00<00:00, 13098.25 examples/s]

Filter:   0%|          | 0/32 [00:00<?, ? examples/s]
Filter: 100%|##########| 32/32 [00:00<00:00, 10050.75 examples/s]

Flattening the indices:   0%|          | 0/32 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 32/32 [00:00<00:00, 8658.09 examples/s]

Map:   0%|          | 0/32 [00:00<?, ? examples/s]
Map: 100%|##########| 32/32 [00:00<00:00, 4541.13 examples/s]

Map:   0%|          | 0/32 [00:00<?, ? examples/s]
Map: 100%|##########| 32/32 [00:00<00:00, 4321.38 examples/s]

Map:   0%|          | 0/32 [00:00<?, ? examples/s]
Map: 100%|##########| 32/32 [00:00<00:00, 4302.68 examples/s]

Filter:   0%|          | 0/32 [00:00<?, ? examples/s]
Filter: 100%|##########| 32/32 [00:00<00:00, 11776.58 examples/s]

Filter:   0%|          | 0/32 [00:00<?, ? examples/s]
Filter: 100%|##########| 32/32 [00:00<00:00, 8747.82 examples/s]

Filter:   0%|          | 0/32 [00:00<?, ? examples/s]
Filter: 100%|##########| 32/32 [00:00<00:00, 8915.75 examples/s]

Casting to class labels:   0%|          | 0/32 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 32/32 [00:00<00:00, 4430.94 examples/s]INFO 2025-05-20 10:33:18,449 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_11_2__About_the_QEMU_guest_agent_information_in_the_web_console.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 631.29ba/s]INFO 2025-05-20 10:33:18,452 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_11_2__About_the_QEMU_guest_agent_information_in_the_web_console_task.yaml
INFO 2025-05-20 10:33:18,461 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:33:18,465 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 16207.32 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 11707.57 examples/s]

Flattening the indices:   0%|          | 0/43 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 43/43 [00:00<00:00, 11001.96 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 5296.30 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 5042.92 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 4957.12 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 14174.40 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 10136.29 examples/s]

Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 9990.50 examples/s]

Flattening the indices:   0%|          | 0/41 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 41/41 [00:00<00:00, 8731.03 examples/s]

Casting to class labels:   0%|          | 0/41 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 41/41 [00:00<00:00, 5091.38 examples/s]INFO 2025-05-20 10:33:26,777 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_8_1__About_reviewing_top_consumers.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 541.20ba/s]INFO 2025-05-20 10:33:26,780 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_8_1__About_reviewing_top_consumers_task.yaml
INFO 2025-05-20 10:33:26,789 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:33:26,793 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/32 [00:00<?, ? examples/s]
Filter: 100%|##########| 32/32 [00:00<00:00, 13175.39 examples/s]

Filter:   0%|          | 0/32 [00:00<?, ? examples/s]
Filter: 100%|##########| 32/32 [00:00<00:00, 9903.91 examples/s]

Flattening the indices:   0%|          | 0/32 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 32/32 [00:00<00:00, 8855.75 examples/s]

Map:   0%|          | 0/32 [00:00<?, ? examples/s]
Map: 100%|##########| 32/32 [00:00<00:00, 4455.66 examples/s]

Map:   0%|          | 0/32 [00:00<?, ? examples/s]
Map: 100%|##########| 32/32 [00:00<00:00, 4423.79 examples/s]

Map:   0%|          | 0/32 [00:00<?, ? examples/s]
Map: 100%|##########| 32/32 [00:00<00:00, 4277.58 examples/s]

Filter:   0%|          | 0/32 [00:00<?, ? examples/s]
Filter: 100%|##########| 32/32 [00:00<00:00, 11926.22 examples/s]

Filter:   0%|          | 0/32 [00:00<?, ? examples/s]
Filter: 100%|##########| 32/32 [00:00<00:00, 8599.29 examples/s]

Filter:   0%|          | 0/31 [00:00<?, ? examples/s]
Filter: 100%|##########| 31/31 [00:00<00:00, 8472.24 examples/s]

Flattening the indices:   0%|          | 0/31 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 31/31 [00:00<00:00, 7043.90 examples/s]

Casting to class labels:   0%|          | 0/31 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 31/31 [00:00<00:00, 4387.05 examples/s]INFO 2025-05-20 10:33:36,404 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_5_3__Installing_OpenShift_Virtualization_using_the_web_console.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 557.16ba/s]INFO 2025-05-20 10:33:36,406 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_3__Installing_OpenShift_Virtualization_using_the_web_console_task.yaml
INFO 2025-05-20 10:33:36,416 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:33:36,420 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/31 [00:00<?, ? examples/s]
Filter: 100%|##########| 31/31 [00:00<00:00, 12703.80 examples/s]

Filter:   0%|          | 0/31 [00:00<?, ? examples/s]
Filter: 100%|##########| 31/31 [00:00<00:00, 9798.30 examples/s]

Flattening the indices:   0%|          | 0/31 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 31/31 [00:00<00:00, 8617.10 examples/s]

Map:   0%|          | 0/31 [00:00<?, ? examples/s]
Map: 100%|##########| 31/31 [00:00<00:00, 4566.87 examples/s]

Map:   0%|          | 0/31 [00:00<?, ? examples/s]
Map: 100%|##########| 31/31 [00:00<00:00, 4293.33 examples/s]

Map:   0%|          | 0/31 [00:00<?, ? examples/s]
Map: 100%|##########| 31/31 [00:00<00:00, 4171.56 examples/s]

Filter:   0%|          | 0/31 [00:00<?, ? examples/s]
Filter: 100%|##########| 31/31 [00:00<00:00, 11518.73 examples/s]

Filter:   0%|          | 0/31 [00:00<?, ? examples/s]
Filter: 100%|##########| 31/31 [00:00<00:00, 8629.68 examples/s]

Filter:   0%|          | 0/30 [00:00<?, ? examples/s]
Filter: 100%|##########| 30/30 [00:00<00:00, 8492.21 examples/s]

Flattening the indices:   0%|          | 0/30 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 30/30 [00:00<00:00, 6808.20 examples/s]

Casting to class labels:   0%|          | 0/30 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 30/30 [00:00<00:00, 4399.62 examples/s]INFO 2025-05-20 10:33:44,162 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_3_4__Viewing_resource_events_in_the_CLI.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 703.98ba/s]INFO 2025-05-20 10:33:44,164 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_3_4__Viewing_resource_events_in_the_CLI_task.yaml
INFO 2025-05-20 10:33:44,174 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:33:44,178 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/54 [00:00<?, ? examples/s]
Filter: 100%|##########| 54/54 [00:00<00:00, 17319.91 examples/s]

Filter:   0%|          | 0/54 [00:00<?, ? examples/s]
Filter: 100%|##########| 54/54 [00:00<00:00, 13066.37 examples/s]

Flattening the indices:   0%|          | 0/54 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 54/54 [00:00<00:00, 12912.17 examples/s]

Map:   0%|          | 0/54 [00:00<?, ? examples/s]
Map: 100%|##########| 54/54 [00:00<00:00, 5741.69 examples/s]

Map:   0%|          | 0/54 [00:00<?, ? examples/s]
Map: 100%|##########| 54/54 [00:00<00:00, 5637.22 examples/s]

Map:   0%|          | 0/54 [00:00<?, ? examples/s]
Map: 100%|##########| 54/54 [00:00<00:00, 5423.80 examples/s]

Filter:   0%|          | 0/54 [00:00<?, ? examples/s]
Filter: 100%|##########| 54/54 [00:00<00:00, 16326.13 examples/s]

Filter:   0%|          | 0/54 [00:00<?, ? examples/s]
Filter: 100%|##########| 54/54 [00:00<00:00, 11320.66 examples/s]

Filter:   0%|          | 0/52 [00:00<?, ? examples/s]
Filter: 100%|##########| 52/52 [00:00<00:00, 11122.64 examples/s]

Flattening the indices:   0%|          | 0/52 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 52/52 [00:00<00:00, 10909.01 examples/s]

Casting to class labels:   0%|          | 0/52 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 52/52 [00:00<00:00, 5772.23 examples/s]INFO 2025-05-20 10:33:54,148 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_1_2__Resources_monitored_actively_in_the_Virtualization_Overview_page.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 415.57ba/s]INFO 2025-05-20 10:33:54,151 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_1_2__Resources_monitored_actively_in_the_Virtualization_Overview_page_task.yaml
INFO 2025-05-20 10:33:54,160 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:33:54,164 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/25 [00:00<?, ? examples/s]
Filter: 100%|##########| 25/25 [00:00<00:00, 10626.02 examples/s]

Filter:   0%|          | 0/25 [00:00<?, ? examples/s]
Filter: 100%|##########| 25/25 [00:00<00:00, 8440.60 examples/s]

Flattening the indices:   0%|          | 0/25 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 25/25 [00:00<00:00, 6879.52 examples/s]

Map:   0%|          | 0/25 [00:00<?, ? examples/s]
Map: 100%|##########| 25/25 [00:00<00:00, 3950.93 examples/s]

Map:   0%|          | 0/25 [00:00<?, ? examples/s]
Map: 100%|##########| 25/25 [00:00<00:00, 3815.50 examples/s]

Map:   0%|          | 0/25 [00:00<?, ? examples/s]
Map: 100%|##########| 25/25 [00:00<00:00, 3725.49 examples/s]

Filter:   0%|          | 0/25 [00:00<?, ? examples/s]
Filter: 100%|##########| 25/25 [00:00<00:00, 9910.93 examples/s]

Filter:   0%|          | 0/25 [00:00<?, ? examples/s]
Filter: 100%|##########| 25/25 [00:00<00:00, 7634.89 examples/s]

Filter:   0%|          | 0/24 [00:00<?, ? examples/s]
Filter: 100%|##########| 24/24 [00:00<00:00, 7287.05 examples/s]

Flattening the indices:   0%|          | 0/24 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 24/24 [00:00<00:00, 5507.35 examples/s]

Casting to class labels:   0%|          | 0/24 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 24/24 [00:00<00:00, 3767.62 examples/s]INFO 2025-05-20 10:34:04,474 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_1_1__What_you_can_do_with_OpenShift_Virtualization.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 661.56ba/s]INFO 2025-05-20 10:34:04,476 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_1_1__What_you_can_do_with_OpenShift_Virtualization_task.yaml
INFO 2025-05-20 10:34:04,486 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:34:04,491 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/134 [00:00<?, ? examples/s]
Filter: 100%|##########| 134/134 [00:00<00:00, 31874.14 examples/s]

Filter:   0%|          | 0/133 [00:00<?, ? examples/s]
Filter: 100%|##########| 133/133 [00:00<00:00, 18541.60 examples/s]

Flattening the indices:   0%|          | 0/133 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 133/133 [00:00<00:00, 24924.82 examples/s]

Map:   0%|          | 0/133 [00:00<?, ? examples/s]
Map: 100%|##########| 133/133 [00:00<00:00, 7852.95 examples/s]

Map:   0%|          | 0/133 [00:00<?, ? examples/s]
Map: 100%|##########| 133/133 [00:00<00:00, 7322.59 examples/s]

Map:   0%|          | 0/133 [00:00<?, ? examples/s]
Map: 100%|##########| 133/133 [00:00<00:00, 7398.24 examples/s]

Filter:   0%|          | 0/133 [00:00<?, ? examples/s]
Filter: 100%|##########| 133/133 [00:00<00:00, 25173.39 examples/s]

Filter:   0%|          | 0/133 [00:00<?, ? examples/s]
Filter: 100%|##########| 133/133 [00:00<00:00, 14879.77 examples/s]

Filter:   0%|          | 0/132 [00:00<?, ? examples/s]
Filter: 100%|##########| 132/132 [00:00<00:00, 15034.98 examples/s]

Flattening the indices:   0%|          | 0/132 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 132/132 [00:00<00:00, 21604.09 examples/s]

Casting to class labels:   0%|          | 0/132 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 132/132 [00:00<00:00, 7718.61 examples/s]INFO 2025-05-20 10:34:25,426 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_2_5__Adding_a_network_interface_to_a_virtual_machine.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 245.34ba/s]INFO 2025-05-20 10:34:25,431 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_2_5__Adding_a_network_interface_to_a_virtual_machine_task.yaml
INFO 2025-05-20 10:34:25,440 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:34:25,445 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/22 [00:00<?, ? examples/s]
Filter: 100%|##########| 22/22 [00:00<00:00, 9557.19 examples/s]

Filter:   0%|          | 0/22 [00:00<?, ? examples/s]
Filter: 100%|##########| 22/22 [00:00<00:00, 7886.73 examples/s]

Flattening the indices:   0%|          | 0/22 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 22/22 [00:00<00:00, 6124.70 examples/s]

Map:   0%|          | 0/22 [00:00<?, ? examples/s]
Map: 100%|##########| 22/22 [00:00<00:00, 3673.36 examples/s]

Map:   0%|          | 0/22 [00:00<?, ? examples/s]
Map: 100%|##########| 22/22 [00:00<00:00, 3475.51 examples/s]

Map:   0%|          | 0/22 [00:00<?, ? examples/s]
Map: 100%|##########| 22/22 [00:00<00:00, 3416.32 examples/s]

Filter:   0%|          | 0/22 [00:00<?, ? examples/s]
Filter: 100%|##########| 22/22 [00:00<00:00, 8847.05 examples/s]

Filter:   0%|          | 0/22 [00:00<?, ? examples/s]
Filter: 100%|##########| 22/22 [00:00<00:00, 6996.87 examples/s]

Filter:   0%|          | 0/22 [00:00<?, ? examples/s]
Filter: 100%|##########| 22/22 [00:00<00:00, 7050.87 examples/s]

Casting to class labels:   0%|          | 0/22 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 22/22 [00:00<00:00, 3661.84 examples/s]INFO 2025-05-20 10:34:34,425 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_4_2_2__Supported_guest_operating_systems.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 745.39ba/s]INFO 2025-05-20 10:34:34,427 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_2_2__Supported_guest_operating_systems_task.yaml
INFO 2025-05-20 10:34:34,437 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:34:34,442 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/100 [00:00<?, ? examples/s]
Filter: 100%|##########| 100/100 [00:00<00:00, 26027.33 examples/s]

Filter:   0%|          | 0/100 [00:00<?, ? examples/s]
Filter: 100%|##########| 100/100 [00:00<00:00, 16847.98 examples/s]

Flattening the indices:   0%|          | 0/100 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 100/100 [00:00<00:00, 20255.49 examples/s]

Map:   0%|          | 0/100 [00:00<?, ? examples/s]
Map: 100%|##########| 100/100 [00:00<00:00, 7234.18 examples/s]

Map:   0%|          | 0/100 [00:00<?, ? examples/s]
Map: 100%|##########| 100/100 [00:00<00:00, 7026.93 examples/s]

Map:   0%|          | 0/100 [00:00<?, ? examples/s]
Map: 100%|##########| 100/100 [00:00<00:00, 6731.78 examples/s]

Filter:   0%|          | 0/100 [00:00<?, ? examples/s]
Filter: 100%|##########| 100/100 [00:00<00:00, 22541.54 examples/s]

Filter:   0%|          | 0/100 [00:00<?, ? examples/s]
Filter: 100%|##########| 100/100 [00:00<00:00, 13686.75 examples/s]

Filter:   0%|          | 0/96 [00:00<?, ? examples/s]
Filter: 100%|##########| 96/96 [00:00<00:00, 13730.25 examples/s]

Flattening the indices:   0%|          | 0/96 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 96/96 [00:00<00:00, 16914.65 examples/s]

Casting to class labels:   0%|          | 0/96 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 96/96 [00:00<00:00, 7127.74 examples/s]INFO 2025-05-20 10:34:49,805 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_6_6__Additional_resources.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 286.67ba/s]INFO 2025-05-20 10:34:49,809 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_6_6__Additional_resources_task.yaml
INFO 2025-05-20 10:34:49,819 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:34:49,823 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/129 [00:00<?, ? examples/s]
Filter: 100%|##########| 129/129 [00:00<00:00, 30637.89 examples/s]

Filter:   0%|          | 0/129 [00:00<?, ? examples/s]
Filter: 100%|##########| 129/129 [00:00<00:00, 18612.49 examples/s]

Flattening the indices:   0%|          | 0/129 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 129/129 [00:00<00:00, 24679.13 examples/s]

Map:   0%|          | 0/129 [00:00<?, ? examples/s]
Map: 100%|##########| 129/129 [00:00<00:00, 7738.35 examples/s]

Map:   0%|          | 0/129 [00:00<?, ? examples/s]
Map: 100%|##########| 129/129 [00:00<00:00, 7598.48 examples/s]

Map:   0%|          | 0/129 [00:00<?, ? examples/s]
Map: 100%|##########| 129/129 [00:00<00:00, 7180.31 examples/s]

Filter:   0%|          | 0/129 [00:00<?, ? examples/s]
Filter: 100%|##########| 129/129 [00:00<00:00, 25443.93 examples/s]

Filter:   0%|          | 0/129 [00:00<?, ? examples/s]
Filter: 100%|##########| 129/129 [00:00<00:00, 14729.28 examples/s]

Filter:   0%|          | 0/127 [00:00<?, ? examples/s]
Filter: 100%|##########| 127/127 [00:00<00:00, 14852.27 examples/s]

Flattening the indices:   0%|          | 0/127 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 127/127 [00:00<00:00, 21637.69 examples/s]

Casting to class labels:   0%|          | 0/127 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 127/127 [00:00<00:00, 7705.77 examples/s]INFO 2025-05-20 10:35:11,083 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_14_3_3__Additional_resources.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 220.17ba/s]INFO 2025-05-20 10:35:11,088 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_14_3_3__Additional_resources_task.yaml
INFO 2025-05-20 10:35:11,098 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:35:11,103 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/134 [00:00<?, ? examples/s]
Filter: 100%|##########| 134/134 [00:00<00:00, 31306.01 examples/s]

Filter:   0%|          | 0/134 [00:00<?, ? examples/s]
Filter: 100%|##########| 134/134 [00:00<00:00, 18681.63 examples/s]

Flattening the indices:   0%|          | 0/134 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 134/134 [00:00<00:00, 26443.81 examples/s]

Map:   0%|          | 0/134 [00:00<?, ? examples/s]
Map: 100%|##########| 134/134 [00:00<00:00, 8021.76 examples/s]

Map:   0%|          | 0/134 [00:00<?, ? examples/s]
Map: 100%|##########| 134/134 [00:00<00:00, 7483.05 examples/s]

Map:   0%|          | 0/134 [00:00<?, ? examples/s]
Map: 100%|##########| 134/134 [00:00<00:00, 7414.73 examples/s]

Filter:   0%|          | 0/134 [00:00<?, ? examples/s]
Filter: 100%|##########| 134/134 [00:00<00:00, 25194.40 examples/s]

Filter:   0%|          | 0/132 [00:00<?, ? examples/s]
Filter: 100%|##########| 132/132 [00:00<00:00, 14869.42 examples/s]

Filter:   0%|          | 0/128 [00:00<?, ? examples/s]
Filter: 100%|##########| 128/128 [00:00<00:00, 14646.59 examples/s]

Flattening the indices:   0%|          | 0/128 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 128/128 [00:00<00:00, 21263.90 examples/s]

Casting to class labels:   0%|          | 0/128 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 128/128 [00:00<00:00, 7805.85 examples/s]INFO 2025-05-20 10:35:31,021 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_11_4__Additional_resources.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 233.91ba/s]INFO 2025-05-20 10:35:31,026 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_11_4__Additional_resources_task.yaml
INFO 2025-05-20 10:35:31,035 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:35:31,039 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/13 [00:00<?, ? examples/s]
Filter: 100%|##########| 13/13 [00:00<00:00, 6196.84 examples/s]

Filter:   0%|          | 0/13 [00:00<?, ? examples/s]
Filter: 100%|##########| 13/13 [00:00<00:00, 5344.11 examples/s]

Flattening the indices:   0%|          | 0/13 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 13/13 [00:00<00:00, 3764.56 examples/s]

Map:   0%|          | 0/13 [00:00<?, ? examples/s]
Map: 100%|##########| 13/13 [00:00<00:00, 2634.61 examples/s]

Map:   0%|          | 0/13 [00:00<?, ? examples/s]
Map: 100%|##########| 13/13 [00:00<00:00, 2404.04 examples/s]

Map:   0%|          | 0/13 [00:00<?, ? examples/s]
Map: 100%|##########| 13/13 [00:00<00:00, 2370.18 examples/s]

Filter:   0%|          | 0/13 [00:00<?, ? examples/s]
Filter: 100%|##########| 13/13 [00:00<00:00, 6150.70 examples/s]

Filter:   0%|          | 0/13 [00:00<?, ? examples/s]
Filter: 100%|##########| 13/13 [00:00<00:00, 5024.97 examples/s]

Filter:   0%|          | 0/10 [00:00<?, ? examples/s]
Filter: 100%|##########| 10/10 [00:00<00:00, 3962.12 examples/s]

Flattening the indices:   0%|          | 0/10 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 10/10 [00:00<00:00, 2474.37 examples/s]

Casting to class labels:   0%|          | 0/10 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 10/10 [00:00<00:00, 2069.42 examples/s]INFO 2025-05-20 10:35:34,886 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_10_5__Deleting_virtual_machine_templates.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 1073.54ba/s]INFO 2025-05-20 10:35:34,888 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_5__Deleting_virtual_machine_templates_task.yaml
INFO 2025-05-20 10:35:34,897 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:35:34,901 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/5 [00:00<?, ? examples/s]
Filter: 100%|##########| 5/5 [00:00<00:00, 2577.62 examples/s]

Filter:   0%|          | 0/5 [00:00<?, ? examples/s]
Filter: 100%|##########| 5/5 [00:00<00:00, 2391.01 examples/s]

Flattening the indices:   0%|          | 0/5 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 5/5 [00:00<00:00, 1497.43 examples/s]

Map:   0%|          | 0/5 [00:00<?, ? examples/s]
Map: 100%|##########| 5/5 [00:00<00:00, 1184.90 examples/s]

Map:   0%|          | 0/5 [00:00<?, ? examples/s]
Map: 100%|##########| 5/5 [00:00<00:00, 1136.85 examples/s]

Map:   0%|          | 0/5 [00:00<?, ? examples/s]
Map: 100%|##########| 5/5 [00:00<00:00, 1080.78 examples/s]

Filter:   0%|          | 0/5 [00:00<?, ? examples/s]
Filter: 100%|##########| 5/5 [00:00<00:00, 2534.94 examples/s]

Filter:   0%|          | 0/5 [00:00<?, ? examples/s]
Filter: 100%|##########| 5/5 [00:00<00:00, 2222.74 examples/s]

Filter:   0%|          | 0/5 [00:00<?, ? examples/s]
Filter: 100%|##########| 5/5 [00:00<00:00, 2233.39 examples/s]

Casting to class labels:   0%|          | 0/5 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 5/5 [00:00<00:00, 1091.81 examples/s]INFO 2025-05-20 10:35:39,265 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_2_7__About_the_tekton_tasks_operator.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 1072.71ba/s]INFO 2025-05-20 10:35:39,267 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_7__About_the_tekton_tasks_operator_task.yaml
INFO 2025-05-20 10:35:39,277 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:35:39,281 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/137 [00:00<?, ? examples/s]
Filter: 100%|##########| 137/137 [00:00<00:00, 29248.68 examples/s]

Filter:   0%|          | 0/137 [00:00<?, ? examples/s]
Filter: 100%|##########| 137/137 [00:00<00:00, 18609.96 examples/s]

Flattening the indices:   0%|          | 0/137 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 137/137 [00:00<00:00, 25220.31 examples/s]

Map:   0%|          | 0/137 [00:00<?, ? examples/s]
Map: 100%|##########| 137/137 [00:00<00:00, 7990.37 examples/s]

Map:   0%|          | 0/137 [00:00<?, ? examples/s]
Map: 100%|##########| 137/137 [00:00<00:00, 7376.47 examples/s]

Map:   0%|          | 0/137 [00:00<?, ? examples/s]
Map: 100%|##########| 137/137 [00:00<00:00, 7457.56 examples/s]

Filter:   0%|          | 0/137 [00:00<?, ? examples/s]
Filter: 100%|##########| 137/137 [00:00<00:00, 26215.60 examples/s]

Filter:   0%|          | 0/137 [00:00<?, ? examples/s]
Filter: 100%|##########| 137/137 [00:00<00:00, 14833.10 examples/s]

Filter:   0%|          | 0/136 [00:00<?, ? examples/s]
Filter: 100%|##########| 136/136 [00:00<00:00, 15140.69 examples/s]

Flattening the indices:   0%|          | 0/136 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 136/136 [00:00<00:00, 21341.86 examples/s]

Casting to class labels:   0%|          | 0/136 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 136/136 [00:00<00:00, 7755.82 examples/s]INFO 2025-05-20 10:35:59,989 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_2_2__Editing_a_virtual_machine_YAML_configuration_using_the_web_console.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 240.80ba/s]INFO 2025-05-20 10:35:59,994 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_2_2__Editing_a_virtual_machine_YAML_configuration_using_the_web_console_task.yaml
INFO 2025-05-20 10:36:00,003 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:36:00,007 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/12 [00:00<?, ? examples/s]
Filter: 100%|##########| 12/12 [00:00<00:00, 5686.55 examples/s]

Filter:   0%|          | 0/12 [00:00<?, ? examples/s]
Filter: 100%|##########| 12/12 [00:00<00:00, 5061.51 examples/s]

Flattening the indices:   0%|          | 0/12 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 12/12 [00:00<00:00, 3600.00 examples/s]

Map:   0%|          | 0/12 [00:00<?, ? examples/s]
Map: 100%|##########| 12/12 [00:00<00:00, 2443.17 examples/s]

Map:   0%|          | 0/12 [00:00<?, ? examples/s]
Map: 100%|##########| 12/12 [00:00<00:00, 2287.28 examples/s]

Map:   0%|          | 0/12 [00:00<?, ? examples/s]
Map: 100%|##########| 12/12 [00:00<00:00, 2223.13 examples/s]

Filter:   0%|          | 0/12 [00:00<?, ? examples/s]
Filter: 100%|##########| 12/12 [00:00<00:00, 5494.72 examples/s]

Filter:   0%|          | 0/12 [00:00<?, ? examples/s]
Filter: 100%|##########| 12/12 [00:00<00:00, 4489.09 examples/s]

Filter:   0%|          | 0/12 [00:00<?, ? examples/s]
Filter: 100%|##########| 12/12 [00:00<00:00, 4484.69 examples/s]

Casting to class labels:   0%|          | 0/12 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 12/12 [00:00<00:00, 2336.23 examples/s]INFO 2025-05-20 10:36:08,860 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_5__Viewing_information_about_virtual_machine_workloads.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 983.19ba/s]INFO 2025-05-20 10:36:08,861 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_5__Viewing_information_about_virtual_machine_workloads_task.yaml
INFO 2025-05-20 10:36:08,871 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:36:08,876 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/113 [00:00<?, ? examples/s]
Filter: 100%|##########| 113/113 [00:00<00:00, 27284.34 examples/s]

Filter:   0%|          | 0/113 [00:00<?, ? examples/s]
Filter: 100%|##########| 113/113 [00:00<00:00, 17807.87 examples/s]

Flattening the indices:   0%|          | 0/113 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 113/113 [00:00<00:00, 22138.18 examples/s]

Map:   0%|          | 0/113 [00:00<?, ? examples/s]
Map: 100%|##########| 113/113 [00:00<00:00, 7500.73 examples/s]

Map:   0%|          | 0/113 [00:00<?, ? examples/s]
Map: 100%|##########| 113/113 [00:00<00:00, 7146.29 examples/s]

Map:   0%|          | 0/113 [00:00<?, ? examples/s]
Map: 100%|##########| 113/113 [00:00<00:00, 6967.18 examples/s]

Filter:   0%|          | 0/113 [00:00<?, ? examples/s]
Filter: 100%|##########| 113/113 [00:00<00:00, 23886.52 examples/s]

Filter:   0%|          | 0/113 [00:00<?, ? examples/s]
Filter: 100%|##########| 113/113 [00:00<00:00, 14083.63 examples/s]

Filter:   0%|          | 0/110 [00:00<?, ? examples/s]
Filter: 100%|##########| 110/110 [00:00<00:00, 14191.74 examples/s]

Flattening the indices:   0%|          | 0/109 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 109/109 [00:00<00:00, 18279.12 examples/s]

Casting to class labels:   0%|          | 0/109 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 109/109 [00:00<00:00, 7353.93 examples/s]INFO 2025-05-20 10:36:25,694 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_10_1__About_the_OpenShift_Container_Platform_cluster_checkup_framework.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 240.91ba/s]INFO 2025-05-20 10:36:25,699 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_10_1__About_the_OpenShift_Container_Platform_cluster_checkup_framework_task.yaml
INFO 2025-05-20 10:36:25,709 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:36:25,713 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/83 [00:00<?, ? examples/s]
Filter: 100%|##########| 83/83 [00:00<00:00, 24862.68 examples/s]

Filter:   0%|          | 0/83 [00:00<?, ? examples/s]
Filter: 100%|##########| 83/83 [00:00<00:00, 15934.78 examples/s]

Flattening the indices:   0%|          | 0/83 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 83/83 [00:00<00:00, 19367.30 examples/s]

Map:   0%|          | 0/83 [00:00<?, ? examples/s]
Map: 100%|##########| 83/83 [00:00<00:00, 6986.30 examples/s]

Map:   0%|          | 0/83 [00:00<?, ? examples/s]
Map: 100%|##########| 83/83 [00:00<00:00, 6611.60 examples/s]

Map:   0%|          | 0/83 [00:00<?, ? examples/s]
Map: 100%|##########| 83/83 [00:00<00:00, 6388.83 examples/s]

Filter:   0%|          | 0/83 [00:00<?, ? examples/s]
Filter: 100%|##########| 83/83 [00:00<00:00, 21005.69 examples/s]

Filter:   0%|          | 0/83 [00:00<?, ? examples/s]
Filter: 100%|##########| 83/83 [00:00<00:00, 13267.55 examples/s]

Filter:   0%|          | 0/81 [00:00<?, ? examples/s]
Filter: 100%|##########| 81/81 [00:00<00:00, 13087.00 examples/s]

Flattening the indices:   0%|          | 0/81 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 81/81 [00:00<00:00, 15528.78 examples/s]

Casting to class labels:   0%|          | 0/81 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 81/81 [00:00<00:00, 6747.41 examples/s]INFO 2025-05-20 10:36:37,317 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_9_3__About_Telemetry.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 314.79ba/s]INFO 2025-05-20 10:36:37,321 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_9_3__About_Telemetry_task.yaml
INFO 2025-05-20 10:36:37,330 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:36:37,334 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/20 [00:00<?, ? examples/s]
Filter: 100%|##########| 20/20 [00:00<00:00, 8873.08 examples/s]

Filter:   0%|          | 0/20 [00:00<?, ? examples/s]
Filter: 100%|##########| 20/20 [00:00<00:00, 7252.82 examples/s]

Flattening the indices:   0%|          | 0/20 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 20/20 [00:00<00:00, 5728.36 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 3498.31 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 3355.98 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 3249.51 examples/s]

Filter:   0%|          | 0/20 [00:00<?, ? examples/s]
Filter: 100%|##########| 20/20 [00:00<00:00, 8280.14 examples/s]

Filter:   0%|          | 0/20 [00:00<?, ? examples/s]
Filter: 100%|##########| 20/20 [00:00<00:00, 6537.26 examples/s]

Filter:   0%|          | 0/20 [00:00<?, ? examples/s]
Filter: 100%|##########| 20/20 [00:00<00:00, 6528.10 examples/s]

Casting to class labels:   0%|          | 0/20 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 20/20 [00:00<00:00, 3382.09 examples/s]INFO 2025-05-20 10:36:44,397 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_4_4_1__Deprecated_features.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 826.30ba/s]INFO 2025-05-20 10:36:44,399 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_4_1__Deprecated_features_task.yaml
INFO 2025-05-20 10:36:44,408 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:36:44,413 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/68 [00:00<?, ? examples/s]
Filter: 100%|##########| 68/68 [00:00<00:00, 22263.11 examples/s]

Filter:   0%|          | 0/68 [00:00<?, ? examples/s]
Filter: 100%|##########| 68/68 [00:00<00:00, 14624.04 examples/s]

Flattening the indices:   0%|          | 0/68 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 68/68 [00:00<00:00, 16130.11 examples/s]

Map:   0%|          | 0/68 [00:00<?, ? examples/s]
Map: 100%|##########| 68/68 [00:00<00:00, 6369.77 examples/s]

Map:   0%|          | 0/68 [00:00<?, ? examples/s]
Map: 100%|##########| 68/68 [00:00<00:00, 6163.83 examples/s]

Map:   0%|          | 0/68 [00:00<?, ? examples/s]
Map: 100%|##########| 68/68 [00:00<00:00, 6092.60 examples/s]

Filter:   0%|          | 0/68 [00:00<?, ? examples/s]
Filter: 100%|##########| 68/68 [00:00<00:00, 18607.30 examples/s]

Filter:   0%|          | 0/68 [00:00<?, ? examples/s]
Filter: 100%|##########| 68/68 [00:00<00:00, 12284.65 examples/s]

Filter:   0%|          | 0/66 [00:00<?, ? examples/s]
Filter: 100%|##########| 66/66 [00:00<00:00, 12138.74 examples/s]

Flattening the indices:   0%|          | 0/66 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 66/66 [00:00<00:00, 12821.86 examples/s]

Casting to class labels:   0%|          | 0/66 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 66/66 [00:00<00:00, 6287.74 examples/s]INFO 2025-05-20 10:36:53,963 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_3_1__Adding_items_to_a_boot_order_list_in_the_web_console.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 426.94ba/s]INFO 2025-05-20 10:36:53,965 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_3_1__Adding_items_to_a_boot_order_list_in_the_web_console_task.yaml
INFO 2025-05-20 10:36:53,977 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:36:53,981 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/224 [00:00<?, ? examples/s]
Filter: 100%|##########| 224/224 [00:00<00:00, 35736.94 examples/s]

Filter:   0%|          | 0/224 [00:00<?, ? examples/s]
Filter: 100%|##########| 224/224 [00:00<00:00, 20113.12 examples/s]

Flattening the indices:   0%|          | 0/224 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 224/224 [00:00<00:00, 32702.99 examples/s]

Map:   0%|          | 0/224 [00:00<?, ? examples/s]
Map: 100%|##########| 224/224 [00:00<00:00, 8606.85 examples/s]

Map:   0%|          | 0/224 [00:00<?, ? examples/s]
Map: 100%|##########| 224/224 [00:00<00:00, 8218.87 examples/s]

Map:   0%|          | 0/224 [00:00<?, ? examples/s]
Map: 100%|##########| 224/224 [00:00<00:00, 8074.29 examples/s]

Filter:   0%|          | 0/224 [00:00<?, ? examples/s]
Filter: 100%|##########| 224/224 [00:00<00:00, 29764.74 examples/s]

Filter:   0%|          | 0/224 [00:00<?, ? examples/s]
Filter: 100%|##########| 224/224 [00:00<00:00, 16246.87 examples/s]

Filter:   0%|          | 0/220 [00:00<?, ? examples/s]
Filter: 100%|##########| 220/220 [00:00<00:00, 16479.68 examples/s]

Flattening the indices:   0%|          | 0/220 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 220/220 [00:00<00:00, 28392.21 examples/s]

Casting to class labels:   0%|          | 0/220 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 220/220 [00:00<00:00, 8597.21 examples/s]INFO 2025-05-20 10:37:14,375 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_17_1__TLS_certificates_for_data_volume_imports.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 164.10ba/s]INFO 2025-05-20 10:37:14,382 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_17_1__TLS_certificates_for_data_volume_imports_task.yaml
INFO 2025-05-20 10:37:14,391 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:37:14,396 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 8139.89 examples/s]

Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 6806.48 examples/s]

Flattening the indices:   0%|          | 0/18 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 18/18 [00:00<00:00, 5096.70 examples/s]

Map:   0%|          | 0/18 [00:00<?, ? examples/s]
Map: 100%|##########| 18/18 [00:00<00:00, 3182.86 examples/s]

Map:   0%|          | 0/18 [00:00<?, ? examples/s]
Map: 100%|##########| 18/18 [00:00<00:00, 3206.25 examples/s]

Map:   0%|          | 0/18 [00:00<?, ? examples/s]
Map: 100%|##########| 18/18 [00:00<00:00, 3008.95 examples/s]

Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 7733.81 examples/s]

Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 6002.34 examples/s]

Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 6069.42 examples/s]

Casting to class labels:   0%|          | 0/18 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 18/18 [00:00<00:00, 3136.71 examples/s]INFO 2025-05-20 10:37:20,565 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_11_3__Migrating_a_virtual_machine_instance_to_another_node.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 859.66ba/s]INFO 2025-05-20 10:37:20,567 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_11_3__Migrating_a_virtual_machine_instance_to_another_node_task.yaml
INFO 2025-05-20 10:37:20,576 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:37:20,581 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/26 [00:00<?, ? examples/s]
Filter: 100%|##########| 26/26 [00:00<00:00, 11156.21 examples/s]

Filter:   0%|          | 0/26 [00:00<?, ? examples/s]
Filter: 100%|##########| 26/26 [00:00<00:00, 8725.55 examples/s]

Flattening the indices:   0%|          | 0/26 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 26/26 [00:00<00:00, 7342.08 examples/s]

Map:   0%|          | 0/26 [00:00<?, ? examples/s]
Map: 100%|##########| 26/26 [00:00<00:00, 4141.11 examples/s]

Map:   0%|          | 0/26 [00:00<?, ? examples/s]
Map: 100%|##########| 26/26 [00:00<00:00, 3903.91 examples/s]

Map:   0%|          | 0/26 [00:00<?, ? examples/s]
Map: 100%|##########| 26/26 [00:00<00:00, 3771.33 examples/s]

Filter:   0%|          | 0/26 [00:00<?, ? examples/s]
Filter: 100%|##########| 26/26 [00:00<00:00, 10188.91 examples/s]

Filter:   0%|          | 0/25 [00:00<?, ? examples/s]
Filter: 100%|##########| 25/25 [00:00<00:00, 7658.31 examples/s]

Filter:   0%|          | 0/25 [00:00<?, ? examples/s]
Filter: 100%|##########| 25/25 [00:00<00:00, 7718.06 examples/s]

Flattening the indices:   0%|          | 0/25 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 25/25 [00:00<00:00, 5742.48 examples/s]

Casting to class labels:   0%|          | 0/25 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 25/25 [00:00<00:00, 3915.67 examples/s]INFO 2025-05-20 10:37:27,241 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_4_2__About_Red_Hat_OpenShift_Virtualization.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 564.66ba/s]INFO 2025-05-20 10:37:27,243 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_2__About_Red_Hat_OpenShift_Virtualization_task.yaml
INFO 2025-05-20 10:37:27,252 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:37:27,257 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 8191.11 examples/s]

Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 6889.08 examples/s]

Flattening the indices:   0%|          | 0/18 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 18/18 [00:00<00:00, 5157.99 examples/s]

Map:   0%|          | 0/18 [00:00<?, ? examples/s]
Map: 100%|##########| 18/18 [00:00<00:00, 3241.21 examples/s]

Map:   0%|          | 0/18 [00:00<?, ? examples/s]
Map: 100%|##########| 18/18 [00:00<00:00, 3062.41 examples/s]

Map:   0%|          | 0/18 [00:00<?, ? examples/s]
Map: 100%|##########| 18/18 [00:00<00:00, 3034.22 examples/s]

Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 7625.24 examples/s]

Filter:   0%|          | 0/16 [00:00<?, ? examples/s]
Filter: 100%|##########| 16/16 [00:00<00:00, 5437.88 examples/s]

Filter:   0%|          | 0/16 [00:00<?, ? examples/s]
Filter: 100%|##########| 16/16 [00:00<00:00, 5651.27 examples/s]

Flattening the indices:   0%|          | 0/16 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 16/16 [00:00<00:00, 3812.35 examples/s]

Casting to class labels:   0%|          | 0/16 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 16/16 [00:00<00:00, 2868.39 examples/s]INFO 2025-05-20 10:37:34,661 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_8_2__OpenShift_Container_Platform_client_commands.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 826.79ba/s]INFO 2025-05-20 10:37:34,663 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_8_2__OpenShift_Container_Platform_client_commands_task.yaml
INFO 2025-05-20 10:37:34,672 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:37:34,677 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/57 [00:00<?, ? examples/s]
Filter: 100%|##########| 57/57 [00:00<00:00, 18301.72 examples/s]

Filter:   0%|          | 0/57 [00:00<?, ? examples/s]
Filter: 100%|##########| 57/57 [00:00<00:00, 13408.60 examples/s]

Flattening the indices:   0%|          | 0/57 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 57/57 [00:00<00:00, 13434.22 examples/s]

Map:   0%|          | 0/57 [00:00<?, ? examples/s]
Map: 100%|##########| 57/57 [00:00<00:00, 5884.06 examples/s]

Map:   0%|          | 0/57 [00:00<?, ? examples/s]
Map: 100%|##########| 57/57 [00:00<00:00, 5693.76 examples/s]

Map:   0%|          | 0/57 [00:00<?, ? examples/s]
Map: 100%|##########| 57/57 [00:00<00:00, 5641.76 examples/s]

Filter:   0%|          | 0/57 [00:00<?, ? examples/s]
Filter: 100%|##########| 57/57 [00:00<00:00, 17014.83 examples/s]

Filter:   0%|          | 0/57 [00:00<?, ? examples/s]
Filter: 100%|##########| 57/57 [00:00<00:00, 11319.85 examples/s]

Filter:   0%|          | 0/57 [00:00<?, ? examples/s]
Filter: 100%|##########| 57/57 [00:00<00:00, 11578.62 examples/s]

Casting to class labels:   0%|          | 0/57 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 57/57 [00:00<00:00, 5983.02 examples/s]INFO 2025-05-20 10:37:46,907 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_12_1__Adding_a_secret__config_map__or_service_account_to_a_virtual_machine.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 402.64ba/s]INFO 2025-05-20 10:37:46,910 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_12_1__Adding_a_secret__config_map__or_service_account_to_a_virtual_machine_task.yaml
INFO 2025-05-20 10:37:46,921 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:37:46,925 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/116 [00:00<?, ? examples/s]
Filter: 100%|##########| 116/116 [00:00<00:00, 29641.72 examples/s]

Filter:   0%|          | 0/116 [00:00<?, ? examples/s]
Filter: 100%|##########| 116/116 [00:00<00:00, 17950.17 examples/s]

Flattening the indices:   0%|          | 0/116 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 116/116 [00:00<00:00, 23552.10 examples/s]

Map:   0%|          | 0/116 [00:00<?, ? examples/s]
Map: 100%|##########| 116/116 [00:00<00:00, 7545.82 examples/s]

Map:   0%|          | 0/116 [00:00<?, ? examples/s]
Map: 100%|##########| 116/116 [00:00<00:00, 7137.77 examples/s]

Map:   0%|          | 0/116 [00:00<?, ? examples/s]
Map: 100%|##########| 116/116 [00:00<00:00, 7088.48 examples/s]

Filter:   0%|          | 0/116 [00:00<?, ? examples/s]
Filter: 100%|##########| 116/116 [00:00<00:00, 24505.86 examples/s]

Filter:   0%|          | 0/115 [00:00<?, ? examples/s]
Filter: 100%|##########| 115/115 [00:00<00:00, 14584.25 examples/s]

Filter:   0%|          | 0/112 [00:00<?, ? examples/s]
Filter: 100%|##########| 112/112 [00:00<00:00, 14109.09 examples/s]

Flattening the indices:   0%|          | 0/112 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 112/112 [00:00<00:00, 19183.36 examples/s]

Casting to class labels:   0%|          | 0/112 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 112/112 [00:00<00:00, 7441.89 examples/s]INFO 2025-05-20 10:38:03,381 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_11_3__Virtualization_metrics.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 281.95ba/s]INFO 2025-05-20 10:38:03,385 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_11_3__Virtualization_metrics_task.yaml
INFO 2025-05-20 10:38:03,394 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:38:03,399 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/34 [00:00<?, ? examples/s]
Filter: 100%|##########| 34/34 [00:00<00:00, 13065.17 examples/s]

Filter:   0%|          | 0/34 [00:00<?, ? examples/s]
Filter: 100%|##########| 34/34 [00:00<00:00, 10348.04 examples/s]

Flattening the indices:   0%|          | 0/34 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 34/34 [00:00<00:00, 9456.65 examples/s]

Map:   0%|          | 0/34 [00:00<?, ? examples/s]
Map: 100%|##########| 34/34 [00:00<00:00, 4731.94 examples/s]

Map:   0%|          | 0/34 [00:00<?, ? examples/s]
Map: 100%|##########| 34/34 [00:00<00:00, 4549.28 examples/s]

Map:   0%|          | 0/34 [00:00<?, ? examples/s]
Map: 100%|##########| 34/34 [00:00<00:00, 4432.48 examples/s]

Filter:   0%|          | 0/34 [00:00<?, ? examples/s]
Filter: 100%|##########| 34/34 [00:00<00:00, 12628.97 examples/s]

Filter:   0%|          | 0/34 [00:00<?, ? examples/s]
Filter: 100%|##########| 34/34 [00:00<00:00, 8901.77 examples/s]

Filter:   0%|          | 0/27 [00:00<?, ? examples/s]
Filter: 100%|##########| 27/27 [00:00<00:00, 7859.95 examples/s]

Flattening the indices:   0%|          | 0/26 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 26/26 [00:00<00:00, 5893.74 examples/s]

Casting to class labels:   0%|          | 0/26 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 26/26 [00:00<00:00, 4015.17 examples/s]INFO 2025-05-20 10:38:13,399 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_12_3__Managing_node_labeling_for_obsolete_CPU_models.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 724.91ba/s]INFO 2025-05-20 10:38:13,401 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_12_3__Managing_node_labeling_for_obsolete_CPU_models_task.yaml
INFO 2025-05-20 10:38:13,411 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:38:13,416 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/123 [00:00<?, ? examples/s]
Filter: 100%|##########| 123/123 [00:00<00:00, 30314.92 examples/s]

Filter:   0%|          | 0/122 [00:00<?, ? examples/s]
Filter: 100%|##########| 122/122 [00:00<00:00, 18181.68 examples/s]

Flattening the indices:   0%|          | 0/122 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 122/122 [00:00<00:00, 23151.98 examples/s]

Map:   0%|          | 0/122 [00:00<?, ? examples/s]
Map: 100%|##########| 122/122 [00:00<00:00, 7645.26 examples/s]

Map:   0%|          | 0/122 [00:00<?, ? examples/s]
Map: 100%|##########| 122/122 [00:00<00:00, 7352.82 examples/s]

Map:   0%|          | 0/122 [00:00<?, ? examples/s]
Map: 100%|##########| 122/122 [00:00<00:00, 7102.28 examples/s]

Filter:   0%|          | 0/122 [00:00<?, ? examples/s]
Filter: 100%|##########| 122/122 [00:00<00:00, 24662.86 examples/s]

Filter:   0%|          | 0/120 [00:00<?, ? examples/s]
Filter: 100%|##########| 120/120 [00:00<00:00, 14329.29 examples/s]

Filter:   0%|          | 0/113 [00:00<?, ? examples/s]
Filter: 100%|##########| 113/113 [00:00<00:00, 14376.69 examples/s]

Flattening the indices:   0%|          | 0/113 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 113/113 [00:00<00:00, 19103.44 examples/s]

Casting to class labels:   0%|          | 0/113 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 113/113 [00:00<00:00, 7476.24 examples/s]INFO 2025-05-20 10:38:28,373 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_12_5__Additional_resources.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 263.59ba/s]INFO 2025-05-20 10:38:28,377 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_12_5__Additional_resources_task.yaml
INFO 2025-05-20 10:38:28,386 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:38:28,391 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/17 [00:00<?, ? examples/s]
Filter: 100%|##########| 17/17 [00:00<00:00, 7820.90 examples/s]

Filter:   0%|          | 0/17 [00:00<?, ? examples/s]
Filter: 100%|##########| 17/17 [00:00<00:00, 6632.85 examples/s]

Flattening the indices:   0%|          | 0/17 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 17/17 [00:00<00:00, 5047.65 examples/s]

Map:   0%|          | 0/17 [00:00<?, ? examples/s]
Map: 100%|##########| 17/17 [00:00<00:00, 3162.00 examples/s]

Map:   0%|          | 0/17 [00:00<?, ? examples/s]
Map: 100%|##########| 17/17 [00:00<00:00, 3003.25 examples/s]

Map:   0%|          | 0/17 [00:00<?, ? examples/s]
Map: 100%|##########| 17/17 [00:00<00:00, 2932.60 examples/s]

Filter:   0%|          | 0/17 [00:00<?, ? examples/s]
Filter: 100%|##########| 17/17 [00:00<00:00, 7448.36 examples/s]

Filter:   0%|          | 0/17 [00:00<?, ? examples/s]
Filter: 100%|##########| 17/17 [00:00<00:00, 5793.71 examples/s]

Filter:   0%|          | 0/17 [00:00<?, ? examples/s]
Filter: 100%|##########| 17/17 [00:00<00:00, 5862.30 examples/s]

Casting to class labels:   0%|          | 0/17 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 17/17 [00:00<00:00, 3006.42 examples/s]INFO 2025-05-20 10:38:34,634 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_12_4_1__Using_skip_node_annotation.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 929.59ba/s]INFO 2025-05-20 10:38:34,635 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_12_4_1__Using_skip_node_annotation_task.yaml
INFO 2025-05-20 10:38:34,645 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:38:34,650 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/113 [00:00<?, ? examples/s]
Filter: 100%|##########| 113/113 [00:00<00:00, 28599.83 examples/s]

Filter:   0%|          | 0/113 [00:00<?, ? examples/s]
Filter: 100%|##########| 113/113 [00:00<00:00, 17703.43 examples/s]

Flattening the indices:   0%|          | 0/113 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 113/113 [00:00<00:00, 22156.81 examples/s]

Map:   0%|          | 0/113 [00:00<?, ? examples/s]
Map: 100%|##########| 113/113 [00:00<00:00, 7744.64 examples/s]

Map:   0%|          | 0/113 [00:00<?, ? examples/s]
Map: 100%|##########| 113/113 [00:00<00:00, 7115.71 examples/s]

Map:   0%|          | 0/113 [00:00<?, ? examples/s]
Map: 100%|##########| 113/113 [00:00<00:00, 7177.24 examples/s]

Filter:   0%|          | 0/113 [00:00<?, ? examples/s]
Filter: 100%|##########| 113/113 [00:00<00:00, 24362.93 examples/s]

Filter:   0%|          | 0/112 [00:00<?, ? examples/s]
Filter: 100%|##########| 112/112 [00:00<00:00, 14242.55 examples/s]

Filter:   0%|          | 0/109 [00:00<?, ? examples/s]
Filter: 100%|##########| 109/109 [00:00<00:00, 14129.22 examples/s]

Flattening the indices:   0%|          | 0/109 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 109/109 [00:00<00:00, 18439.85 examples/s]

Casting to class labels:   0%|          | 0/109 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 109/109 [00:00<00:00, 7324.59 examples/s]INFO 2025-05-20 10:38:52,798 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_6__Monitoring_virtual_machine_health.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 261.51ba/s]INFO 2025-05-20 10:38:52,802 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_6__Monitoring_virtual_machine_health_task.yaml
INFO 2025-05-20 10:38:52,812 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:38:52,816 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/91 [00:00<?, ? examples/s]
Filter: 100%|##########| 91/91 [00:00<00:00, 25260.20 examples/s]

Filter:   0%|          | 0/91 [00:00<?, ? examples/s]
Filter: 100%|##########| 91/91 [00:00<00:00, 16538.77 examples/s]

Flattening the indices:   0%|          | 0/91 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 91/91 [00:00<00:00, 20394.43 examples/s]

Map:   0%|          | 0/91 [00:00<?, ? examples/s]
Map: 100%|##########| 91/91 [00:00<00:00, 7087.74 examples/s]

Map:   0%|          | 0/91 [00:00<?, ? examples/s]
Map: 100%|##########| 91/91 [00:00<00:00, 6751.36 examples/s]

Map:   0%|          | 0/91 [00:00<?, ? examples/s]
Map: 100%|##########| 91/91 [00:00<00:00, 6806.14 examples/s]

Filter:   0%|          | 0/91 [00:00<?, ? examples/s]
Filter: 100%|##########| 91/91 [00:00<00:00, 21107.21 examples/s]

Filter:   0%|          | 0/91 [00:00<?, ? examples/s]
Filter: 100%|##########| 91/91 [00:00<00:00, 13246.86 examples/s]

Filter:   0%|          | 0/90 [00:00<?, ? examples/s]
Filter: 100%|##########| 90/90 [00:00<00:00, 13456.22 examples/s]

Flattening the indices:   0%|          | 0/90 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 90/90 [00:00<00:00, 16694.12 examples/s]

Casting to class labels:   0%|          | 0/90 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 90/90 [00:00<00:00, 7027.07 examples/s]INFO 2025-05-20 10:39:05,469 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_10_1_2__About_virtual_machines_and_boot_sources.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 295.69ba/s]INFO 2025-05-20 10:39:05,473 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_1_2__About_virtual_machines_and_boot_sources_task.yaml
INFO 2025-05-20 10:39:05,482 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:39:05,487 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 17805.22 examples/s]

Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 13352.84 examples/s]

Flattening the indices:   0%|          | 0/53 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 53/53 [00:00<00:00, 13306.48 examples/s]

Map:   0%|          | 0/53 [00:00<?, ? examples/s]
Map: 100%|##########| 53/53 [00:00<00:00, 5806.10 examples/s]

Map:   0%|          | 0/53 [00:00<?, ? examples/s]
Map: 100%|##########| 53/53 [00:00<00:00, 5620.69 examples/s]

Map:   0%|          | 0/53 [00:00<?, ? examples/s]
Map: 100%|##########| 53/53 [00:00<00:00, 5513.48 examples/s]

Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 16640.33 examples/s]

Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 11141.09 examples/s]

Filter:   0%|          | 0/51 [00:00<?, ? examples/s]
Filter: 100%|##########| 51/51 [00:00<00:00, 11112.18 examples/s]

Flattening the indices:   0%|          | 0/51 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 51/51 [00:00<00:00, 10676.26 examples/s]

Casting to class labels:   0%|          | 0/51 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 51/51 [00:00<00:00, 5727.93 examples/s]INFO 2025-05-20 10:39:16,604 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_2__Viewing_OpenShift_Virtualization_logs.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 434.69ba/s]INFO 2025-05-20 10:39:16,607 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_2__Viewing_OpenShift_Virtualization_logs_task.yaml
INFO 2025-05-20 10:39:16,616 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:39:16,620 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/20 [00:00<?, ? examples/s]
Filter: 100%|##########| 20/20 [00:00<00:00, 8941.17 examples/s]

Filter:   0%|          | 0/20 [00:00<?, ? examples/s]
Filter: 100%|##########| 20/20 [00:00<00:00, 7501.89 examples/s]

Flattening the indices:   0%|          | 0/20 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 20/20 [00:00<00:00, 5856.74 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 3497.15 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 3407.93 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 3228.13 examples/s]

Filter:   0%|          | 0/20 [00:00<?, ? examples/s]
Filter: 100%|##########| 20/20 [00:00<00:00, 8383.58 examples/s]

Filter:   0%|          | 0/20 [00:00<?, ? examples/s]
Filter: 100%|##########| 20/20 [00:00<00:00, 6622.93 examples/s]

Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 6106.73 examples/s]

Flattening the indices:   0%|          | 0/18 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 18/18 [00:00<00:00, 4248.11 examples/s]

Casting to class labels:   0%|          | 0/18 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 18/18 [00:00<00:00, 3200.40 examples/s]INFO 2025-05-20 10:39:23,704 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_2_8__About_the_virt_operator.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 912.20ba/s]INFO 2025-05-20 10:39:23,706 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_8__About_the_virt_operator_task.yaml
INFO 2025-05-20 10:39:23,715 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:39:23,720 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/46 [00:00<?, ? examples/s]
Filter: 100%|##########| 46/46 [00:00<00:00, 17153.09 examples/s]

Filter:   0%|          | 0/46 [00:00<?, ? examples/s]
Filter: 100%|##########| 46/46 [00:00<00:00, 12161.23 examples/s]

Flattening the indices:   0%|          | 0/46 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 46/46 [00:00<00:00, 11610.88 examples/s]

Map:   0%|          | 0/46 [00:00<?, ? examples/s]
Map: 100%|##########| 46/46 [00:00<00:00, 5478.86 examples/s]

Map:   0%|          | 0/46 [00:00<?, ? examples/s]
Map: 100%|##########| 46/46 [00:00<00:00, 5203.15 examples/s]

Map:   0%|          | 0/46 [00:00<?, ? examples/s]
Map: 100%|##########| 46/46 [00:00<00:00, 5071.18 examples/s]

Filter:   0%|          | 0/46 [00:00<?, ? examples/s]
Filter: 100%|##########| 46/46 [00:00<00:00, 15103.96 examples/s]

Filter:   0%|          | 0/46 [00:00<?, ? examples/s]
Filter: 100%|##########| 46/46 [00:00<00:00, 10235.98 examples/s]

Filter:   0%|          | 0/45 [00:00<?, ? examples/s]
Filter: 100%|##########| 45/45 [00:00<00:00, 10454.40 examples/s]

Flattening the indices:   0%|          | 0/45 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 45/45 [00:00<00:00, 9390.70 examples/s]

Casting to class labels:   0%|          | 0/45 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 45/45 [00:00<00:00, 5422.11 examples/s]INFO 2025-05-20 10:39:34,801 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_14_4__Adding_VirtIO_drivers_container_disk_to_a_virtual_machine.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 477.87ba/s]INFO 2025-05-20 10:39:34,804 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_14_4__Adding_VirtIO_drivers_container_disk_to_a_virtual_machine_task.yaml
INFO 2025-05-20 10:39:34,814 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:39:34,818 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 15872.14 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 11937.72 examples/s]

Flattening the indices:   0%|          | 0/43 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 43/43 [00:00<00:00, 11246.89 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 5325.86 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 5054.94 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 4916.85 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 14670.17 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 9978.70 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 10248.03 examples/s]

Casting to class labels:   0%|          | 0/43 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 43/43 [00:00<00:00, 5177.26 examples/s]INFO 2025-05-20 10:39:46,789 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_5__Managing_virtual_machine_instances.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 473.56ba/s]INFO 2025-05-20 10:39:46,791 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_5__Managing_virtual_machine_instances_task.yaml
INFO 2025-05-20 10:39:46,802 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:39:46,806 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/120 [00:00<?, ? examples/s]
Filter: 100%|##########| 120/120 [00:00<00:00, 29925.47 examples/s]

Filter:   0%|          | 0/119 [00:00<?, ? examples/s]
Filter: 100%|##########| 119/119 [00:00<00:00, 18182.96 examples/s]

Flattening the indices:   0%|          | 0/119 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 119/119 [00:00<00:00, 23849.49 examples/s]

Map:   0%|          | 0/119 [00:00<?, ? examples/s]
Map: 100%|##########| 119/119 [00:00<00:00, 7689.92 examples/s]

Map:   0%|          | 0/119 [00:00<?, ? examples/s]
Map: 100%|##########| 119/119 [00:00<00:00, 7151.67 examples/s]

Map:   0%|          | 0/119 [00:00<?, ? examples/s]
Map: 100%|##########| 119/119 [00:00<00:00, 7161.52 examples/s]

Filter:   0%|          | 0/119 [00:00<?, ? examples/s]
Filter: 100%|##########| 119/119 [00:00<00:00, 24397.41 examples/s]

Filter:   0%|          | 0/119 [00:00<?, ? examples/s]
Filter: 100%|##########| 119/119 [00:00<00:00, 14845.99 examples/s]

Filter:   0%|          | 0/119 [00:00<?, ? examples/s]
Filter: 100%|##########| 119/119 [00:00<00:00, 14527.10 examples/s]

Casting to class labels:   0%|          | 0/119 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 119/119 [00:00<00:00, 7617.86 examples/s]INFO 2025-05-20 10:40:06,832 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_11_1__Prerequisites.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 273.83ba/s]INFO 2025-05-20 10:40:06,836 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_11_1__Prerequisites_task.yaml
INFO 2025-05-20 10:40:06,846 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:40:06,851 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/130 [00:00<?, ? examples/s]
Filter: 100%|##########| 130/130 [00:00<00:00, 31028.25 examples/s]

Filter:   0%|          | 0/129 [00:00<?, ? examples/s]
Filter: 100%|##########| 129/129 [00:00<00:00, 18171.19 examples/s]

Flattening the indices:   0%|          | 0/129 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 129/129 [00:00<00:00, 24813.81 examples/s]

Map:   0%|          | 0/129 [00:00<?, ? examples/s]
Map: 100%|##########| 129/129 [00:00<00:00, 7822.82 examples/s]

Map:   0%|          | 0/129 [00:00<?, ? examples/s]
Map: 100%|##########| 129/129 [00:00<00:00, 7329.62 examples/s]

Map:   0%|          | 0/129 [00:00<?, ? examples/s]
Map: 100%|##########| 129/129 [00:00<00:00, 7446.94 examples/s]

Filter:   0%|          | 0/129 [00:00<?, ? examples/s]
Filter: 100%|##########| 129/129 [00:00<00:00, 24886.86 examples/s]

Filter:   0%|          | 0/129 [00:00<?, ? examples/s]
Filter: 100%|##########| 129/129 [00:00<00:00, 14779.57 examples/s]

Filter:   0%|          | 0/125 [00:00<?, ? examples/s]
Filter: 100%|##########| 125/125 [00:00<00:00, 14583.81 examples/s]

Flattening the indices:   0%|          | 0/125 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 125/125 [00:00<00:00, 20847.27 examples/s]

Casting to class labels:   0%|          | 0/125 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 125/125 [00:00<00:00, 7742.57 examples/s]INFO 2025-05-20 10:40:21,610 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_5_1_3__Object_maximums.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 216.95ba/s]INFO 2025-05-20 10:40:21,615 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_1_3__Object_maximums_task.yaml
INFO 2025-05-20 10:40:21,625 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:40:21,629 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/45 [00:00<?, ? examples/s]
Filter: 100%|##########| 45/45 [00:00<00:00, 16700.02 examples/s]

Filter:   0%|          | 0/45 [00:00<?, ? examples/s]
Filter: 100%|##########| 45/45 [00:00<00:00, 12089.65 examples/s]

Flattening the indices:   0%|          | 0/45 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 45/45 [00:00<00:00, 11827.53 examples/s]

Map:   0%|          | 0/45 [00:00<?, ? examples/s]
Map: 100%|##########| 45/45 [00:00<00:00, 5531.92 examples/s]

Map:   0%|          | 0/45 [00:00<?, ? examples/s]
Map: 100%|##########| 45/45 [00:00<00:00, 5227.34 examples/s]

Map:   0%|          | 0/45 [00:00<?, ? examples/s]
Map: 100%|##########| 45/45 [00:00<00:00, 5125.42 examples/s]

Filter:   0%|          | 0/45 [00:00<?, ? examples/s]
Filter: 100%|##########| 45/45 [00:00<00:00, 14980.85 examples/s]

Filter:   0%|          | 0/45 [00:00<?, ? examples/s]
Filter: 100%|##########| 45/45 [00:00<00:00, 10189.14 examples/s]

Filter:   0%|          | 0/45 [00:00<?, ? examples/s]
Filter: 100%|##########| 45/45 [00:00<00:00, 10461.93 examples/s]

Casting to class labels:   0%|          | 0/45 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 45/45 [00:00<00:00, 5370.12 examples/s]INFO 2025-05-20 10:40:32,701 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_14_5__Installing_VirtIO_drivers_during_Windows_installation.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 561.34ba/s]INFO 2025-05-20 10:40:32,704 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_14_5__Installing_VirtIO_drivers_during_Windows_installation_task.yaml
INFO 2025-05-20 10:40:32,728 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:40:32,734 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge
INFO 2025-05-20 10:40:32,738 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/896 [00:00<?, ? examples/s]
Filter: 100%|##########| 896/896 [00:00<00:00, 51636.39 examples/s]

Filter:   0%|          | 0/896 [00:00<?, ? examples/s]
Filter: 100%|##########| 896/896 [00:00<00:00, 23157.53 examples/s]

Flattening the indices:   0%|          | 0/895 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 895/895 [00:00<00:00, 49945.48 examples/s]

Map:   0%|          | 0/895 [00:00<?, ? examples/s]
Map: 100%|##########| 895/895 [00:00<00:00, 2484.72 examples/s]
Map: 100%|##########| 895/895 [00:00<00:00, 2473.42 examples/s]

Map:   0%|          | 0/895 [00:00<?, ? examples/s]
Map: 100%|##########| 895/895 [00:00<00:00, 8968.14 examples/s]

Map:   0%|          | 0/895 [00:00<?, ? examples/s]
Map: 100%|##########| 895/895 [00:00<00:00, 9045.70 examples/s]

Filter:   0%|          | 0/895 [00:00<?, ? examples/s]
Filter: 100%|##########| 895/895 [00:00<00:00, 36403.24 examples/s]

Filter:   0%|          | 0/889 [00:00<?, ? examples/s]
Filter: 100%|##########| 889/889 [00:00<00:00, 17876.00 examples/s]

Filter:   0%|          | 0/882 [00:00<?, ? examples/s]
Filter: 100%|##########| 882/882 [00:00<00:00, 18042.66 examples/s]

Flattening the indices:   0%|          | 0/882 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 882/882 [00:00<00:00, 41634.79 examples/s]

Casting to class labels:   0%|          | 0/882 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 882/882 [00:00<00:00, 9469.72 examples/s]INFO 2025-05-20 10:41:21,597 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_16_12__Configuring_a_watchdog.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 42.75ba/s]INFO 2025-05-20 10:41:21,621 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_16_12__Configuring_a_watchdog_task.yaml
INFO 2025-05-20 10:41:21,631 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:41:21,636 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/10 [00:00<?, ? examples/s]
Filter: 100%|##########| 10/10 [00:00<00:00, 4912.51 examples/s]

Filter:   0%|          | 0/10 [00:00<?, ? examples/s]
Filter: 100%|##########| 10/10 [00:00<00:00, 4238.38 examples/s]

Flattening the indices:   0%|          | 0/10 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 10/10 [00:00<00:00, 2933.28 examples/s]

Map:   0%|          | 0/10 [00:00<?, ? examples/s]
Map: 100%|##########| 10/10 [00:00<00:00, 2119.41 examples/s]

Map:   0%|          | 0/10 [00:00<?, ? examples/s]
Map: 100%|##########| 10/10 [00:00<00:00, 2015.62 examples/s]

Map:   0%|          | 0/10 [00:00<?, ? examples/s]
Map: 100%|##########| 10/10 [00:00<00:00, 1913.90 examples/s]

Filter:   0%|          | 0/10 [00:00<?, ? examples/s]
Filter: 100%|##########| 10/10 [00:00<00:00, 4720.13 examples/s]

Filter:   0%|          | 0/10 [00:00<?, ? examples/s]
Filter: 100%|##########| 10/10 [00:00<00:00, 3992.29 examples/s]

Filter:   0%|          | 0/10 [00:00<?, ? examples/s]
Filter: 100%|##########| 10/10 [00:00<00:00, 3988.12 examples/s]

Casting to class labels:   0%|          | 0/10 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 10/10 [00:00<00:00, 2008.00 examples/s]INFO 2025-05-20 10:41:27,149 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_2_4__About_the_cluster_network_addons_operator.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 1034.10ba/s]INFO 2025-05-20 10:41:27,151 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_4__About_the_cluster_network_addons_operator_task.yaml
INFO 2025-05-20 10:41:27,160 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:41:27,164 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/19 [00:00<?, ? examples/s]
Filter: 100%|##########| 19/19 [00:00<00:00, 8026.16 examples/s]

Filter:   0%|          | 0/19 [00:00<?, ? examples/s]
Filter: 100%|##########| 19/19 [00:00<00:00, 6732.43 examples/s]

Flattening the indices:   0%|          | 0/19 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 19/19 [00:00<00:00, 5251.17 examples/s]

Map:   0%|          | 0/19 [00:00<?, ? examples/s]
Map: 100%|##########| 19/19 [00:00<00:00, 3302.46 examples/s]

Map:   0%|          | 0/19 [00:00<?, ? examples/s]
Map: 100%|##########| 19/19 [00:00<00:00, 3227.43 examples/s]

Map:   0%|          | 0/19 [00:00<?, ? examples/s]
Map: 100%|##########| 19/19 [00:00<00:00, 3037.96 examples/s]

Filter:   0%|          | 0/19 [00:00<?, ? examples/s]
Filter: 100%|##########| 19/19 [00:00<00:00, 8097.11 examples/s]

Filter:   0%|          | 0/19 [00:00<?, ? examples/s]
Filter: 100%|##########| 19/19 [00:00<00:00, 6327.25 examples/s]

Filter:   0%|          | 0/19 [00:00<?, ? examples/s]
Filter: 100%|##########| 19/19 [00:00<00:00, 6357.03 examples/s]

Casting to class labels:   0%|          | 0/19 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 19/19 [00:00<00:00, 3206.40 examples/s]INFO 2025-05-20 10:41:34,382 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_15_1__About_vTPM_devices.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 694.54ba/s]INFO 2025-05-20 10:41:34,384 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_15_1__About_vTPM_devices_task.yaml
INFO 2025-05-20 10:41:34,393 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:41:34,398 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/15 [00:00<?, ? examples/s]
Filter: 100%|##########| 15/15 [00:00<00:00, 7116.23 examples/s]

Filter:   0%|          | 0/15 [00:00<?, ? examples/s]
Filter: 100%|##########| 15/15 [00:00<00:00, 6066.98 examples/s]

Flattening the indices:   0%|          | 0/15 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 15/15 [00:00<00:00, 4392.86 examples/s]

Map:   0%|          | 0/15 [00:00<?, ? examples/s]
Map: 100%|##########| 15/15 [00:00<00:00, 2826.99 examples/s]

Map:   0%|          | 0/15 [00:00<?, ? examples/s]
Map: 100%|##########| 15/15 [00:00<00:00, 2746.88 examples/s]

Map:   0%|          | 0/15 [00:00<?, ? examples/s]
Map: 100%|##########| 15/15 [00:00<00:00, 2648.37 examples/s]

Filter:   0%|          | 0/15 [00:00<?, ? examples/s]
Filter: 100%|##########| 15/15 [00:00<00:00, 6662.56 examples/s]

Filter:   0%|          | 0/13 [00:00<?, ? examples/s]
Filter: 100%|##########| 13/13 [00:00<00:00, 4878.84 examples/s]

Filter:   0%|          | 0/13 [00:00<?, ? examples/s]
Filter: 100%|##########| 13/13 [00:00<00:00, 4886.28 examples/s]

Flattening the indices:   0%|          | 0/13 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 13/13 [00:00<00:00, 3068.95 examples/s]

Casting to class labels:   0%|          | 0/13 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 13/13 [00:00<00:00, 2463.34 examples/s]INFO 2025-05-20 10:41:40,249 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_12_4__Preventing_node_reconciliation.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 915.39ba/s]INFO 2025-05-20 10:41:40,251 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_12_4__Preventing_node_reconciliation_task.yaml
INFO 2025-05-20 10:41:40,270 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:41:40,274 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/433 [00:00<?, ? examples/s]
Filter: 100%|##########| 433/433 [00:00<00:00, 46531.74 examples/s]

Filter:   0%|          | 0/433 [00:00<?, ? examples/s]
Filter: 100%|##########| 433/433 [00:00<00:00, 21716.55 examples/s]

Flattening the indices:   0%|          | 0/433 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 433/433 [00:00<00:00, 38817.89 examples/s]

Map:   0%|          | 0/433 [00:00<?, ? examples/s]
Map: 100%|##########| 433/433 [00:00<00:00, 9293.92 examples/s]

Map:   0%|          | 0/433 [00:00<?, ? examples/s]
Map: 100%|##########| 433/433 [00:00<00:00, 8779.87 examples/s]

Map:   0%|          | 0/433 [00:00<?, ? examples/s]
Map: 100%|##########| 433/433 [00:00<00:00, 8666.45 examples/s]

Filter:   0%|          | 0/433 [00:00<?, ? examples/s]
Filter: 100%|##########| 433/433 [00:00<00:00, 34113.48 examples/s]

Filter:   0%|          | 0/429 [00:00<?, ? examples/s]
Filter: 100%|##########| 429/429 [00:00<00:00, 17054.86 examples/s]

Filter:   0%|          | 0/421 [00:00<?, ? examples/s]
Filter: 100%|##########| 421/421 [00:00<00:00, 17301.78 examples/s]

Flattening the indices:   0%|          | 0/421 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 421/421 [00:00<00:00, 36441.35 examples/s]

Casting to class labels:   0%|          | 0/421 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 421/421 [00:00<00:00, 9213.49 examples/s]INFO 2025-05-20 10:42:09,694 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_19__Virtual_machine_networking.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 68.39ba/s]INFO 2025-05-20 10:42:09,709 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_19__Virtual_machine_networking_task.yaml
INFO 2025-05-20 10:42:09,719 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:42:09,724 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/23 [00:00<?, ? examples/s]
Filter: 100%|##########| 23/23 [00:00<00:00, 9947.31 examples/s]

Filter:   0%|          | 0/23 [00:00<?, ? examples/s]
Filter: 100%|##########| 23/23 [00:00<00:00, 7893.71 examples/s]

Flattening the indices:   0%|          | 0/23 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 23/23 [00:00<00:00, 6328.33 examples/s]

Map:   0%|          | 0/23 [00:00<?, ? examples/s]
Map: 100%|##########| 23/23 [00:00<00:00, 3775.69 examples/s]

Map:   0%|          | 0/23 [00:00<?, ? examples/s]
Map: 100%|##########| 23/23 [00:00<00:00, 3637.05 examples/s]

Map:   0%|          | 0/23 [00:00<?, ? examples/s]
Map: 100%|##########| 23/23 [00:00<00:00, 3489.44 examples/s]

Filter:   0%|          | 0/23 [00:00<?, ? examples/s]
Filter: 100%|##########| 23/23 [00:00<00:00, 9605.60 examples/s]

Filter:   0%|          | 0/23 [00:00<?, ? examples/s]
Filter: 100%|##########| 23/23 [00:00<00:00, 7313.80 examples/s]

Filter:   0%|          | 0/23 [00:00<?, ? examples/s]
Filter: 100%|##########| 23/23 [00:00<00:00, 7182.56 examples/s]

Casting to class labels:   0%|          | 0/23 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 23/23 [00:00<00:00, 3596.10 examples/s]INFO 2025-05-20 10:42:20,628 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_1_1_1__OpenShift_Virtualization_supported_cluster_version.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 724.03ba/s]INFO 2025-05-20 10:42:20,630 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_1_1_1__OpenShift_Virtualization_supported_cluster_version_task.yaml
INFO 2025-05-20 10:42:20,639 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:42:20,644 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/62 [00:00<?, ? examples/s]
Filter: 100%|##########| 62/62 [00:00<00:00, 20715.91 examples/s]

Filter:   0%|          | 0/62 [00:00<?, ? examples/s]
Filter: 100%|##########| 62/62 [00:00<00:00, 14139.90 examples/s]

Flattening the indices:   0%|          | 0/62 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 62/62 [00:00<00:00, 14742.72 examples/s]

Map:   0%|          | 0/62 [00:00<?, ? examples/s]
Map: 100%|##########| 62/62 [00:00<00:00, 6184.96 examples/s]

Map:   0%|          | 0/62 [00:00<?, ? examples/s]
Map: 100%|##########| 62/62 [00:00<00:00, 5900.90 examples/s]

Map:   0%|          | 0/62 [00:00<?, ? examples/s]
Map: 100%|##########| 62/62 [00:00<00:00, 5824.25 examples/s]

Filter:   0%|          | 0/62 [00:00<?, ? examples/s]
Filter: 100%|##########| 62/62 [00:00<00:00, 18134.37 examples/s]

Filter:   0%|          | 0/61 [00:00<?, ? examples/s]
Filter: 100%|##########| 61/61 [00:00<00:00, 12049.19 examples/s]

Filter:   0%|          | 0/61 [00:00<?, ? examples/s]
Filter: 100%|##########| 61/61 [00:00<00:00, 11865.90 examples/s]

Flattening the indices:   0%|          | 0/61 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 61/61 [00:00<00:00, 12397.76 examples/s]

Casting to class labels:   0%|          | 0/61 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 61/61 [00:00<00:00, 6122.78 examples/s]INFO 2025-05-20 10:42:34,699 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_10_4_1__Creating_a_custom_namespace_for_templates.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 474.74ba/s]INFO 2025-05-20 10:42:34,702 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_4_1__Creating_a_custom_namespace_for_templates_task.yaml
INFO 2025-05-20 10:42:34,711 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:42:34,715 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/15 [00:00<?, ? examples/s]
Filter: 100%|##########| 15/15 [00:00<00:00, 6892.48 examples/s]

Filter:   0%|          | 0/15 [00:00<?, ? examples/s]
Filter: 100%|##########| 15/15 [00:00<00:00, 5906.92 examples/s]

Flattening the indices:   0%|          | 0/15 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 15/15 [00:00<00:00, 4540.60 examples/s]

Map:   0%|          | 0/15 [00:00<?, ? examples/s]
Map: 100%|##########| 15/15 [00:00<00:00, 2897.95 examples/s]

Map:   0%|          | 0/15 [00:00<?, ? examples/s]
Map: 100%|##########| 15/15 [00:00<00:00, 2673.69 examples/s]

Map:   0%|          | 0/15 [00:00<?, ? examples/s]
Map: 100%|##########| 15/15 [00:00<00:00, 2704.26 examples/s]

Filter:   0%|          | 0/15 [00:00<?, ? examples/s]
Filter: 100%|##########| 15/15 [00:00<00:00, 6591.36 examples/s]

Filter:   0%|          | 0/15 [00:00<?, ? examples/s]
Filter: 100%|##########| 15/15 [00:00<00:00, 5307.01 examples/s]

Filter:   0%|          | 0/15 [00:00<?, ? examples/s]
Filter: 100%|##########| 15/15 [00:00<00:00, 5394.83 examples/s]

Casting to class labels:   0%|          | 0/15 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 15/15 [00:00<00:00, 2769.86 examples/s]INFO 2025-05-20 10:42:43,824 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_4_5__Technology_Preview_features.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 813.80ba/s]INFO 2025-05-20 10:42:43,826 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_5__Technology_Preview_features_task.yaml
INFO 2025-05-20 10:42:43,835 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:42:43,840 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/65 [00:00<?, ? examples/s]
Filter: 100%|##########| 65/65 [00:00<00:00, 21161.98 examples/s]

Filter:   0%|          | 0/65 [00:00<?, ? examples/s]
Filter: 100%|##########| 65/65 [00:00<00:00, 14364.05 examples/s]

Flattening the indices:   0%|          | 0/65 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 65/65 [00:00<00:00, 15305.96 examples/s]

Map:   0%|          | 0/65 [00:00<?, ? examples/s]
Map: 100%|##########| 65/65 [00:00<00:00, 6210.53 examples/s]

Map:   0%|          | 0/65 [00:00<?, ? examples/s]
Map: 100%|##########| 65/65 [00:00<00:00, 5961.86 examples/s]

Map:   0%|          | 0/65 [00:00<?, ? examples/s]
Map: 100%|##########| 65/65 [00:00<00:00, 5864.14 examples/s]

Filter:   0%|          | 0/65 [00:00<?, ? examples/s]
Filter: 100%|##########| 65/65 [00:00<00:00, 18199.58 examples/s]

Filter:   0%|          | 0/65 [00:00<?, ? examples/s]
Filter: 100%|##########| 65/65 [00:00<00:00, 11849.86 examples/s]

Filter:   0%|          | 0/65 [00:00<?, ? examples/s]
Filter: 100%|##########| 65/65 [00:00<00:00, 12110.42 examples/s]

Casting to class labels:   0%|          | 0/65 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 65/65 [00:00<00:00, 6199.09 examples/s]INFO 2025-05-20 10:42:58,834 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_10_1__Installing_QEMU_guest_agent_on_a_Linux_virtual_machine.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 396.55ba/s]INFO 2025-05-20 10:42:58,837 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_10_1__Installing_QEMU_guest_agent_on_a_Linux_virtual_machine_task.yaml
INFO 2025-05-20 10:42:58,846 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:42:58,851 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/33 [00:00<?, ? examples/s]
Filter: 100%|##########| 33/33 [00:00<00:00, 12701.85 examples/s]

Filter:   0%|          | 0/33 [00:00<?, ? examples/s]
Filter: 100%|##########| 33/33 [00:00<00:00, 10056.82 examples/s]

Flattening the indices:   0%|          | 0/33 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 33/33 [00:00<00:00, 8967.99 examples/s]

Map:   0%|          | 0/33 [00:00<?, ? examples/s]
Map: 100%|##########| 33/33 [00:00<00:00, 4655.17 examples/s]

Map:   0%|          | 0/33 [00:00<?, ? examples/s]
Map: 100%|##########| 33/33 [00:00<00:00, 4436.85 examples/s]

Map:   0%|          | 0/33 [00:00<?, ? examples/s]
Map: 100%|##########| 33/33 [00:00<00:00, 4362.46 examples/s]

Filter:   0%|          | 0/33 [00:00<?, ? examples/s]
Filter: 100%|##########| 33/33 [00:00<00:00, 12056.80 examples/s]

Filter:   0%|          | 0/33 [00:00<?, ? examples/s]
Filter: 100%|##########| 33/33 [00:00<00:00, 8732.07 examples/s]

Filter:   0%|          | 0/33 [00:00<?, ? examples/s]
Filter: 100%|##########| 33/33 [00:00<00:00, 8895.95 examples/s]

Casting to class labels:   0%|          | 0/33 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 33/33 [00:00<00:00, 4506.77 examples/s]INFO 2025-05-20 10:43:09,015 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_9_2__Deleting_nodes_from_a_bare_metal_cluster.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 569.96ba/s]INFO 2025-05-20 10:43:09,017 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_9_2__Deleting_nodes_from_a_bare_metal_cluster_task.yaml
INFO 2025-05-20 10:43:09,027 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:43:09,031 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 16184.05 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 12007.66 examples/s]

Flattening the indices:   0%|          | 0/43 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 43/43 [00:00<00:00, 11359.52 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 5281.88 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 5129.12 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 5005.83 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 14451.53 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 10012.49 examples/s]

Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 10135.35 examples/s]

Flattening the indices:   0%|          | 0/41 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 41/41 [00:00<00:00, 8731.92 examples/s]

Casting to class labels:   0%|          | 0/41 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 41/41 [00:00<00:00, 5138.70 examples/s]INFO 2025-05-20 10:43:18,336 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_11_2__Live_migration_limits_and_timeouts.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 619.45ba/s]INFO 2025-05-20 10:43:18,338 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_11_2__Live_migration_limits_and_timeouts_task.yaml
INFO 2025-05-20 10:43:18,349 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:43:18,354 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/182 [00:00<?, ? examples/s]
Filter: 100%|##########| 182/182 [00:00<00:00, 33172.40 examples/s]

Filter:   0%|          | 0/182 [00:00<?, ? examples/s]
Filter: 100%|##########| 182/182 [00:00<00:00, 20060.00 examples/s]

Flattening the indices:   0%|          | 0/182 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 182/182 [00:00<00:00, 30342.77 examples/s]

Map:   0%|          | 0/182 [00:00<?, ? examples/s]
Map: 100%|##########| 182/182 [00:00<00:00, 8323.76 examples/s]

Map:   0%|          | 0/182 [00:00<?, ? examples/s]
Map: 100%|##########| 182/182 [00:00<00:00, 7937.15 examples/s]

Map:   0%|          | 0/182 [00:00<?, ? examples/s]
Map: 100%|##########| 182/182 [00:00<00:00, 8005.07 examples/s]

Filter:   0%|          | 0/182 [00:00<?, ? examples/s]
Filter: 100%|##########| 182/182 [00:00<00:00, 28570.06 examples/s]

Filter:   0%|          | 0/182 [00:00<?, ? examples/s]
Filter: 100%|##########| 182/182 [00:00<00:00, 16050.53 examples/s]

Filter:   0%|          | 0/179 [00:00<?, ? examples/s]
Filter: 100%|##########| 179/179 [00:00<00:00, 15989.70 examples/s]

Flattening the indices:   0%|          | 0/179 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 179/179 [00:00<00:00, 26919.34 examples/s]

Casting to class labels:   0%|          | 0/179 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 179/179 [00:00<00:00, 8433.75 examples/s]INFO 2025-05-20 10:43:36,062 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_1_7__Additional_resources.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 177.40ba/s]INFO 2025-05-20 10:43:36,068 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1_7__Additional_resources_task.yaml
INFO 2025-05-20 10:43:36,079 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:43:36,083 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/134 [00:00<?, ? examples/s]
Filter: 100%|##########| 134/134 [00:00<00:00, 28936.66 examples/s]

Filter:   0%|          | 0/134 [00:00<?, ? examples/s]
Filter: 100%|##########| 134/134 [00:00<00:00, 18317.53 examples/s]

Flattening the indices:   0%|          | 0/134 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 134/134 [00:00<00:00, 26794.28 examples/s]

Map:   0%|          | 0/134 [00:00<?, ? examples/s]
Map: 100%|##########| 134/134 [00:00<00:00, 7922.93 examples/s]

Map:   0%|          | 0/134 [00:00<?, ? examples/s]
Map: 100%|##########| 134/134 [00:00<00:00, 7551.72 examples/s]

Map:   0%|          | 0/134 [00:00<?, ? examples/s]
Map: 100%|##########| 134/134 [00:00<00:00, 7432.28 examples/s]

Filter:   0%|          | 0/134 [00:00<?, ? examples/s]
Filter: 100%|##########| 134/134 [00:00<00:00, 25809.92 examples/s]

Filter:   0%|          | 0/133 [00:00<?, ? examples/s]
Filter: 100%|##########| 133/133 [00:00<00:00, 14852.43 examples/s]

Filter:   0%|          | 0/132 [00:00<?, ? examples/s]
Filter: 100%|##########| 132/132 [00:00<00:00, 14954.97 examples/s]

Flattening the indices:   0%|          | 0/132 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 132/132 [00:00<00:00, 21879.00 examples/s]

Casting to class labels:   0%|          | 0/132 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 132/132 [00:00<00:00, 7798.74 examples/s]INFO 2025-05-20 10:43:52,445 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_14_1_1__Installing_the_OADP_Operator.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 266.98ba/s]INFO 2025-05-20 10:43:52,449 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_14_1_1__Installing_the_OADP_Operator_task.yaml
INFO 2025-05-20 10:43:52,459 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:43:52,464 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/58 [00:00<?, ? examples/s]
Filter: 100%|##########| 58/58 [00:00<00:00, 20099.94 examples/s]

Filter:   0%|          | 0/58 [00:00<?, ? examples/s]
Filter: 100%|##########| 58/58 [00:00<00:00, 13825.28 examples/s]

Flattening the indices:   0%|          | 0/58 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 58/58 [00:00<00:00, 14173.25 examples/s]

Map:   0%|          | 0/58 [00:00<?, ? examples/s]
Map: 100%|##########| 58/58 [00:00<00:00, 6181.73 examples/s]

Map:   0%|          | 0/58 [00:00<?, ? examples/s]
Map: 100%|##########| 58/58 [00:00<00:00, 5745.08 examples/s]

Map:   0%|          | 0/58 [00:00<?, ? examples/s]
Map: 100%|##########| 58/58 [00:00<00:00, 5756.91 examples/s]

Filter:   0%|          | 0/58 [00:00<?, ? examples/s]
Filter: 100%|##########| 58/58 [00:00<00:00, 17466.23 examples/s]

Filter:   0%|          | 0/57 [00:00<?, ? examples/s]
Filter: 100%|##########| 57/57 [00:00<00:00, 11699.88 examples/s]

Filter:   0%|          | 0/57 [00:00<?, ? examples/s]
Filter: 100%|##########| 57/57 [00:00<00:00, 11826.05 examples/s]

Flattening the indices:   0%|          | 0/57 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 57/57 [00:00<00:00, 11442.84 examples/s]

Casting to class labels:   0%|          | 0/57 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 57/57 [00:00<00:00, 5995.92 examples/s]INFO 2025-05-20 10:44:08,098 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_5_4_2__Subscribing_to_the_OpenShift_Virtualization_catalog_by_using_the_CLI.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 455.85ba/s]INFO 2025-05-20 10:44:08,101 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_4_2__Subscribing_to_the_OpenShift_Virtualization_catalog_by_using_the_CLI_task.yaml
INFO 2025-05-20 10:44:08,110 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:44:08,115 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/35 [00:00<?, ? examples/s]
Filter: 100%|##########| 35/35 [00:00<00:00, 14033.14 examples/s]

Filter:   0%|          | 0/35 [00:00<?, ? examples/s]
Filter: 100%|##########| 35/35 [00:00<00:00, 10639.27 examples/s]

Flattening the indices:   0%|          | 0/35 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 35/35 [00:00<00:00, 9489.99 examples/s]

Map:   0%|          | 0/35 [00:00<?, ? examples/s]
Map: 100%|##########| 35/35 [00:00<00:00, 4871.59 examples/s]

Map:   0%|          | 0/35 [00:00<?, ? examples/s]
Map: 100%|##########| 35/35 [00:00<00:00, 4643.24 examples/s]

Map:   0%|          | 0/35 [00:00<?, ? examples/s]
Map: 100%|##########| 35/35 [00:00<00:00, 4468.95 examples/s]

Filter:   0%|          | 0/35 [00:00<?, ? examples/s]
Filter: 100%|##########| 35/35 [00:00<00:00, 12852.45 examples/s]

Filter:   0%|          | 0/35 [00:00<?, ? examples/s]
Filter: 100%|##########| 35/35 [00:00<00:00, 9101.09 examples/s]

Filter:   0%|          | 0/35 [00:00<?, ? examples/s]
Filter: 100%|##########| 35/35 [00:00<00:00, 9429.03 examples/s]

Casting to class labels:   0%|          | 0/35 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 35/35 [00:00<00:00, 4755.45 examples/s]INFO 2025-05-20 10:44:15,647 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_4_3__New_and_changed_features.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 693.96ba/s]INFO 2025-05-20 10:44:15,649 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_3__New_and_changed_features_task.yaml
INFO 2025-05-20 10:44:15,659 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:44:15,663 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/47 [00:00<?, ? examples/s]
Filter: 100%|##########| 47/47 [00:00<00:00, 17152.38 examples/s]

Filter:   0%|          | 0/47 [00:00<?, ? examples/s]
Filter: 100%|##########| 47/47 [00:00<00:00, 12512.36 examples/s]

Flattening the indices:   0%|          | 0/47 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 47/47 [00:00<00:00, 12219.95 examples/s]

Map:   0%|          | 0/47 [00:00<?, ? examples/s]
Map: 100%|##########| 47/47 [00:00<00:00, 5536.96 examples/s]

Map:   0%|          | 0/47 [00:00<?, ? examples/s]
Map: 100%|##########| 47/47 [00:00<00:00, 5297.97 examples/s]

Map:   0%|          | 0/47 [00:00<?, ? examples/s]
Map: 100%|##########| 47/47 [00:00<00:00, 5231.05 examples/s]

Filter:   0%|          | 0/47 [00:00<?, ? examples/s]
Filter: 100%|##########| 47/47 [00:00<00:00, 15479.57 examples/s]

Filter:   0%|          | 0/47 [00:00<?, ? examples/s]
Filter: 100%|##########| 47/47 [00:00<00:00, 10672.53 examples/s]

Filter:   0%|          | 0/45 [00:00<?, ? examples/s]
Filter: 100%|##########| 45/45 [00:00<00:00, 10286.88 examples/s]

Flattening the indices:   0%|          | 0/45 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 45/45 [00:00<00:00, 9495.10 examples/s]

Casting to class labels:   0%|          | 0/45 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 45/45 [00:00<00:00, 5457.39 examples/s]INFO 2025-05-20 10:44:24,229 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_6_4__Unpausing_a_virtual_machine.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 571.35ba/s]INFO 2025-05-20 10:44:24,232 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_6_4__Unpausing_a_virtual_machine_task.yaml
INFO 2025-05-20 10:44:24,243 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:44:24,247 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/180 [00:00<?, ? examples/s]
Filter: 100%|##########| 180/180 [00:00<00:00, 35777.40 examples/s]

Filter:   0%|          | 0/180 [00:00<?, ? examples/s]
Filter: 100%|##########| 180/180 [00:00<00:00, 20048.19 examples/s]

Flattening the indices:   0%|          | 0/180 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 180/180 [00:00<00:00, 30165.20 examples/s]

Map:   0%|          | 0/180 [00:00<?, ? examples/s]
Map: 100%|##########| 180/180 [00:00<00:00, 8324.33 examples/s]

Map:   0%|          | 0/180 [00:00<?, ? examples/s]
Map: 100%|##########| 180/180 [00:00<00:00, 7950.37 examples/s]

Map:   0%|          | 0/180 [00:00<?, ? examples/s]
Map: 100%|##########| 180/180 [00:00<00:00, 7735.00 examples/s]

Filter:   0%|          | 0/180 [00:00<?, ? examples/s]
Filter: 100%|##########| 180/180 [00:00<00:00, 28317.57 examples/s]

Filter:   0%|          | 0/180 [00:00<?, ? examples/s]
Filter: 100%|##########| 180/180 [00:00<00:00, 15618.98 examples/s]

Filter:   0%|          | 0/177 [00:00<?, ? examples/s]
Filter: 100%|##########| 177/177 [00:00<00:00, 15741.31 examples/s]

Flattening the indices:   0%|          | 0/177 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 177/177 [00:00<00:00, 25304.79 examples/s]

Casting to class labels:   0%|          | 0/177 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 177/177 [00:00<00:00, 8297.57 examples/s]INFO 2025-05-20 10:44:41,818 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_1_1__Using_a_Quick_Start_to_create_a_virtual_machine.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 195.67ba/s]INFO 2025-05-20 10:44:41,824 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1_1__Using_a_Quick_Start_to_create_a_virtual_machine_task.yaml
INFO 2025-05-20 10:44:41,833 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:44:41,837 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/5 [00:00<?, ? examples/s]
Filter: 100%|##########| 5/5 [00:00<00:00, 2540.16 examples/s]

Filter:   0%|          | 0/5 [00:00<?, ? examples/s]
Filter: 100%|##########| 5/5 [00:00<00:00, 2333.28 examples/s]

Flattening the indices:   0%|          | 0/5 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 5/5 [00:00<00:00, 1513.75 examples/s]

Map:   0%|          | 0/5 [00:00<?, ? examples/s]
Map: 100%|##########| 5/5 [00:00<00:00, 1180.63 examples/s]

Map:   0%|          | 0/5 [00:00<?, ? examples/s]
Map: 100%|##########| 5/5 [00:00<00:00, 1153.80 examples/s]

Map:   0%|          | 0/5 [00:00<?, ? examples/s]
Map: 100%|##########| 5/5 [00:00<00:00, 1095.63 examples/s]

Filter:   0%|          | 0/5 [00:00<?, ? examples/s]
Filter: 100%|##########| 5/5 [00:00<00:00, 2541.08 examples/s]

Filter:   0%|          | 0/5 [00:00<?, ? examples/s]
Filter: 100%|##########| 5/5 [00:00<00:00, 2220.62 examples/s]

Filter:   0%|          | 0/5 [00:00<?, ? examples/s]
Filter: 100%|##########| 5/5 [00:00<00:00, 2230.54 examples/s]

Casting to class labels:   0%|          | 0/5 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 5/5 [00:00<00:00, 1101.45 examples/s]INFO 2025-05-20 10:44:44,800 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_2_6__About_the_ssp_operator.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 1131.46ba/s]INFO 2025-05-20 10:44:44,801 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_6__About_the_ssp_operator_task.yaml
INFO 2025-05-20 10:44:44,811 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:44:44,816 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/130 [00:00<?, ? examples/s]
Filter: 100%|##########| 130/130 [00:00<00:00, 31623.91 examples/s]

Filter:   0%|          | 0/130 [00:00<?, ? examples/s]
Filter: 100%|##########| 130/130 [00:00<00:00, 18847.55 examples/s]

Flattening the indices:   0%|          | 0/130 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 130/130 [00:00<00:00, 24642.27 examples/s]

Map:   0%|          | 0/130 [00:00<?, ? examples/s]
Map: 100%|##########| 130/130 [00:00<00:00, 7686.75 examples/s]

Map:   0%|          | 0/130 [00:00<?, ? examples/s]
Map: 100%|##########| 130/130 [00:00<00:00, 7394.85 examples/s]

Map:   0%|          | 0/130 [00:00<?, ? examples/s]
Map: 100%|##########| 130/130 [00:00<00:00, 7390.44 examples/s]

Filter:   0%|          | 0/130 [00:00<?, ? examples/s]
Filter: 100%|##########| 130/130 [00:00<00:00, 25529.52 examples/s]

Filter:   0%|          | 0/130 [00:00<?, ? examples/s]
Filter: 100%|##########| 130/130 [00:00<00:00, 14940.66 examples/s]

Filter:   0%|          | 0/130 [00:00<?, ? examples/s]
Filter: 100%|##########| 130/130 [00:00<00:00, 15077.83 examples/s]

Casting to class labels:   0%|          | 0/130 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 130/130 [00:00<00:00, 7670.64 examples/s]INFO 2025-05-20 10:45:02,629 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_2_3__Editing_a_virtual_machine_YAML_configuration_using_the_CLI.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 243.57ba/s]INFO 2025-05-20 10:45:02,634 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_2_3__Editing_a_virtual_machine_YAML_configuration_using_the_CLI_task.yaml
INFO 2025-05-20 10:45:02,643 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:45:02,647 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/20 [00:00<?, ? examples/s]
Filter: 100%|##########| 20/20 [00:00<00:00, 8883.41 examples/s]

Filter:   0%|          | 0/20 [00:00<?, ? examples/s]
Filter: 100%|##########| 20/20 [00:00<00:00, 7373.95 examples/s]

Flattening the indices:   0%|          | 0/20 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 20/20 [00:00<00:00, 5754.69 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 3474.98 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 3360.15 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 3293.14 examples/s]

Filter:   0%|          | 0/20 [00:00<?, ? examples/s]
Filter: 100%|##########| 20/20 [00:00<00:00, 8467.35 examples/s]

Filter:   0%|          | 0/20 [00:00<?, ? examples/s]
Filter: 100%|##########| 20/20 [00:00<00:00, 6568.48 examples/s]

Filter:   0%|          | 0/20 [00:00<?, ? examples/s]
Filter: 100%|##########| 20/20 [00:00<00:00, 6644.97 examples/s]

Casting to class labels:   0%|          | 0/20 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 20/20 [00:00<00:00, 3396.34 examples/s]INFO 2025-05-20 10:45:11,056 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_7_3_1__Viewing_the_SCC_and_RBAC_definitions_for_the_kubevirt_controller.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 791.53ba/s]INFO 2025-05-20 10:45:11,058 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_7_3_1__Viewing_the_SCC_and_RBAC_definitions_for_the_kubevirt_controller_task.yaml
INFO 2025-05-20 10:45:11,067 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:45:11,072 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 9257.00 examples/s]

Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 7645.87 examples/s]

Flattening the indices:   0%|          | 0/21 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 21/21 [00:00<00:00, 6305.42 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 3675.22 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 3456.98 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 3339.92 examples/s]

Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 9007.10 examples/s]

Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 6972.25 examples/s]

Filter:   0%|          | 0/20 [00:00<?, ? examples/s]
Filter: 100%|##########| 20/20 [00:00<00:00, 6584.46 examples/s]

Flattening the indices:   0%|          | 0/20 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 20/20 [00:00<00:00, 4730.51 examples/s]

Casting to class labels:   0%|          | 0/20 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 20/20 [00:00<00:00, 3460.36 examples/s]INFO 2025-05-20 10:45:17,476 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_4_2_1__OpenShift_Virtualization_supported_cluster_version.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 759.98ba/s]INFO 2025-05-20 10:45:17,478 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_2_1__OpenShift_Virtualization_supported_cluster_version_task.yaml
INFO 2025-05-20 10:45:17,487 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:45:17,492 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/32 [00:00<?, ? examples/s]
Filter: 100%|##########| 32/32 [00:00<00:00, 12224.95 examples/s]

Filter:   0%|          | 0/32 [00:00<?, ? examples/s]
Filter: 100%|##########| 32/32 [00:00<00:00, 10010.27 examples/s]

Flattening the indices:   0%|          | 0/32 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 32/32 [00:00<00:00, 8283.51 examples/s]

Map:   0%|          | 0/32 [00:00<?, ? examples/s]
Map: 100%|##########| 32/32 [00:00<00:00, 4556.40 examples/s]

Map:   0%|          | 0/32 [00:00<?, ? examples/s]
Map: 100%|##########| 32/32 [00:00<00:00, 4404.05 examples/s]

Map:   0%|          | 0/32 [00:00<?, ? examples/s]
Map: 100%|##########| 32/32 [00:00<00:00, 4318.60 examples/s]

Filter:   0%|          | 0/32 [00:00<?, ? examples/s]
Filter: 100%|##########| 32/32 [00:00<00:00, 11966.63 examples/s]

Filter:   0%|          | 0/32 [00:00<?, ? examples/s]
Filter: 100%|##########| 32/32 [00:00<00:00, 8760.95 examples/s]

Filter:   0%|          | 0/31 [00:00<?, ? examples/s]
Filter: 100%|##########| 31/31 [00:00<00:00, 8527.24 examples/s]

Flattening the indices:   0%|          | 0/31 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 31/31 [00:00<00:00, 6979.25 examples/s]

Casting to class labels:   0%|          | 0/31 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 31/31 [00:00<00:00, 4408.02 examples/s]INFO 2025-05-20 10:45:26,832 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_12_3_3__Configuring_obsolete_CPU_models.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 608.75ba/s]INFO 2025-05-20 10:45:26,834 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_12_3_3__Configuring_obsolete_CPU_models_task.yaml
INFO 2025-05-20 10:45:26,843 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:45:26,848 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 9141.71 examples/s]

Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 7657.83 examples/s]

Flattening the indices:   0%|          | 0/21 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 21/21 [00:00<00:00, 5970.74 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 3576.43 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 3438.89 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 3312.41 examples/s]

Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 8711.34 examples/s]

Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 6832.70 examples/s]

Filter:   0%|          | 0/20 [00:00<?, ? examples/s]
Filter: 100%|##########| 20/20 [00:00<00:00, 6681.49 examples/s]

Flattening the indices:   0%|          | 0/20 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 20/20 [00:00<00:00, 4708.47 examples/s]

Casting to class labels:   0%|          | 0/20 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 20/20 [00:00<00:00, 3370.81 examples/s]INFO 2025-05-20 10:45:35,259 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_7__Using_the_OpenShift_Container_Platform_dashboard_to_get_cluster_information.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 742.75ba/s]INFO 2025-05-20 10:45:35,261 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_7__Using_the_OpenShift_Container_Platform_dashboard_to_get_cluster_information_task.yaml
INFO 2025-05-20 10:45:35,271 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:45:35,275 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/30 [00:00<?, ? examples/s]
Filter: 100%|##########| 30/30 [00:00<00:00, 12421.43 examples/s]

Filter:   0%|          | 0/30 [00:00<?, ? examples/s]
Filter: 100%|##########| 30/30 [00:00<00:00, 9555.67 examples/s]

Flattening the indices:   0%|          | 0/30 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 30/30 [00:00<00:00, 8128.50 examples/s]

Map:   0%|          | 0/30 [00:00<?, ? examples/s]
Map: 100%|##########| 30/30 [00:00<00:00, 4458.23 examples/s]

Map:   0%|          | 0/30 [00:00<?, ? examples/s]
Map: 100%|##########| 30/30 [00:00<00:00, 4244.39 examples/s]

Map:   0%|          | 0/30 [00:00<?, ? examples/s]
Map: 100%|##########| 30/30 [00:00<00:00, 4030.14 examples/s]

Filter:   0%|          | 0/30 [00:00<?, ? examples/s]
Filter: 100%|##########| 30/30 [00:00<00:00, 11253.83 examples/s]

Filter:   0%|          | 0/30 [00:00<?, ? examples/s]
Filter: 100%|##########| 30/30 [00:00<00:00, 8351.86 examples/s]

Filter:   0%|          | 0/30 [00:00<?, ? examples/s]
Filter: 100%|##########| 30/30 [00:00<00:00, 8445.47 examples/s]

Casting to class labels:   0%|          | 0/30 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 30/30 [00:00<00:00, 4404.70 examples/s]INFO 2025-05-20 10:45:44,554 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_9_1__Prerequisites.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 616.81ba/s]INFO 2025-05-20 10:45:44,556 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_9_1__Prerequisites_task.yaml
INFO 2025-05-20 10:45:44,587 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:45:44,595 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge
INFO 2025-05-20 10:45:44,598 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge
INFO 2025-05-20 10:46:35,735 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/1393 [00:00<?, ? examples/s]
Filter: 100%|##########| 1393/1393 [00:00<00:00, 51704.55 examples/s]

Filter:   0%|          | 0/1388 [00:00<?, ? examples/s]
Filter: 100%|##########| 1388/1388 [00:00<00:00, 22732.29 examples/s]

Flattening the indices:   0%|          | 0/1386 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 1386/1386 [00:00<00:00, 27259.49 examples/s]

Map:   0%|          | 0/1386 [00:00<?, ? examples/s]
Map:  72%|#######2  | 1003/1386 [00:00<00:00, 9950.13 examples/s]
Map: 100%|##########| 1386/1386 [00:00<00:00, 9972.80 examples/s]

Map:   0%|          | 0/1386 [00:00<?, ? examples/s]
Map:  72%|#######2  | 1000/1386 [00:00<00:00, 9211.49 examples/s]
Map: 100%|##########| 1386/1386 [00:00<00:00, 9108.02 examples/s]

Map:   0%|          | 0/1386 [00:00<?, ? examples/s]
Map:  72%|#######2  | 1000/1386 [00:00<00:00, 9109.62 examples/s]
Map: 100%|##########| 1386/1386 [00:00<00:00, 8926.98 examples/s]

Filter:   0%|          | 0/1386 [00:00<?, ? examples/s]
Filter: 100%|##########| 1386/1386 [00:00<00:00, 37106.36 examples/s]

Filter:   0%|          | 0/1379 [00:00<?, ? examples/s]
Filter: 100%|##########| 1379/1379 [00:00<00:00, 17843.92 examples/s]

Filter:   0%|          | 0/1358 [00:00<?, ? examples/s]
Filter: 100%|##########| 1358/1358 [00:00<00:00, 17976.14 examples/s]

Flattening the indices:   0%|          | 0/1358 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 1358/1358 [00:00<00:00, 21827.67 examples/s]

Casting to class labels:   0%|          | 0/1358 [00:00<?, ? examples/s]
Casting to class labels:  74%|#######3  | 1000/1358 [00:00<00:00, 9817.53 examples/s]
Casting to class labels: 100%|##########| 1358/1358 [00:00<00:00, 9476.00 examples/s]INFO 2025-05-20 10:47:00,843 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_20_19__Re_using_persistent_volumes.jsonl


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 51.10ba/s]INFO 2025-05-20 10:47:00,883 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_20_19__Re_using_persistent_volumes_task.yaml
INFO 2025-05-20 10:47:00,894 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:47:00,899 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/42 [00:00<?, ? examples/s]
Filter: 100%|##########| 42/42 [00:00<00:00, 15880.35 examples/s]

Filter:   0%|          | 0/42 [00:00<?, ? examples/s]
Filter: 100%|##########| 42/42 [00:00<00:00, 11319.93 examples/s]

Flattening the indices:   0%|          | 0/42 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 42/42 [00:00<00:00, 10920.64 examples/s]

Map:   0%|          | 0/42 [00:00<?, ? examples/s]
Map: 100%|##########| 42/42 [00:00<00:00, 5190.51 examples/s]

Map:   0%|          | 0/42 [00:00<?, ? examples/s]
Map: 100%|##########| 42/42 [00:00<00:00, 4956.97 examples/s]

Map:   0%|          | 0/42 [00:00<?, ? examples/s]
Map: 100%|##########| 42/42 [00:00<00:00, 4763.42 examples/s]

Filter:   0%|          | 0/42 [00:00<?, ? examples/s]
Filter: 100%|##########| 42/42 [00:00<00:00, 14294.12 examples/s]

Filter:   0%|          | 0/42 [00:00<?, ? examples/s]
Filter: 100%|##########| 42/42 [00:00<00:00, 10006.86 examples/s]

Filter:   0%|          | 0/42 [00:00<?, ? examples/s]
Filter: 100%|##########| 42/42 [00:00<00:00, 10140.50 examples/s]

Casting to class labels:   0%|          | 0/42 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 42/42 [00:00<00:00, 5151.50 examples/s]INFO 2025-05-20 10:47:11,050 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_5_5__Enabling_the_virtctl_client.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 561.64ba/s]INFO 2025-05-20 10:47:11,053 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_5__Enabling_the_virtctl_client_task.yaml
INFO 2025-05-20 10:47:11,062 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:47:11,066 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/54 [00:00<?, ? examples/s]
Filter: 100%|##########| 54/54 [00:00<00:00, 19039.38 examples/s]

Filter:   0%|          | 0/54 [00:00<?, ? examples/s]
Filter: 100%|##########| 54/54 [00:00<00:00, 13136.09 examples/s]

Flattening the indices:   0%|          | 0/54 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 54/54 [00:00<00:00, 13626.06 examples/s]

Map:   0%|          | 0/54 [00:00<?, ? examples/s]
Map: 100%|##########| 54/54 [00:00<00:00, 5841.35 examples/s]

Map:   0%|          | 0/54 [00:00<?, ? examples/s]
Map: 100%|##########| 54/54 [00:00<00:00, 5518.96 examples/s]

Map:   0%|          | 0/54 [00:00<?, ? examples/s]
Map: 100%|##########| 54/54 [00:00<00:00, 5427.31 examples/s]

Filter:   0%|          | 0/54 [00:00<?, ? examples/s]
Filter: 100%|##########| 54/54 [00:00<00:00, 16197.70 examples/s]

Filter:   0%|          | 0/54 [00:00<?, ? examples/s]
Filter: 100%|##########| 54/54 [00:00<00:00, 11491.24 examples/s]

Filter:   0%|          | 0/52 [00:00<?, ? examples/s]
Filter: 100%|##########| 52/52 [00:00<00:00, 11161.34 examples/s]

Flattening the indices:   0%|          | 0/52 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 52/52 [00:00<00:00, 10780.67 examples/s]

Casting to class labels:   0%|          | 0/52 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 52/52 [00:00<00:00, 5827.91 examples/s]INFO 2025-05-20 10:47:21,123 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_1_1__Prerequisites.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 523.11ba/s]INFO 2025-05-20 10:47:21,125 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_1_1__Prerequisites_task.yaml
INFO 2025-05-20 10:47:21,136 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:47:21,140 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/135 [00:00<?, ? examples/s]
Filter: 100%|##########| 135/135 [00:00<00:00, 31617.12 examples/s]

Filter:   0%|          | 0/135 [00:00<?, ? examples/s]
Filter: 100%|##########| 135/135 [00:00<00:00, 18739.44 examples/s]

Flattening the indices:   0%|          | 0/135 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 135/135 [00:00<00:00, 25438.30 examples/s]

Map:   0%|          | 0/135 [00:00<?, ? examples/s]
Map: 100%|##########| 135/135 [00:00<00:00, 7902.85 examples/s]

Map:   0%|          | 0/135 [00:00<?, ? examples/s]
Map: 100%|##########| 135/135 [00:00<00:00, 7493.89 examples/s]

Map:   0%|          | 0/135 [00:00<?, ? examples/s]
Map: 100%|##########| 135/135 [00:00<00:00, 7437.49 examples/s]

Filter:   0%|          | 0/135 [00:00<?, ? examples/s]
Filter: 100%|##########| 135/135 [00:00<00:00, 25246.61 examples/s]

Filter:   0%|          | 0/135 [00:00<?, ? examples/s]
Filter: 100%|##########| 135/135 [00:00<00:00, 14966.99 examples/s]

Filter:   0%|          | 0/132 [00:00<?, ? examples/s]
Filter: 100%|##########| 132/132 [00:00<00:00, 15270.95 examples/s]

Flattening the indices:   0%|          | 0/132 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 132/132 [00:00<00:00, 21816.07 examples/s]

Casting to class labels:   0%|          | 0/132 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 132/132 [00:00<00:00, 7973.50 examples/s]INFO 2025-05-20 10:47:36,176 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_5_1__Preparing_your_cluster_for_OpenShift_Virtualization.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 231.27ba/s]INFO 2025-05-20 10:47:36,181 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_1__Preparing_your_cluster_for_OpenShift_Virtualization_task.yaml
INFO 2025-05-20 10:47:36,192 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:47:36,197 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/145 [00:00<?, ? examples/s]
Filter: 100%|##########| 145/145 [00:00<00:00, 33115.93 examples/s]

Filter:   0%|          | 0/145 [00:00<?, ? examples/s]
Filter: 100%|##########| 145/145 [00:00<00:00, 19206.51 examples/s]

Flattening the indices:   0%|          | 0/145 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 145/145 [00:00<00:00, 27545.36 examples/s]

Map:   0%|          | 0/145 [00:00<?, ? examples/s]
Map: 100%|##########| 145/145 [00:00<00:00, 8137.96 examples/s]

Map:   0%|          | 0/145 [00:00<?, ? examples/s]
Map: 100%|##########| 145/145 [00:00<00:00, 7631.46 examples/s]

Map:   0%|          | 0/145 [00:00<?, ? examples/s]
Map: 100%|##########| 145/145 [00:00<00:00, 7591.26 examples/s]

Filter:   0%|          | 0/145 [00:00<?, ? examples/s]
Filter: 100%|##########| 145/145 [00:00<00:00, 27300.54 examples/s]

Filter:   0%|          | 0/145 [00:00<?, ? examples/s]
Filter: 100%|##########| 145/145 [00:00<00:00, 15237.11 examples/s]

Filter:   0%|          | 0/142 [00:00<?, ? examples/s]
Filter: 100%|##########| 142/142 [00:00<00:00, 15454.65 examples/s]

Flattening the indices:   0%|          | 0/142 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 142/142 [00:00<00:00, 23025.14 examples/s]

Casting to class labels:   0%|          | 0/142 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 142/142 [00:00<00:00, 7960.64 examples/s]INFO 2025-05-20 10:47:52,217 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_14_1__Installing_and_configuring_OADP.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 240.79ba/s]INFO 2025-05-20 10:47:52,222 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_14_1__Installing_and_configuring_OADP_task.yaml
INFO 2025-05-20 10:47:52,231 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:47:52,236 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/81 [00:00<?, ? examples/s]
Filter: 100%|##########| 81/81 [00:00<00:00, 23040.94 examples/s]

Filter:   0%|          | 0/81 [00:00<?, ? examples/s]
Filter: 100%|##########| 81/81 [00:00<00:00, 15703.20 examples/s]

Flattening the indices:   0%|          | 0/81 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 81/81 [00:00<00:00, 19006.36 examples/s]

Map:   0%|          | 0/81 [00:00<?, ? examples/s]
Map: 100%|##########| 81/81 [00:00<00:00, 6858.83 examples/s]

Map:   0%|          | 0/81 [00:00<?, ? examples/s]
Map: 100%|##########| 81/81 [00:00<00:00, 6493.97 examples/s]

Map:   0%|          | 0/81 [00:00<?, ? examples/s]
Map: 100%|##########| 81/81 [00:00<00:00, 6362.86 examples/s]

Filter:   0%|          | 0/81 [00:00<?, ? examples/s]
Filter: 100%|##########| 81/81 [00:00<00:00, 20961.17 examples/s]

Filter:   0%|          | 0/81 [00:00<?, ? examples/s]
Filter: 100%|##########| 81/81 [00:00<00:00, 13048.80 examples/s]

Filter:   0%|          | 0/80 [00:00<?, ? examples/s]
Filter: 100%|##########| 80/80 [00:00<00:00, 12964.89 examples/s]

Flattening the indices:   0%|          | 0/80 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 80/80 [00:00<00:00, 15666.46 examples/s]

Casting to class labels:   0%|          | 0/80 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 80/80 [00:00<00:00, 6760.51 examples/s]INFO 2025-05-20 10:48:03,094 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_9__OpenShift_Container_Platform_cluster_monitoring__logging__and_Telemetry.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 337.54ba/s]INFO 2025-05-20 10:48:03,097 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_9__OpenShift_Container_Platform_cluster_monitoring__logging__and_Telemetry_task.yaml
INFO 2025-05-20 10:48:03,107 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:48:03,111 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/28 [00:00<?, ? examples/s]
Filter: 100%|##########| 28/28 [00:00<00:00, 11545.47 examples/s]

Filter:   0%|          | 0/28 [00:00<?, ? examples/s]
Filter: 100%|##########| 28/28 [00:00<00:00, 9284.57 examples/s]

Flattening the indices:   0%|          | 0/28 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 28/28 [00:00<00:00, 7980.46 examples/s]

Map:   0%|          | 0/28 [00:00<?, ? examples/s]
Map: 100%|##########| 28/28 [00:00<00:00, 4277.88 examples/s]

Map:   0%|          | 0/28 [00:00<?, ? examples/s]
Map: 100%|##########| 28/28 [00:00<00:00, 3990.10 examples/s]

Map:   0%|          | 0/28 [00:00<?, ? examples/s]
Map: 100%|##########| 28/28 [00:00<00:00, 3879.77 examples/s]

Filter:   0%|          | 0/28 [00:00<?, ? examples/s]
Filter: 100%|##########| 28/28 [00:00<00:00, 10893.29 examples/s]

Filter:   0%|          | 0/28 [00:00<?, ? examples/s]
Filter: 100%|##########| 28/28 [00:00<00:00, 8178.88 examples/s]

Filter:   0%|          | 0/26 [00:00<?, ? examples/s]
Filter: 100%|##########| 26/26 [00:00<00:00, 7902.31 examples/s]

Flattening the indices:   0%|          | 0/26 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 26/26 [00:00<00:00, 5859.86 examples/s]

Casting to class labels:   0%|          | 0/26 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 26/26 [00:00<00:00, 3997.36 examples/s]INFO 2025-05-20 10:48:11,069 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_5_3_1__Installing_the_OpenShift_Virtualization_Operator.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 672.38ba/s]INFO 2025-05-20 10:48:11,071 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_3_1__Installing_the_OpenShift_Virtualization_Operator_task.yaml
INFO 2025-05-20 10:48:11,088 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:48:11,093 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/435 [00:00<?, ? examples/s]
Filter: 100%|##########| 435/435 [00:00<00:00, 46719.13 examples/s]

Filter:   0%|          | 0/435 [00:00<?, ? examples/s]
Filter: 100%|##########| 435/435 [00:00<00:00, 22090.52 examples/s]

Flattening the indices:   0%|          | 0/435 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 435/435 [00:00<00:00, 41915.10 examples/s]

Map:   0%|          | 0/435 [00:00<?, ? examples/s]
Map: 100%|##########| 435/435 [00:00<00:00, 8980.45 examples/s]

Map:   0%|          | 0/435 [00:00<?, ? examples/s]
Map: 100%|##########| 435/435 [00:00<00:00, 8763.48 examples/s]

Map:   0%|          | 0/435 [00:00<?, ? examples/s]
Map: 100%|##########| 435/435 [00:00<00:00, 8810.24 examples/s]

Filter:   0%|          | 0/435 [00:00<?, ? examples/s]
Filter: 100%|##########| 435/435 [00:00<00:00, 33779.94 examples/s]

Filter:   0%|          | 0/431 [00:00<?, ? examples/s]
Filter: 100%|##########| 431/431 [00:00<00:00, 17460.40 examples/s]

Filter:   0%|          | 0/425 [00:00<?, ? examples/s]
Filter: 100%|##########| 425/425 [00:00<00:00, 17515.25 examples/s]

Flattening the indices:   0%|          | 0/425 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 425/425 [00:00<00:00, 36792.14 examples/s]

Casting to class labels:   0%|          | 0/425 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 425/425 [00:00<00:00, 9254.33 examples/s]INFO 2025-05-20 10:48:42,169 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_19_1__Configuring_the_virtual_machine_for_the_default_pod_network.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 77.28ba/s]INFO 2025-05-20 10:48:42,183 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_19_1__Configuring_the_virtual_machine_for_the_default_pod_network_task.yaml
INFO 2025-05-20 10:48:42,192 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:48:42,197 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/16 [00:00<?, ? examples/s]
Filter: 100%|##########| 16/16 [00:00<00:00, 7465.67 examples/s]

Filter:   0%|          | 0/16 [00:00<?, ? examples/s]
Filter: 100%|##########| 16/16 [00:00<00:00, 6032.80 examples/s]

Flattening the indices:   0%|          | 0/16 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 16/16 [00:00<00:00, 4583.00 examples/s]

Map:   0%|          | 0/16 [00:00<?, ? examples/s]
Map: 100%|##########| 16/16 [00:00<00:00, 3077.68 examples/s]

Map:   0%|          | 0/16 [00:00<?, ? examples/s]
Map: 100%|##########| 16/16 [00:00<00:00, 2877.74 examples/s]

Map:   0%|          | 0/16 [00:00<?, ? examples/s]
Map: 100%|##########| 16/16 [00:00<00:00, 2748.79 examples/s]

Filter:   0%|          | 0/16 [00:00<?, ? examples/s]
Filter: 100%|##########| 16/16 [00:00<00:00, 7146.84 examples/s]

Filter:   0%|          | 0/16 [00:00<?, ? examples/s]
Filter: 100%|##########| 16/16 [00:00<00:00, 5679.01 examples/s]

Filter:   0%|          | 0/16 [00:00<?, ? examples/s]
Filter: 100%|##########| 16/16 [00:00<00:00, 5767.35 examples/s]

Casting to class labels:   0%|          | 0/16 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 16/16 [00:00<00:00, 2870.97 examples/s]INFO 2025-05-20 10:48:48,529 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_11_7__Configuring_virtual_machine_eviction_strategy.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 873.63ba/s]INFO 2025-05-20 10:48:48,530 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_11_7__Configuring_virtual_machine_eviction_strategy_task.yaml
INFO 2025-05-20 10:48:48,539 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:48:48,544 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 9241.46 examples/s]

Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 7688.58 examples/s]

Flattening the indices:   0%|          | 0/21 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 21/21 [00:00<00:00, 5997.98 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 3641.34 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 3507.36 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 3340.68 examples/s]

Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 8889.82 examples/s]

Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 6856.64 examples/s]

Filter:   0%|          | 0/21 [00:00<?, ? examples/s]
Filter: 100%|##########| 21/21 [00:00<00:00, 6979.43 examples/s]

Casting to class labels:   0%|          | 0/21 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 21/21 [00:00<00:00, 3539.35 examples/s]INFO 2025-05-20 10:48:55,150 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_14_2_1__Additional_resources.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 790.78ba/s]INFO 2025-05-20 10:48:55,152 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_14_2_1__Additional_resources_task.yaml
INFO 2025-05-20 10:48:55,162 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:48:55,166 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/113 [00:00<?, ? examples/s]
Filter: 100%|##########| 113/113 [00:00<00:00, 28334.81 examples/s]

Filter:   0%|          | 0/111 [00:00<?, ? examples/s]
Filter: 100%|##########| 111/111 [00:00<00:00, 17653.87 examples/s]

Flattening the indices:   0%|          | 0/111 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 111/111 [00:00<00:00, 21780.95 examples/s]

Map:   0%|          | 0/111 [00:00<?, ? examples/s]
Map: 100%|##########| 111/111 [00:00<00:00, 7470.36 examples/s]

Map:   0%|          | 0/111 [00:00<?, ? examples/s]
Map: 100%|##########| 111/111 [00:00<00:00, 7111.92 examples/s]

Map:   0%|          | 0/111 [00:00<?, ? examples/s]
Map: 100%|##########| 111/111 [00:00<00:00, 6762.55 examples/s]

Filter:   0%|          | 0/111 [00:00<?, ? examples/s]
Filter: 100%|##########| 111/111 [00:00<00:00, 23782.58 examples/s]

Filter:   0%|          | 0/111 [00:00<?, ? examples/s]
Filter: 100%|##########| 111/111 [00:00<00:00, 14225.36 examples/s]

Filter:   0%|          | 0/109 [00:00<?, ? examples/s]
Filter: 100%|##########| 109/109 [00:00<00:00, 14047.17 examples/s]

Flattening the indices:   0%|          | 0/109 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 109/109 [00:00<00:00, 17928.59 examples/s]

Casting to class labels:   0%|          | 0/109 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 109/109 [00:00<00:00, 7445.79 examples/s]INFO 2025-05-20 10:49:11,709 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_10__Running_cluster_checkups.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 278.14ba/s]INFO 2025-05-20 10:49:11,713 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_10__Running_cluster_checkups_task.yaml
INFO 2025-05-20 10:49:11,723 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:49:11,728 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/143 [00:00<?, ? examples/s]
Filter: 100%|##########| 143/143 [00:00<00:00, 33293.67 examples/s]

Filter:   0%|          | 0/143 [00:00<?, ? examples/s]
Filter: 100%|##########| 143/143 [00:00<00:00, 19145.96 examples/s]

Flattening the indices:   0%|          | 0/143 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 143/143 [00:00<00:00, 26061.77 examples/s]

Map:   0%|          | 0/143 [00:00<?, ? examples/s]
Map: 100%|##########| 143/143 [00:00<00:00, 8094.16 examples/s]

Map:   0%|          | 0/143 [00:00<?, ? examples/s]
Map: 100%|##########| 143/143 [00:00<00:00, 7632.41 examples/s]

Map:   0%|          | 0/143 [00:00<?, ? examples/s]
Map: 100%|##########| 143/143 [00:00<00:00, 7693.60 examples/s]

Filter:   0%|          | 0/143 [00:00<?, ? examples/s]
Filter: 100%|##########| 143/143 [00:00<00:00, 27315.12 examples/s]

Filter:   0%|          | 0/143 [00:00<?, ? examples/s]
Filter: 100%|##########| 143/143 [00:00<00:00, 15686.00 examples/s]

Filter:   0%|          | 0/138 [00:00<?, ? examples/s]
Filter: 100%|##########| 138/138 [00:00<00:00, 15375.18 examples/s]

Flattening the indices:   0%|          | 0/138 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 138/138 [00:00<00:00, 22999.84 examples/s]

Casting to class labels:   0%|          | 0/138 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 138/138 [00:00<00:00, 8033.95 examples/s]INFO 2025-05-20 10:49:27,413 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_12_2__Configuring_a_virtual_machine_with_the_node_exporter_service.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 213.36ba/s]INFO 2025-05-20 10:49:27,418 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_12_2__Configuring_a_virtual_machine_with_the_node_exporter_service_task.yaml
INFO 2025-05-20 10:49:27,427 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:49:27,432 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/19 [00:00<?, ? examples/s]
Filter: 100%|##########| 19/19 [00:00<00:00, 8513.17 examples/s]

Filter:   0%|          | 0/19 [00:00<?, ? examples/s]
Filter: 100%|##########| 19/19 [00:00<00:00, 7077.42 examples/s]

Flattening the indices:   0%|          | 0/19 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 19/19 [00:00<00:00, 5451.62 examples/s]

Map:   0%|          | 0/19 [00:00<?, ? examples/s]
Map: 100%|##########| 19/19 [00:00<00:00, 3401.56 examples/s]

Map:   0%|          | 0/19 [00:00<?, ? examples/s]
Map: 100%|##########| 19/19 [00:00<00:00, 3181.56 examples/s]

Map:   0%|          | 0/19 [00:00<?, ? examples/s]
Map: 100%|##########| 19/19 [00:00<00:00, 3120.40 examples/s]

Filter:   0%|          | 0/19 [00:00<?, ? examples/s]
Filter: 100%|##########| 19/19 [00:00<00:00, 8215.65 examples/s]

Filter:   0%|          | 0/19 [00:00<?, ? examples/s]
Filter: 100%|##########| 19/19 [00:00<00:00, 6327.76 examples/s]

Filter:   0%|          | 0/19 [00:00<?, ? examples/s]
Filter: 100%|##########| 19/19 [00:00<00:00, 6357.03 examples/s]

Casting to class labels:   0%|          | 0/19 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 19/19 [00:00<00:00, 3253.66 examples/s]INFO 2025-05-20 10:49:34,300 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_7_3__Additional_OpenShift_Container_Platform_security_context_constraints_and_Linux_capabilities_for_the_kubevirt_controller_service_account.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 805.51ba/s]INFO 2025-05-20 10:49:34,302 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_7_3__Additional_OpenShift_Container_Platform_security_context_constraints_and_Linux_capabilities_for_the_kubevirt_controller_service_account_task.yaml
INFO 2025-05-20 10:49:34,311 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:49:34,316 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/55 [00:00<?, ? examples/s]
Filter: 100%|##########| 55/55 [00:00<00:00, 18502.30 examples/s]

Filter:   0%|          | 0/55 [00:00<?, ? examples/s]
Filter: 100%|##########| 55/55 [00:00<00:00, 12781.14 examples/s]

Flattening the indices:   0%|          | 0/55 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 55/55 [00:00<00:00, 13473.12 examples/s]

Map:   0%|          | 0/55 [00:00<?, ? examples/s]
Map: 100%|##########| 55/55 [00:00<00:00, 5970.93 examples/s]

Map:   0%|          | 0/55 [00:00<?, ? examples/s]
Map: 100%|##########| 55/55 [00:00<00:00, 5646.20 examples/s]

Map:   0%|          | 0/55 [00:00<?, ? examples/s]
Map: 100%|##########| 55/55 [00:00<00:00, 5546.69 examples/s]

Filter:   0%|          | 0/55 [00:00<?, ? examples/s]
Filter: 100%|##########| 55/55 [00:00<00:00, 16959.76 examples/s]

Filter:   0%|          | 0/55 [00:00<?, ? examples/s]
Filter: 100%|##########| 55/55 [00:00<00:00, 11097.11 examples/s]

Filter:   0%|          | 0/54 [00:00<?, ? examples/s]
Filter: 100%|##########| 54/54 [00:00<00:00, 11549.84 examples/s]

Flattening the indices:   0%|          | 0/54 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 54/54 [00:00<00:00, 10791.01 examples/s]

Casting to class labels:   0%|          | 0/54 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 54/54 [00:00<00:00, 5828.12 examples/s]INFO 2025-05-20 10:49:48,988 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_4_7__Known_issues.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 382.66ba/s]INFO 2025-05-20 10:49:48,992 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_7__Known_issues_task.yaml
INFO 2025-05-20 10:49:49,001 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:49:49,005 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/40 [00:00<?, ? examples/s]
Filter: 100%|##########| 40/40 [00:00<00:00, 15203.64 examples/s]

Filter:   0%|          | 0/40 [00:00<?, ? examples/s]
Filter: 100%|##########| 40/40 [00:00<00:00, 11212.47 examples/s]

Flattening the indices:   0%|          | 0/40 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 40/40 [00:00<00:00, 10416.10 examples/s]

Map:   0%|          | 0/40 [00:00<?, ? examples/s]
Map: 100%|##########| 40/40 [00:00<00:00, 5191.93 examples/s]

Map:   0%|          | 0/40 [00:00<?, ? examples/s]
Map: 100%|##########| 40/40 [00:00<00:00, 4861.69 examples/s]

Map:   0%|          | 0/40 [00:00<?, ? examples/s]
Map: 100%|##########| 40/40 [00:00<00:00, 4795.00 examples/s]

Filter:   0%|          | 0/40 [00:00<?, ? examples/s]
Filter: 100%|##########| 40/40 [00:00<00:00, 13921.85 examples/s]

Filter:   0%|          | 0/40 [00:00<?, ? examples/s]
Filter: 100%|##########| 40/40 [00:00<00:00, 9348.20 examples/s]

Filter:   0%|          | 0/39 [00:00<?, ? examples/s]
Filter: 100%|##########| 39/39 [00:00<00:00, 9925.24 examples/s]

Flattening the indices:   0%|          | 0/39 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 39/39 [00:00<00:00, 8167.05 examples/s]

Casting to class labels:   0%|          | 0/39 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 39/39 [00:00<00:00, 4992.61 examples/s]INFO 2025-05-20 10:49:59,262 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_11_4__Migrating_a_virtual_machine_over_a_dedicated_additional_network.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 531.93ba/s]INFO 2025-05-20 10:49:59,264 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_11_4__Migrating_a_virtual_machine_over_a_dedicated_additional_network_task.yaml
INFO 2025-05-20 10:49:59,274 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:49:59,279 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/80 [00:00<?, ? examples/s]
Filter: 100%|##########| 80/80 [00:00<00:00, 22615.38 examples/s]

Filter:   0%|          | 0/80 [00:00<?, ? examples/s]
Filter: 100%|##########| 80/80 [00:00<00:00, 15169.96 examples/s]

Flattening the indices:   0%|          | 0/80 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 80/80 [00:00<00:00, 18164.04 examples/s]

Map:   0%|          | 0/80 [00:00<?, ? examples/s]
Map: 100%|##########| 80/80 [00:00<00:00, 6796.80 examples/s]

Map:   0%|          | 0/80 [00:00<?, ? examples/s]
Map: 100%|##########| 80/80 [00:00<00:00, 6370.81 examples/s]

Map:   0%|          | 0/80 [00:00<?, ? examples/s]
Map: 100%|##########| 80/80 [00:00<00:00, 6348.75 examples/s]

Filter:   0%|          | 0/80 [00:00<?, ? examples/s]
Filter: 100%|##########| 80/80 [00:00<00:00, 20445.06 examples/s]

Filter:   0%|          | 0/80 [00:00<?, ? examples/s]
Filter: 100%|##########| 80/80 [00:00<00:00, 12795.31 examples/s]

Filter:   0%|          | 0/79 [00:00<?, ? examples/s]
Filter: 100%|##########| 79/79 [00:00<00:00, 12678.89 examples/s]

Flattening the indices:   0%|          | 0/79 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 79/79 [00:00<00:00, 15099.80 examples/s]

Casting to class labels:   0%|          | 0/79 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 79/79 [00:00<00:00, 6630.98 examples/s]INFO 2025-05-20 10:50:13,075 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_10_1_3__Creating_a_virtual_machine_template_in_the_web_console.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 292.86ba/s]INFO 2025-05-20 10:50:13,079 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_1_3__Creating_a_virtual_machine_template_in_the_web_console_task.yaml
INFO 2025-05-20 10:50:13,089 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:50:13,094 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/140 [00:00<?, ? examples/s]
Filter: 100%|##########| 140/140 [00:00<00:00, 28866.51 examples/s]

Filter:   0%|          | 0/140 [00:00<?, ? examples/s]
Filter: 100%|##########| 140/140 [00:00<00:00, 18341.48 examples/s]

Flattening the indices:   0%|          | 0/140 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 140/140 [00:00<00:00, 26790.88 examples/s]

Map:   0%|          | 0/140 [00:00<?, ? examples/s]
Map: 100%|##########| 140/140 [00:00<00:00, 7900.26 examples/s]

Map:   0%|          | 0/140 [00:00<?, ? examples/s]
Map: 100%|##########| 140/140 [00:00<00:00, 7482.96 examples/s]

Map:   0%|          | 0/140 [00:00<?, ? examples/s]
Map: 100%|##########| 140/140 [00:00<00:00, 7470.96 examples/s]

Filter:   0%|          | 0/140 [00:00<?, ? examples/s]
Filter: 100%|##########| 140/140 [00:00<00:00, 26591.91 examples/s]

Filter:   0%|          | 0/140 [00:00<?, ? examples/s]
Filter: 100%|##########| 140/140 [00:00<00:00, 15226.70 examples/s]

Filter:   0%|          | 0/133 [00:00<?, ? examples/s]
Filter: 100%|##########| 133/133 [00:00<00:00, 14953.96 examples/s]

Flattening the indices:   0%|          | 0/133 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 133/133 [00:00<00:00, 22513.62 examples/s]

Casting to class labels:   0%|          | 0/133 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 133/133 [00:00<00:00, 7937.43 examples/s]INFO 2025-05-20 10:50:27,884 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_5_1_1__Hardware_and_operating_system_requirements.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 206.33ba/s]INFO 2025-05-20 10:50:27,890 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_1_1__Hardware_and_operating_system_requirements_task.yaml
INFO 2025-05-20 10:50:27,899 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:50:27,904 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 17679.19 examples/s]

Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 12926.56 examples/s]

Flattening the indices:   0%|          | 0/53 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 53/53 [00:00<00:00, 13238.33 examples/s]

Map:   0%|          | 0/53 [00:00<?, ? examples/s]
Map: 100%|##########| 53/53 [00:00<00:00, 5981.22 examples/s]

Map:   0%|          | 0/53 [00:00<?, ? examples/s]
Map: 100%|##########| 53/53 [00:00<00:00, 5490.74 examples/s]

Map:   0%|          | 0/53 [00:00<?, ? examples/s]
Map: 100%|##########| 53/53 [00:00<00:00, 5473.17 examples/s]

Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 16927.97 examples/s]

Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 11196.64 examples/s]

Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 11234.55 examples/s]

Casting to class labels:   0%|          | 0/53 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 53/53 [00:00<00:00, 5803.07 examples/s]INFO 2025-05-20 10:50:40,949 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_13_1__About_VirtIO_drivers.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 456.75ba/s]INFO 2025-05-20 10:50:40,952 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_13_1__About_VirtIO_drivers_task.yaml
INFO 2025-05-20 10:50:40,961 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:50:40,965 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/14 [00:00<?, ? examples/s]
Filter: 100%|##########| 14/14 [00:00<00:00, 6601.49 examples/s]

Filter:   0%|          | 0/14 [00:00<?, ? examples/s]
Filter: 100%|##########| 14/14 [00:00<00:00, 5643.47 examples/s]

Flattening the indices:   0%|          | 0/14 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 14/14 [00:00<00:00, 4135.52 examples/s]

Map:   0%|          | 0/14 [00:00<?, ? examples/s]
Map: 100%|##########| 14/14 [00:00<00:00, 2720.17 examples/s]

Map:   0%|          | 0/14 [00:00<?, ? examples/s]
Map: 100%|##########| 14/14 [00:00<00:00, 2592.62 examples/s]

Map:   0%|          | 0/14 [00:00<?, ? examples/s]
Map: 100%|##########| 14/14 [00:00<00:00, 2513.60 examples/s]

Filter:   0%|          | 0/14 [00:00<?, ? examples/s]
Filter: 100%|##########| 14/14 [00:00<00:00, 6424.54 examples/s]

Filter:   0%|          | 0/14 [00:00<?, ? examples/s]
Filter: 100%|##########| 14/14 [00:00<00:00, 5024.84 examples/s]

Filter:   0%|          | 0/12 [00:00<?, ? examples/s]
Filter: 100%|##########| 12/12 [00:00<00:00, 4476.31 examples/s]

Flattening the indices:   0%|          | 0/12 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 12/12 [00:00<00:00, 2910.35 examples/s]

Casting to class labels:   0%|          | 0/12 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 12/12 [00:00<00:00, 2374.47 examples/s]INFO 2025-05-20 10:50:45,200 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_10_5_1__Deleting_a_virtual_machine_template_in_the_web_console.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 1003.90ba/s]INFO 2025-05-20 10:50:45,202 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_5_1__Deleting_a_virtual_machine_template_in_the_web_console_task.yaml
INFO 2025-05-20 10:50:45,213 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:50:45,218 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/158 [00:00<?, ? examples/s]
Filter: 100%|##########| 158/158 [00:00<00:00, 32876.92 examples/s]

Filter:   0%|          | 0/158 [00:00<?, ? examples/s]
Filter: 100%|##########| 158/158 [00:00<00:00, 19146.54 examples/s]

Flattening the indices:   0%|          | 0/158 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 158/158 [00:00<00:00, 27268.24 examples/s]

Map:   0%|          | 0/158 [00:00<?, ? examples/s]
Map: 100%|##########| 158/158 [00:00<00:00, 8087.53 examples/s]

Map:   0%|          | 0/158 [00:00<?, ? examples/s]
Map: 100%|##########| 158/158 [00:00<00:00, 7638.58 examples/s]

Map:   0%|          | 0/158 [00:00<?, ? examples/s]
Map: 100%|##########| 158/158 [00:00<00:00, 7538.73 examples/s]

Filter:   0%|          | 0/158 [00:00<?, ? examples/s]
Filter: 100%|##########| 158/158 [00:00<00:00, 25795.03 examples/s]

Filter:   0%|          | 0/158 [00:00<?, ? examples/s]
Filter: 100%|##########| 158/158 [00:00<00:00, 15585.61 examples/s]

Filter:   0%|          | 0/153 [00:00<?, ? examples/s]
Filter: 100%|##########| 153/153 [00:00<00:00, 15396.93 examples/s]

Flattening the indices:   0%|          | 0/153 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 153/153 [00:00<00:00, 23943.31 examples/s]

Casting to class labels:   0%|          | 0/153 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 153/153 [00:00<00:00, 8130.45 examples/s]INFO 2025-05-20 10:51:01,002 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_7_1__Accessing_virtual_machine_consoles_in_the_OpenShift_Container_Platform_web_console.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 178.12ba/s]INFO 2025-05-20 10:51:01,008 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_7_1__Accessing_virtual_machine_consoles_in_the_OpenShift_Container_Platform_web_console_task.yaml
INFO 2025-05-20 10:51:01,018 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:51:01,022 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 7949.61 examples/s]

Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 6784.46 examples/s]

Flattening the indices:   0%|          | 0/18 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 18/18 [00:00<00:00, 5190.61 examples/s]

Map:   0%|          | 0/18 [00:00<?, ? examples/s]
Map: 100%|##########| 18/18 [00:00<00:00, 3237.04 examples/s]

Map:   0%|          | 0/18 [00:00<?, ? examples/s]
Map: 100%|##########| 18/18 [00:00<00:00, 3130.47 examples/s]

Map:   0%|          | 0/18 [00:00<?, ? examples/s]
Map: 100%|##########| 18/18 [00:00<00:00, 3103.19 examples/s]

Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 7574.74 examples/s]

Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 6266.91 examples/s]

Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 6086.54 examples/s]

Casting to class labels:   0%|          | 0/18 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 18/18 [00:00<00:00, 3155.33 examples/s]INFO 2025-05-20 10:51:07,935 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_15__Using_virtual_Trusted_Platform_Module_devices.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 892.41ba/s]INFO 2025-05-20 10:51:07,937 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_15__Using_virtual_Trusted_Platform_Module_devices_task.yaml
INFO 2025-05-20 10:51:07,946 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:51:07,950 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 15534.46 examples/s]

Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 11506.62 examples/s]

Flattening the indices:   0%|          | 0/41 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 41/41 [00:00<00:00, 11010.08 examples/s]

Map:   0%|          | 0/41 [00:00<?, ? examples/s]
Map: 100%|##########| 41/41 [00:00<00:00, 5232.51 examples/s]

Map:   0%|          | 0/41 [00:00<?, ? examples/s]
Map: 100%|##########| 41/41 [00:00<00:00, 5051.89 examples/s]

Map:   0%|          | 0/41 [00:00<?, ? examples/s]
Map: 100%|##########| 41/41 [00:00<00:00, 4792.56 examples/s]

Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 13994.67 examples/s]

Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 10011.44 examples/s]

Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 9935.09 examples/s]

Casting to class labels:   0%|          | 0/41 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 41/41 [00:00<00:00, 5196.46 examples/s]INFO 2025-05-20 10:51:19,213 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_11_2_2__Cluster_wide_live_migration_limits_and_timeouts.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 571.90ba/s]INFO 2025-05-20 10:51:19,216 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_11_2_2__Cluster_wide_live_migration_limits_and_timeouts_task.yaml
INFO 2025-05-20 10:51:19,225 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:51:19,229 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/49 [00:00<?, ? examples/s]
Filter: 100%|##########| 49/49 [00:00<00:00, 17571.90 examples/s]

Filter:   0%|          | 0/49 [00:00<?, ? examples/s]
Filter: 100%|##########| 49/49 [00:00<00:00, 12493.67 examples/s]

Flattening the indices:   0%|          | 0/48 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 48/48 [00:00<00:00, 12811.11 examples/s]

Map:   0%|          | 0/48 [00:00<?, ? examples/s]
Map: 100%|##########| 48/48 [00:00<00:00, 5740.38 examples/s]

Map:   0%|          | 0/48 [00:00<?, ? examples/s]
Map: 100%|##########| 48/48 [00:00<00:00, 5356.14 examples/s]

Map:   0%|          | 0/48 [00:00<?, ? examples/s]
Map: 100%|##########| 48/48 [00:00<00:00, 5287.35 examples/s]

Filter:   0%|          | 0/48 [00:00<?, ? examples/s]
Filter: 100%|##########| 48/48 [00:00<00:00, 15606.71 examples/s]

Filter:   0%|          | 0/47 [00:00<?, ? examples/s]
Filter: 100%|##########| 47/47 [00:00<00:00, 10593.38 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 10415.52 examples/s]

Flattening the indices:   0%|          | 0/43 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 43/43 [00:00<00:00, 9282.78 examples/s]

Casting to class labels:   0%|          | 0/43 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 43/43 [00:00<00:00, 5372.03 examples/s]INFO 2025-05-20 10:51:28,717 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_8_3__Additional_resources.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 600.30ba/s]INFO 2025-05-20 10:51:28,719 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_8_3__Additional_resources_task.yaml
INFO 2025-05-20 10:51:28,728 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:51:28,733 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/42 [00:00<?, ? examples/s]
Filter: 100%|##########| 42/42 [00:00<00:00, 16080.40 examples/s]

Filter:   0%|          | 0/42 [00:00<?, ? examples/s]
Filter: 100%|##########| 42/42 [00:00<00:00, 11916.44 examples/s]

Flattening the indices:   0%|          | 0/42 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 42/42 [00:00<00:00, 11240.48 examples/s]

Map:   0%|          | 0/42 [00:00<?, ? examples/s]
Map: 100%|##########| 42/42 [00:00<00:00, 5269.38 examples/s]

Map:   0%|          | 0/42 [00:00<?, ? examples/s]
Map: 100%|##########| 42/42 [00:00<00:00, 4962.41 examples/s]

Map:   0%|          | 0/42 [00:00<?, ? examples/s]
Map: 100%|##########| 42/42 [00:00<00:00, 4932.96 examples/s]

Filter:   0%|          | 0/42 [00:00<?, ? examples/s]
Filter: 100%|##########| 42/42 [00:00<00:00, 14490.48 examples/s]

Filter:   0%|          | 0/42 [00:00<?, ? examples/s]
Filter: 100%|##########| 42/42 [00:00<00:00, 9828.21 examples/s]

Filter:   0%|          | 0/42 [00:00<?, ? examples/s]
Filter: 100%|##########| 42/42 [00:00<00:00, 10103.86 examples/s]

Casting to class labels:   0%|          | 0/42 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 42/42 [00:00<00:00, 5145.48 examples/s]INFO 2025-05-20 10:51:40,236 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_14_2__About_VirtIO_drivers.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 520.00ba/s]INFO 2025-05-20 10:51:40,238 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_14_2__About_VirtIO_drivers_task.yaml
INFO 2025-05-20 10:51:40,248 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:51:40,253 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 15626.21 examples/s]

Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 11542.15 examples/s]

Flattening the indices:   0%|          | 0/41 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 41/41 [00:00<00:00, 10674.52 examples/s]

Map:   0%|          | 0/41 [00:00<?, ? examples/s]
Map: 100%|##########| 41/41 [00:00<00:00, 5153.79 examples/s]

Map:   0%|          | 0/41 [00:00<?, ? examples/s]
Map: 100%|##########| 41/41 [00:00<00:00, 4976.46 examples/s]

Map:   0%|          | 0/41 [00:00<?, ? examples/s]
Map: 100%|##########| 41/41 [00:00<00:00, 4779.77 examples/s]

Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 13543.87 examples/s]

Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 9713.42 examples/s]

Filter:   0%|          | 0/40 [00:00<?, ? examples/s]
Filter: 100%|##########| 40/40 [00:00<00:00, 9664.29 examples/s]

Flattening the indices:   0%|          | 0/40 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 40/40 [00:00<00:00, 8529.78 examples/s]

Casting to class labels:   0%|          | 0/40 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 40/40 [00:00<00:00, 4982.84 examples/s]INFO 2025-05-20 10:51:50,312 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_5_4__Editing_a_standalone_virtual_machine_instance_using_the_web_console.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 570.34ba/s]INFO 2025-05-20 10:51:50,314 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_5_4__Editing_a_standalone_virtual_machine_instance_using_the_web_console_task.yaml
INFO 2025-05-20 10:51:50,324 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:51:50,328 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 15984.67 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 11745.69 examples/s]

Flattening the indices:   0%|          | 0/43 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 43/43 [00:00<00:00, 11073.56 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 5381.48 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 5034.76 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 4999.59 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 14577.68 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 10037.01 examples/s]

Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 9974.27 examples/s]

Flattening the indices:   0%|          | 0/40 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 40/40 [00:00<00:00, 8889.06 examples/s]

Casting to class labels:   0%|          | 0/40 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 40/40 [00:00<00:00, 5036.54 examples/s]INFO 2025-05-20 10:51:59,183 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_6_1__Starting_a_virtual_machine.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 545.71ba/s]INFO 2025-05-20 10:51:59,186 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_6_1__Starting_a_virtual_machine_task.yaml
INFO 2025-05-20 10:51:59,195 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:51:59,199 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/56 [00:00<?, ? examples/s]
Filter: 100%|##########| 56/56 [00:00<00:00, 19376.43 examples/s]

Filter:   0%|          | 0/56 [00:00<?, ? examples/s]
Filter: 100%|##########| 56/56 [00:00<00:00, 13476.45 examples/s]

Flattening the indices:   0%|          | 0/56 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 56/56 [00:00<00:00, 13703.68 examples/s]

Map:   0%|          | 0/56 [00:00<?, ? examples/s]
Map: 100%|##########| 56/56 [00:00<00:00, 5969.78 examples/s]

Map:   0%|          | 0/56 [00:00<?, ? examples/s]
Map: 100%|##########| 56/56 [00:00<00:00, 5677.02 examples/s]

Map:   0%|          | 0/56 [00:00<?, ? examples/s]
Map: 100%|##########| 56/56 [00:00<00:00, 5560.63 examples/s]

Filter:   0%|          | 0/56 [00:00<?, ? examples/s]
Filter: 100%|##########| 56/56 [00:00<00:00, 17389.58 examples/s]

Filter:   0%|          | 0/56 [00:00<?, ? examples/s]
Filter: 100%|##########| 56/56 [00:00<00:00, 11295.07 examples/s]

Filter:   0%|          | 0/56 [00:00<?, ? examples/s]
Filter: 100%|##########| 56/56 [00:00<00:00, 11356.78 examples/s]

Casting to class labels:   0%|          | 0/56 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 56/56 [00:00<00:00, 5868.65 examples/s]INFO 2025-05-20 10:52:10,107 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_13_5__Removing_the_VirtIO_container_disk_from_a_virtual_machine.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 474.42ba/s]INFO 2025-05-20 10:52:10,110 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_13_5__Removing_the_VirtIO_container_disk_from_a_virtual_machine_task.yaml
INFO 2025-05-20 10:52:10,119 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:52:10,124 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/62 [00:00<?, ? examples/s]
Filter: 100%|##########| 62/62 [00:00<00:00, 20932.69 examples/s]

Filter:   0%|          | 0/62 [00:00<?, ? examples/s]
Filter: 100%|##########| 62/62 [00:00<00:00, 14195.47 examples/s]

Flattening the indices:   0%|          | 0/62 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 62/62 [00:00<00:00, 14969.31 examples/s]

Map:   0%|          | 0/62 [00:00<?, ? examples/s]
Map: 100%|##########| 62/62 [00:00<00:00, 6215.42 examples/s]

Map:   0%|          | 0/62 [00:00<?, ? examples/s]
Map: 100%|##########| 62/62 [00:00<00:00, 5953.45 examples/s]

Map:   0%|          | 0/62 [00:00<?, ? examples/s]
Map: 100%|##########| 62/62 [00:00<00:00, 5816.17 examples/s]

Filter:   0%|          | 0/62 [00:00<?, ? examples/s]
Filter: 100%|##########| 62/62 [00:00<00:00, 18120.47 examples/s]

Filter:   0%|          | 0/62 [00:00<?, ? examples/s]
Filter: 100%|##########| 62/62 [00:00<00:00, 11886.23 examples/s]

Filter:   0%|          | 0/62 [00:00<?, ? examples/s]
Filter: 100%|##########| 62/62 [00:00<00:00, 11914.00 examples/s]

Casting to class labels:   0%|          | 0/62 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 62/62 [00:00<00:00, 6227.03 examples/s]INFO 2025-05-20 10:52:24,668 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_13_4__Installing_VirtIO_drivers_on_an_existing_Windows_virtual_machine.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 412.74ba/s]INFO 2025-05-20 10:52:24,671 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_13_4__Installing_VirtIO_drivers_on_an_existing_Windows_virtual_machine_task.yaml
INFO 2025-05-20 10:52:24,680 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:52:24,685 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/6 [00:00<?, ? examples/s]
Filter: 100%|##########| 6/6 [00:00<00:00, 2973.28 examples/s]

Filter:   0%|          | 0/6 [00:00<?, ? examples/s]
Filter: 100%|##########| 6/6 [00:00<00:00, 2783.83 examples/s]

Flattening the indices:   0%|          | 0/6 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 6/6 [00:00<00:00, 1798.59 examples/s]

Map:   0%|          | 0/6 [00:00<?, ? examples/s]
Map: 100%|##########| 6/6 [00:00<00:00, 1386.39 examples/s]

Map:   0%|          | 0/6 [00:00<?, ? examples/s]
Map: 100%|##########| 6/6 [00:00<00:00, 1313.18 examples/s]

Map:   0%|          | 0/6 [00:00<?, ? examples/s]
Map: 100%|##########| 6/6 [00:00<00:00, 1281.62 examples/s]

Filter:   0%|          | 0/6 [00:00<?, ? examples/s]
Filter: 100%|##########| 6/6 [00:00<00:00, 2976.79 examples/s]

Filter:   0%|          | 0/6 [00:00<?, ? examples/s]
Filter: 100%|##########| 6/6 [00:00<00:00, 2622.81 examples/s]

Filter:   0%|          | 0/5 [00:00<?, ? examples/s]
Filter: 100%|##########| 5/5 [00:00<00:00, 2235.77 examples/s]

Flattening the indices:   0%|          | 0/5 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 5/5 [00:00<00:00, 1213.21 examples/s]

Casting to class labels:   0%|          | 0/5 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 5/5 [00:00<00:00, 1118.84 examples/s]INFO 2025-05-20 10:52:31,428 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_2_1__How_OpenShift_Virtualization_architecture_works.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 1071.89ba/s]INFO 2025-05-20 10:52:31,430 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_1__How_OpenShift_Virtualization_architecture_works_task.yaml
INFO 2025-05-20 10:52:31,439 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:52:31,444 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/17 [00:00<?, ? examples/s]
Filter: 100%|##########| 17/17 [00:00<00:00, 7579.80 examples/s]

Filter:   0%|          | 0/17 [00:00<?, ? examples/s]
Filter: 100%|##########| 17/17 [00:00<00:00, 6500.43 examples/s]

Flattening the indices:   0%|          | 0/17 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 17/17 [00:00<00:00, 4845.28 examples/s]

Map:   0%|          | 0/17 [00:00<?, ? examples/s]
Map: 100%|##########| 17/17 [00:00<00:00, 3052.23 examples/s]

Map:   0%|          | 0/17 [00:00<?, ? examples/s]
Map: 100%|##########| 17/17 [00:00<00:00, 2966.39 examples/s]

Map:   0%|          | 0/17 [00:00<?, ? examples/s]
Map: 100%|##########| 17/17 [00:00<00:00, 2888.99 examples/s]

Filter:   0%|          | 0/17 [00:00<?, ? examples/s]
Filter: 100%|##########| 17/17 [00:00<00:00, 7398.90 examples/s]

Filter:   0%|          | 0/17 [00:00<?, ? examples/s]
Filter: 100%|##########| 17/17 [00:00<00:00, 5920.71 examples/s]

Filter:   0%|          | 0/17 [00:00<?, ? examples/s]
Filter: 100%|##########| 17/17 [00:00<00:00, 6016.13 examples/s]

Casting to class labels:   0%|          | 0/17 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 17/17 [00:00<00:00, 2998.70 examples/s]INFO 2025-05-20 10:52:41,054 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_7_1__About_the_OpenShift_Container_Platform_dashboards_page.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 805.51ba/s]INFO 2025-05-20 10:52:41,056 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_7_1__About_the_OpenShift_Container_Platform_dashboards_page_task.yaml
INFO 2025-05-20 10:52:41,066 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:52:41,070 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 18605.47 examples/s]

Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 13594.55 examples/s]

Flattening the indices:   0%|          | 0/53 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 53/53 [00:00<00:00, 13456.30 examples/s]

Map:   0%|          | 0/53 [00:00<?, ? examples/s]
Map: 100%|##########| 53/53 [00:00<00:00, 5832.15 examples/s]

Map:   0%|          | 0/53 [00:00<?, ? examples/s]
Map: 100%|##########| 53/53 [00:00<00:00, 5526.78 examples/s]

Map:   0%|          | 0/53 [00:00<?, ? examples/s]
Map: 100%|##########| 53/53 [00:00<00:00, 5398.73 examples/s]

Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 16305.88 examples/s]

Filter:   0%|          | 0/53 [00:00<?, ? examples/s]
Filter: 100%|##########| 53/53 [00:00<00:00, 11066.77 examples/s]

Filter:   0%|          | 0/51 [00:00<?, ? examples/s]
Filter: 100%|##########| 51/51 [00:00<00:00, 11086.84 examples/s]

Flattening the indices:   0%|          | 0/51 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 51/51 [00:00<00:00, 10567.61 examples/s]

Casting to class labels:   0%|          | 0/51 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 51/51 [00:00<00:00, 5659.58 examples/s]INFO 2025-05-20 10:52:51,086 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_10_4__Deploying_a_virtual_machine_template_to_a_custom_namespace.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 481.33ba/s]INFO 2025-05-20 10:52:51,089 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_4__Deploying_a_virtual_machine_template_to_a_custom_namespace_task.yaml
INFO 2025-05-20 10:52:51,102 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:52:51,107 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/278 [00:00<?, ? examples/s]
Filter: 100%|##########| 278/278 [00:00<00:00, 35761.89 examples/s]

Filter:   0%|          | 0/277 [00:00<?, ? examples/s]
Filter: 100%|##########| 277/277 [00:00<00:00, 20710.58 examples/s]

Flattening the indices:   0%|          | 0/277 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 277/277 [00:00<00:00, 34234.68 examples/s]

Map:   0%|          | 0/277 [00:00<?, ? examples/s]
Map: 100%|##########| 277/277 [00:00<00:00, 9085.54 examples/s]

Map:   0%|          | 0/277 [00:00<?, ? examples/s]
Map: 100%|##########| 277/277 [00:00<00:00, 8477.17 examples/s]

Map:   0%|          | 0/277 [00:00<?, ? examples/s]
Map: 100%|##########| 277/277 [00:00<00:00, 8534.71 examples/s]

Filter:   0%|          | 0/277 [00:00<?, ? examples/s]
Filter: 100%|##########| 277/277 [00:00<00:00, 31850.82 examples/s]

Filter:   0%|          | 0/277 [00:00<?, ? examples/s]
Filter: 100%|##########| 277/277 [00:00<00:00, 16261.54 examples/s]

Filter:   0%|          | 0/271 [00:00<?, ? examples/s]
Filter: 100%|##########| 271/271 [00:00<00:00, 16213.63 examples/s]

Flattening the indices:   0%|          | 0/271 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 271/271 [00:00<00:00, 30106.91 examples/s]

Casting to class labels:   0%|          | 0/271 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 271/271 [00:00<00:00, 8924.12 examples/s]INFO 2025-05-20 10:53:12,552 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_13_4__Additional_resources.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 96.61ba/s]INFO 2025-05-20 10:53:12,563 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_13_4__Additional_resources_task.yaml
INFO 2025-05-20 10:53:12,573 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:53:12,577 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/63 [00:00<?, ? examples/s]
Filter: 100%|##########| 63/63 [00:00<00:00, 21147.75 examples/s]

Filter:   0%|          | 0/63 [00:00<?, ? examples/s]
Filter: 100%|##########| 63/63 [00:00<00:00, 14325.90 examples/s]

Flattening the indices:   0%|          | 0/63 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 63/63 [00:00<00:00, 15243.22 examples/s]

Map:   0%|          | 0/63 [00:00<?, ? examples/s]
Map: 100%|##########| 63/63 [00:00<00:00, 6352.71 examples/s]

Map:   0%|          | 0/63 [00:00<?, ? examples/s]
Map: 100%|##########| 63/63 [00:00<00:00, 6026.02 examples/s]

Map:   0%|          | 0/63 [00:00<?, ? examples/s]
Map: 100%|##########| 63/63 [00:00<00:00, 5860.05 examples/s]

Filter:   0%|          | 0/63 [00:00<?, ? examples/s]
Filter: 100%|##########| 63/63 [00:00<00:00, 18364.11 examples/s]

Filter:   0%|          | 0/63 [00:00<?, ? examples/s]
Filter: 100%|##########| 63/63 [00:00<00:00, 11681.75 examples/s]

Filter:   0%|          | 0/60 [00:00<?, ? examples/s]
Filter: 100%|##########| 60/60 [00:00<00:00, 11663.80 examples/s]

Flattening the indices:   0%|          | 0/60 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 60/60 [00:00<00:00, 12327.73 examples/s]

Casting to class labels:   0%|          | 0/60 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 60/60 [00:00<00:00, 6007.31 examples/s]INFO 2025-05-20 10:53:24,033 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_1_4__Reviewing_top_consumers_for_projects__virtual_machines__and_nodes.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 498.79ba/s]INFO 2025-05-20 10:53:24,035 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_1_4__Reviewing_top_consumers_for_projects__virtual_machines__and_nodes_task.yaml
INFO 2025-05-20 10:53:24,045 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:53:24,050 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/151 [00:00<?, ? examples/s]
Filter: 100%|##########| 151/151 [00:00<00:00, 33951.96 examples/s]

Filter:   0%|          | 0/151 [00:00<?, ? examples/s]
Filter: 100%|##########| 151/151 [00:00<00:00, 19242.26 examples/s]

Flattening the indices:   0%|          | 0/151 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 151/151 [00:00<00:00, 26876.30 examples/s]

Map:   0%|          | 0/151 [00:00<?, ? examples/s]
Map: 100%|##########| 151/151 [00:00<00:00, 8184.59 examples/s]

Map:   0%|          | 0/151 [00:00<?, ? examples/s]
Map: 100%|##########| 151/151 [00:00<00:00, 7604.12 examples/s]

Map:   0%|          | 0/151 [00:00<?, ? examples/s]
Map: 100%|##########| 151/151 [00:00<00:00, 7526.77 examples/s]

Filter:   0%|          | 0/151 [00:00<?, ? examples/s]
Filter: 100%|##########| 151/151 [00:00<00:00, 27307.37 examples/s]

Filter:   0%|          | 0/151 [00:00<?, ? examples/s]
Filter: 100%|##########| 151/151 [00:00<00:00, 15368.60 examples/s]

Filter:   0%|          | 0/148 [00:00<?, ? examples/s]
Filter: 100%|##########| 148/148 [00:00<00:00, 15546.91 examples/s]

Flattening the indices:   0%|          | 0/148 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 148/148 [00:00<00:00, 23422.14 examples/s]

Casting to class labels:   0%|          | 0/148 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 148/148 [00:00<00:00, 7912.57 examples/s]INFO 2025-05-20 10:53:39,099 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_7__Accessing_virtual_machine_consoles.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 258.40ba/s]INFO 2025-05-20 10:53:39,103 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_7__Accessing_virtual_machine_consoles_task.yaml
INFO 2025-05-20 10:53:39,112 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:53:39,117 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/11 [00:00<?, ? examples/s]
Filter: 100%|##########| 11/11 [00:00<00:00, 5371.05 examples/s]

Filter:   0%|          | 0/11 [00:00<?, ? examples/s]
Filter: 100%|##########| 11/11 [00:00<00:00, 4764.29 examples/s]

Flattening the indices:   0%|          | 0/11 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 11/11 [00:00<00:00, 3319.95 examples/s]

Map:   0%|          | 0/11 [00:00<?, ? examples/s]
Map: 100%|##########| 11/11 [00:00<00:00, 2280.87 examples/s]

Map:   0%|          | 0/11 [00:00<?, ? examples/s]
Map: 100%|##########| 11/11 [00:00<00:00, 2149.12 examples/s]

Map:   0%|          | 0/11 [00:00<?, ? examples/s]
Map: 100%|##########| 11/11 [00:00<00:00, 2046.00 examples/s]

Filter:   0%|          | 0/11 [00:00<?, ? examples/s]
Filter: 100%|##########| 11/11 [00:00<00:00, 5251.23 examples/s]

Filter:   0%|          | 0/11 [00:00<?, ? examples/s]
Filter: 100%|##########| 11/11 [00:00<00:00, 4297.84 examples/s]

Filter:   0%|          | 0/10 [00:00<?, ? examples/s]
Filter: 100%|##########| 10/10 [00:00<00:00, 4038.81 examples/s]

Flattening the indices:   0%|          | 0/10 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 10/10 [00:00<00:00, 2433.60 examples/s]

Casting to class labels:   0%|          | 0/10 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 10/10 [00:00<00:00, 2046.60 examples/s]INFO 2025-05-20 10:53:46,574 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_2_5__About_the_hostpath_provisioner_operator.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 1158.33ba/s]INFO 2025-05-20 10:53:46,575 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_5__About_the_hostpath_provisioner_operator_task.yaml
INFO 2025-05-20 10:53:46,586 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:53:46,591 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/157 [00:00<?, ? examples/s]
Filter: 100%|##########| 157/157 [00:00<00:00, 34038.34 examples/s]

Filter:   0%|          | 0/154 [00:00<?, ? examples/s]
Filter: 100%|##########| 154/154 [00:00<00:00, 19413.41 examples/s]

Flattening the indices:   0%|          | 0/154 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 154/154 [00:00<00:00, 27369.61 examples/s]

Map:   0%|          | 0/154 [00:00<?, ? examples/s]
Map: 100%|##########| 154/154 [00:00<00:00, 8075.65 examples/s]

Map:   0%|          | 0/154 [00:00<?, ? examples/s]
Map: 100%|##########| 154/154 [00:00<00:00, 7580.36 examples/s]

Map:   0%|          | 0/154 [00:00<?, ? examples/s]
Map: 100%|##########| 154/154 [00:00<00:00, 7680.41 examples/s]

Filter:   0%|          | 0/154 [00:00<?, ? examples/s]
Filter: 100%|##########| 154/154 [00:00<00:00, 27397.47 examples/s]

Filter:   0%|          | 0/154 [00:00<?, ? examples/s]
Filter: 100%|##########| 154/154 [00:00<00:00, 15345.50 examples/s]

Filter:   0%|          | 0/153 [00:00<?, ? examples/s]
Filter: 100%|##########| 153/153 [00:00<00:00, 15714.77 examples/s]

Flattening the indices:   0%|          | 0/153 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 153/153 [00:00<00:00, 23116.19 examples/s]

Casting to class labels:   0%|          | 0/153 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 153/153 [00:00<00:00, 8005.89 examples/s]INFO 2025-05-20 10:54:02,518 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_1_4__Using_the_CLI_to_create_a_virtual_machine.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 217.82ba/s]INFO 2025-05-20 10:54:02,523 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1_4__Using_the_CLI_to_create_a_virtual_machine_task.yaml
INFO 2025-05-20 10:54:02,533 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:54:02,538 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/102 [00:00<?, ? examples/s]
Filter: 100%|##########| 102/102 [00:00<00:00, 28013.29 examples/s]

Filter:   0%|          | 0/102 [00:00<?, ? examples/s]
Filter: 100%|##########| 102/102 [00:00<00:00, 17284.22 examples/s]

Flattening the indices:   0%|          | 0/102 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 102/102 [00:00<00:00, 22489.57 examples/s]

Map:   0%|          | 0/102 [00:00<?, ? examples/s]
Map: 100%|##########| 102/102 [00:00<00:00, 7487.86 examples/s]

Map:   0%|          | 0/102 [00:00<?, ? examples/s]
Map: 100%|##########| 102/102 [00:00<00:00, 6999.77 examples/s]

Map:   0%|          | 0/102 [00:00<?, ? examples/s]
Map: 100%|##########| 102/102 [00:00<00:00, 6865.76 examples/s]

Filter:   0%|          | 0/102 [00:00<?, ? examples/s]
Filter: 100%|##########| 102/102 [00:00<00:00, 22966.45 examples/s]

Filter:   0%|          | 0/100 [00:00<?, ? examples/s]
Filter: 100%|##########| 100/100 [00:00<00:00, 13854.02 examples/s]

Filter:   0%|          | 0/99 [00:00<?, ? examples/s]
Filter: 100%|##########| 99/99 [00:00<00:00, 14142.92 examples/s]

Flattening the indices:   0%|          | 0/99 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 99/99 [00:00<00:00, 18193.76 examples/s]

Casting to class labels:   0%|          | 0/99 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 99/99 [00:00<00:00, 7142.25 examples/s]INFO 2025-05-20 10:54:16,689 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_14_2__Collecting_data_about_virtual_machines.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 342.31ba/s]INFO 2025-05-20 10:54:16,693 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_14_2__Collecting_data_about_virtual_machines_task.yaml
INFO 2025-05-20 10:54:16,702 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:54:16,706 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/52 [00:00<?, ? examples/s]
Filter: 100%|##########| 52/52 [00:00<00:00, 17752.22 examples/s]

Filter:   0%|          | 0/52 [00:00<?, ? examples/s]
Filter: 100%|##########| 52/52 [00:00<00:00, 12946.15 examples/s]

Flattening the indices:   0%|          | 0/52 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 52/52 [00:00<00:00, 13260.20 examples/s]

Map:   0%|          | 0/52 [00:00<?, ? examples/s]
Map: 100%|##########| 52/52 [00:00<00:00, 5798.78 examples/s]

Map:   0%|          | 0/52 [00:00<?, ? examples/s]
Map: 100%|##########| 52/52 [00:00<00:00, 5478.48 examples/s]

Map:   0%|          | 0/52 [00:00<?, ? examples/s]
Map: 100%|##########| 52/52 [00:00<00:00, 5374.66 examples/s]

Filter:   0%|          | 0/52 [00:00<?, ? examples/s]
Filter: 100%|##########| 52/52 [00:00<00:00, 16638.98 examples/s]

Filter:   0%|          | 0/52 [00:00<?, ? examples/s]
Filter: 100%|##########| 52/52 [00:00<00:00, 11012.01 examples/s]

Filter:   0%|          | 0/51 [00:00<?, ? examples/s]
Filter: 100%|##########| 51/51 [00:00<00:00, 10956.79 examples/s]

Flattening the indices:   0%|          | 0/51 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 51/51 [00:00<00:00, 10680.52 examples/s]

Casting to class labels:   0%|          | 0/51 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 51/51 [00:00<00:00, 5751.80 examples/s]INFO 2025-05-20 10:54:28,332 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_14_6__Removing_the_VirtIO_container_disk_from_a_virtual_machine.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 452.56ba/s]INFO 2025-05-20 10:54:28,335 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_14_6__Removing_the_VirtIO_container_disk_from_a_virtual_machine_task.yaml
INFO 2025-05-20 10:54:28,345 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:54:28,349 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/100 [00:00<?, ? examples/s]
Filter: 100%|##########| 100/100 [00:00<00:00, 27815.53 examples/s]

Filter:   0%|          | 0/100 [00:00<?, ? examples/s]
Filter: 100%|##########| 100/100 [00:00<00:00, 17041.01 examples/s]

Flattening the indices:   0%|          | 0/100 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 100/100 [00:00<00:00, 21942.47 examples/s]

Map:   0%|          | 0/100 [00:00<?, ? examples/s]
Map: 100%|##########| 100/100 [00:00<00:00, 7311.10 examples/s]

Map:   0%|          | 0/100 [00:00<?, ? examples/s]
Map: 100%|##########| 100/100 [00:00<00:00, 6804.52 examples/s]

Map:   0%|          | 0/100 [00:00<?, ? examples/s]
Map: 100%|##########| 100/100 [00:00<00:00, 6879.63 examples/s]

Filter:   0%|          | 0/100 [00:00<?, ? examples/s]
Filter: 100%|##########| 100/100 [00:00<00:00, 22729.66 examples/s]

Filter:   0%|          | 0/99 [00:00<?, ? examples/s]
Filter: 100%|##########| 99/99 [00:00<00:00, 13950.01 examples/s]

Filter:   0%|          | 0/98 [00:00<?, ? examples/s]
Filter: 100%|##########| 98/98 [00:00<00:00, 13919.94 examples/s]

Flattening the indices:   0%|          | 0/97 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 97/97 [00:00<00:00, 17212.31 examples/s]

Casting to class labels:   0%|          | 0/97 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 97/97 [00:00<00:00, 7153.24 examples/s]INFO 2025-05-20 10:54:46,720 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_14_1__Collecting_data_about_your_environment.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 318.50ba/s]INFO 2025-05-20 10:54:46,724 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_14_1__Collecting_data_about_your_environment_task.yaml
INFO 2025-05-20 10:54:46,732 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:54:46,737 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/14 [00:00<?, ? examples/s]
Filter: 100%|##########| 14/14 [00:00<00:00, 6529.55 examples/s]

Filter:   0%|          | 0/14 [00:00<?, ? examples/s]
Filter: 100%|##########| 14/14 [00:00<00:00, 5724.34 examples/s]

Flattening the indices:   0%|          | 0/14 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 14/14 [00:00<00:00, 4089.44 examples/s]

Map:   0%|          | 0/14 [00:00<?, ? examples/s]
Map: 100%|##########| 14/14 [00:00<00:00, 2748.95 examples/s]

Map:   0%|          | 0/14 [00:00<?, ? examples/s]
Map: 100%|##########| 14/14 [00:00<00:00, 2579.07 examples/s]

Map:   0%|          | 0/14 [00:00<?, ? examples/s]
Map: 100%|##########| 14/14 [00:00<00:00, 2502.25 examples/s]

Filter:   0%|          | 0/14 [00:00<?, ? examples/s]
Filter: 100%|##########| 14/14 [00:00<00:00, 6459.87 examples/s]

Filter:   0%|          | 0/14 [00:00<?, ? examples/s]
Filter: 100%|##########| 14/14 [00:00<00:00, 5260.73 examples/s]

Filter:   0%|          | 0/14 [00:00<?, ? examples/s]
Filter: 100%|##########| 14/14 [00:00<00:00, 5433.04 examples/s]

Casting to class labels:   0%|          | 0/14 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 14/14 [00:00<00:00, 2611.99 examples/s]INFO 2025-05-20 10:54:51,805 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_3_1__Before_you_begin.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 987.13ba/s]INFO 2025-05-20 10:54:51,806 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_3_1__Before_you_begin_task.yaml
INFO 2025-05-20 10:54:51,816 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:54:51,820 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 15427.15 examples/s]

Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 11389.26 examples/s]

Flattening the indices:   0%|          | 0/41 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 41/41 [00:00<00:00, 10675.18 examples/s]

Map:   0%|          | 0/41 [00:00<?, ? examples/s]
Map: 100%|##########| 41/41 [00:00<00:00, 5179.40 examples/s]

Map:   0%|          | 0/41 [00:00<?, ? examples/s]
Map: 100%|##########| 41/41 [00:00<00:00, 4930.09 examples/s]

Map:   0%|          | 0/41 [00:00<?, ? examples/s]
Map: 100%|##########| 41/41 [00:00<00:00, 4851.78 examples/s]

Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 14117.60 examples/s]

Filter:   0%|          | 0/41 [00:00<?, ? examples/s]
Filter: 100%|##########| 41/41 [00:00<00:00, 9708.49 examples/s]

Filter:   0%|          | 0/40 [00:00<?, ? examples/s]
Filter: 100%|##########| 40/40 [00:00<00:00, 9849.25 examples/s]

Flattening the indices:   0%|          | 0/40 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 40/40 [00:00<00:00, 8530.21 examples/s]

Casting to class labels:   0%|          | 0/40 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 40/40 [00:00<00:00, 5040.78 examples/s]INFO 2025-05-20 10:55:00,138 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_5_4_3__Deploying_the_OpenShift_Virtualization_Operator_by_using_the_CLI.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 590.00ba/s]INFO 2025-05-20 10:55:00,140 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_4_3__Deploying_the_OpenShift_Virtualization_Operator_by_using_the_CLI_task.yaml
INFO 2025-05-20 10:55:00,149 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:55:00,154 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/4 [00:00<?, ? examples/s]
Filter: 100%|##########| 4/4 [00:00<00:00, 2105.84 examples/s]

Filter:   0%|          | 0/4 [00:00<?, ? examples/s]
Filter: 100%|##########| 4/4 [00:00<00:00, 1961.10 examples/s]

Flattening the indices:   0%|          | 0/4 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 4/4 [00:00<00:00, 1213.98 examples/s]

Map:   0%|          | 0/4 [00:00<?, ? examples/s]
Map: 100%|##########| 4/4 [00:00<00:00, 953.36 examples/s]

Map:   0%|          | 0/4 [00:00<?, ? examples/s]
Map: 100%|##########| 4/4 [00:00<00:00, 937.59 examples/s]

Map:   0%|          | 0/4 [00:00<?, ? examples/s]
Map: 100%|##########| 4/4 [00:00<00:00, 891.31 examples/s]

Filter:   0%|          | 0/4 [00:00<?, ? examples/s]
Filter: 100%|##########| 4/4 [00:00<00:00, 2096.37 examples/s]

Filter:   0%|          | 0/4 [00:00<?, ? examples/s]
Filter: 100%|##########| 4/4 [00:00<00:00, 1802.84 examples/s]

Filter:   0%|          | 0/4 [00:00<?, ? examples/s]
Filter: 100%|##########| 4/4 [00:00<00:00, 1813.17 examples/s]

Casting to class labels:   0%|          | 0/4 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 4/4 [00:00<00:00, 913.19 examples/s]INFO 2025-05-20 10:55:01,791 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_8_4__Creating_a_container_using_virtctl_guestfs.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 1165.73ba/s]INFO 2025-05-20 10:55:01,792 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_8_4__Creating_a_container_using_virtctl_guestfs_task.yaml
INFO 2025-05-20 10:55:01,803 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:55:01,808 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/146 [00:00<?, ? examples/s]
Filter: 100%|##########| 146/146 [00:00<00:00, 32792.57 examples/s]

Filter:   0%|          | 0/146 [00:00<?, ? examples/s]
Filter: 100%|##########| 146/146 [00:00<00:00, 19110.83 examples/s]

Flattening the indices:   0%|          | 0/146 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 146/146 [00:00<00:00, 26685.04 examples/s]

Map:   0%|          | 0/146 [00:00<?, ? examples/s]
Map: 100%|##########| 146/146 [00:00<00:00, 8080.34 examples/s]

Map:   0%|          | 0/146 [00:00<?, ? examples/s]
Map: 100%|##########| 146/146 [00:00<00:00, 7617.66 examples/s]

Map:   0%|          | 0/146 [00:00<?, ? examples/s]
Map: 100%|##########| 146/146 [00:00<00:00, 7441.95 examples/s]

Filter:   0%|          | 0/146 [00:00<?, ? examples/s]
Filter: 100%|##########| 146/146 [00:00<00:00, 26038.28 examples/s]

Filter:   0%|          | 0/146 [00:00<?, ? examples/s]
Filter: 100%|##########| 146/146 [00:00<00:00, 15344.12 examples/s]

Filter:   0%|          | 0/143 [00:00<?, ? examples/s]
Filter: 100%|##########| 143/143 [00:00<00:00, 15138.83 examples/s]

Flattening the indices:   0%|          | 0/143 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 143/143 [00:00<00:00, 23097.10 examples/s]

Casting to class labels:   0%|          | 0/143 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 143/143 [00:00<00:00, 8120.68 examples/s]INFO 2025-05-20 10:55:18,162 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_5_1_6__Snapshots_and_cloning.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 230.56ba/s]INFO 2025-05-20 10:55:18,167 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_1_6__Snapshots_and_cloning_task.yaml
INFO 2025-05-20 10:55:18,177 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:55:18,182 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/146 [00:00<?, ? examples/s]
Filter: 100%|##########| 146/146 [00:00<00:00, 32150.39 examples/s]

Filter:   0%|          | 0/146 [00:00<?, ? examples/s]
Filter: 100%|##########| 146/146 [00:00<00:00, 19215.78 examples/s]

Flattening the indices:   0%|          | 0/146 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 146/146 [00:00<00:00, 28552.64 examples/s]

Map:   0%|          | 0/146 [00:00<?, ? examples/s]
Map: 100%|##########| 146/146 [00:00<00:00, 8038.44 examples/s]

Map:   0%|          | 0/146 [00:00<?, ? examples/s]
Map: 100%|##########| 146/146 [00:00<00:00, 7665.72 examples/s]

Map:   0%|          | 0/146 [00:00<?, ? examples/s]
Map: 100%|##########| 146/146 [00:00<00:00, 7507.92 examples/s]

Filter:   0%|          | 0/146 [00:00<?, ? examples/s]
Filter: 100%|##########| 146/146 [00:00<00:00, 26685.04 examples/s]

Filter:   0%|          | 0/146 [00:00<?, ? examples/s]
Filter: 100%|##########| 146/146 [00:00<00:00, 15357.97 examples/s]

Filter:   0%|          | 0/142 [00:00<?, ? examples/s]
Filter: 100%|##########| 142/142 [00:00<00:00, 15373.27 examples/s]

Flattening the indices:   0%|          | 0/142 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 142/142 [00:00<00:00, 22828.33 examples/s]

Casting to class labels:   0%|          | 0/142 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 142/142 [00:00<00:00, 7956.49 examples/s]INFO 2025-05-20 10:55:35,458 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_2_1__Editing_a_virtual_machine_in_the_web_console.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 218.81ba/s]INFO 2025-05-20 10:55:35,463 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_2_1__Editing_a_virtual_machine_in_the_web_console_task.yaml
INFO 2025-05-20 10:55:35,473 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:55:35,477 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/85 [00:00<?, ? examples/s]
Filter: 100%|##########| 85/85 [00:00<00:00, 24825.28 examples/s]

Filter:   0%|          | 0/85 [00:00<?, ? examples/s]
Filter: 100%|##########| 85/85 [00:00<00:00, 15625.01 examples/s]

Flattening the indices:   0%|          | 0/85 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 85/85 [00:00<00:00, 19877.11 examples/s]

Map:   0%|          | 0/85 [00:00<?, ? examples/s]
Map: 100%|##########| 85/85 [00:00<00:00, 7092.02 examples/s]

Map:   0%|          | 0/85 [00:00<?, ? examples/s]
Map: 100%|##########| 85/85 [00:00<00:00, 6647.20 examples/s]

Map:   0%|          | 0/85 [00:00<?, ? examples/s]
Map: 100%|##########| 85/85 [00:00<00:00, 6471.40 examples/s]

Filter:   0%|          | 0/85 [00:00<?, ? examples/s]
Filter: 100%|##########| 85/85 [00:00<00:00, 21326.54 examples/s]

Filter:   0%|          | 0/85 [00:00<?, ? examples/s]
Filter: 100%|##########| 85/85 [00:00<00:00, 13101.90 examples/s]

Filter:   0%|          | 0/84 [00:00<?, ? examples/s]
Filter: 100%|##########| 84/84 [00:00<00:00, 13236.71 examples/s]

Flattening the indices:   0%|          | 0/84 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 84/84 [00:00<00:00, 15906.16 examples/s]

Casting to class labels:   0%|          | 0/84 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 84/84 [00:00<00:00, 6929.60 examples/s]INFO 2025-05-20 10:55:50,349 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_9_1__About_OpenShift_Container_Platform_monitoring.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 362.23ba/s]INFO 2025-05-20 10:55:50,352 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_9_1__About_OpenShift_Container_Platform_monitoring_task.yaml
INFO 2025-05-20 10:55:50,361 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:55:50,366 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/13 [00:00<?, ? examples/s]
Filter: 100%|##########| 13/13 [00:00<00:00, 6267.35 examples/s]

Filter:   0%|          | 0/13 [00:00<?, ? examples/s]
Filter: 100%|##########| 13/13 [00:00<00:00, 5355.13 examples/s]

Flattening the indices:   0%|          | 0/13 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 13/13 [00:00<00:00, 3801.57 examples/s]

Map:   0%|          | 0/13 [00:00<?, ? examples/s]
Map: 100%|##########| 13/13 [00:00<00:00, 2560.75 examples/s]

Map:   0%|          | 0/13 [00:00<?, ? examples/s]
Map: 100%|##########| 13/13 [00:00<00:00, 2440.40 examples/s]

Map:   0%|          | 0/13 [00:00<?, ? examples/s]
Map: 100%|##########| 13/13 [00:00<00:00, 2390.96 examples/s]

Filter:   0%|          | 0/13 [00:00<?, ? examples/s]
Filter: 100%|##########| 13/13 [00:00<00:00, 5743.81 examples/s]

Filter:   0%|          | 0/13 [00:00<?, ? examples/s]
Filter: 100%|##########| 13/13 [00:00<00:00, 4824.02 examples/s]

Filter:   0%|          | 0/12 [00:00<?, ? examples/s]
Filter: 100%|##########| 12/12 [00:00<00:00, 4523.38 examples/s]

Flattening the indices:   0%|          | 0/12 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 12/12 [00:00<00:00, 2830.48 examples/s]

Casting to class labels:   0%|          | 0/12 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 12/12 [00:00<00:00, 2354.92 examples/s]INFO 2025-05-20 10:55:55,895 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_7_1__About_workload_security.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 1010.43ba/s]INFO 2025-05-20 10:55:55,897 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_7_1__About_workload_security_task.yaml
INFO 2025-05-20 10:55:55,907 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:55:55,911 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/75 [00:00<?, ? examples/s]
Filter: 100%|##########| 75/75 [00:00<00:00, 22341.82 examples/s]

Filter:   0%|          | 0/75 [00:00<?, ? examples/s]
Filter: 100%|##########| 75/75 [00:00<00:00, 14917.15 examples/s]

Flattening the indices:   0%|          | 0/75 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 75/75 [00:00<00:00, 17019.57 examples/s]

Map:   0%|          | 0/75 [00:00<?, ? examples/s]
Map: 100%|##########| 75/75 [00:00<00:00, 6730.99 examples/s]

Map:   0%|          | 0/75 [00:00<?, ? examples/s]
Map: 100%|##########| 75/75 [00:00<00:00, 6321.04 examples/s]

Map:   0%|          | 0/75 [00:00<?, ? examples/s]
Map: 100%|##########| 75/75 [00:00<00:00, 6320.66 examples/s]

Filter:   0%|          | 0/75 [00:00<?, ? examples/s]
Filter: 100%|##########| 75/75 [00:00<00:00, 19921.02 examples/s]

Filter:   0%|          | 0/75 [00:00<?, ? examples/s]
Filter: 100%|##########| 75/75 [00:00<00:00, 12313.49 examples/s]

Filter:   0%|          | 0/74 [00:00<?, ? examples/s]
Filter: 100%|##########| 74/74 [00:00<00:00, 12647.35 examples/s]

Flattening the indices:   0%|          | 0/74 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 74/74 [00:00<00:00, 13589.25 examples/s]

Casting to class labels:   0%|          | 0/74 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 74/74 [00:00<00:00, 6552.49 examples/s]INFO 2025-05-20 10:56:09,553 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_10_1_5__Additional_resources.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 327.53ba/s]INFO 2025-05-20 10:56:09,557 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_1_5__Additional_resources_task.yaml
INFO 2025-05-20 10:56:09,565 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:56:09,570 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/10 [00:00<?, ? examples/s]
Filter: 100%|##########| 10/10 [00:00<00:00, 4854.52 examples/s]

Filter:   0%|          | 0/10 [00:00<?, ? examples/s]
Filter: 100%|##########| 10/10 [00:00<00:00, 4268.58 examples/s]

Flattening the indices:   0%|          | 0/10 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 10/10 [00:00<00:00, 3049.52 examples/s]

Map:   0%|          | 0/10 [00:00<?, ? examples/s]
Map: 100%|##########| 10/10 [00:00<00:00, 2055.12 examples/s]

Map:   0%|          | 0/10 [00:00<?, ? examples/s]
Map: 100%|##########| 10/10 [00:00<00:00, 2055.93 examples/s]

Map:   0%|          | 0/10 [00:00<?, ? examples/s]
Map: 100%|##########| 10/10 [00:00<00:00, 1947.67 examples/s]

Filter:   0%|          | 0/10 [00:00<?, ? examples/s]
Filter: 100%|##########| 10/10 [00:00<00:00, 4763.55 examples/s]

Filter:   0%|          | 0/10 [00:00<?, ? examples/s]
Filter: 100%|##########| 10/10 [00:00<00:00, 3943.50 examples/s]

Filter:   0%|          | 0/10 [00:00<?, ? examples/s]
Filter: 100%|##########| 10/10 [00:00<00:00, 4059.92 examples/s]

Casting to class labels:   0%|          | 0/10 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 10/10 [00:00<00:00, 1996.81 examples/s]INFO 2025-05-20 10:56:18,473 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_2_3__About_the_cdi_operator.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 1036.40ba/s]INFO 2025-05-20 10:56:18,475 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_3__About_the_cdi_operator_task.yaml
INFO 2025-05-20 10:56:18,486 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:56:18,491 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/172 [00:00<?, ? examples/s]
Filter: 100%|##########| 172/172 [00:00<00:00, 34436.98 examples/s]

Filter:   0%|          | 0/170 [00:00<?, ? examples/s]
Filter: 100%|##########| 170/170 [00:00<00:00, 20095.02 examples/s]

Flattening the indices:   0%|          | 0/170 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 170/170 [00:00<00:00, 28607.09 examples/s]

Map:   0%|          | 0/170 [00:00<?, ? examples/s]
Map: 100%|##########| 170/170 [00:00<00:00, 8424.49 examples/s]

Map:   0%|          | 0/170 [00:00<?, ? examples/s]
Map: 100%|##########| 170/170 [00:00<00:00, 7812.68 examples/s]

Map:   0%|          | 0/170 [00:00<?, ? examples/s]
Map: 100%|##########| 170/170 [00:00<00:00, 7761.73 examples/s]

Filter:   0%|          | 0/170 [00:00<?, ? examples/s]
Filter: 100%|##########| 170/170 [00:00<00:00, 27224.30 examples/s]

Filter:   0%|          | 0/170 [00:00<?, ? examples/s]
Filter: 100%|##########| 170/170 [00:00<00:00, 15575.52 examples/s]

Filter:   0%|          | 0/169 [00:00<?, ? examples/s]
Filter: 100%|##########| 169/169 [00:00<00:00, 15691.96 examples/s]

Flattening the indices:   0%|          | 0/169 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 169/169 [00:00<00:00, 24231.27 examples/s]

Casting to class labels:   0%|          | 0/169 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 169/169 [00:00<00:00, 8206.51 examples/s]INFO 2025-05-20 10:56:36,071 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_1_5__Virtual_machine_storage_volume_types.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 175.41ba/s]INFO 2025-05-20 10:56:36,077 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1_5__Virtual_machine_storage_volume_types_task.yaml
INFO 2025-05-20 10:56:36,088 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:56:36,092 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/186 [00:00<?, ? examples/s]
Filter: 100%|##########| 186/186 [00:00<00:00, 31767.27 examples/s]

Filter:   0%|          | 0/185 [00:00<?, ? examples/s]
Filter: 100%|##########| 185/185 [00:00<00:00, 19961.57 examples/s]

Flattening the indices:   0%|          | 0/185 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 185/185 [00:00<00:00, 32981.10 examples/s]

Map:   0%|          | 0/185 [00:00<?, ? examples/s]
Map: 100%|##########| 185/185 [00:00<00:00, 8606.23 examples/s]

Map:   0%|          | 0/185 [00:00<?, ? examples/s]
Map: 100%|##########| 185/185 [00:00<00:00, 8078.82 examples/s]

Map:   0%|          | 0/185 [00:00<?, ? examples/s]
Map: 100%|##########| 185/185 [00:00<00:00, 8069.66 examples/s]

Filter:   0%|          | 0/185 [00:00<?, ? examples/s]
Filter: 100%|##########| 185/185 [00:00<00:00, 29093.26 examples/s]

Filter:   0%|          | 0/185 [00:00<?, ? examples/s]
Filter: 100%|##########| 185/185 [00:00<00:00, 16119.87 examples/s]

Filter:   0%|          | 0/184 [00:00<?, ? examples/s]
Filter: 100%|##########| 184/184 [00:00<00:00, 16060.43 examples/s]

Flattening the indices:   0%|          | 0/184 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 184/184 [00:00<00:00, 26439.82 examples/s]

Casting to class labels:   0%|          | 0/184 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 184/184 [00:00<00:00, 8361.07 examples/s]INFO 2025-05-20 10:56:56,352 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_1__Creating_virtual_machines.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 178.65ba/s]INFO 2025-05-20 10:56:56,358 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1__Creating_virtual_machines_task.yaml
INFO 2025-05-20 10:56:56,368 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:56:56,372 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 8101.46 examples/s]

Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 6795.45 examples/s]

Flattening the indices:   0%|          | 0/18 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 18/18 [00:00<00:00, 5159.05 examples/s]

Map:   0%|          | 0/18 [00:00<?, ? examples/s]
Map: 100%|##########| 18/18 [00:00<00:00, 3286.07 examples/s]

Map:   0%|          | 0/18 [00:00<?, ? examples/s]
Map: 100%|##########| 18/18 [00:00<00:00, 3189.58 examples/s]

Map:   0%|          | 0/18 [00:00<?, ? examples/s]
Map: 100%|##########| 18/18 [00:00<00:00, 3011.83 examples/s]

Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 7614.47 examples/s]

Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 6083.11 examples/s]

Filter:   0%|          | 0/18 [00:00<?, ? examples/s]
Filter: 100%|##########| 18/18 [00:00<00:00, 6128.04 examples/s]

Casting to class labels:   0%|          | 0/18 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 18/18 [00:00<00:00, 3149.01 examples/s]INFO 2025-05-20 10:57:04,452 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_4_1__Deleting_a_virtual_machine_using_the_web_console.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 864.63ba/s]INFO 2025-05-20 10:57:04,454 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_4_1__Deleting_a_virtual_machine_using_the_web_console_task.yaml
INFO 2025-05-20 10:57:04,463 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:57:04,468 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 16081.59 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 12319.34 examples/s]

Flattening the indices:   0%|          | 0/43 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 43/43 [00:00<00:00, 11223.79 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 5289.16 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 5116.31 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 4878.94 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 14731.28 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 9998.62 examples/s]

Filter:   0%|          | 0/43 [00:00<?, ? examples/s]
Filter: 100%|##########| 43/43 [00:00<00:00, 10399.30 examples/s]

Casting to class labels:   0%|          | 0/43 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 43/43 [00:00<00:00, 5301.91 examples/s]INFO 2025-05-20 10:57:14,868 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_12_1_1__About_node_maintenance_mode.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 515.78ba/s]INFO 2025-05-20 10:57:14,871 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_12_1_1__About_node_maintenance_mode_task.yaml
INFO 2025-05-20 10:57:14,881 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:57:14,885 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/99 [00:00<?, ? examples/s]
Filter: 100%|##########| 99/99 [00:00<00:00, 27298.41 examples/s]

Filter:   0%|          | 0/99 [00:00<?, ? examples/s]
Filter: 100%|##########| 99/99 [00:00<00:00, 17033.23 examples/s]

Flattening the indices:   0%|          | 0/99 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 99/99 [00:00<00:00, 20694.55 examples/s]

Map:   0%|          | 0/99 [00:00<?, ? examples/s]
Map: 100%|##########| 99/99 [00:00<00:00, 7353.09 examples/s]

Map:   0%|          | 0/99 [00:00<?, ? examples/s]
Map: 100%|##########| 99/99 [00:00<00:00, 6865.34 examples/s]

Map:   0%|          | 0/99 [00:00<?, ? examples/s]
Map: 100%|##########| 99/99 [00:00<00:00, 6908.51 examples/s]

Filter:   0%|          | 0/99 [00:00<?, ? examples/s]
Filter: 100%|##########| 99/99 [00:00<00:00, 22915.90 examples/s]

Filter:   0%|          | 0/98 [00:00<?, ? examples/s]
Filter: 100%|##########| 98/98 [00:00<00:00, 13816.53 examples/s]

Filter:   0%|          | 0/98 [00:00<?, ? examples/s]
Filter: 100%|##########| 98/98 [00:00<00:00, 13860.79 examples/s]

Flattening the indices:   0%|          | 0/98 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 98/98 [00:00<00:00, 17810.98 examples/s]

Casting to class labels:   0%|          | 0/98 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 98/98 [00:00<00:00, 7258.38 examples/s]INFO 2025-05-20 10:57:28,910 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_13_14_3__Using_the_must_gather_tool_for_OpenShift_Virtualization.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 305.08ba/s]INFO 2025-05-20 10:57:28,913 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_14_3__Using_the_must_gather_tool_for_OpenShift_Virtualization_task.yaml
INFO 2025-05-20 10:57:28,923 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:57:28,928 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/70 [00:00<?, ? examples/s]
Filter: 100%|##########| 70/70 [00:00<00:00, 22220.64 examples/s]

Filter:   0%|          | 0/70 [00:00<?, ? examples/s]
Filter: 100%|##########| 70/70 [00:00<00:00, 15044.90 examples/s]

Flattening the indices:   0%|          | 0/70 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 70/70 [00:00<00:00, 16922.26 examples/s]

Map:   0%|          | 0/70 [00:00<?, ? examples/s]
Map: 100%|##########| 70/70 [00:00<00:00, 6510.87 examples/s]

Map:   0%|          | 0/70 [00:00<?, ? examples/s]
Map: 100%|##########| 70/70 [00:00<00:00, 6204.46 examples/s]

Map:   0%|          | 0/70 [00:00<?, ? examples/s]
Map: 100%|##########| 70/70 [00:00<00:00, 6011.00 examples/s]

Filter:   0%|          | 0/70 [00:00<?, ? examples/s]
Filter: 100%|##########| 70/70 [00:00<00:00, 19253.81 examples/s]

Filter:   0%|          | 0/70 [00:00<?, ? examples/s]
Filter: 100%|##########| 70/70 [00:00<00:00, 12395.56 examples/s]

Filter:   0%|          | 0/70 [00:00<?, ? examples/s]
Filter: 100%|##########| 70/70 [00:00<00:00, 12885.73 examples/s]

Casting to class labels:   0%|          | 0/70 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 70/70 [00:00<00:00, 6509.43 examples/s]INFO 2025-05-20 10:57:40,446 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_5_6__Uninstalling_OpenShift_Virtualization.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 416.27ba/s]INFO 2025-05-20 10:57:40,448 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_6__Uninstalling_OpenShift_Virtualization_task.yaml
INFO 2025-05-20 10:57:40,458 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:57:40,463 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/47 [00:00<?, ? examples/s]
Filter: 100%|##########| 47/47 [00:00<00:00, 16651.09 examples/s]

Filter:   0%|          | 0/47 [00:00<?, ? examples/s]
Filter: 100%|##########| 47/47 [00:00<00:00, 12075.48 examples/s]

Flattening the indices:   0%|          | 0/47 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 47/47 [00:00<00:00, 11646.03 examples/s]

Map:   0%|          | 0/47 [00:00<?, ? examples/s]
Map: 100%|##########| 47/47 [00:00<00:00, 5518.51 examples/s]

Map:   0%|          | 0/47 [00:00<?, ? examples/s]
Map: 100%|##########| 47/47 [00:00<00:00, 5250.28 examples/s]

Map:   0%|          | 0/47 [00:00<?, ? examples/s]
Map: 100%|##########| 47/47 [00:00<00:00, 5091.49 examples/s]

Filter:   0%|          | 0/47 [00:00<?, ? examples/s]
Filter: 100%|##########| 47/47 [00:00<00:00, 15357.77 examples/s]

Filter:   0%|          | 0/47 [00:00<?, ? examples/s]
Filter: 100%|##########| 47/47 [00:00<00:00, 10442.99 examples/s]

Filter:   0%|          | 0/46 [00:00<?, ? examples/s]
Filter: 100%|##########| 46/46 [00:00<00:00, 10514.33 examples/s]

Flattening the indices:   0%|          | 0/46 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 46/46 [00:00<00:00, 9466.10 examples/s]

Casting to class labels:   0%|          | 0/46 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 46/46 [00:00<00:00, 5405.94 examples/s]INFO 2025-05-20 10:57:51,656 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_14_1__Prerequisites.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 457.79ba/s]INFO 2025-05-20 10:57:51,659 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_14_1__Prerequisites_task.yaml
INFO 2025-05-20 10:57:51,668 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:57:51,673 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/58 [00:00<?, ? examples/s]
Filter: 100%|##########| 58/58 [00:00<00:00, 19642.28 examples/s]

Filter:   0%|          | 0/58 [00:00<?, ? examples/s]
Filter: 100%|##########| 58/58 [00:00<00:00, 13789.23 examples/s]

Flattening the indices:   0%|          | 0/58 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 58/58 [00:00<00:00, 13890.01 examples/s]

Map:   0%|          | 0/58 [00:00<?, ? examples/s]
Map: 100%|##########| 58/58 [00:00<00:00, 6000.14 examples/s]

Map:   0%|          | 0/58 [00:00<?, ? examples/s]
Map: 100%|##########| 58/58 [00:00<00:00, 5776.04 examples/s]

Map:   0%|          | 0/58 [00:00<?, ? examples/s]
Map: 100%|##########| 58/58 [00:00<00:00, 5593.95 examples/s]

Filter:   0%|          | 0/58 [00:00<?, ? examples/s]
Filter: 100%|##########| 58/58 [00:00<00:00, 17578.56 examples/s]

Filter:   0%|          | 0/58 [00:00<?, ? examples/s]
Filter: 100%|##########| 58/58 [00:00<00:00, 11186.87 examples/s]

Filter:   0%|          | 0/57 [00:00<?, ? examples/s]
Filter: 100%|##########| 57/57 [00:00<00:00, 11515.04 examples/s]

Flattening the indices:   0%|          | 0/57 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 57/57 [00:00<00:00, 11569.09 examples/s]

Casting to class labels:   0%|          | 0/57 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 57/57 [00:00<00:00, 5902.66 examples/s]INFO 2025-05-20 10:58:02,237 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_3_2__Editing_a_boot_order_list_in_the_web_console.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 430.36ba/s]INFO 2025-05-20 10:58:02,240 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_3_2__Editing_a_boot_order_list_in_the_web_console_task.yaml
INFO 2025-05-20 10:58:02,264 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:58:02,270 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge
INFO 2025-05-20 10:58:02,274 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/917 [00:00<?, ? examples/s]
Filter: 100%|##########| 917/917 [00:00<00:00, 51830.38 examples/s]

Filter:   0%|          | 0/916 [00:00<?, ? examples/s]
Filter: 100%|##########| 916/916 [00:00<00:00, 22931.05 examples/s]

Flattening the indices:   0%|          | 0/916 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 916/916 [00:00<00:00, 47771.00 examples/s]

Map:   0%|          | 0/916 [00:00<?, ? examples/s]
Map: 100%|##########| 916/916 [00:00<00:00, 9619.12 examples/s]

Map:   0%|          | 0/916 [00:00<?, ? examples/s]
Map: 100%|##########| 916/916 [00:00<00:00, 9155.64 examples/s]
Map: 100%|##########| 916/916 [00:00<00:00, 9016.88 examples/s]

Map:   0%|          | 0/916 [00:00<?, ? examples/s]
Map: 100%|##########| 916/916 [00:00<00:00, 9052.46 examples/s]

Filter:   0%|          | 0/916 [00:00<?, ? examples/s]
Filter: 100%|##########| 916/916 [00:00<00:00, 37232.48 examples/s]

Filter:   0%|          | 0/911 [00:00<?, ? examples/s]
Filter: 100%|##########| 911/911 [00:00<00:00, 18060.70 examples/s]

Filter:   0%|          | 0/901 [00:00<?, ? examples/s]
Filter: 100%|##########| 901/901 [00:00<00:00, 17988.45 examples/s]

Flattening the indices:   0%|          | 0/901 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 901/901 [00:00<00:00, 43272.43 examples/s]

Casting to class labels:   0%|          | 0/901 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 901/901 [00:00<00:00, 9617.98 examples/s]INFO 2025-05-20 10:58:54,742 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_9_16_5__Configuring_PXE_booting_for_virtual_machines.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 39.98ba/s]INFO 2025-05-20 10:58:54,768 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_16_5__Configuring_PXE_booting_for_virtual_machines_task.yaml
INFO 2025-05-20 10:58:54,779 instructlab.sdg.pipeline:161: Running pipeline with multi-threaded batching. Using 2 workers for batches of size 256
INFO 2025-05-20 10:58:54,783 instructlab.sdg.pipeline:199: Running block: gen_mmlu_knowledge


Filter:   0%|          | 0/25 [00:00<?, ? examples/s]
Filter: 100%|##########| 25/25 [00:00<00:00, 10595.96 examples/s]

Filter:   0%|          | 0/25 [00:00<?, ? examples/s]
Filter: 100%|##########| 25/25 [00:00<00:00, 8459.67 examples/s]

Flattening the indices:   0%|          | 0/25 [00:00<?, ? examples/s]
Flattening the indices: 100%|##########| 25/25 [00:00<00:00, 6775.50 examples/s]

Map:   0%|          | 0/25 [00:00<?, ? examples/s]
Map: 100%|##########| 25/25 [00:00<00:00, 3975.34 examples/s]

Map:   0%|          | 0/25 [00:00<?, ? examples/s]
Map: 100%|##########| 25/25 [00:00<00:00, 3816.75 examples/s]

Map:   0%|          | 0/25 [00:00<?, ? examples/s]
Map: 100%|##########| 25/25 [00:00<00:00, 3630.17 examples/s]

Filter:   0%|          | 0/25 [00:00<?, ? examples/s]
Filter: 100%|##########| 25/25 [00:00<00:00, 9763.28 examples/s]

Filter:   0%|          | 0/25 [00:00<?, ? examples/s]
Filter: 100%|##########| 25/25 [00:00<00:00, 7553.49 examples/s]

Filter:   0%|          | 0/25 [00:00<?, ? examples/s]
Filter: 100%|##########| 25/25 [00:00<00:00, 7734.00 examples/s]

Casting to class labels:   0%|          | 0/25 [00:00<?, ? examples/s]
Casting to class labels: 100%|##########| 25/25 [00:00<00:00, 3907.49 examples/s]INFO 2025-05-20 10:59:03,545 instructlab.sdg.eval_data:126: Saving MMLU Dataset /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/mmlubench_knowledge_technology_cloud-computing_6_4_1__Monitoring_OpenShift_Virtualization_upgrade_status.jsonl


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 681.67ba/s]INFO 2025-05-20 10:59:03,547 instructlab.sdg.eval_data:130: Saving MMLU Task yaml /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_6_4_1__Monitoring_OpenShift_Virtualization_upgrade_status_task.yaml


Map:   0%|          | 0/623 [00:00<?, ? examples/s]
Map: 100%|##########| 623/623 [00:00<00:00, 4743.15 examples/s]
Map: 100%|##########| 623/623 [00:00<00:00, 4702.53 examples/s]

Map:   0%|          | 0/623 [00:00<?, ? examples/s]
Map: 100%|##########| 623/623 [00:00<00:00, 27720.86 examples/s]

Filter:   0%|          | 0/623 [00:00<?, ? examples/s]
Filter: 100%|##########| 623/623 [00:00<00:00, 42443.78 examples/s]

Map:   0%|          | 0/26 [00:00<?, ? examples/s]
Map: 100%|##########| 26/26 [00:00<00:00, 5940.62 examples/s]

Map:   0%|          | 0/26 [00:00<?, ? examples/s]
Map: 100%|##########| 26/26 [00:00<00:00, 8645.99 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 46.88ba/s]

Map:   0%|          | 0/623 [00:00<?, ? examples/s]
Map: 100%|##########| 623/623 [00:00<00:00, 7036.87 examples/s]

Map:   0%|          | 0/623 [00:00<?, ? examples/s]
Map: 100%|##########| 623/623 [00:00<00:00, 7250.78 examples/s]

Map:   0%|          | 0/623 [00:00<?, ? examples/s]
Map: 100%|##########| 623/623 [00:00<00:00, 7226.56 examples/s]

Map:   0%|          | 0/623 [00:00<?, ? examples/s]
Map: 100%|##########| 623/623 [00:00<00:00, 29063.29 examples/s]

Filter:   0%|          | 0/623 [00:00<?, ? examples/s]
Filter: 100%|##########| 623/623 [00:00<00:00, 43678.98 examples/s]

Map:   0%|          | 0/26 [00:00<?, ? examples/s]
Map: 100%|##########| 26/26 [00:00<00:00, 5935.12 examples/s]INFO 2025-05-20 11:00:08,451 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.32%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 24.15ba/s]

Map:   0%|          | 0/2175 [00:00<?, ? examples/s]
Map:  36%|###5      | 782/2175 [00:00<00:00, 7765.39 examples/s]
Map:  87%|########6 | 1890/2175 [00:00<00:00, 7504.67 examples/s]
Map: 100%|##########| 2175/2175 [00:00<00:00, 7230.30 examples/s]

Map:   0%|          | 0/2175 [00:00<?, ? examples/s]
Map: 100%|##########| 2175/2175 [00:00<00:00, 29225.93 examples/s]

Filter:   0%|          | 0/2175 [00:00<?, ? examples/s]
Filter: 100%|##########| 2175/2175 [00:00<00:00, 46663.69 examples/s]

Map:   0%|          | 0/166 [00:00<?, ? examples/s]
Map: 100%|##########| 166/166 [00:00<00:00, 10588.78 examples/s]

Map:   0%|          | 0/166 [00:00<?, ? examples/s]
Map: 100%|##########| 166/166 [00:00<00:00, 22838.50 examples/s]

Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 43.21ba/s]

Map:   0%|          | 0/2175 [00:00<?, ? examples/s]
Map:  36%|###6      | 791/2175 [00:00<00:00, 7844.94 examples/s]
Map:  88%|########8 | 1914/2175 [00:00<00:00, 7595.32 examples/s]
Map: 100%|##########| 2175/2175 [00:00<00:00, 7294.74 examples/s]

Map:   0%|          | 0/2175 [00:00<?, ? examples/s]
Map:  40%|###9      | 861/2175 [00:00<00:00, 8562.02 examples/s]
Map:  88%|########8 | 1916/2175 [00:00<00:00, 7518.94 examples/s]
Map: 100%|##########| 2175/2175 [00:00<00:00, 6917.38 examples/s]

Map:   0%|          | 0/2175 [00:00<?, ? examples/s]
Map:  36%|###6      | 785/2175 [00:00<00:00, 7791.49 examples/s]
Map:  88%|########7 | 1910/2175 [00:00<00:00, 7587.60 examples/s]
Map: 100%|##########| 2175/2175 [00:00<00:00, 7249.19 examples/s]

Map:   0%|          | 0/2175 [00:00<?, ? examples/s]
Map: 100%|##########| 2175/2175 [00:00<00:00, 29625.99 examples/s]

Filter:   0%|          | 0/2175 [00:00<?, ? examples/s]
Filter: 100%|##########| 2175/2175 [00:00<00:00, 47280.40 examples/s]

Map:   0%|          | 0/166 [00:00<?, ? examples/s]
Map: 100%|##########| 166/166 [00:00<00:00, 10604.91 examples/s]INFO 2025-05-20 11:00:10,304 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.15%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/5 [00:00<?, ?ba/s]
Creating json from Arrow format:  40%|####      | 2/5 [00:00<00:00, 17.22ba/s]
Creating json from Arrow format: 100%|##########| 5/5 [00:00<00:00, 25.46ba/s]

Map:   0%|          | 0/95 [00:00<?, ? examples/s]
Map: 100%|##########| 95/95 [00:00<00:00, 6138.64 examples/s]

Map:   0%|          | 0/95 [00:00<?, ? examples/s]
Map: 100%|##########| 95/95 [00:00<00:00, 18336.81 examples/s]

Filter:   0%|          | 0/95 [00:00<?, ? examples/s]
Filter: 100%|##########| 95/95 [00:00<00:00, 25978.54 examples/s]

Map:   0%|          | 0/26 [00:00<?, ? examples/s]
Map: 100%|##########| 26/26 [00:00<00:00, 6175.43 examples/s]

Map:   0%|          | 0/26 [00:00<?, ? examples/s]
Map: 100%|##########| 26/26 [00:00<00:00, 8961.45 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 379.85ba/s]

Map:   0%|          | 0/95 [00:00<?, ? examples/s]
Map: 100%|##########| 95/95 [00:00<00:00, 6167.04 examples/s]

Map:   0%|          | 0/95 [00:00<?, ? examples/s]
Map: 100%|##########| 95/95 [00:00<00:00, 6897.21 examples/s]

Map:   0%|          | 0/95 [00:00<?, ? examples/s]
Map: 100%|##########| 95/95 [00:00<00:00, 6308.92 examples/s]

Map:   0%|          | 0/95 [00:00<?, ? examples/s]
Map: 100%|##########| 95/95 [00:00<00:00, 18660.56 examples/s]

Filter:   0%|          | 0/95 [00:00<?, ? examples/s]
Filter: 100%|##########| 95/95 [00:00<00:00, 26365.31 examples/s]

Map:   0%|          | 0/26 [00:00<?, ? examples/s]
Map: 100%|##########| 26/26 [00:00<00:00, 6296.30 examples/s]INFO 2025-05-20 11:00:10,662 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.05%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 181.89ba/s]

Map:   0%|          | 0/260 [00:00<?, ? examples/s]
Map: 100%|##########| 260/260 [00:00<00:00, 6683.66 examples/s]

Map:   0%|          | 0/260 [00:00<?, ? examples/s]
Map: 100%|##########| 260/260 [00:00<00:00, 24487.89 examples/s]

Filter:   0%|          | 0/260 [00:00<?, ? examples/s]
Filter: 100%|##########| 260/260 [00:00<00:00, 36524.74 examples/s]

Map:   0%|          | 0/14 [00:00<?, ? examples/s]
Map: 100%|##########| 14/14 [00:00<00:00, 4096.86 examples/s]

Map:   0%|          | 0/14 [00:00<?, ? examples/s]
Map: 100%|##########| 14/14 [00:00<00:00, 5549.07 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 117.59ba/s]

Map:   0%|          | 0/260 [00:00<?, ? examples/s]
Map: 100%|##########| 260/260 [00:00<00:00, 6650.07 examples/s]

Map:   0%|          | 0/260 [00:00<?, ? examples/s]
Map: 100%|##########| 260/260 [00:00<00:00, 7045.15 examples/s]

Map:   0%|          | 0/260 [00:00<?, ? examples/s]
Map: 100%|##########| 260/260 [00:00<00:00, 6715.35 examples/s]

Map:   0%|          | 0/260 [00:00<?, ? examples/s]
Map: 100%|##########| 260/260 [00:00<00:00, 25202.66 examples/s]

Filter:   0%|          | 0/260 [00:00<?, ? examples/s]
Filter: 100%|##########| 260/260 [00:00<00:00, 37117.73 examples/s]

Map:   0%|          | 0/14 [00:00<?, ? examples/s]
Map: 100%|##########| 14/14 [00:00<00:00, 4125.93 examples/s]INFO 2025-05-20 11:00:10,950 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.14%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 44.64ba/s]

Map:   0%|          | 0/692 [00:00<?, ? examples/s]
Map: 100%|##########| 692/692 [00:00<00:00, 7301.73 examples/s]

Map:   0%|          | 0/692 [00:00<?, ? examples/s]
Map: 100%|##########| 692/692 [00:00<00:00, 28921.62 examples/s]

Filter:   0%|          | 0/692 [00:00<?, ? examples/s]
Filter: 100%|##########| 692/692 [00:00<00:00, 45169.53 examples/s]

Map:   0%|          | 0/47 [00:00<?, ? examples/s]
Map: 100%|##########| 47/47 [00:00<00:00, 7874.27 examples/s]

Map:   0%|          | 0/47 [00:00<?, ? examples/s]
Map: 100%|##########| 47/47 [00:00<00:00, 12663.47 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 54.27ba/s]

Map:   0%|          | 0/692 [00:00<?, ? examples/s]
Map: 100%|##########| 692/692 [00:00<00:00, 7320.14 examples/s]

Map:   0%|          | 0/692 [00:00<?, ? examples/s]
Map: 100%|##########| 692/692 [00:00<00:00, 7658.52 examples/s]

Map:   0%|          | 0/692 [00:00<?, ? examples/s]
Map: 100%|##########| 692/692 [00:00<00:00, 7293.32 examples/s]

Map:   0%|          | 0/692 [00:00<?, ? examples/s]
Map: 100%|##########| 692/692 [00:00<00:00, 29253.93 examples/s]

Filter:   0%|          | 0/692 [00:00<?, ? examples/s]
Filter: 100%|##########| 692/692 [00:00<00:00, 44898.42 examples/s]

Map:   0%|          | 0/47 [00:00<?, ? examples/s]
Map: 100%|##########| 47/47 [00:00<00:00, 7895.08 examples/s]INFO 2025-05-20 11:00:11,564 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.36%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 33.12ba/s]

Map:   0%|          | 0/9459 [00:00<?, ? examples/s]
Map:   8%|8         | 782/9459 [00:00<00:01, 7759.75 examples/s]
Map:  20%|##        | 1904/9459 [00:00<00:00, 7563.37 examples/s]
Map:  32%|###1      | 3000/9459 [00:00<00:00, 7209.39 examples/s]
Map:  40%|####      | 3805/9459 [00:00<00:00, 7471.12 examples/s]
Map:  52%|#####1    | 4912/9459 [00:00<00:00, 7430.86 examples/s]
Map:  63%|######3   | 6000/9459 [00:00<00:00, 7294.87 examples/s]
Map:  72%|#######1  | 6780/9459 [00:00<00:00, 7421.50 examples/s]
Map:  83%|########3 | 7895/9459 [00:01<00:00, 7423.87 examples/s]
Map:  95%|#########5| 9000/9459 [00:01<00:00, 7337.50 examples/s]
Map: 100%|##########| 9459/9459 [00:01<00:00, 7365.32 examples/s]

Map:   0%|          | 0/9459 [00:00<?, ? examples/s]
Map:  32%|###1      | 3000/9459 [00:00<00:00, 29204.58 examples/s]
Map:  64%|######3   | 6020/9459 [00:00<00:00, 29779.19 examples/s]
Map:  98%|#########7| 9248/9459 [00:00<00:00, 30910.27 examples/s]
Map: 100%|##########| 9459/9459 [00:00<00:00, 30187.80 examples/s]

Filter:   0%|          | 0/9459 [00:00<?, ? examples/s]
Filter:  63%|######3   | 6000/9459 [00:00<00:00, 49985.45 examples/s]
Filter: 100%|##########| 9459/9459 [00:00<00:00, 50189.15 examples/s]

Map:   0%|          | 0/847 [00:00<?, ? examples/s]
Map: 100%|##########| 847/847 [00:00<00:00, 11916.48 examples/s]

Map:   0%|          | 0/847 [00:00<?, ? examples/s]
Map: 100%|##########| 847/847 [00:00<00:00, 28863.49 examples/s]

Creating json from Arrow format:   0%|          | 0/11 [00:00<?, ?ba/s]
Creating json from Arrow format:  36%|###6      | 4/11 [00:00<00:00, 38.29ba/s]
Creating json from Arrow format:  73%|#######2  | 8/11 [00:00<00:00, 39.09ba/s]
Creating json from Arrow format: 100%|##########| 11/11 [00:00<00:00, 40.65ba/s]

Map:   0%|          | 0/9459 [00:00<?, ? examples/s]
Map:   8%|8         | 785/9459 [00:00<00:01, 7782.36 examples/s]
Map:  20%|##        | 1910/9459 [00:00<00:00, 7584.23 examples/s]
Map:  32%|###1      | 3000/9459 [00:00<00:00, 7207.83 examples/s]
Map:  40%|####      | 3805/9459 [00:00<00:00, 7469.00 examples/s]
Map:  52%|#####2    | 4938/9459 [00:00<00:00, 7499.06 examples/s]
Map:  63%|######3   | 6000/9459 [00:00<00:00, 7323.87 examples/s]
Map:  72%|#######1  | 6789/9459 [00:00<00:00, 7467.98 examples/s]
Map:  84%|########3 | 7914/9459 [00:01<00:00, 7476.65 examples/s]
Map:  95%|#########5| 9000/9459 [00:01<00:00, 7377.57 examples/s]
Map: 100%|##########| 9459/9459 [00:01<00:00, 7401.69 examples/s]

Map:   0%|          | 0/9459 [00:00<?, ? examples/s]
Map:   7%|7         | 685/9459 [00:00<00:01, 6796.47 examples/s]
Map:  18%|#7        | 1689/9459 [00:00<00:01, 6217.35 examples/s]
Map:  25%|##4       | 2352/9459 [00:00<00:01, 5946.24 examples/s]
Map:  32%|###1      | 3000/9459 [00:00<00:01, 5776.95 examples/s]
Map:  39%|###9      | 3705/9459 [00:00<00:00, 6167.06 examples/s]
Map:  46%|####5     | 4339/9459 [00:00<00:00, 6043.44 examples/s]
Map:  53%|#####2    | 5000/9459 [00:00<00:00, 5955.05 examples/s]
Map:  60%|######    | 5682/9459 [00:00<00:00, 6203.89 examples/s]
Map:  67%|######6   | 6326/9459 [00:01<00:00, 5788.93 examples/s]
Map:  74%|#######3  | 6984/9459 [00:01<00:00, 6006.25 examples/s]
Map:  83%|########2 | 7843/9459 [00:01<00:00, 5896.42 examples/s]
Map:  92%|#########1| 8680/9459 [00:01<00:00, 5784.06 examples/s]
Map:  99%|#########8| 9321/9459 [00:01<00:00, 5693.14 examples/s]
Map: 100%|##########| 9459/9459 [00:01<00:00, 5879.37 examples/s]

Map:   0%|          | 0/9459 [00:00<?, ? examples/s]
Map:   8%|8         | 783/9459 [00:00<00:01, 7768.94 examples/s]
Map:  20%|##        | 1893/9459 [00:00<00:01, 7507.33 examples/s]
Map:  31%|###1      | 2970/9459 [00:00<00:00, 7342.03 examples/s]
Map:  42%|####2     | 4008/9459 [00:00<00:00, 7161.31 examples/s]
Map:  51%|#####     | 4796/9459 [00:00<00:00, 7363.07 examples/s]
Map:  63%|######2   | 5937/9459 [00:00<00:00, 7450.87 examples/s]
Map:  74%|#######4  | 7000/9459 [00:00<00:00, 7246.48 examples/s]
Map:  82%|########2 | 7778/9459 [00:01<00:00, 7376.79 examples/s]
Map:  94%|#########4| 8901/9459 [00:01<00:00, 7411.23 examples/s]
Map: 100%|##########| 9459/9459 [00:01<00:00, 7321.83 examples/s]

Map:   0%|          | 0/9459 [00:00<?, ? examples/s]
Map:  21%|##1       | 2000/9459 [00:00<00:01, 6027.34 examples/s]
Map:  53%|#####2    | 5000/9459 [00:00<00:00, 13194.17 examples/s]
Map:  85%|########4 | 8008/9459 [00:00<00:00, 18258.21 examples/s]
Map: 100%|##########| 9459/9459 [00:00<00:00, 16281.88 examples/s]

Filter:   0%|          | 0/9459 [00:00<?, ? examples/s]
Filter:  63%|######3   | 6000/9459 [00:00<00:00, 48795.09 examples/s]
Filter: 100%|##########| 9459/9459 [00:00<00:00, 48936.77 examples/s]

Map:   0%|          | 0/847 [00:00<?, ? examples/s]
Map: 100%|##########| 847/847 [00:00<00:00, 11897.32 examples/s]

Creating json from Arrow format:   0%|          | 0/20 [00:00<?, ?ba/s]
Creating json from Arrow format:  10%|#         | 2/20 [00:00<00:00, 18.53ba/s]
Creating json from Arrow format:  20%|##        | 4/20 [00:00<00:00, 18.85ba/s]
Creating json from Arrow format:  30%|###       | 6/20 [00:00<00:00, 19.14ba/s]
Creating json from Arrow format:  40%|####      | 8/20 [00:00<00:00, 18.69ba/s]
Creating json from Arrow format:  55%|#####5    | 11/20 [00:00<00:00, 21.07ba/s]
Creating json from Arrow format:  80%|########  | 16/20 [00:00<00:00, 28.98ba/s]
Creating json from Arrow format: 100%|##########| 20/20 [00:00<00:00, 26.46ba/s]

Map:   0%|          | 0/272 [00:00<?, ? examples/s]
Map: 100%|##########| 272/272 [00:00<00:00, 7226.06 examples/s]

Map:   0%|          | 0/272 [00:00<?, ? examples/s]
Map: 100%|##########| 272/272 [00:00<00:00, 24190.55 examples/s]

Filter:   0%|          | 0/272 [00:00<?, ? examples/s]
Filter: 100%|##########| 272/272 [00:00<00:00, 38273.31 examples/s]

Map:   0%|          | 0/38 [00:00<?, ? examples/s]
Map: 100%|##########| 38/38 [00:00<00:00, 7502.87 examples/s]

Map:   0%|          | 0/38 [00:00<?, ? examples/s]
Map: 100%|##########| 38/38 [00:00<00:00, 11346.45 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 210.94ba/s]

Map:   0%|          | 0/272 [00:00<?, ? examples/s]
Map: 100%|##########| 272/272 [00:00<00:00, 7326.20 examples/s]

Map:   0%|          | 0/272 [00:00<?, ? examples/s]
Map: 100%|##########| 272/272 [00:00<00:00, 8286.37 examples/s]

Map:   0%|          | 0/272 [00:00<?, ? examples/s]
Map: 100%|##########| 272/272 [00:00<00:00, 7364.98 examples/s]

Map:   0%|          | 0/272 [00:00<?, ? examples/s]
Map: 100%|##########| 272/272 [00:00<00:00, 26014.15 examples/s]

Filter:   0%|          | 0/272 [00:00<?, ? examples/s]
Filter: 100%|##########| 272/272 [00:00<00:00, 39688.67 examples/s]

Map:   0%|          | 0/38 [00:00<?, ? examples/s]
Map: 100%|##########| 38/38 [00:00<00:00, 7584.64 examples/s]INFO 2025-05-20 11:00:20,669 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.15%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 88.31ba/s]

Map:   0%|          | 0/1310 [00:00<?, ? examples/s]
Map:  61%|######    | 794/1310 [00:00<00:00, 7876.22 examples/s]
Map: 100%|##########| 1310/1310 [00:00<00:00, 7282.73 examples/s]

Map:   0%|          | 0/1310 [00:00<?, ? examples/s]
Map: 100%|##########| 1310/1310 [00:00<00:00, 29163.60 examples/s]

Filter:   0%|          | 0/1310 [00:00<?, ? examples/s]
Filter: 100%|##########| 1310/1310 [00:00<00:00, 47725.89 examples/s]

Map:   0%|          | 0/50 [00:00<?, ? examples/s]
Map: 100%|##########| 50/50 [00:00<00:00, 7838.06 examples/s]

Map:   0%|          | 0/50 [00:00<?, ? examples/s]
Map: 100%|##########| 50/50 [00:00<00:00, 13260.52 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 59.53ba/s]

Map:   0%|          | 0/1310 [00:00<?, ? examples/s]
Map:  60%|######    | 789/1310 [00:00<00:00, 7831.92 examples/s]
Map: 100%|##########| 1310/1310 [00:00<00:00, 7261.36 examples/s]

Map:   0%|          | 0/1310 [00:00<?, ? examples/s]
Map:  70%|######9   | 914/1310 [00:00<00:00, 9094.47 examples/s]
Map: 100%|##########| 1310/1310 [00:00<00:00, 7358.93 examples/s]

Map:   0%|          | 0/1310 [00:00<?, ? examples/s]
Map:  60%|######    | 790/1310 [00:00<00:00, 7842.12 examples/s]
Map: 100%|##########| 1310/1310 [00:00<00:00, 7303.57 examples/s]

Map:   0%|          | 0/1310 [00:00<?, ? examples/s]
Map: 100%|##########| 1310/1310 [00:00<00:00, 29192.57 examples/s]

Filter:   0%|          | 0/1310 [00:00<?, ? examples/s]
Filter: 100%|##########| 1310/1310 [00:00<00:00, 46826.59 examples/s]

Map:   0%|          | 0/50 [00:00<?, ? examples/s]
Map: 100%|##########| 50/50 [00:00<00:00, 8113.72 examples/s]INFO 2025-05-20 11:00:21,729 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.68%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format:  67%|######6   | 2/3 [00:00<00:00, 19.51ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 25.30ba/s]

Map:   0%|          | 0/221 [00:00<?, ? examples/s]
Map: 100%|##########| 221/221 [00:00<00:00, 6857.04 examples/s]

Map:   0%|          | 0/221 [00:00<?, ? examples/s]
Map: 100%|##########| 221/221 [00:00<00:00, 23654.30 examples/s]

Filter:   0%|          | 0/221 [00:00<?, ? examples/s]
Filter: 100%|##########| 221/221 [00:00<00:00, 35255.64 examples/s]

Map:   0%|          | 0/34 [00:00<?, ? examples/s]
Map: 100%|##########| 34/34 [00:00<00:00, 6972.39 examples/s]

Map:   0%|          | 0/34 [00:00<?, ? examples/s]
Map: 100%|##########| 34/34 [00:00<00:00, 10609.80 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 168.05ba/s]

Map:   0%|          | 0/221 [00:00<?, ? examples/s]
Map: 100%|##########| 221/221 [00:00<00:00, 6942.24 examples/s]

Map:   0%|          | 0/221 [00:00<?, ? examples/s]
Map: 100%|##########| 221/221 [00:00<00:00, 7517.95 examples/s]

Map:   0%|          | 0/221 [00:00<?, ? examples/s]
Map: 100%|##########| 221/221 [00:00<00:00, 7040.79 examples/s]

Map:   0%|          | 0/221 [00:00<?, ? examples/s]
Map: 100%|##########| 221/221 [00:00<00:00, 24951.98 examples/s]

Filter:   0%|          | 0/221 [00:00<?, ? examples/s]
Filter: 100%|##########| 221/221 [00:00<00:00, 36661.18 examples/s]

Map:   0%|          | 0/34 [00:00<?, ? examples/s]
Map: 100%|##########| 34/34 [00:00<00:00, 6995.99 examples/s]INFO 2025-05-20 11:00:22,099 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.12%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 65.71ba/s]

Map:   0%|          | 0/2501 [00:00<?, ? examples/s]
Map:  31%|###1      | 786/2501 [00:00<00:00, 7797.71 examples/s]
Map:  76%|#######6  | 1913/2501 [00:00<00:00, 7596.35 examples/s]
Map: 100%|##########| 2501/2501 [00:00<00:00, 7265.49 examples/s]

Map:   0%|          | 0/2501 [00:00<?, ? examples/s]
Map: 100%|##########| 2501/2501 [00:00<00:00, 28708.93 examples/s]

Filter:   0%|          | 0/2501 [00:00<?, ? examples/s]
Filter: 100%|##########| 2501/2501 [00:00<00:00, 48295.62 examples/s]

Map:   0%|          | 0/151 [00:00<?, ? examples/s]
Map: 100%|##########| 151/151 [00:00<00:00, 10414.89 examples/s]

Map:   0%|          | 0/151 [00:00<?, ? examples/s]
Map: 100%|##########| 151/151 [00:00<00:00, 21523.14 examples/s]

Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 44.56ba/s]

Map:   0%|          | 0/2501 [00:00<?, ? examples/s]
Map:  32%|###1      | 790/2501 [00:00<00:00, 7833.52 examples/s]
Map:  76%|#######6  | 1913/2501 [00:00<00:00, 7588.44 examples/s]
Map: 100%|##########| 2501/2501 [00:00<00:00, 7200.91 examples/s]

Map:   0%|          | 0/2501 [00:00<?, ? examples/s]
Map:  34%|###4      | 854/2501 [00:00<00:00, 8485.65 examples/s]
Map:  76%|#######6  | 1910/2501 [00:00<00:00, 7494.26 examples/s]
Map: 100%|##########| 2501/2501 [00:00<00:00, 7121.30 examples/s]

Map:   0%|          | 0/2501 [00:00<?, ? examples/s]
Map:  32%|###1      | 790/2501 [00:00<00:00, 7831.67 examples/s]
Map:  77%|#######7  | 1926/2501 [00:00<00:00, 7652.67 examples/s]
Map: 100%|##########| 2501/2501 [00:00<00:00, 7303.03 examples/s]

Map:   0%|          | 0/2501 [00:00<?, ? examples/s]
Map: 100%|##########| 2501/2501 [00:00<00:00, 28902.64 examples/s]

Filter:   0%|          | 0/2501 [00:00<?, ? examples/s]
Filter: 100%|##########| 2501/2501 [00:00<00:00, 47670.13 examples/s]

Map:   0%|          | 0/151 [00:00<?, ? examples/s]
Map: 100%|##########| 151/151 [00:00<00:00, 10342.95 examples/s]INFO 2025-05-20 11:00:24,110 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.31%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/6 [00:00<?, ?ba/s]
Creating json from Arrow format:  33%|###3      | 2/6 [00:00<00:00, 16.41ba/s]
Creating json from Arrow format: 100%|##########| 6/6 [00:00<00:00, 28.70ba/s]
Creating json from Arrow format: 100%|##########| 6/6 [00:00<00:00, 26.68ba/s]

Map:   0%|          | 0/222 [00:00<?, ? examples/s]
Map: 100%|##########| 222/222 [00:00<00:00, 6207.78 examples/s]

Map:   0%|          | 0/222 [00:00<?, ? examples/s]
Map: 100%|##########| 222/222 [00:00<00:00, 23527.19 examples/s]

Filter:   0%|          | 0/222 [00:00<?, ? examples/s]
Filter: 100%|##########| 222/222 [00:00<00:00, 33923.62 examples/s]

Map:   0%|          | 0/50 [00:00<?, ? examples/s]
Map: 100%|##########| 50/50 [00:00<00:00, 7939.55 examples/s]

Map:   0%|          | 0/50 [00:00<?, ? examples/s]
Map: 100%|##########| 50/50 [00:00<00:00, 13661.34 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 73.48ba/s]

Map:   0%|          | 0/222 [00:00<?, ? examples/s]
Map: 100%|##########| 222/222 [00:00<00:00, 6270.78 examples/s]

Map:   0%|          | 0/222 [00:00<?, ? examples/s]
Map: 100%|##########| 222/222 [00:00<00:00, 6507.89 examples/s]

Map:   0%|          | 0/222 [00:00<?, ? examples/s]
Map: 100%|##########| 222/222 [00:00<00:00, 6290.44 examples/s]

Map:   0%|          | 0/222 [00:00<?, ? examples/s]
Map: 100%|##########| 222/222 [00:00<00:00, 23861.81 examples/s]

Filter:   0%|          | 0/222 [00:00<?, ? examples/s]
Filter: 100%|##########| 222/222 [00:00<00:00, 34246.77 examples/s]

Map:   0%|          | 0/50 [00:00<?, ? examples/s]
Map: 100%|##########| 50/50 [00:00<00:00, 8138.27 examples/s]INFO 2025-05-20 11:00:24,621 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.13%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 24.41ba/s]

Map:   0%|          | 0/986 [00:00<?, ? examples/s]
Map:  81%|########1 | 799/986 [00:00<00:00, 7929.75 examples/s]
Map: 100%|##########| 986/986 [00:00<00:00, 7439.93 examples/s]

Map:   0%|          | 0/986 [00:00<?, ? examples/s]
Map: 100%|##########| 986/986 [00:00<00:00, 29265.62 examples/s]

Filter:   0%|          | 0/986 [00:00<?, ? examples/s]
Filter: 100%|##########| 986/986 [00:00<00:00, 45893.82 examples/s]

Map:   0%|          | 0/48 [00:00<?, ? examples/s]
Map: 100%|##########| 48/48 [00:00<00:00, 7771.73 examples/s]

Map:   0%|          | 0/48 [00:00<?, ? examples/s]
Map: 100%|##########| 48/48 [00:00<00:00, 13085.05 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 79.96ba/s]

Map:   0%|          | 0/986 [00:00<?, ? examples/s]
Map:  81%|########  | 797/986 [00:00<00:00, 7909.19 examples/s]
Map: 100%|##########| 986/986 [00:00<00:00, 7360.64 examples/s]

Map:   0%|          | 0/986 [00:00<?, ? examples/s]
Map:  92%|#########1| 907/986 [00:00<00:00, 9021.10 examples/s]
Map: 100%|##########| 986/986 [00:00<00:00, 7524.77 examples/s]

Map:   0%|          | 0/986 [00:00<?, ? examples/s]
Map:  81%|########1 | 800/986 [00:00<00:00, 7938.59 examples/s]
Map: 100%|##########| 986/986 [00:00<00:00, 7465.80 examples/s]

Map:   0%|          | 0/986 [00:00<?, ? examples/s]
Map: 100%|##########| 986/986 [00:00<00:00, 29784.33 examples/s]

Filter:   0%|          | 0/986 [00:00<?, ? examples/s]
Filter: 100%|##########| 986/986 [00:00<00:00, 46449.49 examples/s]

Map:   0%|          | 0/48 [00:00<?, ? examples/s]
Map: 100%|##########| 48/48 [00:00<00:00, 7801.54 examples/s]INFO 2025-05-20 11:00:25,457 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.51%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 35.06ba/s]

Map:   0%|          | 0/1407 [00:00<?, ? examples/s]
Map:  57%|#####7    | 805/1407 [00:00<00:00, 7991.85 examples/s]
Map: 100%|##########| 1407/1407 [00:00<00:00, 7537.86 examples/s]

Map:   0%|          | 0/1407 [00:00<?, ? examples/s]
Map: 100%|##########| 1407/1407 [00:00<00:00, 30025.77 examples/s]

Filter:   0%|          | 0/1407 [00:00<?, ? examples/s]
Filter: 100%|##########| 1407/1407 [00:00<00:00, 46294.09 examples/s]

Map:   0%|          | 0/73 [00:00<?, ? examples/s]
Map: 100%|##########| 73/73 [00:00<00:00, 8970.06 examples/s]

Map:   0%|          | 0/73 [00:00<?, ? examples/s]
Map: 100%|##########| 73/73 [00:00<00:00, 15816.12 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 68.38ba/s]

Map:   0%|          | 0/1407 [00:00<?, ? examples/s]
Map:  57%|#####7    | 802/1407 [00:00<00:00, 7965.60 examples/s]
Map: 100%|##########| 1407/1407 [00:00<00:00, 7489.59 examples/s]

Map:   0%|          | 0/1407 [00:00<?, ? examples/s]
Map:  66%|######5   | 922/1407 [00:00<00:00, 9168.42 examples/s]
Map: 100%|##########| 1407/1407 [00:00<00:00, 8039.14 examples/s]

Map:   0%|          | 0/1407 [00:00<?, ? examples/s]
Map:  57%|#####7    | 805/1407 [00:00<00:00, 7994.41 examples/s]
Map: 100%|##########| 1407/1407 [00:00<00:00, 7565.53 examples/s]

Map:   0%|          | 0/1407 [00:00<?, ? examples/s]
Map: 100%|##########| 1407/1407 [00:00<00:00, 30431.12 examples/s]

Filter:   0%|          | 0/1407 [00:00<?, ? examples/s]
Filter: 100%|##########| 1407/1407 [00:00<00:00, 46452.97 examples/s]

Map:   0%|          | 0/73 [00:00<?, ? examples/s]
Map: 100%|##########| 73/73 [00:00<00:00, 9225.75 examples/s]INFO 2025-05-20 11:00:26,626 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.73%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 36.15ba/s]

Map:   0%|          | 0/138 [00:00<?, ? examples/s]
Map: 100%|##########| 138/138 [00:00<00:00, 6471.24 examples/s]

Map:   0%|          | 0/138 [00:00<?, ? examples/s]
Map: 100%|##########| 138/138 [00:00<00:00, 20649.80 examples/s]

Filter:   0%|          | 0/138 [00:00<?, ? examples/s]
Filter: 100%|##########| 138/138 [00:00<00:00, 29637.17 examples/s]

Map:   0%|          | 0/12 [00:00<?, ? examples/s]
Map: 100%|##########| 12/12 [00:00<00:00, 3799.76 examples/s]

Map:   0%|          | 0/12 [00:00<?, ? examples/s]
Map: 100%|##########| 12/12 [00:00<00:00, 4852.65 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 262.72ba/s]

Map:   0%|          | 0/138 [00:00<?, ? examples/s]
Map: 100%|##########| 138/138 [00:00<00:00, 6423.20 examples/s]

Map:   0%|          | 0/138 [00:00<?, ? examples/s]
Map: 100%|##########| 138/138 [00:00<00:00, 7293.89 examples/s]

Map:   0%|          | 0/138 [00:00<?, ? examples/s]
Map: 100%|##########| 138/138 [00:00<00:00, 6588.74 examples/s]

Map:   0%|          | 0/138 [00:00<?, ? examples/s]
Map: 100%|##########| 138/138 [00:00<00:00, 21103.03 examples/s]

Filter:   0%|          | 0/138 [00:00<?, ? examples/s]
Filter: 100%|##########| 138/138 [00:00<00:00, 30396.70 examples/s]

Map:   0%|          | 0/12 [00:00<?, ? examples/s]
Map: 100%|##########| 12/12 [00:00<00:00, 3909.86 examples/s]INFO 2025-05-20 11:00:26,897 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.07%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 101.62ba/s]

Map:   0%|          | 0/245 [00:00<?, ? examples/s]
Map: 100%|##########| 245/245 [00:00<00:00, 7085.12 examples/s]

Map:   0%|          | 0/245 [00:00<?, ? examples/s]
Map: 100%|##########| 245/245 [00:00<00:00, 25237.11 examples/s]

Filter:   0%|          | 0/245 [00:00<?, ? examples/s]
Filter: 100%|##########| 245/245 [00:00<00:00, 36941.60 examples/s]

Map:   0%|          | 0/6 [00:00<?, ? examples/s]
Map: 100%|##########| 6/6 [00:00<00:00, 2306.04 examples/s]

Map:   0%|          | 0/6 [00:00<?, ? examples/s]
Map: 100%|##########| 6/6 [00:00<00:00, 2601.66 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 233.90ba/s]

Map:   0%|          | 0/245 [00:00<?, ? examples/s]
Map: 100%|##########| 245/245 [00:00<00:00, 7212.37 examples/s]

Map:   0%|          | 0/245 [00:00<?, ? examples/s]
Map: 100%|##########| 245/245 [00:00<00:00, 8128.37 examples/s]

Map:   0%|          | 0/245 [00:00<?, ? examples/s]
Map: 100%|##########| 245/245 [00:00<00:00, 7179.27 examples/s]

Map:   0%|          | 0/245 [00:00<?, ? examples/s]
Map: 100%|##########| 245/245 [00:00<00:00, 25188.85 examples/s]

Filter:   0%|          | 0/245 [00:00<?, ? examples/s]
Filter: 100%|##########| 245/245 [00:00<00:00, 37682.60 examples/s]

Map:   0%|          | 0/6 [00:00<?, ? examples/s]
Map: 100%|##########| 6/6 [00:00<00:00, 2286.76 examples/s]INFO 2025-05-20 11:00:27,155 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.13%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 98.95ba/s]

Map:   0%|          | 0/15052 [00:00<?, ? examples/s]
Map:   5%|5         | 779/15052 [00:00<00:01, 7727.84 examples/s]
Map:  13%|#2        | 1896/15052 [00:00<00:01, 7530.86 examples/s]
Map:  20%|#9        | 2997/15052 [00:00<00:01, 7430.11 examples/s]
Map:  27%|##6       | 4029/15052 [00:00<00:01, 7199.48 examples/s]
Map:  32%|###1      | 4815/15052 [00:00<00:01, 7382.37 examples/s]
Map:  39%|###9      | 5892/15052 [00:00<00:01, 7302.05 examples/s]
Map:  47%|####6     | 7000/15052 [00:00<00:01, 7257.88 examples/s]
Map:  52%|#####1    | 7795/15052 [00:01<00:00, 7425.72 examples/s]
Map:  59%|#####9    | 8911/15052 [00:01<00:00, 7427.37 examples/s]
Map:  66%|######6   | 10000/15052 [00:01<00:00, 7111.47 examples/s]
Map:  72%|#######1  | 10778/15052 [00:01<00:00, 7265.42 examples/s]
Map:  79%|#######9  | 11903/15052 [00:01<00:00, 7340.15 examples/s]
Map:  84%|########4 | 12648/15052 [00:01<00:00, 7365.07 examples/s]
Map:  89%|########9 | 13403/15052 [00:01<00:00, 7389.61 examples/s]
Map:  94%|#########4| 14149/15052 [00:01<00:00, 7406.80 examples/s]
Map:  99%|#########9| 14941/15052 [00:02<00:00, 7546.18 examples/s]
Map: 100%|##########| 15052/15052 [00:02<00:00, 7346.29 examples/s]

Map:   0%|          | 0/15052 [00:00<?, ? examples/s]
Map:  20%|#9        | 3000/15052 [00:00<00:00, 28575.32 examples/s]
Map:  40%|###9      | 6000/15052 [00:00<00:00, 28985.68 examples/s]
Map:  61%|######    | 9150/15052 [00:00<00:00, 30111.97 examples/s]
Map:  92%|#########1| 13801/15052 [00:00<00:00, 30154.25 examples/s]
Map: 100%|##########| 15052/15052 [00:00<00:00, 29545.23 examples/s]

Filter:   0%|          | 0/15052 [00:00<?, ? examples/s]
Filter:  40%|###9      | 6000/15052 [00:00<00:00, 50075.66 examples/s]
Filter:  80%|#######9  | 12000/15052 [00:00<00:00, 51115.69 examples/s]
Filter: 100%|##########| 15052/15052 [00:00<00:00, 51001.70 examples/s]

Map:   0%|          | 0/1248 [00:00<?, ? examples/s]
Map:  97%|#########7| 1211/1248 [00:00<00:00, 12053.51 examples/s]
Map: 100%|##########| 1248/1248 [00:00<00:00, 11729.43 examples/s]

Map:   0%|          | 0/1248 [00:00<?, ? examples/s]
Map: 100%|##########| 1248/1248 [00:00<00:00, 29630.65 examples/s]

Creating json from Arrow format:   0%|          | 0/17 [00:00<?, ?ba/s]
Creating json from Arrow format:  24%|##3       | 4/17 [00:00<00:00, 35.80ba/s]
Creating json from Arrow format:  47%|####7     | 8/17 [00:00<00:00, 37.48ba/s]
Creating json from Arrow format:  71%|#######   | 12/17 [00:00<00:00, 36.90ba/s]
Creating json from Arrow format: 100%|##########| 17/17 [00:00<00:00, 40.92ba/s]

Map:   0%|          | 0/15052 [00:00<?, ? examples/s]
Map:   5%|5         | 782/15052 [00:00<00:01, 7753.46 examples/s]
Map:  13%|#2        | 1908/15052 [00:00<00:01, 7581.58 examples/s]
Map:  20%|#9        | 2988/15052 [00:00<00:01, 7389.22 examples/s]
Map:  27%|##6       | 4004/15052 [00:00<00:01, 7128.74 examples/s]
Map:  32%|###1      | 4795/15052 [00:00<00:01, 7345.35 examples/s]
Map:  39%|###8      | 5866/15052 [00:00<00:01, 7265.62 examples/s]
Map:  46%|####6     | 6972/15052 [00:00<00:01, 7300.17 examples/s]
Map:  53%|#####3    | 8027/15052 [00:01<00:00, 7206.06 examples/s]
Map:  59%|#####8    | 8822/15052 [00:01<00:00, 7382.40 examples/s]
Map:  66%|######6   | 9950/15052 [00:01<00:00, 7427.08 examples/s]
Map:  73%|#######2  | 10961/15052 [00:01<00:00, 7199.50 examples/s]
Map:  78%|#######7  | 11687/15052 [00:01<00:00, 7211.56 examples/s]
Map:  83%|########2 | 12433/15052 [00:01<00:00, 7271.56 examples/s]
Map:  88%|########7 | 13178/15052 [00:01<00:00, 7317.18 examples/s]
Map:  93%|#########2| 13988/15052 [00:01<00:00, 7528.63 examples/s]
Map: 100%|##########| 15052/15052 [00:02<00:00, 7398.78 examples/s]
Map: 100%|##########| 15052/15052 [00:02<00:00, 7332.19 examples/s]

Map:   0%|          | 0/15052 [00:00<?, ? examples/s]
Map:   4%|3         | 554/15052 [00:00<00:02, 5499.62 examples/s]
Map:   8%|8         | 1274/15052 [00:00<00:05, 2440.91 examples/s]
Map:  12%|#2        | 1824/15052 [00:00<00:04, 3171.43 examples/s]
Map:  15%|#5        | 2274/15052 [00:00<00:03, 3463.55 examples/s]
Map:  19%|#8        | 2827/15052 [00:00<00:03, 4008.35 examples/s]
Map:  24%|##3       | 3559/15052 [00:00<00:02, 4255.67 examples/s]
Map:  29%|##8       | 4307/15052 [00:01<00:02, 4499.47 examples/s]
Map:  32%|###2      | 4852/15052 [00:01<00:02, 4728.50 examples/s]
Map:  37%|###7      | 5585/15052 [00:01<00:01, 4780.26 examples/s]
Map:  42%|####1     | 6273/15052 [00:01<00:01, 4618.70 examples/s]
Map:  45%|####5     | 6831/15052 [00:01<00:01, 4841.39 examples/s]
Map:  50%|#####     | 7559/15052 [00:01<00:01, 4842.95 examples/s]
Map:  55%|#####5    | 8288/15052 [00:01<00:01, 4846.73 examples/s]
Map:  59%|#####8    | 8839/15052 [00:02<00:01, 5001.85 examples/s]
Map:  63%|######3   | 9555/15052 [00:02<00:01, 4897.18 examples/s]
Map:  68%|######8   | 10268/15052 [00:02<00:00, 4824.48 examples/s]
Map:  72%|#######1  | 10820/15052 [00:02<00:00, 4985.58 examples/s]
Map:  77%|#######6  | 11560/15052 [00:02<00:00, 4932.88 examples/s]
Map:  82%|########1 | 12280/15052 [00:02<00:00, 4676.05 examples/s]
Map:  85%|########5 | 12842/15052 [00:02<00:00, 4886.87 examples/s]
Map:  90%|######### | 13576/15052 [00:02<00:00, 4886.02 examples/s]
Map:  95%|#########5| 14319/15052 [00:03<00:00, 4906.06 examples/s]
Map:  99%|#########8| 14877/15052 [00:03<00:00, 5061.07 examples/s]
Map: 100%|##########| 15052/15052 [00:03<00:00, 4574.98 examples/s]

Map:   0%|          | 0/15052 [00:00<?, ? examples/s]
Map:   5%|5         | 773/15052 [00:00<00:01, 7663.95 examples/s]
Map:  13%|#2        | 1885/15052 [00:00<00:01, 7489.08 examples/s]
Map:  20%|#9        | 2990/15052 [00:00<00:01, 7424.90 examples/s]
Map:  27%|##6       | 4011/15052 [00:00<00:01, 7166.73 examples/s]
Map:  32%|###1      | 4797/15052 [00:00<00:01, 7359.95 examples/s]
Map:  39%|###9      | 5871/15052 [00:00<00:01, 7281.26 examples/s]
Map:  46%|####6     | 6999/15052 [00:00<00:01, 7362.55 examples/s]
Map:  51%|#####1    | 7739/15052 [00:01<00:00, 7370.24 examples/s]
Map:  59%|#####8    | 8863/15052 [00:01<00:00, 7411.86 examples/s]
Map:  66%|######6   | 9982/15052 [00:01<00:00, 7424.04 examples/s]
Map:  73%|#######3  | 11000/15052 [00:01<00:00, 7094.84 examples/s]
Map:  78%|#######8  | 11799/15052 [00:01<00:00, 7299.62 examples/s]
Map:  83%|########3 | 12551/15052 [00:01<00:00, 7352.97 examples/s]
Map:  91%|#########1| 13699/15052 [00:01<00:00, 7454.62 examples/s]
Map:  96%|#########6| 14457/15052 [00:01<00:00, 7483.04 examples/s]
Map: 100%|##########| 15052/15052 [00:02<00:00, 7364.37 examples/s]

Map:   0%|          | 0/15052 [00:00<?, ? examples/s]
Map:  20%|#9        | 3000/15052 [00:00<00:00, 28389.13 examples/s]
Map:  40%|###9      | 6000/15052 [00:00<00:00, 28805.64 examples/s]
Map:  61%|######    | 9134/15052 [00:00<00:00, 29933.25 examples/s]
Map:  92%|#########1| 13811/15052 [00:00<00:00, 30126.58 examples/s]
Map: 100%|##########| 15052/15052 [00:00<00:00, 29481.97 examples/s]

Filter:   0%|          | 0/15052 [00:00<?, ? examples/s]
Filter:  40%|###9      | 6000/15052 [00:00<00:00, 50066.40 examples/s]
Filter:  80%|#######9  | 12000/15052 [00:00<00:00, 50964.43 examples/s]
Filter: 100%|##########| 15052/15052 [00:00<00:00, 50994.33 examples/s]

Map:   0%|          | 0/1248 [00:00<?, ? examples/s]
Map:  98%|#########7| 1223/1248 [00:00<00:00, 12180.93 examples/s]
Map: 100%|##########| 1248/1248 [00:00<00:00, 11819.42 examples/s]

Creating json from Arrow format:   0%|          | 0/32 [00:00<?, ?ba/s]
Creating json from Arrow format:   6%|6         | 2/32 [00:00<00:01, 17.39ba/s]
Creating json from Arrow format:  12%|#2        | 4/32 [00:00<00:01, 18.18ba/s]
Creating json from Arrow format:  19%|#8        | 6/32 [00:00<00:01, 17.55ba/s]
Creating json from Arrow format:  25%|##5       | 8/32 [00:00<00:01, 17.82ba/s]
Creating json from Arrow format:  31%|###1      | 10/32 [00:00<00:01, 17.52ba/s]
Creating json from Arrow format:  38%|###7      | 12/32 [00:00<00:01, 18.14ba/s]
Creating json from Arrow format:  47%|####6     | 15/32 [00:00<00:00, 19.00ba/s]
Creating json from Arrow format:  59%|#####9    | 19/32 [00:00<00:00, 23.72ba/s]
Creating json from Arrow format:  72%|#######1  | 23/32 [00:01<00:00, 27.58ba/s]
Creating json from Arrow format:  84%|########4 | 27/32 [00:01<00:00, 30.28ba/s]
Creating json from Arrow format: 100%|##########| 32/32 [00:01<00:00, 25.78ba/s]

Map:   0%|          | 0/1283 [00:00<?, ? examples/s]
Map:  62%|######2   | 797/1283 [00:00<00:00, 7900.18 examples/s]
Map: 100%|##########| 1283/1283 [00:00<00:00, 7453.07 examples/s]

Map:   0%|          | 0/1283 [00:00<?, ? examples/s]
Map: 100%|##########| 1283/1283 [00:00<00:00, 29661.74 examples/s]

Filter:   0%|          | 0/1283 [00:00<?, ? examples/s]
Filter: 100%|##########| 1283/1283 [00:00<00:00, 46868.02 examples/s]

Map:   0%|          | 0/66 [00:00<?, ? examples/s]
Map: 100%|##########| 66/66 [00:00<00:00, 8861.21 examples/s]

Map:   0%|          | 0/66 [00:00<?, ? examples/s]
Map: 100%|##########| 66/66 [00:00<00:00, 15497.06 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 71.75ba/s]

Map:   0%|          | 0/1283 [00:00<?, ? examples/s]
Map:  62%|######2   | 801/1283 [00:00<00:00, 7948.14 examples/s]
Map: 100%|##########| 1283/1283 [00:00<00:00, 7471.85 examples/s]

Map:   0%|          | 0/1283 [00:00<?, ? examples/s]
Map:  72%|#######1  | 922/1283 [00:00<00:00, 9168.16 examples/s]
Map: 100%|##########| 1283/1283 [00:00<00:00, 7712.78 examples/s]

Map:   0%|          | 0/1283 [00:00<?, ? examples/s]
Map:  63%|######3   | 812/1283 [00:00<00:00, 8053.15 examples/s]
Map: 100%|##########| 1283/1283 [00:00<00:00, 7616.44 examples/s]

Map:   0%|          | 0/1283 [00:00<?, ? examples/s]
Map: 100%|##########| 1283/1283 [00:00<00:00, 29241.07 examples/s]

Filter:   0%|          | 0/1283 [00:00<?, ? examples/s]
Filter: 100%|##########| 1283/1283 [00:00<00:00, 46550.57 examples/s]

Map:   0%|          | 0/66 [00:00<?, ? examples/s]
Map: 100%|##########| 66/66 [00:00<00:00, 8962.77 examples/s]INFO 2025-05-20 11:00:42,467 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.67%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 31.91ba/s]

Map:   0%|          | 0/7808 [00:00<?, ? examples/s]
Map:  10%|#         | 782/7808 [00:00<00:00, 7761.00 examples/s]
Map:  24%|##4       | 1908/7808 [00:00<00:00, 7585.14 examples/s]
Map:  38%|###8      | 3000/7808 [00:00<00:00, 7247.36 examples/s]
Map:  49%|####8     | 3794/7808 [00:00<00:00, 7461.62 examples/s]
Map:  63%|######3   | 4931/7808 [00:00<00:00, 7504.81 examples/s]
Map:  77%|#######6  | 6000/7808 [00:00<00:00, 7305.81 examples/s]
Map:  87%|########7 | 6800/7808 [00:00<00:00, 7481.54 examples/s]
Map: 100%|##########| 7808/7808 [00:01<00:00, 7432.18 examples/s]
Map: 100%|##########| 7808/7808 [00:01<00:00, 7427.74 examples/s]

Map:   0%|          | 0/7808 [00:00<?, ? examples/s]
Map:  38%|###8      | 3000/7808 [00:00<00:00, 29199.57 examples/s]
Map:  77%|#######7  | 6031/7808 [00:00<00:00, 29841.06 examples/s]
Map: 100%|##########| 7808/7808 [00:00<00:00, 30160.07 examples/s]

Filter:   0%|          | 0/7808 [00:00<?, ? examples/s]
Filter:  77%|#######6  | 6000/7808 [00:00<00:00, 50087.12 examples/s]
Filter: 100%|##########| 7808/7808 [00:00<00:00, 49614.78 examples/s]

Map:   0%|          | 0/484 [00:00<?, ? examples/s]
Map: 100%|##########| 484/484 [00:00<00:00, 11734.49 examples/s]

Map:   0%|          | 0/484 [00:00<?, ? examples/s]
Map: 100%|##########| 484/484 [00:00<00:00, 27244.51 examples/s]

Creating json from Arrow format:   0%|          | 0/9 [00:00<?, ?ba/s]
Creating json from Arrow format:  44%|####4     | 4/9 [00:00<00:00, 37.88ba/s]
Creating json from Arrow format: 100%|##########| 9/9 [00:00<00:00, 44.98ba/s]

Map:   0%|          | 0/7808 [00:00<?, ? examples/s]
Map:  10%|9         | 761/7808 [00:00<00:00, 7549.60 examples/s]
Map:  24%|##4       | 1877/7808 [00:00<00:00, 7468.57 examples/s]
Map:  38%|###8      | 2983/7808 [00:00<00:00, 7415.70 examples/s]
Map:  52%|#####1    | 4022/7808 [00:00<00:00, 7208.71 examples/s]
Map:  62%|######1   | 4819/7808 [00:00<00:00, 7421.87 examples/s]
Map:  76%|#######5  | 5909/7808 [00:00<00:00, 7358.53 examples/s]
Map:  85%|########5 | 6653/7808 [00:00<00:00, 7378.30 examples/s]
Map:  95%|#########4| 7402/7808 [00:01<00:00, 7387.43 examples/s]
Map: 100%|##########| 7808/7808 [00:01<00:00, 7361.97 examples/s]

Map:   0%|          | 0/7808 [00:00<?, ? examples/s]
Map:   9%|9         | 737/7808 [00:00<00:00, 7329.05 examples/s]
Map:  22%|##2       | 1750/7808 [00:00<00:00, 6672.94 examples/s]
Map:  35%|###5      | 2765/7808 [00:00<00:00, 6595.94 examples/s]
Map:  48%|####8     | 3760/7808 [00:00<00:00, 6454.99 examples/s]
Map:  61%|######    | 4749/7808 [00:00<00:00, 6493.21 examples/s]
Map:  74%|#######3  | 5753/7808 [00:00<00:00, 6539.91 examples/s]
Map:  87%|########6 | 6757/7808 [00:01<00:00, 6423.15 examples/s]
Map: 100%|#########9| 7773/7808 [00:01<00:00, 6527.80 examples/s]
Map: 100%|##########| 7808/7808 [00:01<00:00, 6472.40 examples/s]

Map:   0%|          | 0/7808 [00:00<?, ? examples/s]
Map:  10%|#         | 785/7808 [00:00<00:00, 7797.95 examples/s]
Map:  24%|##4       | 1911/7808 [00:00<00:00, 7594.58 examples/s]
Map:  38%|###8      | 3000/7808 [00:00<00:00, 7218.22 examples/s]
Map:  49%|####8     | 3795/7808 [00:00<00:00, 7442.90 examples/s]
Map:  63%|######2   | 4890/7808 [00:00<00:00, 4626.99 examples/s]
Map:  71%|#######1  | 5557/7808 [00:00<00:00, 5020.11 examples/s]
Map:  81%|########  | 6289/7808 [00:01<00:00, 5515.90 examples/s]
Map:  90%|########9 | 7015/7808 [00:01<00:00, 5925.78 examples/s]
Map: 100%|##########| 7808/7808 [00:01<00:00, 6332.91 examples/s]
Map: 100%|##########| 7808/7808 [00:01<00:00, 6050.75 examples/s]

Map:   0%|          | 0/7808 [00:00<?, ? examples/s]
Map:  38%|###8      | 3000/7808 [00:00<00:00, 28855.65 examples/s]
Map:  77%|#######6  | 6000/7808 [00:00<00:00, 29396.91 examples/s]
Map: 100%|##########| 7808/7808 [00:00<00:00, 29722.75 examples/s]

Filter:   0%|          | 0/7808 [00:00<?, ? examples/s]
Filter:  77%|#######6  | 6000/7808 [00:00<00:00, 49821.67 examples/s]
Filter: 100%|##########| 7808/7808 [00:00<00:00, 49921.46 examples/s]

Map:   0%|          | 0/484 [00:00<?, ? examples/s]
Map: 100%|##########| 484/484 [00:00<00:00, 11475.20 examples/s]

Creating json from Arrow format:   0%|          | 0/17 [00:00<?, ?ba/s]
Creating json from Arrow format:  12%|#1        | 2/17 [00:00<00:00, 17.20ba/s]
Creating json from Arrow format:  24%|##3       | 4/17 [00:00<00:00, 17.39ba/s]
Creating json from Arrow format:  35%|###5      | 6/17 [00:00<00:00, 17.66ba/s]
Creating json from Arrow format:  47%|####7     | 8/17 [00:00<00:00, 18.22ba/s]
Creating json from Arrow format:  71%|#######   | 12/17 [00:00<00:00, 24.14ba/s]
Creating json from Arrow format: 100%|##########| 17/17 [00:00<00:00, 25.74ba/s]

Map:   0%|          | 0/2149 [00:00<?, ? examples/s]
Map:  36%|###5      | 766/2149 [00:00<00:00, 7600.12 examples/s]
Map:  86%|########6 | 1855/2149 [00:00<00:00, 7363.78 examples/s]
Map: 100%|##########| 2149/2149 [00:00<00:00, 7078.72 examples/s]

Map:   0%|          | 0/2149 [00:00<?, ? examples/s]
Map: 100%|##########| 2149/2149 [00:00<00:00, 28966.30 examples/s]

Filter:   0%|          | 0/2149 [00:00<?, ? examples/s]
Filter: 100%|##########| 2149/2149 [00:00<00:00, 46923.87 examples/s]

Map:   0%|          | 0/111 [00:00<?, ? examples/s]
Map: 100%|##########| 111/111 [00:00<00:00, 9366.43 examples/s]

Map:   0%|          | 0/111 [00:00<?, ? examples/s]
Map: 100%|##########| 111/111 [00:00<00:00, 18774.41 examples/s]

Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 49.33ba/s]

Map:   0%|          | 0/2149 [00:00<?, ? examples/s]
Map:  36%|###5      | 773/2149 [00:00<00:00, 7675.56 examples/s]
Map:  86%|########6 | 1856/2149 [00:00<00:00, 7357.41 examples/s]
Map: 100%|##########| 2149/2149 [00:00<00:00, 7077.99 examples/s]

Map:   0%|          | 0/2149 [00:00<?, ? examples/s]
Map:  40%|###9      | 859/2149 [00:00<00:00, 8536.87 examples/s]
Map:  92%|#########1| 1967/2149 [00:00<00:00, 7749.19 examples/s]
Map: 100%|##########| 2149/2149 [00:00<00:00, 7139.57 examples/s]

Map:   0%|          | 0/2149 [00:00<?, ? examples/s]
Map:  36%|###6      | 777/2149 [00:00<00:00, 7700.81 examples/s]
Map:  88%|########7 | 1890/2149 [00:00<00:00, 7499.43 examples/s]
Map: 100%|##########| 2149/2149 [00:00<00:00, 7226.21 examples/s]

Map:   0%|          | 0/2149 [00:00<?, ? examples/s]
Map: 100%|##########| 2149/2149 [00:00<00:00, 28857.98 examples/s]

Filter:   0%|          | 0/2149 [00:00<?, ? examples/s]
Filter: 100%|##########| 2149/2149 [00:00<00:00, 46648.24 examples/s]

Map:   0%|          | 0/111 [00:00<?, ? examples/s]
Map: 100%|##########| 111/111 [00:00<00:00, 9484.16 examples/s]INFO 2025-05-20 11:00:51,419 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.12%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/5 [00:00<?, ?ba/s]
Creating json from Arrow format:  40%|####      | 2/5 [00:00<00:00, 17.37ba/s]
Creating json from Arrow format: 100%|##########| 5/5 [00:00<00:00, 26.61ba/s]

Map:   0%|          | 0/1298 [00:00<?, ? examples/s]
Map:  61%|######1   | 796/1298 [00:00<00:00, 7893.87 examples/s]
Map: 100%|##########| 1298/1298 [00:00<00:00, 7439.05 examples/s]

Map:   0%|          | 0/1298 [00:00<?, ? examples/s]
Map: 100%|##########| 1298/1298 [00:00<00:00, 29641.40 examples/s]

Filter:   0%|          | 0/1298 [00:00<?, ? examples/s]
Filter: 100%|##########| 1298/1298 [00:00<00:00, 47131.09 examples/s]

Map:   0%|          | 0/68 [00:00<?, ? examples/s]
Map: 100%|##########| 68/68 [00:00<00:00, 8978.83 examples/s]

Map:   0%|          | 0/68 [00:00<?, ? examples/s]
Map: 100%|##########| 68/68 [00:00<00:00, 15344.74 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 69.56ba/s]

Map:   0%|          | 0/1298 [00:00<?, ? examples/s]
Map:  62%|######1   | 803/1298 [00:00<00:00, 7975.08 examples/s]
Map: 100%|##########| 1298/1298 [00:00<00:00, 7487.56 examples/s]

Map:   0%|          | 0/1298 [00:00<?, ? examples/s]
Map:  71%|#######   | 918/1298 [00:00<00:00, 9135.21 examples/s]
Map: 100%|##########| 1298/1298 [00:00<00:00, 7592.85 examples/s]

Map:   0%|          | 0/1298 [00:00<?, ? examples/s]
Map:  62%|######2   | 806/1298 [00:00<00:00, 8005.52 examples/s]
Map: 100%|##########| 1298/1298 [00:00<00:00, 7540.49 examples/s]

Map:   0%|          | 0/1298 [00:00<?, ? examples/s]
Map: 100%|##########| 1298/1298 [00:00<00:00, 30048.28 examples/s]

Filter:   0%|          | 0/1298 [00:00<?, ? examples/s]
Filter: 100%|##########| 1298/1298 [00:00<00:00, 48131.54 examples/s]

Map:   0%|          | 0/68 [00:00<?, ? examples/s]
Map: 100%|##########| 68/68 [00:00<00:00, 9030.00 examples/s]INFO 2025-05-20 11:00:52,625 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.68%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 32.26ba/s]

Map:   0%|          | 0/713 [00:00<?, ? examples/s]
Map: 100%|##########| 713/713 [00:00<00:00, 7485.35 examples/s]

Map:   0%|          | 0/713 [00:00<?, ? examples/s]
Map: 100%|##########| 713/713 [00:00<00:00, 29320.16 examples/s]

Filter:   0%|          | 0/713 [00:00<?, ? examples/s]
Filter: 100%|##########| 713/713 [00:00<00:00, 45158.61 examples/s]

Map:   0%|          | 0/37 [00:00<?, ? examples/s]
Map: 100%|##########| 37/37 [00:00<00:00, 7445.63 examples/s]

Map:   0%|          | 0/37 [00:00<?, ? examples/s]
Map: 100%|##########| 37/37 [00:00<00:00, 11467.47 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 68.65ba/s]

Map:   0%|          | 0/713 [00:00<?, ? examples/s]
Map: 100%|##########| 713/713 [00:00<00:00, 7511.03 examples/s]

Map:   0%|          | 0/713 [00:00<?, ? examples/s]
Map: 100%|##########| 713/713 [00:00<00:00, 8170.51 examples/s]

Map:   0%|          | 0/713 [00:00<?, ? examples/s]
Map: 100%|##########| 713/713 [00:00<00:00, 7479.47 examples/s]

Map:   0%|          | 0/713 [00:00<?, ? examples/s]
Map: 100%|##########| 713/713 [00:00<00:00, 28877.35 examples/s]

Filter:   0%|          | 0/713 [00:00<?, ? examples/s]
Filter: 100%|##########| 713/713 [00:00<00:00, 45393.73 examples/s]

Map:   0%|          | 0/37 [00:00<?, ? examples/s]
Map: 100%|##########| 37/37 [00:00<00:00, 7054.06 examples/s]INFO 2025-05-20 11:00:53,296 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.37%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 44.47ba/s]

Map:   0%|          | 0/686 [00:00<?, ? examples/s]
Map: 100%|##########| 686/686 [00:00<00:00, 7456.60 examples/s]

Map:   0%|          | 0/686 [00:00<?, ? examples/s]
Map: 100%|##########| 686/686 [00:00<00:00, 28412.93 examples/s]

Filter:   0%|          | 0/686 [00:00<?, ? examples/s]
Filter: 100%|##########| 686/686 [00:00<00:00, 44686.78 examples/s]

Map:   0%|          | 0/36 [00:00<?, ? examples/s]
Map: 100%|##########| 36/36 [00:00<00:00, 7214.28 examples/s]

Map:   0%|          | 0/36 [00:00<?, ? examples/s]
Map: 100%|##########| 36/36 [00:00<00:00, 11037.64 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 74.33ba/s]

Map:   0%|          | 0/686 [00:00<?, ? examples/s]
Map: 100%|##########| 686/686 [00:00<00:00, 7389.82 examples/s]

Map:   0%|          | 0/686 [00:00<?, ? examples/s]
Map: 100%|##########| 686/686 [00:00<00:00, 8212.67 examples/s]

Map:   0%|          | 0/686 [00:00<?, ? examples/s]
Map: 100%|##########| 686/686 [00:00<00:00, 7485.15 examples/s]

Map:   0%|          | 0/686 [00:00<?, ? examples/s]
Map: 100%|##########| 686/686 [00:00<00:00, 29295.86 examples/s]

Filter:   0%|          | 0/686 [00:00<?, ? examples/s]
Filter: 100%|##########| 686/686 [00:00<00:00, 45799.26 examples/s]

Map:   0%|          | 0/36 [00:00<?, ? examples/s]
Map: 100%|##########| 36/36 [00:00<00:00, 7319.90 examples/s]INFO 2025-05-20 11:00:53,900 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.36%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 52.62ba/s]

Map:   0%|          | 0/688 [00:00<?, ? examples/s]
Map: 100%|##########| 688/688 [00:00<00:00, 7168.11 examples/s]

Map:   0%|          | 0/688 [00:00<?, ? examples/s]
Map: 100%|##########| 688/688 [00:00<00:00, 28165.08 examples/s]

Filter:   0%|          | 0/688 [00:00<?, ? examples/s]
Filter: 100%|##########| 688/688 [00:00<00:00, 43260.34 examples/s]

Map:   0%|          | 0/37 [00:00<?, ? examples/s]
Map: 100%|##########| 37/37 [00:00<00:00, 7060.48 examples/s]

Map:   0%|          | 0/37 [00:00<?, ? examples/s]
Map: 100%|##########| 37/37 [00:00<00:00, 10977.52 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 51.91ba/s]

Map:   0%|          | 0/688 [00:00<?, ? examples/s]
Map: 100%|##########| 688/688 [00:00<00:00, 7253.77 examples/s]

Map:   0%|          | 0/688 [00:00<?, ? examples/s]
Map: 100%|##########| 688/688 [00:00<00:00, 7555.23 examples/s]

Map:   0%|          | 0/688 [00:00<?, ? examples/s]
Map: 100%|##########| 688/688 [00:00<00:00, 7283.10 examples/s]

Map:   0%|          | 0/688 [00:00<?, ? examples/s]
Map: 100%|##########| 688/688 [00:00<00:00, 27724.01 examples/s]

Filter:   0%|          | 0/688 [00:00<?, ? examples/s]
Filter: 100%|##########| 688/688 [00:00<00:00, 44909.83 examples/s]

Map:   0%|          | 0/37 [00:00<?, ? examples/s]
Map: 100%|##########| 37/37 [00:00<00:00, 7094.37 examples/s]INFO 2025-05-20 11:00:54,529 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.36%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 28.13ba/s]

Map:   0%|          | 0/162 [00:00<?, ? examples/s]
Map: 100%|##########| 162/162 [00:00<00:00, 6662.46 examples/s]

Map:   0%|          | 0/162 [00:00<?, ? examples/s]
Map: 100%|##########| 162/162 [00:00<00:00, 21945.52 examples/s]

Filter:   0%|          | 0/162 [00:00<?, ? examples/s]
Filter: 100%|##########| 162/162 [00:00<00:00, 30753.93 examples/s]

Map:   0%|          | 0/36 [00:00<?, ? examples/s]
Map: 100%|##########| 36/36 [00:00<00:00, 7099.63 examples/s]

Map:   0%|          | 0/36 [00:00<?, ? examples/s]
Map: 100%|##########| 36/36 [00:00<00:00, 10972.67 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 269.18ba/s]

Map:   0%|          | 0/162 [00:00<?, ? examples/s]
Map: 100%|##########| 162/162 [00:00<00:00, 6674.50 examples/s]

Map:   0%|          | 0/162 [00:00<?, ? examples/s]
Map: 100%|##########| 162/162 [00:00<00:00, 7433.45 examples/s]

Map:   0%|          | 0/162 [00:00<?, ? examples/s]
Map: 100%|##########| 162/162 [00:00<00:00, 6815.77 examples/s]

Map:   0%|          | 0/162 [00:00<?, ? examples/s]
Map: 100%|##########| 162/162 [00:00<00:00, 22406.50 examples/s]

Filter:   0%|          | 0/162 [00:00<?, ? examples/s]
Filter: 100%|##########| 162/162 [00:00<00:00, 32990.74 examples/s]

Map:   0%|          | 0/36 [00:00<?, ? examples/s]
Map: 100%|##########| 36/36 [00:00<00:00, 7375.32 examples/s]INFO 2025-05-20 11:00:54,809 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.09%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 113.70ba/s]

Map:   0%|          | 0/832 [00:00<?, ? examples/s]
Map:  96%|#########5| 798/832 [00:00<00:00, 7928.16 examples/s]
Map: 100%|##########| 832/832 [00:00<00:00, 7409.46 examples/s]

Map:   0%|          | 0/832 [00:00<?, ? examples/s]
Map: 100%|##########| 832/832 [00:00<00:00, 28993.29 examples/s]

Filter:   0%|          | 0/832 [00:00<?, ? examples/s]
Filter: 100%|##########| 832/832 [00:00<00:00, 45770.25 examples/s]

Map:   0%|          | 0/49 [00:00<?, ? examples/s]
Map: 100%|##########| 49/49 [00:00<00:00, 8074.84 examples/s]

Map:   0%|          | 0/49 [00:00<?, ? examples/s]
Map: 100%|##########| 49/49 [00:00<00:00, 13180.33 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 55.18ba/s]

Map:   0%|          | 0/832 [00:00<?, ? examples/s]
Map:  96%|#########5| 798/832 [00:00<00:00, 7917.13 examples/s]
Map: 100%|##########| 832/832 [00:00<00:00, 7394.15 examples/s]

Map:   0%|          | 0/832 [00:00<?, ? examples/s]
Map: 100%|##########| 832/832 [00:00<00:00, 8052.64 examples/s]
Map: 100%|##########| 832/832 [00:00<00:00, 7980.05 examples/s]

Map:   0%|          | 0/832 [00:00<?, ? examples/s]
Map:  97%|#########6| 807/832 [00:00<00:00, 8010.48 examples/s]
Map: 100%|##########| 832/832 [00:00<00:00, 7525.62 examples/s]

Map:   0%|          | 0/832 [00:00<?, ? examples/s]
Map: 100%|##########| 832/832 [00:00<00:00, 29061.86 examples/s]

Filter:   0%|          | 0/832 [00:00<?, ? examples/s]
Filter: 100%|##########| 832/832 [00:00<00:00, 45951.05 examples/s]

Map:   0%|          | 0/49 [00:00<?, ? examples/s]
Map: 100%|##########| 49/49 [00:00<00:00, 8060.91 examples/s]INFO 2025-05-20 11:00:55,488 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.43%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 34.83ba/s]

Map:   0%|          | 0/138 [00:00<?, ? examples/s]
Map: 100%|##########| 138/138 [00:00<00:00, 6024.98 examples/s]

Map:   0%|          | 0/138 [00:00<?, ? examples/s]
Map: 100%|##########| 138/138 [00:00<00:00, 20515.86 examples/s]

Filter:   0%|          | 0/138 [00:00<?, ? examples/s]
Filter: 100%|##########| 138/138 [00:00<00:00, 29572.06 examples/s]

Map:   0%|          | 0/26 [00:00<?, ? examples/s]
Map: 100%|##########| 26/26 [00:00<00:00, 6034.97 examples/s]

Map:   0%|          | 0/26 [00:00<?, ? examples/s]
Map: 100%|##########| 26/26 [00:00<00:00, 8843.00 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 153.78ba/s]

Map:   0%|          | 0/138 [00:00<?, ? examples/s]
Map: 100%|##########| 138/138 [00:00<00:00, 6145.17 examples/s]

Map:   0%|          | 0/138 [00:00<?, ? examples/s]
Map: 100%|##########| 138/138 [00:00<00:00, 6461.78 examples/s]

Map:   0%|          | 0/138 [00:00<?, ? examples/s]
Map: 100%|##########| 138/138 [00:00<00:00, 6286.95 examples/s]

Map:   0%|          | 0/138 [00:00<?, ? examples/s]
Map: 100%|##########| 138/138 [00:00<00:00, 20486.81 examples/s]

Filter:   0%|          | 0/138 [00:00<?, ? examples/s]
Filter: 100%|##########| 138/138 [00:00<00:00, 30691.66 examples/s]

Map:   0%|          | 0/26 [00:00<?, ? examples/s]
Map: 100%|##########| 26/26 [00:00<00:00, 6051.38 examples/s]INFO 2025-05-20 11:00:55,747 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.08%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 62.34ba/s]

Map:   0%|          | 0/2122 [00:00<?, ? examples/s]
Map:  37%|###7      | 787/2122 [00:00<00:00, 7816.04 examples/s]
Map:  90%|########9 | 1904/2122 [00:00<00:00, 7559.08 examples/s]
Map: 100%|##########| 2122/2122 [00:00<00:00, 7280.77 examples/s]

Map:   0%|          | 0/2122 [00:00<?, ? examples/s]
Map: 100%|##########| 2122/2122 [00:00<00:00, 28753.07 examples/s]

Filter:   0%|          | 0/2122 [00:00<?, ? examples/s]
Filter: 100%|##########| 2122/2122 [00:00<00:00, 48684.01 examples/s]

Map:   0%|          | 0/121 [00:00<?, ? examples/s]
Map: 100%|##########| 121/121 [00:00<00:00, 9992.73 examples/s]

Map:   0%|          | 0/121 [00:00<?, ? examples/s]
Map: 100%|##########| 121/121 [00:00<00:00, 19928.17 examples/s]

Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 51.61ba/s]

Map:   0%|          | 0/2122 [00:00<?, ? examples/s]
Map:  37%|###6      | 784/2122 [00:00<00:00, 7787.98 examples/s]
Map:  90%|########9 | 1908/2122 [00:00<00:00, 7579.07 examples/s]
Map: 100%|##########| 2122/2122 [00:00<00:00, 7293.69 examples/s]

Map:   0%|          | 0/2122 [00:00<?, ? examples/s]
Map:  41%|####      | 862/2122 [00:00<00:00, 8577.86 examples/s]
Map:  94%|#########4| 2000/2122 [00:00<00:00, 7215.12 examples/s]
Map: 100%|##########| 2122/2122 [00:00<00:00, 7337.90 examples/s]

Map:   0%|          | 0/2122 [00:00<?, ? examples/s]
Map:  37%|###6      | 781/2122 [00:00<00:00, 7745.52 examples/s]
Map:  90%|########9 | 1907/2122 [00:00<00:00, 7576.78 examples/s]
Map: 100%|##########| 2122/2122 [00:00<00:00, 7333.64 examples/s]

Map:   0%|          | 0/2122 [00:00<?, ? examples/s]
Map: 100%|##########| 2122/2122 [00:00<00:00, 29529.61 examples/s]

Filter:   0%|          | 0/2122 [00:00<?, ? examples/s]
Filter: 100%|##########| 2122/2122 [00:00<00:00, 48752.54 examples/s]

Map:   0%|          | 0/121 [00:00<?, ? examples/s]
Map: 100%|##########| 121/121 [00:00<00:00, 10110.18 examples/s]INFO 2025-05-20 11:00:57,448 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.11%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/5 [00:00<?, ?ba/s]
Creating json from Arrow format:  40%|####      | 2/5 [00:00<00:00, 16.64ba/s]
Creating json from Arrow format: 100%|##########| 5/5 [00:00<00:00, 27.01ba/s]

Map:   0%|          | 0/243 [00:00<?, ? examples/s]
Map: 100%|##########| 243/243 [00:00<00:00, 6955.73 examples/s]

Map:   0%|          | 0/243 [00:00<?, ? examples/s]
Map: 100%|##########| 243/243 [00:00<00:00, 24336.00 examples/s]

Filter:   0%|          | 0/243 [00:00<?, ? examples/s]
Filter: 100%|##########| 243/243 [00:00<00:00, 36983.05 examples/s]

Map:   0%|          | 0/26 [00:00<?, ? examples/s]
Map: 100%|##########| 26/26 [00:00<00:00, 6276.01 examples/s]

Map:   0%|          | 0/26 [00:00<?, ? examples/s]
Map: 100%|##########| 26/26 [00:00<00:00, 8757.78 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 208.62ba/s]

Map:   0%|          | 0/243 [00:00<?, ? examples/s]
Map: 100%|##########| 243/243 [00:00<00:00, 7040.15 examples/s]

Map:   0%|          | 0/243 [00:00<?, ? examples/s]
Map: 100%|##########| 243/243 [00:00<00:00, 8071.14 examples/s]

Map:   0%|          | 0/243 [00:00<?, ? examples/s]
Map: 100%|##########| 243/243 [00:00<00:00, 7148.43 examples/s]

Map:   0%|          | 0/243 [00:00<?, ? examples/s]
Map: 100%|##########| 243/243 [00:00<00:00, 24557.05 examples/s]

Filter:   0%|          | 0/243 [00:00<?, ? examples/s]
Filter: 100%|##########| 243/243 [00:00<00:00, 37226.19 examples/s]

Map:   0%|          | 0/26 [00:00<?, ? examples/s]
Map: 100%|##########| 26/26 [00:00<00:00, 6098.08 examples/s]INFO 2025-05-20 11:00:57,894 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.13%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 84.53ba/s]

Map:   0%|          | 0/1425 [00:00<?, ? examples/s]
Map:  56%|#####5    | 791/1425 [00:00<00:00, 7845.09 examples/s]
Map: 100%|##########| 1425/1425 [00:00<00:00, 7258.80 examples/s]

Map:   0%|          | 0/1425 [00:00<?, ? examples/s]
Map: 100%|##########| 1425/1425 [00:00<00:00, 28466.50 examples/s]

Filter:   0%|          | 0/1425 [00:00<?, ? examples/s]
Filter: 100%|##########| 1425/1425 [00:00<00:00, 45748.69 examples/s]

Map:   0%|          | 0/79 [00:00<?, ? examples/s]
Map: 100%|##########| 79/79 [00:00<00:00, 9117.55 examples/s]

Map:   0%|          | 0/79 [00:00<?, ? examples/s]
Map: 100%|##########| 79/79 [00:00<00:00, 17008.88 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 49.19ba/s]

Map:   0%|          | 0/1425 [00:00<?, ? examples/s]
Map:  55%|#####5    | 789/1425 [00:00<00:00, 7837.67 examples/s]
Map: 100%|##########| 1425/1425 [00:00<00:00, 7222.65 examples/s]

Map:   0%|          | 0/1425 [00:00<?, ? examples/s]
Map:  62%|######2   | 885/1425 [00:00<00:00, 8806.78 examples/s]
Map: 100%|##########| 1425/1425 [00:00<00:00, 7303.37 examples/s]

Map:   0%|          | 0/1425 [00:00<?, ? examples/s]
Map:  55%|#####5    | 785/1425 [00:00<00:00, 7795.00 examples/s]
Map: 100%|##########| 1425/1425 [00:00<00:00, 7276.13 examples/s]

Map:   0%|          | 0/1425 [00:00<?, ? examples/s]
Map: 100%|##########| 1425/1425 [00:00<00:00, 28031.40 examples/s]

Filter:   0%|          | 0/1425 [00:00<?, ? examples/s]
Filter: 100%|##########| 1425/1425 [00:00<00:00, 45328.52 examples/s]

Map:   0%|          | 0/79 [00:00<?, ? examples/s]
Map: 100%|##########| 79/79 [00:00<00:00, 9249.13 examples/s]INFO 2025-05-20 11:00:59,082 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.74%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format:  67%|######6   | 2/3 [00:00<00:00, 18.21ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 21.68ba/s]

Map:   0%|          | 0/1687 [00:00<?, ? examples/s]
Map:  47%|####6     | 787/1687 [00:00<00:00, 7800.23 examples/s]
Map: 100%|##########| 1687/1687 [00:00<00:00, 7215.09 examples/s]
Map: 100%|##########| 1687/1687 [00:00<00:00, 7257.68 examples/s]

Map:   0%|          | 0/1687 [00:00<?, ? examples/s]
Map: 100%|##########| 1687/1687 [00:00<00:00, 28385.48 examples/s]

Filter:   0%|          | 0/1687 [00:00<?, ? examples/s]
Filter: 100%|##########| 1687/1687 [00:00<00:00, 47911.70 examples/s]

Map:   0%|          | 0/123 [00:00<?, ? examples/s]
Map: 100%|##########| 123/123 [00:00<00:00, 10041.84 examples/s]

Map:   0%|          | 0/123 [00:00<?, ? examples/s]
Map: 100%|##########| 123/123 [00:00<00:00, 20181.49 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 42.09ba/s]

Map:   0%|          | 0/1687 [00:00<?, ? examples/s]
Map:  46%|####6     | 784/1687 [00:00<00:00, 7778.71 examples/s]
Map: 100%|##########| 1687/1687 [00:00<00:00, 7200.65 examples/s]
Map: 100%|##########| 1687/1687 [00:00<00:00, 7240.48 examples/s]

Map:   0%|          | 0/1687 [00:00<?, ? examples/s]
Map:  52%|#####1    | 870/1687 [00:00<00:00, 8656.11 examples/s]
Map: 100%|##########| 1687/1687 [00:00<00:00, 7363.63 examples/s]

Map:   0%|          | 0/1687 [00:00<?, ? examples/s]
Map:  47%|####6     | 791/1687 [00:00<00:00, 7855.80 examples/s]
Map: 100%|##########| 1687/1687 [00:00<00:00, 7322.41 examples/s]
Map: 100%|##########| 1687/1687 [00:00<00:00, 7361.17 examples/s]

Map:   0%|          | 0/1687 [00:00<?, ? examples/s]
Map: 100%|##########| 1687/1687 [00:00<00:00, 29043.30 examples/s]

Filter:   0%|          | 0/1687 [00:00<?, ? examples/s]
Filter: 100%|##########| 1687/1687 [00:00<00:00, 47713.33 examples/s]

Map:   0%|          | 0/123 [00:00<?, ? examples/s]
Map: 100%|##########| 123/123 [00:00<00:00, 10065.54 examples/s]INFO 2025-05-20 11:01:00,584 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.89%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format:  50%|#####     | 2/4 [00:00<00:00, 20.00ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 28.37ba/s]

Map:   0%|          | 0/1526 [00:00<?, ? examples/s]
Map:  52%|#####1    | 792/1526 [00:00<00:00, 7851.85 examples/s]
Map: 100%|##########| 1526/1526 [00:00<00:00, 7380.67 examples/s]

Map:   0%|          | 0/1526 [00:00<?, ? examples/s]
Map: 100%|##########| 1526/1526 [00:00<00:00, 28466.57 examples/s]

Filter:   0%|          | 0/1526 [00:00<?, ? examples/s]
Filter: 100%|##########| 1526/1526 [00:00<00:00, 46434.67 examples/s]

Map:   0%|          | 0/120 [00:00<?, ? examples/s]
Map: 100%|##########| 120/120 [00:00<00:00, 10080.64 examples/s]

Map:   0%|          | 0/120 [00:00<?, ? examples/s]
Map: 100%|##########| 120/120 [00:00<00:00, 20065.24 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 51.32ba/s]

Map:   0%|          | 0/1526 [00:00<?, ? examples/s]
Map:  52%|#####2    | 797/1526 [00:00<00:00, 7906.74 examples/s]
Map: 100%|##########| 1526/1526 [00:00<00:00, 7422.94 examples/s]

Map:   0%|          | 0/1526 [00:00<?, ? examples/s]
Map:  59%|#####8    | 893/1526 [00:00<00:00, 8878.23 examples/s]
Map: 100%|##########| 1526/1526 [00:00<00:00, 7689.24 examples/s]

Map:   0%|          | 0/1526 [00:00<?, ? examples/s]
Map:  52%|#####1    | 788/1526 [00:00<00:00, 7826.40 examples/s]
Map: 100%|##########| 1526/1526 [00:00<00:00, 7313.25 examples/s]

Map:   0%|          | 0/1526 [00:00<?, ? examples/s]
Map: 100%|##########| 1526/1526 [00:00<00:00, 29285.97 examples/s]

Filter:   0%|          | 0/1526 [00:00<?, ? examples/s]
Filter: 100%|##########| 1526/1526 [00:00<00:00, 46294.28 examples/s]

Map:   0%|          | 0/120 [00:00<?, ? examples/s]
Map: 100%|##########| 120/120 [00:00<00:00, 10261.71 examples/s]INFO 2025-05-20 11:01:01,956 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.80%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format:  75%|#######5  | 3/4 [00:00<00:00, 26.68ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 34.30ba/s]

Map:   0%|          | 0/649 [00:00<?, ? examples/s]
Map: 100%|##########| 649/649 [00:00<00:00, 7625.43 examples/s]

Map:   0%|          | 0/649 [00:00<?, ? examples/s]
Map: 100%|##########| 649/649 [00:00<00:00, 28221.19 examples/s]

Filter:   0%|          | 0/649 [00:00<?, ? examples/s]
Filter: 100%|##########| 649/649 [00:00<00:00, 45677.47 examples/s]

Map:   0%|          | 0/18 [00:00<?, ? examples/s]
Map: 100%|##########| 18/18 [00:00<00:00, 5005.14 examples/s]

Map:   0%|          | 0/18 [00:00<?, ? examples/s]
Map: 100%|##########| 18/18 [00:00<00:00, 6632.48 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 100.32ba/s]

Map:   0%|          | 0/649 [00:00<?, ? examples/s]
Map: 100%|##########| 649/649 [00:00<00:00, 7661.27 examples/s]

Map:   0%|          | 0/649 [00:00<?, ? examples/s]
Map: 100%|##########| 649/649 [00:00<00:00, 8590.44 examples/s]

Map:   0%|          | 0/649 [00:00<?, ? examples/s]
Map: 100%|##########| 649/649 [00:00<00:00, 7678.24 examples/s]

Map:   0%|          | 0/649 [00:00<?, ? examples/s]
Map: 100%|##########| 649/649 [00:00<00:00, 29629.30 examples/s]

Filter:   0%|          | 0/649 [00:00<?, ? examples/s]
Filter: 100%|##########| 649/649 [00:00<00:00, 46490.36 examples/s]

Map:   0%|          | 0/18 [00:00<?, ? examples/s]
Map: 100%|##########| 18/18 [00:00<00:00, 5106.70 examples/s]INFO 2025-05-20 11:01:02,585 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.33%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 57.59ba/s]

Map:   0%|          | 0/113 [00:00<?, ? examples/s]
Map: 100%|##########| 113/113 [00:00<00:00, 5814.92 examples/s]

Map:   0%|          | 0/113 [00:00<?, ? examples/s]
Map: 100%|##########| 113/113 [00:00<00:00, 18666.31 examples/s]

Filter:   0%|          | 0/113 [00:00<?, ? examples/s]
Filter: 100%|##########| 113/113 [00:00<00:00, 26843.93 examples/s]

Map:   0%|          | 0/7 [00:00<?, ? examples/s]
Map: 100%|##########| 7/7 [00:00<00:00, 2532.79 examples/s]

Map:   0%|          | 0/7 [00:00<?, ? examples/s]
Map: 100%|##########| 7/7 [00:00<00:00, 3069.22 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 166.12ba/s]

Map:   0%|          | 0/113 [00:00<?, ? examples/s]
Map: 100%|##########| 113/113 [00:00<00:00, 5746.32 examples/s]

Map:   0%|          | 0/113 [00:00<?, ? examples/s]
Map: 100%|##########| 113/113 [00:00<00:00, 5635.90 examples/s]

Map:   0%|          | 0/113 [00:00<?, ? examples/s]
Map: 100%|##########| 113/113 [00:00<00:00, 5943.77 examples/s]

Map:   0%|          | 0/113 [00:00<?, ? examples/s]
Map: 100%|##########| 113/113 [00:00<00:00, 19166.01 examples/s]

Filter:   0%|          | 0/113 [00:00<?, ? examples/s]
Filter: 100%|##########| 113/113 [00:00<00:00, 27745.95 examples/s]

Map:   0%|          | 0/7 [00:00<?, ? examples/s]
Map: 100%|##########| 7/7 [00:00<00:00, 2560.40 examples/s]INFO 2025-05-20 11:01:02,802 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.06%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 44.15ba/s]

Map:   0%|          | 0/2434 [00:00<?, ? examples/s]
Map:  32%|###1      | 774/2434 [00:00<00:00, 7675.61 examples/s]
Map:  78%|#######7  | 1895/2434 [00:00<00:00, 7533.70 examples/s]
Map: 100%|##########| 2434/2434 [00:00<00:00, 7287.03 examples/s]

Map:   0%|          | 0/2434 [00:00<?, ? examples/s]
Map: 100%|##########| 2434/2434 [00:00<00:00, 29413.11 examples/s]

Filter:   0%|          | 0/2434 [00:00<?, ? examples/s]
Filter: 100%|##########| 2434/2434 [00:00<00:00, 49056.89 examples/s]

Map:   0%|          | 0/110 [00:00<?, ? examples/s]
Map: 100%|##########| 110/110 [00:00<00:00, 9855.25 examples/s]

Map:   0%|          | 0/110 [00:00<?, ? examples/s]
Map: 100%|##########| 110/110 [00:00<00:00, 19172.77 examples/s]

Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 48.50ba/s]

Map:   0%|          | 0/2434 [00:00<?, ? examples/s]
Map:  32%|###2      | 783/2434 [00:00<00:00, 7769.09 examples/s]
Map:  78%|#######8  | 1906/2434 [00:00<00:00, 7570.42 examples/s]
Map: 100%|##########| 2434/2434 [00:00<00:00, 7323.88 examples/s]

Map:   0%|          | 0/2434 [00:00<?, ? examples/s]
Map:  36%|###5      | 872/2434 [00:00<00:00, 8676.88 examples/s]
Map:  82%|########2 | 2000/2434 [00:00<00:00, 7289.76 examples/s]
Map: 100%|##########| 2434/2434 [00:00<00:00, 7479.07 examples/s]

Map:   0%|          | 0/2434 [00:00<?, ? examples/s]
Map:  32%|###2      | 786/2434 [00:00<00:00, 7795.77 examples/s]
Map:  79%|#######8  | 1917/2434 [00:00<00:00, 7616.59 examples/s]
Map: 100%|##########| 2434/2434 [00:00<00:00, 7391.91 examples/s]

Map:   0%|          | 0/2434 [00:00<?, ? examples/s]
Map: 100%|##########| 2434/2434 [00:00<00:00, 29444.58 examples/s]

Filter:   0%|          | 0/2434 [00:00<?, ? examples/s]
Filter: 100%|##########| 2434/2434 [00:00<00:00, 49011.68 examples/s]

Map:   0%|          | 0/110 [00:00<?, ? examples/s]
Map: 100%|##########| 110/110 [00:00<00:00, 9836.97 examples/s]INFO 2025-05-20 11:01:04,737 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.26%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/5 [00:00<?, ?ba/s]
Creating json from Arrow format:  40%|####      | 2/5 [00:00<00:00, 16.93ba/s]
Creating json from Arrow format: 100%|##########| 5/5 [00:00<00:00, 24.14ba/s]

Map:   0%|          | 0/375 [00:00<?, ? examples/s]
Map: 100%|##########| 375/375 [00:00<00:00, 7187.90 examples/s]

Map:   0%|          | 0/375 [00:00<?, ? examples/s]
Map: 100%|##########| 375/375 [00:00<00:00, 26894.38 examples/s]

Filter:   0%|          | 0/375 [00:00<?, ? examples/s]
Filter: 100%|##########| 375/375 [00:00<00:00, 40590.04 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 5368.14 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 7607.56 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 106.16ba/s]

Map:   0%|          | 0/375 [00:00<?, ? examples/s]
Map: 100%|##########| 375/375 [00:00<00:00, 7083.19 examples/s]

Map:   0%|          | 0/375 [00:00<?, ? examples/s]
Map: 100%|##########| 375/375 [00:00<00:00, 7880.12 examples/s]

Map:   0%|          | 0/375 [00:00<?, ? examples/s]
Map: 100%|##########| 375/375 [00:00<00:00, 7237.25 examples/s]

Map:   0%|          | 0/375 [00:00<?, ? examples/s]
Map: 100%|##########| 375/375 [00:00<00:00, 26959.38 examples/s]

Filter:   0%|          | 0/375 [00:00<?, ? examples/s]
Filter: 100%|##########| 375/375 [00:00<00:00, 40389.91 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 5492.32 examples/s]INFO 2025-05-20 11:01:05,302 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.20%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 32.84ba/s]

Map:   0%|          | 0/1467 [00:00<?, ? examples/s]
Map:  52%|#####2    | 765/1467 [00:00<00:00, 7593.41 examples/s]
Map: 100%|##########| 1467/1467 [00:00<00:00, 6982.23 examples/s]

Map:   0%|          | 0/1467 [00:00<?, ? examples/s]
Map:  68%|######8   | 1000/1467 [00:00<00:00, 3795.71 examples/s]
Map: 100%|##########| 1467/1467 [00:00<00:00, 5178.66 examples/s]

Filter:   0%|          | 0/1467 [00:00<?, ? examples/s]
Filter: 100%|##########| 1467/1467 [00:00<00:00, 45245.30 examples/s]

Map:   0%|          | 0/91 [00:00<?, ? examples/s]
Map: 100%|##########| 91/91 [00:00<00:00, 9082.47 examples/s]

Map:   0%|          | 0/91 [00:00<?, ? examples/s]
Map: 100%|##########| 91/91 [00:00<00:00, 17030.24 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 40.34ba/s]

Map:   0%|          | 0/1467 [00:00<?, ? examples/s]
Map:  53%|#####2    | 771/1467 [00:00<00:00, 7641.49 examples/s]
Map: 100%|##########| 1467/1467 [00:00<00:00, 7093.70 examples/s]

Map:   0%|          | 0/1467 [00:00<?, ? examples/s]
Map:  58%|#####8    | 858/1467 [00:00<00:00, 8526.44 examples/s]
Map: 100%|##########| 1467/1467 [00:00<00:00, 7022.52 examples/s]

Map:   0%|          | 0/1467 [00:00<?, ? examples/s]
Map:  53%|#####2    | 772/1467 [00:00<00:00, 7657.28 examples/s]
Map: 100%|##########| 1467/1467 [00:00<00:00, 7119.30 examples/s]

Map:   0%|          | 0/1467 [00:00<?, ? examples/s]
Map: 100%|##########| 1467/1467 [00:00<00:00, 28102.25 examples/s]

Filter:   0%|          | 0/1467 [00:00<?, ? examples/s]
Filter: 100%|##########| 1467/1467 [00:00<00:00, 45646.74 examples/s]

Map:   0%|          | 0/91 [00:00<?, ? examples/s]
Map: 100%|##########| 91/91 [00:00<00:00, 8893.48 examples/s]INFO 2025-05-20 11:01:06,824 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.77%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format:  50%|#####     | 2/4 [00:00<00:00, 17.07ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 26.71ba/s]

Map:   0%|          | 0/1581 [00:00<?, ? examples/s]
Map:  50%|#####     | 793/1581 [00:00<00:00, 7864.01 examples/s]
Map: 100%|##########| 1581/1581 [00:00<00:00, 7310.87 examples/s]
Map: 100%|##########| 1581/1581 [00:00<00:00, 7352.34 examples/s]

Map:   0%|          | 0/1581 [00:00<?, ? examples/s]
Map: 100%|##########| 1581/1581 [00:00<00:00, 28417.50 examples/s]

Filter:   0%|          | 0/1581 [00:00<?, ? examples/s]
Filter: 100%|##########| 1581/1581 [00:00<00:00, 46759.80 examples/s]

Map:   0%|          | 0/72 [00:00<?, ? examples/s]
Map: 100%|##########| 72/72 [00:00<00:00, 8903.53 examples/s]

Map:   0%|          | 0/72 [00:00<?, ? examples/s]
Map: 100%|##########| 72/72 [00:00<00:00, 15244.32 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 49.08ba/s]

Map:   0%|          | 0/1581 [00:00<?, ? examples/s]
Map:  50%|####9     | 790/1581 [00:00<00:00, 7839.28 examples/s]
Map: 100%|##########| 1581/1581 [00:00<00:00, 7297.44 examples/s]
Map: 100%|##########| 1581/1581 [00:00<00:00, 7336.82 examples/s]

Map:   0%|          | 0/1581 [00:00<?, ? examples/s]
Map:  57%|#####7    | 905/1581 [00:00<00:00, 8996.66 examples/s]
Map: 100%|##########| 1581/1581 [00:00<00:00, 7546.64 examples/s]

Map:   0%|          | 0/1581 [00:00<?, ? examples/s]
Map:  50%|#####     | 798/1581 [00:00<00:00, 7926.07 examples/s]
Map: 100%|##########| 1581/1581 [00:00<00:00, 7318.87 examples/s]

Map:   0%|          | 0/1581 [00:00<?, ? examples/s]
Map: 100%|##########| 1581/1581 [00:00<00:00, 28793.72 examples/s]

Filter:   0%|          | 0/1581 [00:00<?, ? examples/s]
Filter: 100%|##########| 1581/1581 [00:00<00:00, 47264.06 examples/s]

Map:   0%|          | 0/72 [00:00<?, ? examples/s]
Map: 100%|##########| 72/72 [00:00<00:00, 9027.56 examples/s]INFO 2025-05-20 11:01:08,247 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.82%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format:  75%|#######5  | 3/4 [00:00<00:00, 25.15ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 31.91ba/s]

Map:   0%|          | 0/327 [00:00<?, ? examples/s]
Map: 100%|##########| 327/327 [00:00<00:00, 7101.44 examples/s]

Map:   0%|          | 0/327 [00:00<?, ? examples/s]
Map: 100%|##########| 327/327 [00:00<00:00, 25777.87 examples/s]

Filter:   0%|          | 0/327 [00:00<?, ? examples/s]
Filter: 100%|##########| 327/327 [00:00<00:00, 38943.11 examples/s]

Map:   0%|          | 0/16 [00:00<?, ? examples/s]
Map: 100%|##########| 16/16 [00:00<00:00, 4627.24 examples/s]

Map:   0%|          | 0/16 [00:00<?, ? examples/s]
Map: 100%|##########| 16/16 [00:00<00:00, 6126.98 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 135.69ba/s]

Map:   0%|          | 0/327 [00:00<?, ? examples/s]
Map: 100%|##########| 327/327 [00:00<00:00, 7272.51 examples/s]

Map:   0%|          | 0/327 [00:00<?, ? examples/s]
Map: 100%|##########| 327/327 [00:00<00:00, 8070.86 examples/s]

Map:   0%|          | 0/327 [00:00<?, ? examples/s]
Map: 100%|##########| 327/327 [00:00<00:00, 7323.03 examples/s]

Map:   0%|          | 0/327 [00:00<?, ? examples/s]
Map: 100%|##########| 327/327 [00:00<00:00, 26750.74 examples/s]

Filter:   0%|          | 0/327 [00:00<?, ? examples/s]
Filter: 100%|##########| 327/327 [00:00<00:00, 40178.62 examples/s]

Map:   0%|          | 0/16 [00:00<?, ? examples/s]
Map: 100%|##########| 16/16 [00:00<00:00, 4724.65 examples/s]INFO 2025-05-20 11:01:08,686 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.17%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 41.18ba/s]

Map:   0%|          | 0/1113 [00:00<?, ? examples/s]
Map:  70%|#######   | 784/1113 [00:00<00:00, 7783.65 examples/s]
Map: 100%|##########| 1113/1113 [00:00<00:00, 7173.13 examples/s]

Map:   0%|          | 0/1113 [00:00<?, ? examples/s]
Map: 100%|##########| 1113/1113 [00:00<00:00, 27995.06 examples/s]

Filter:   0%|          | 0/1113 [00:00<?, ? examples/s]
Filter: 100%|##########| 1113/1113 [00:00<00:00, 47010.74 examples/s]

Map:   0%|          | 0/50 [00:00<?, ? examples/s]
Map: 100%|##########| 50/50 [00:00<00:00, 7749.15 examples/s]

Map:   0%|          | 0/50 [00:00<?, ? examples/s]
Map: 100%|##########| 50/50 [00:00<00:00, 13319.48 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 61.95ba/s]

Map:   0%|          | 0/1113 [00:00<?, ? examples/s]
Map:  70%|######9   | 779/1113 [00:00<00:00, 7735.58 examples/s]
Map: 100%|##########| 1113/1113 [00:00<00:00, 7179.56 examples/s]

Map:   0%|          | 0/1113 [00:00<?, ? examples/s]
Map:  82%|########1 | 912/1113 [00:00<00:00, 9066.96 examples/s]
Map: 100%|##########| 1113/1113 [00:00<00:00, 7549.92 examples/s]

Map:   0%|          | 0/1113 [00:00<?, ? examples/s]
Map:  72%|#######1  | 800/1113 [00:00<00:00, 7938.19 examples/s]
Map: 100%|##########| 1113/1113 [00:00<00:00, 7456.52 examples/s]

Map:   0%|          | 0/1113 [00:00<?, ? examples/s]
Map: 100%|##########| 1113/1113 [00:00<00:00, 28685.74 examples/s]

Filter:   0%|          | 0/1113 [00:00<?, ? examples/s]
Filter: 100%|##########| 1113/1113 [00:00<00:00, 47067.14 examples/s]

Map:   0%|          | 0/50 [00:00<?, ? examples/s]
Map: 100%|##########| 50/50 [00:00<00:00, 8096.49 examples/s]INFO 2025-05-20 11:01:09,622 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.58%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 34.32ba/s]

Map:   0%|          | 0/835 [00:00<?, ? examples/s]
Map:  93%|#########3| 778/835 [00:00<00:00, 7721.50 examples/s]
Map: 100%|##########| 835/835 [00:00<00:00, 7135.76 examples/s]

Map:   0%|          | 0/835 [00:00<?, ? examples/s]
Map: 100%|##########| 835/835 [00:00<00:00, 27502.44 examples/s]

Filter:   0%|          | 0/835 [00:00<?, ? examples/s]
Filter: 100%|##########| 835/835 [00:00<00:00, 44776.57 examples/s]

Map:   0%|          | 0/200 [00:00<?, ? examples/s]
Map: 100%|##########| 200/200 [00:00<00:00, 10723.14 examples/s]

Map:   0%|          | 0/200 [00:00<?, ? examples/s]
Map: 100%|##########| 200/200 [00:00<00:00, 23658.54 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 61.90ba/s]

Map:   0%|          | 0/835 [00:00<?, ? examples/s]
Map:  93%|#########3| 777/835 [00:00<00:00, 7713.66 examples/s]
Map: 100%|##########| 835/835 [00:00<00:00, 6959.11 examples/s]

Map:   0%|          | 0/835 [00:00<?, ? examples/s]
Map:  99%|#########9| 828/835 [00:00<00:00, 8227.16 examples/s]
Map: 100%|##########| 835/835 [00:00<00:00, 6718.10 examples/s]

Map:   0%|          | 0/835 [00:00<?, ? examples/s]
Map:  94%|#########4| 785/835 [00:00<00:00, 7797.88 examples/s]
Map: 100%|##########| 835/835 [00:00<00:00, 7287.58 examples/s]

Map:   0%|          | 0/835 [00:00<?, ? examples/s]
Map: 100%|##########| 835/835 [00:00<00:00, 28142.23 examples/s]

Filter:   0%|          | 0/835 [00:00<?, ? examples/s]
Filter: 100%|##########| 835/835 [00:00<00:00, 44975.52 examples/s]

Map:   0%|          | 0/200 [00:00<?, ? examples/s]
Map: 100%|##########| 200/200 [00:00<00:00, 10890.05 examples/s]INFO 2025-05-20 11:01:10,487 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.47%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 23.27ba/s]

Map:   0%|          | 0/115 [00:00<?, ? examples/s]
Map: 100%|##########| 115/115 [00:00<00:00, 6354.25 examples/s]

Map:   0%|          | 0/115 [00:00<?, ? examples/s]
Map: 100%|##########| 115/115 [00:00<00:00, 19727.81 examples/s]

Filter:   0%|          | 0/115 [00:00<?, ? examples/s]
Filter: 100%|##########| 115/115 [00:00<00:00, 27617.80 examples/s]

Map:   0%|          | 0/32 [00:00<?, ? examples/s]
Map: 100%|##########| 32/32 [00:00<00:00, 6672.52 examples/s]

Map:   0%|          | 0/32 [00:00<?, ? examples/s]
Map: 100%|##########| 32/32 [00:00<00:00, 10310.17 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 300.11ba/s]

Map:   0%|          | 0/115 [00:00<?, ? examples/s]
Map: 100%|##########| 115/115 [00:00<00:00, 6295.78 examples/s]

Map:   0%|          | 0/115 [00:00<?, ? examples/s]
Map: 100%|##########| 115/115 [00:00<00:00, 7019.60 examples/s]

Map:   0%|          | 0/115 [00:00<?, ? examples/s]
Map: 100%|##########| 115/115 [00:00<00:00, 6476.78 examples/s]

Map:   0%|          | 0/115 [00:00<?, ? examples/s]
Map: 100%|##########| 115/115 [00:00<00:00, 20054.26 examples/s]

Filter:   0%|          | 0/115 [00:00<?, ? examples/s]
Filter: 100%|##########| 115/115 [00:00<00:00, 28736.67 examples/s]

Map:   0%|          | 0/32 [00:00<?, ? examples/s]
Map: 100%|##########| 32/32 [00:00<00:00, 6794.80 examples/s]INFO 2025-05-20 11:01:10,749 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.07%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 115.15ba/s]

Map:   0%|          | 0/330 [00:00<?, ? examples/s]
Map: 100%|##########| 330/330 [00:00<00:00, 7232.47 examples/s]

Map:   0%|          | 0/330 [00:00<?, ? examples/s]
Map: 100%|##########| 330/330 [00:00<00:00, 26169.30 examples/s]

Filter:   0%|          | 0/330 [00:00<?, ? examples/s]
Filter: 100%|##########| 330/330 [00:00<00:00, 39879.00 examples/s]

Map:   0%|          | 0/27 [00:00<?, ? examples/s]
Map: 100%|##########| 27/27 [00:00<00:00, 6402.79 examples/s]

Map:   0%|          | 0/27 [00:00<?, ? examples/s]
Map: 100%|##########| 27/27 [00:00<00:00, 9057.52 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 146.56ba/s]

Map:   0%|          | 0/330 [00:00<?, ? examples/s]
Map: 100%|##########| 330/330 [00:00<00:00, 7278.37 examples/s]

Map:   0%|          | 0/330 [00:00<?, ? examples/s]
Map: 100%|##########| 330/330 [00:00<00:00, 8228.33 examples/s]

Map:   0%|          | 0/330 [00:00<?, ? examples/s]
Map: 100%|##########| 330/330 [00:00<00:00, 7256.20 examples/s]

Map:   0%|          | 0/330 [00:00<?, ? examples/s]
Map: 100%|##########| 330/330 [00:00<00:00, 26131.75 examples/s]

Filter:   0%|          | 0/330 [00:00<?, ? examples/s]
Filter: 100%|##########| 330/330 [00:00<00:00, 40210.34 examples/s]

Map:   0%|          | 0/27 [00:00<?, ? examples/s]
Map: 100%|##########| 27/27 [00:00<00:00, 6152.34 examples/s]INFO 2025-05-20 11:01:11,070 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.17%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 45.90ba/s]

Map:   0%|          | 0/395 [00:00<?, ? examples/s]
Map: 100%|##########| 395/395 [00:00<00:00, 7000.93 examples/s]

Map:   0%|          | 0/395 [00:00<?, ? examples/s]
Map: 100%|##########| 395/395 [00:00<00:00, 26466.92 examples/s]

Filter:   0%|          | 0/395 [00:00<?, ? examples/s]
Filter: 100%|##########| 395/395 [00:00<00:00, 40616.57 examples/s]

Map:   0%|          | 0/13 [00:00<?, ? examples/s]
Map: 100%|##########| 13/13 [00:00<00:00, 4101.55 examples/s]

Map:   0%|          | 0/13 [00:00<?, ? examples/s]
Map: 100%|##########| 13/13 [00:00<00:00, 5308.73 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 74.69ba/s]

Map:   0%|          | 0/395 [00:00<?, ? examples/s]
Map: 100%|##########| 395/395 [00:00<00:00, 7068.02 examples/s]

Map:   0%|          | 0/395 [00:00<?, ? examples/s]
Map: 100%|##########| 395/395 [00:00<00:00, 7326.31 examples/s]

Map:   0%|          | 0/395 [00:00<?, ? examples/s]
Map: 100%|##########| 395/395 [00:00<00:00, 7040.80 examples/s]

Map:   0%|          | 0/395 [00:00<?, ? examples/s]
Map: 100%|##########| 395/395 [00:00<00:00, 25568.31 examples/s]

Filter:   0%|          | 0/395 [00:00<?, ? examples/s]
Filter: 100%|##########| 395/395 [00:00<00:00, 41446.73 examples/s]

Map:   0%|          | 0/13 [00:00<?, ? examples/s]
Map: 100%|##########| 13/13 [00:00<00:00, 3969.57 examples/s]INFO 2025-05-20 11:01:11,470 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.20%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 19.93ba/s]

Map:   0%|          | 0/747 [00:00<?, ? examples/s]
Map: 100%|##########| 747/747 [00:00<00:00, 7438.72 examples/s]
Map: 100%|##########| 747/747 [00:00<00:00, 7363.77 examples/s]

Map:   0%|          | 0/747 [00:00<?, ? examples/s]
Map: 100%|##########| 747/747 [00:00<00:00, 28734.70 examples/s]

Filter:   0%|          | 0/747 [00:00<?, ? examples/s]
Filter: 100%|##########| 747/747 [00:00<00:00, 44440.51 examples/s]

Map:   0%|          | 0/51 [00:00<?, ? examples/s]
Map: 100%|##########| 51/51 [00:00<00:00, 8152.35 examples/s]

Map:   0%|          | 0/51 [00:00<?, ? examples/s]
Map: 100%|##########| 51/51 [00:00<00:00, 13530.01 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 48.45ba/s]

Map:   0%|          | 0/747 [00:00<?, ? examples/s]
Map: 100%|##########| 747/747 [00:00<00:00, 7345.15 examples/s]
Map: 100%|##########| 747/747 [00:00<00:00, 7272.79 examples/s]

Map:   0%|          | 0/747 [00:00<?, ? examples/s]
Map: 100%|##########| 747/747 [00:00<00:00, 7321.12 examples/s]
Map: 100%|##########| 747/747 [00:00<00:00, 7254.57 examples/s]

Map:   0%|          | 0/747 [00:00<?, ? examples/s]
Map: 100%|##########| 747/747 [00:00<00:00, 7246.90 examples/s]
Map: 100%|##########| 747/747 [00:00<00:00, 7182.50 examples/s]

Map:   0%|          | 0/747 [00:00<?, ? examples/s]
Map: 100%|##########| 747/747 [00:00<00:00, 28235.69 examples/s]

Filter:   0%|          | 0/747 [00:00<?, ? examples/s]
Filter: 100%|##########| 747/747 [00:00<00:00, 43910.49 examples/s]

Map:   0%|          | 0/51 [00:00<?, ? examples/s]
Map: 100%|##########| 51/51 [00:00<00:00, 7917.88 examples/s]INFO 2025-05-20 11:01:12,163 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.39%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 30.06ba/s]

Map:   0%|          | 0/1391 [00:00<?, ? examples/s]
Map:  57%|#####7    | 794/1391 [00:00<00:00, 7884.96 examples/s]
Map: 100%|##########| 1391/1391 [00:00<00:00, 7388.12 examples/s]

Map:   0%|          | 0/1391 [00:00<?, ? examples/s]
Map: 100%|##########| 1391/1391 [00:00<00:00, 28503.55 examples/s]

Filter:   0%|          | 0/1391 [00:00<?, ? examples/s]
Filter: 100%|##########| 1391/1391 [00:00<00:00, 46622.74 examples/s]

Map:   0%|          | 0/115 [00:00<?, ? examples/s]
Map: 100%|##########| 115/115 [00:00<00:00, 9915.82 examples/s]

Map:   0%|          | 0/115 [00:00<?, ? examples/s]
Map: 100%|##########| 115/115 [00:00<00:00, 19131.56 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 56.92ba/s]

Map:   0%|          | 0/1391 [00:00<?, ? examples/s]
Map:  57%|#####6    | 791/1391 [00:00<00:00, 7846.98 examples/s]
Map: 100%|##########| 1391/1391 [00:00<00:00, 7398.64 examples/s]

Map:   0%|          | 0/1391 [00:00<?, ? examples/s]
Map:  64%|######4   | 893/1391 [00:00<00:00, 8876.82 examples/s]
Map: 100%|##########| 1391/1391 [00:00<00:00, 7545.91 examples/s]

Map:   0%|          | 0/1391 [00:00<?, ? examples/s]
Map:  57%|#####6    | 790/1391 [00:00<00:00, 7839.71 examples/s]
Map: 100%|##########| 1391/1391 [00:00<00:00, 7373.61 examples/s]

Map:   0%|          | 0/1391 [00:00<?, ? examples/s]
Map: 100%|##########| 1391/1391 [00:00<00:00, 29260.92 examples/s]

Filter:   0%|          | 0/1391 [00:00<?, ? examples/s]
Filter: 100%|##########| 1391/1391 [00:00<00:00, 47307.76 examples/s]

Map:   0%|          | 0/115 [00:00<?, ? examples/s]
Map: 100%|##########| 115/115 [00:00<00:00, 10146.51 examples/s]INFO 2025-05-20 11:01:13,349 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.74%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 32.30ba/s]

Map:   0%|          | 0/118 [00:00<?, ? examples/s]
Map: 100%|##########| 118/118 [00:00<00:00, 6248.14 examples/s]

Map:   0%|          | 0/118 [00:00<?, ? examples/s]
Map: 100%|##########| 118/118 [00:00<00:00, 19558.50 examples/s]

Filter:   0%|          | 0/118 [00:00<?, ? examples/s]
Filter: 100%|##########| 118/118 [00:00<00:00, 28225.14 examples/s]

Map:   0%|          | 0/41 [00:00<?, ? examples/s]
Map: 100%|##########| 41/41 [00:00<00:00, 7409.15 examples/s]

Map:   0%|          | 0/41 [00:00<?, ? examples/s]
Map: 100%|##########| 41/41 [00:00<00:00, 11938.80 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 241.76ba/s]

Map:   0%|          | 0/118 [00:00<?, ? examples/s]
Map: 100%|##########| 118/118 [00:00<00:00, 6366.12 examples/s]

Map:   0%|          | 0/118 [00:00<?, ? examples/s]
Map: 100%|##########| 118/118 [00:00<00:00, 7034.32 examples/s]

Map:   0%|          | 0/118 [00:00<?, ? examples/s]
Map: 100%|##########| 118/118 [00:00<00:00, 6497.85 examples/s]

Map:   0%|          | 0/118 [00:00<?, ? examples/s]
Map: 100%|##########| 118/118 [00:00<00:00, 19810.59 examples/s]

Filter:   0%|          | 0/118 [00:00<?, ? examples/s]
Filter: 100%|##########| 118/118 [00:00<00:00, 28769.86 examples/s]

Map:   0%|          | 0/41 [00:00<?, ? examples/s]
Map: 100%|##########| 41/41 [00:00<00:00, 7419.06 examples/s]INFO 2025-05-20 11:01:13,638 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.07%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 128.52ba/s]

Map:   0%|          | 0/714 [00:00<?, ? examples/s]
Map: 100%|##########| 714/714 [00:00<00:00, 7086.13 examples/s]
Map: 100%|##########| 714/714 [00:00<00:00, 7015.37 examples/s]

Map:   0%|          | 0/714 [00:00<?, ? examples/s]
Map: 100%|##########| 714/714 [00:00<00:00, 28123.52 examples/s]

Filter:   0%|          | 0/714 [00:00<?, ? examples/s]
Filter: 100%|##########| 714/714 [00:00<00:00, 44714.19 examples/s]

Map:   0%|          | 0/134 [00:00<?, ? examples/s]
Map: 100%|##########| 134/134 [00:00<00:00, 10013.13 examples/s]

Map:   0%|          | 0/134 [00:00<?, ? examples/s]
Map: 100%|##########| 134/134 [00:00<00:00, 20385.81 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 33.19ba/s]

Map:   0%|          | 0/714 [00:00<?, ? examples/s]
Map: 100%|##########| 714/714 [00:00<00:00, 6984.93 examples/s]
Map: 100%|##########| 714/714 [00:00<00:00, 6912.33 examples/s]

Map:   0%|          | 0/714 [00:00<?, ? examples/s]
Map: 100%|##########| 714/714 [00:00<00:00, 6704.89 examples/s]
Map: 100%|##########| 714/714 [00:00<00:00, 6644.70 examples/s]

Map:   0%|          | 0/714 [00:00<?, ? examples/s]
Map: 100%|##########| 714/714 [00:00<00:00, 7118.74 examples/s]
Map: 100%|##########| 714/714 [00:00<00:00, 7053.47 examples/s]

Map:   0%|          | 0/714 [00:00<?, ? examples/s]
Map: 100%|##########| 714/714 [00:00<00:00, 28925.12 examples/s]

Filter:   0%|          | 0/714 [00:00<?, ? examples/s]
Filter: 100%|##########| 714/714 [00:00<00:00, 44230.77 examples/s]

Map:   0%|          | 0/134 [00:00<?, ? examples/s]
Map: 100%|##########| 134/134 [00:00<00:00, 10043.72 examples/s]INFO 2025-05-20 11:01:14,323 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.40%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 24.09ba/s]

Map:   0%|          | 0/126 [00:00<?, ? examples/s]
Map: 100%|##########| 126/126 [00:00<00:00, 6512.09 examples/s]

Map:   0%|          | 0/126 [00:00<?, ? examples/s]
Map: 100%|##########| 126/126 [00:00<00:00, 20139.56 examples/s]

Filter:   0%|          | 0/126 [00:00<?, ? examples/s]
Filter: 100%|##########| 126/126 [00:00<00:00, 27310.34 examples/s]

Map:   0%|          | 0/30 [00:00<?, ? examples/s]
Map: 100%|##########| 30/30 [00:00<00:00, 6104.06 examples/s]

Map:   0%|          | 0/30 [00:00<?, ? examples/s]
Map: 100%|##########| 30/30 [00:00<00:00, 9915.61 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 348.91ba/s]

Map:   0%|          | 0/126 [00:00<?, ? examples/s]
Map: 100%|##########| 126/126 [00:00<00:00, 6528.34 examples/s]

Map:   0%|          | 0/126 [00:00<?, ? examples/s]
Map: 100%|##########| 126/126 [00:00<00:00, 7272.16 examples/s]

Map:   0%|          | 0/126 [00:00<?, ? examples/s]
Map: 100%|##########| 126/126 [00:00<00:00, 6650.09 examples/s]

Map:   0%|          | 0/126 [00:00<?, ? examples/s]
Map: 100%|##########| 126/126 [00:00<00:00, 20994.01 examples/s]

Filter:   0%|          | 0/126 [00:00<?, ? examples/s]
Filter: 100%|##########| 126/126 [00:00<00:00, 29957.62 examples/s]

Map:   0%|          | 0/30 [00:00<?, ? examples/s]
Map: 100%|##########| 30/30 [00:00<00:00, 6835.57 examples/s]INFO 2025-05-20 11:01:14,588 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.07%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 164.56ba/s]

Map:   0%|          | 0/1557 [00:00<?, ? examples/s]
Map:  50%|#####     | 786/1557 [00:00<00:00, 7795.59 examples/s]
Map: 100%|##########| 1557/1557 [00:00<00:00, 7228.12 examples/s]

Map:   0%|          | 0/1557 [00:00<?, ? examples/s]
Map: 100%|##########| 1557/1557 [00:00<00:00, 29301.84 examples/s]

Filter:   0%|          | 0/1557 [00:00<?, ? examples/s]
Filter: 100%|##########| 1557/1557 [00:00<00:00, 47131.77 examples/s]

Map:   0%|          | 0/70 [00:00<?, ? examples/s]
Map: 100%|##########| 70/70 [00:00<00:00, 8661.83 examples/s]

Map:   0%|          | 0/70 [00:00<?, ? examples/s]
Map: 100%|##########| 70/70 [00:00<00:00, 16155.02 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 44.88ba/s]

Map:   0%|          | 0/1557 [00:00<?, ? examples/s]
Map:  51%|#####     | 788/1557 [00:00<00:00, 7815.44 examples/s]
Map: 100%|##########| 1557/1557 [00:00<00:00, 7300.16 examples/s]

Map:   0%|          | 0/1557 [00:00<?, ? examples/s]
Map:  57%|#####7    | 889/1557 [00:00<00:00, 8839.45 examples/s]
Map: 100%|##########| 1557/1557 [00:00<00:00, 7216.60 examples/s]

Map:   0%|          | 0/1557 [00:00<?, ? examples/s]
Map:  50%|#####     | 785/1557 [00:00<00:00, 7781.70 examples/s]
Map: 100%|##########| 1557/1557 [00:00<00:00, 7269.17 examples/s]

Map:   0%|          | 0/1557 [00:00<?, ? examples/s]
Map: 100%|##########| 1557/1557 [00:00<00:00, 29497.99 examples/s]

Filter:   0%|          | 0/1557 [00:00<?, ? examples/s]
Filter: 100%|##########| 1557/1557 [00:00<00:00, 47140.62 examples/s]

Map:   0%|          | 0/70 [00:00<?, ? examples/s]
Map: 100%|##########| 70/70 [00:00<00:00, 8911.59 examples/s]INFO 2025-05-20 11:01:15,858 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.81%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format:  50%|#####     | 2/4 [00:00<00:00, 19.46ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 30.69ba/s]

Map:   0%|          | 0/351 [00:00<?, ? examples/s]
Map: 100%|##########| 351/351 [00:00<00:00, 6641.80 examples/s]

Map:   0%|          | 0/351 [00:00<?, ? examples/s]
Map: 100%|##########| 351/351 [00:00<00:00, 25679.86 examples/s]

Filter:   0%|          | 0/351 [00:00<?, ? examples/s]
Filter: 100%|##########| 351/351 [00:00<00:00, 37845.78 examples/s]

Map:   0%|          | 0/61 [00:00<?, ? examples/s]
Map: 100%|##########| 61/61 [00:00<00:00, 8204.61 examples/s]

Map:   0%|          | 0/61 [00:00<?, ? examples/s]
Map: 100%|##########| 61/61 [00:00<00:00, 14835.47 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 68.22ba/s]

Map:   0%|          | 0/351 [00:00<?, ? examples/s]
Map: 100%|##########| 351/351 [00:00<00:00, 6715.51 examples/s]

Map:   0%|          | 0/351 [00:00<?, ? examples/s]
Map: 100%|##########| 351/351 [00:00<00:00, 6611.02 examples/s]

Map:   0%|          | 0/351 [00:00<?, ? examples/s]
Map: 100%|##########| 351/351 [00:00<00:00, 6656.63 examples/s]

Map:   0%|          | 0/351 [00:00<?, ? examples/s]
Map: 100%|##########| 351/351 [00:00<00:00, 24809.58 examples/s]

Filter:   0%|          | 0/351 [00:00<?, ? examples/s]
Filter: 100%|##########| 351/351 [00:00<00:00, 38386.54 examples/s]

Map:   0%|          | 0/61 [00:00<?, ? examples/s]
Map: 100%|##########| 61/61 [00:00<00:00, 8210.93 examples/s]INFO 2025-05-20 11:01:16,369 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.19%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 21.79ba/s]

Map:   0%|          | 0/254 [00:00<?, ? examples/s]
Map: 100%|##########| 254/254 [00:00<00:00, 6578.69 examples/s]

Map:   0%|          | 0/254 [00:00<?, ? examples/s]
Map: 100%|##########| 254/254 [00:00<00:00, 23639.84 examples/s]

Filter:   0%|          | 0/254 [00:00<?, ? examples/s]
Filter: 100%|##########| 254/254 [00:00<00:00, 36282.17 examples/s]

Map:   0%|          | 0/53 [00:00<?, ? examples/s]
Map: 100%|##########| 53/53 [00:00<00:00, 8044.95 examples/s]

Map:   0%|          | 0/53 [00:00<?, ? examples/s]
Map: 100%|##########| 53/53 [00:00<00:00, 14209.80 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 84.90ba/s]

Map:   0%|          | 0/254 [00:00<?, ? examples/s]
Map: 100%|##########| 254/254 [00:00<00:00, 6589.43 examples/s]

Map:   0%|          | 0/254 [00:00<?, ? examples/s]
Map: 100%|##########| 254/254 [00:00<00:00, 6975.90 examples/s]

Map:   0%|          | 0/254 [00:00<?, ? examples/s]
Map: 100%|##########| 254/254 [00:00<00:00, 6692.13 examples/s]

Map:   0%|          | 0/254 [00:00<?, ? examples/s]
Map: 100%|##########| 254/254 [00:00<00:00, 24928.71 examples/s]

Filter:   0%|          | 0/254 [00:00<?, ? examples/s]
Filter: 100%|##########| 254/254 [00:00<00:00, 36943.97 examples/s]

Map:   0%|          | 0/53 [00:00<?, ? examples/s]
Map: 100%|##########| 53/53 [00:00<00:00, 8109.81 examples/s]INFO 2025-05-20 11:01:16,710 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.14%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 31.00ba/s]

Map:   0%|          | 0/381 [00:00<?, ? examples/s]
Map: 100%|##########| 381/381 [00:00<00:00, 7242.14 examples/s]

Map:   0%|          | 0/381 [00:00<?, ? examples/s]
Map: 100%|##########| 381/381 [00:00<00:00, 26182.62 examples/s]

Filter:   0%|          | 0/381 [00:00<?, ? examples/s]
Filter: 100%|##########| 381/381 [00:00<00:00, 41401.88 examples/s]

Map:   0%|          | 0/16 [00:00<?, ? examples/s]
Map: 100%|##########| 16/16 [00:00<00:00, 4540.21 examples/s]

Map:   0%|          | 0/16 [00:00<?, ? examples/s]
Map: 100%|##########| 16/16 [00:00<00:00, 6104.14 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 109.21ba/s]

Map:   0%|          | 0/381 [00:00<?, ? examples/s]
Map: 100%|##########| 381/381 [00:00<00:00, 7227.05 examples/s]

Map:   0%|          | 0/381 [00:00<?, ? examples/s]
Map: 100%|##########| 381/381 [00:00<00:00, 7660.44 examples/s]

Map:   0%|          | 0/381 [00:00<?, ? examples/s]
Map: 100%|##########| 381/381 [00:00<00:00, 7225.64 examples/s]

Map:   0%|          | 0/381 [00:00<?, ? examples/s]
Map: 100%|##########| 381/381 [00:00<00:00, 26041.39 examples/s]

Filter:   0%|          | 0/381 [00:00<?, ? examples/s]
Filter: 100%|##########| 381/381 [00:00<00:00, 40776.47 examples/s]

Map:   0%|          | 0/16 [00:00<?, ? examples/s]
Map: 100%|##########| 16/16 [00:00<00:00, 4812.74 examples/s]INFO 2025-05-20 11:01:17,095 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.20%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 36.18ba/s]

Map:   0%|          | 0/1368 [00:00<?, ? examples/s]
Map:  57%|#####7    | 781/1368 [00:00<00:00, 7750.29 examples/s]
Map: 100%|##########| 1368/1368 [00:00<00:00, 7100.66 examples/s]

Map:   0%|          | 0/1368 [00:00<?, ? examples/s]
Map: 100%|##########| 1368/1368 [00:00<00:00, 28496.83 examples/s]

Filter:   0%|          | 0/1368 [00:00<?, ? examples/s]
Filter: 100%|##########| 1368/1368 [00:00<00:00, 47397.57 examples/s]

Map:   0%|          | 0/63 [00:00<?, ? examples/s]
Map: 100%|##########| 63/63 [00:00<00:00, 8582.88 examples/s]

Map:   0%|          | 0/63 [00:00<?, ? examples/s]
Map: 100%|##########| 63/63 [00:00<00:00, 15517.13 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 47.13ba/s]

Map:   0%|          | 0/1368 [00:00<?, ? examples/s]
Map:  57%|#####7    | 786/1368 [00:00<00:00, 7796.10 examples/s]
Map: 100%|##########| 1368/1368 [00:00<00:00, 7189.98 examples/s]

Map:   0%|          | 0/1368 [00:00<?, ? examples/s]
Map:  65%|######4   | 887/1368 [00:00<00:00, 8821.68 examples/s]
Map: 100%|##########| 1368/1368 [00:00<00:00, 7117.09 examples/s]

Map:   0%|          | 0/1368 [00:00<?, ? examples/s]
Map:  58%|#####7    | 790/1368 [00:00<00:00, 7838.00 examples/s]
Map: 100%|##########| 1368/1368 [00:00<00:00, 7218.52 examples/s]

Map:   0%|          | 0/1368 [00:00<?, ? examples/s]
Map: 100%|##########| 1368/1368 [00:00<00:00, 28447.80 examples/s]

Filter:   0%|          | 0/1368 [00:00<?, ? examples/s]
Filter: 100%|##########| 1368/1368 [00:00<00:00, 46891.31 examples/s]

Map:   0%|          | 0/63 [00:00<?, ? examples/s]
Map: 100%|##########| 63/63 [00:00<00:00, 8693.29 examples/s]INFO 2025-05-20 11:01:18,243 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.71%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format:  67%|######6   | 2/3 [00:00<00:00, 19.07ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 23.88ba/s]

Map:   0%|          | 0/1476 [00:00<?, ? examples/s]
Map:  54%|#####4    | 803/1476 [00:00<00:00, 7961.35 examples/s]
Map: 100%|##########| 1476/1476 [00:00<00:00, 7527.78 examples/s]

Map:   0%|          | 0/1476 [00:00<?, ? examples/s]
Map: 100%|##########| 1476/1476 [00:00<00:00, 29185.47 examples/s]

Filter:   0%|          | 0/1476 [00:00<?, ? examples/s]
Filter: 100%|##########| 1476/1476 [00:00<00:00, 46967.19 examples/s]

Map:   0%|          | 0/71 [00:00<?, ? examples/s]
Map: 100%|##########| 71/71 [00:00<00:00, 9003.92 examples/s]

Map:   0%|          | 0/71 [00:00<?, ? examples/s]
Map: 100%|##########| 71/71 [00:00<00:00, 16195.10 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 62.37ba/s]

Map:   0%|          | 0/1476 [00:00<?, ? examples/s]
Map:  55%|#####4    | 805/1476 [00:00<00:00, 7997.48 examples/s]
Map: 100%|##########| 1476/1476 [00:00<00:00, 7497.82 examples/s]

Map:   0%|          | 0/1476 [00:00<?, ? examples/s]
Map:  62%|######1   | 914/1476 [00:00<00:00, 9090.96 examples/s]
Map: 100%|##########| 1476/1476 [00:00<00:00, 7989.61 examples/s]

Map:   0%|          | 0/1476 [00:00<?, ? examples/s]
Map:  55%|#####4    | 807/1476 [00:00<00:00, 8003.11 examples/s]
Map: 100%|##########| 1476/1476 [00:00<00:00, 7619.76 examples/s]

Map:   0%|          | 0/1476 [00:00<?, ? examples/s]
Map: 100%|##########| 1476/1476 [00:00<00:00, 29825.95 examples/s]

Filter:   0%|          | 0/1476 [00:00<?, ? examples/s]
Filter: 100%|##########| 1476/1476 [00:00<00:00, 47126.29 examples/s]

Map:   0%|          | 0/71 [00:00<?, ? examples/s]
Map: 100%|##########| 71/71 [00:00<00:00, 9109.96 examples/s]INFO 2025-05-20 11:01:19,504 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.77%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 44.79ba/s]

Map:   0%|          | 0/2315 [00:00<?, ? examples/s]
Map:  35%|###5      | 811/2315 [00:00<00:00, 8038.50 examples/s]
Map:  84%|########4 | 1956/2315 [00:00<00:00, 7757.48 examples/s]
Map: 100%|##########| 2315/2315 [00:00<00:00, 7424.70 examples/s]

Map:   0%|          | 0/2315 [00:00<?, ? examples/s]
Map: 100%|##########| 2315/2315 [00:00<00:00, 29808.39 examples/s]

Filter:   0%|          | 0/2315 [00:00<?, ? examples/s]
Filter: 100%|##########| 2315/2315 [00:00<00:00, 49336.23 examples/s]

Map:   0%|          | 0/127 [00:00<?, ? examples/s]
Map: 100%|##########| 127/127 [00:00<00:00, 10173.54 examples/s]

Map:   0%|          | 0/127 [00:00<?, ? examples/s]
Map: 100%|##########| 127/127 [00:00<00:00, 20500.18 examples/s]

Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 53.48ba/s]

Map:   0%|          | 0/2315 [00:00<?, ? examples/s]
Map:  35%|###5      | 811/2315 [00:00<00:00, 8048.58 examples/s]
Map:  84%|########4 | 1952/2315 [00:00<00:00, 7741.30 examples/s]
Map: 100%|##########| 2315/2315 [00:00<00:00, 7399.19 examples/s]

Map:   0%|          | 0/2315 [00:00<?, ? examples/s]
Map:  39%|###9      | 906/2315 [00:00<00:00, 9000.59 examples/s]
Map:  86%|########6 | 2000/2315 [00:00<00:00, 7460.89 examples/s]
Map: 100%|##########| 2315/2315 [00:00<00:00, 7618.40 examples/s]

Map:   0%|          | 0/2315 [00:00<?, ? examples/s]
Map:  35%|###4      | 807/2315 [00:00<00:00, 8013.99 examples/s]
Map:  84%|########4 | 1955/2315 [00:00<00:00, 7762.14 examples/s]
Map: 100%|##########| 2315/2315 [00:00<00:00, 7444.89 examples/s]

Map:   0%|          | 0/2315 [00:00<?, ? examples/s]
Map: 100%|##########| 2315/2315 [00:00<00:00, 30452.03 examples/s]

Filter:   0%|          | 0/2315 [00:00<?, ? examples/s]
Filter: 100%|##########| 2315/2315 [00:00<00:00, 49886.01 examples/s]

Map:   0%|          | 0/127 [00:00<?, ? examples/s]
Map: 100%|##########| 127/127 [00:00<00:00, 10300.03 examples/s]INFO 2025-05-20 11:01:21,380 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.21%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/5 [00:00<?, ?ba/s]
Creating json from Arrow format:  60%|######    | 3/5 [00:00<00:00, 23.07ba/s]
Creating json from Arrow format: 100%|##########| 5/5 [00:00<00:00, 28.76ba/s]

Map:   0%|          | 0/2213 [00:00<?, ? examples/s]
Map:  36%|###5      | 786/2213 [00:00<00:00, 7794.87 examples/s]
Map:  86%|########5 | 1903/2213 [00:00<00:00, 7548.60 examples/s]
Map: 100%|##########| 2213/2213 [00:00<00:00, 7245.61 examples/s]

Map:   0%|          | 0/2213 [00:00<?, ? examples/s]
Map: 100%|##########| 2213/2213 [00:00<00:00, 29981.67 examples/s]

Filter:   0%|          | 0/2213 [00:00<?, ? examples/s]
Filter: 100%|##########| 2213/2213 [00:00<00:00, 47887.54 examples/s]

Map:   0%|          | 0/121 [00:00<?, ? examples/s]
Map: 100%|##########| 121/121 [00:00<00:00, 9976.23 examples/s]

Map:   0%|          | 0/121 [00:00<?, ? examples/s]
Map: 100%|##########| 121/121 [00:00<00:00, 19934.44 examples/s]

Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 46.40ba/s]

Map:   0%|          | 0/2213 [00:00<?, ? examples/s]
Map:  36%|###5      | 795/2213 [00:00<00:00, 7890.00 examples/s]
Map:  87%|########6 | 1917/2213 [00:00<00:00, 7608.53 examples/s]
Map: 100%|##########| 2213/2213 [00:00<00:00, 7290.87 examples/s]

Map:   0%|          | 0/2213 [00:00<?, ? examples/s]
Map:  38%|###8      | 851/2213 [00:00<00:00, 8463.08 examples/s]
Map:  87%|########6 | 1917/2213 [00:00<00:00, 7533.47 examples/s]
Map: 100%|##########| 2213/2213 [00:00<00:00, 7009.75 examples/s]

Map:   0%|          | 0/2213 [00:00<?, ? examples/s]
Map:  36%|###5      | 792/2213 [00:00<00:00, 7866.01 examples/s]
Map:  86%|########6 | 1904/2213 [00:00<00:00, 7552.35 examples/s]
Map: 100%|##########| 2213/2213 [00:00<00:00, 7273.41 examples/s]

Map:   0%|          | 0/2213 [00:00<?, ? examples/s]
Map: 100%|##########| 2213/2213 [00:00<00:00, 28932.73 examples/s]

Filter:   0%|          | 0/2213 [00:00<?, ? examples/s]
Filter: 100%|##########| 2213/2213 [00:00<00:00, 46533.05 examples/s]

Map:   0%|          | 0/121 [00:00<?, ? examples/s]
Map: 100%|##########| 121/121 [00:00<00:00, 9804.13 examples/s]INFO 2025-05-20 11:01:23,358 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.15%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/5 [00:00<?, ?ba/s]
Creating json from Arrow format:  40%|####      | 2/5 [00:00<00:00, 17.56ba/s]
Creating json from Arrow format: 100%|##########| 5/5 [00:00<00:00, 26.31ba/s]

Map:   0%|          | 0/1454 [00:00<?, ? examples/s]
Map:  54%|#####4    | 787/1454 [00:00<00:00, 7816.82 examples/s]
Map: 100%|##########| 1454/1454 [00:00<00:00, 7202.39 examples/s]

Map:   0%|          | 0/1454 [00:00<?, ? examples/s]
Map: 100%|##########| 1454/1454 [00:00<00:00, 29229.86 examples/s]

Filter:   0%|          | 0/1454 [00:00<?, ? examples/s]
Filter: 100%|##########| 1454/1454 [00:00<00:00, 47065.91 examples/s]

Map:   0%|          | 0/70 [00:00<?, ? examples/s]
Map: 100%|##########| 70/70 [00:00<00:00, 8725.67 examples/s]

Map:   0%|          | 0/70 [00:00<?, ? examples/s]
Map: 100%|##########| 70/70 [00:00<00:00, 16047.29 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 45.67ba/s]

Map:   0%|          | 0/1454 [00:00<?, ? examples/s]
Map:  55%|#####4    | 796/1454 [00:00<00:00, 7894.41 examples/s]
Map: 100%|##########| 1454/1454 [00:00<00:00, 7255.23 examples/s]

Map:   0%|          | 0/1454 [00:00<?, ? examples/s]
Map:  62%|######1   | 901/1454 [00:00<00:00, 8963.06 examples/s]
Map: 100%|##########| 1454/1454 [00:00<00:00, 3340.21 examples/s]

Map:   0%|          | 0/1454 [00:00<?, ? examples/s]
Map:  54%|#####4    | 790/1454 [00:00<00:00, 7837.45 examples/s]
Map: 100%|##########| 1454/1454 [00:00<00:00, 7231.10 examples/s]

Map:   0%|          | 0/1454 [00:00<?, ? examples/s]
Map: 100%|##########| 1454/1454 [00:00<00:00, 28960.71 examples/s]

Filter:   0%|          | 0/1454 [00:00<?, ? examples/s]
Filter: 100%|##########| 1454/1454 [00:00<00:00, 47270.57 examples/s]

Map:   0%|          | 0/70 [00:00<?, ? examples/s]
Map: 100%|##########| 70/70 [00:00<00:00, 8844.48 examples/s]INFO 2025-05-20 11:01:24,964 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.76%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format:  67%|######6   | 2/3 [00:00<00:00, 18.78ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 22.09ba/s]

Map:   0%|          | 0/7161 [00:00<?, ? examples/s]
Map:  11%|#1        | 792/7161 [00:00<00:00, 7857.73 examples/s]
Map:  27%|##6       | 1925/7161 [00:00<00:00, 7641.73 examples/s]
Map:  42%|####1     | 3000/7161 [00:00<00:00, 7279.41 examples/s]
Map:  53%|#####3    | 3798/7161 [00:00<00:00, 7496.02 examples/s]
Map:  69%|######9   | 4948/7161 [00:00<00:00, 7563.15 examples/s]
Map:  84%|########3 | 6000/7161 [00:00<00:00, 7239.13 examples/s]
Map:  95%|#########4| 6791/7161 [00:00<00:00, 7407.81 examples/s]
Map: 100%|##########| 7161/7161 [00:00<00:00, 7359.90 examples/s]

Map:   0%|          | 0/7161 [00:00<?, ? examples/s]
Map:  42%|####1     | 3000/7161 [00:00<00:00, 29276.13 examples/s]
Map:  84%|########3 | 6000/7161 [00:00<00:00, 29507.58 examples/s]
Map: 100%|##########| 7161/7161 [00:00<00:00, 29603.20 examples/s]

Filter:   0%|          | 0/7161 [00:00<?, ? examples/s]
Filter:  84%|########3 | 6000/7161 [00:00<00:00, 50033.55 examples/s]
Filter: 100%|##########| 7161/7161 [00:00<00:00, 49805.26 examples/s]

Map:   0%|          | 0/807 [00:00<?, ? examples/s]
Map: 100%|##########| 807/807 [00:00<00:00, 11913.41 examples/s]

Map:   0%|          | 0/807 [00:00<?, ? examples/s]
Map: 100%|##########| 807/807 [00:00<00:00, 28145.24 examples/s]

Creating json from Arrow format:   0%|          | 0/8 [00:00<?, ?ba/s]
Creating json from Arrow format:  62%|######2   | 5/8 [00:00<00:00, 41.18ba/s]
Creating json from Arrow format: 100%|##########| 8/8 [00:00<00:00, 38.46ba/s]

Map:   0%|          | 0/7161 [00:00<?, ? examples/s]
Map:  11%|#1        | 788/7161 [00:00<00:00, 7817.44 examples/s]
Map:  27%|##6       | 1928/7161 [00:00<00:00, 7662.90 examples/s]
Map:  42%|####1     | 3000/7161 [00:00<00:00, 7299.01 examples/s]
Map:  53%|#####2    | 3792/7161 [00:00<00:00, 7493.08 examples/s]
Map:  69%|######8   | 4931/7161 [00:00<00:00, 7531.29 examples/s]
Map:  84%|########3 | 6000/7161 [00:00<00:00, 7239.44 examples/s]
Map:  95%|#########4| 6786/7161 [00:00<00:00, 7394.82 examples/s]
Map: 100%|##########| 7161/7161 [00:00<00:00, 7351.48 examples/s]

Map:   0%|          | 0/7161 [00:00<?, ? examples/s]
Map:   9%|9         | 670/7161 [00:00<00:00, 6662.16 examples/s]
Map:  19%|#8        | 1340/7161 [00:00<00:00, 6034.60 examples/s]
Map:  28%|##7       | 2000/7161 [00:00<00:00, 5758.35 examples/s]
Map:  37%|###7      | 2677/7161 [00:00<00:00, 6120.18 examples/s]
Map:  46%|####6     | 3329/7161 [00:00<00:00, 5772.13 examples/s]
Map:  56%|#####5    | 3994/7161 [00:00<00:00, 6041.42 examples/s]
Map:  68%|######7   | 4861/7161 [00:00<00:00, 5933.58 examples/s]
Map:  80%|########  | 5729/7161 [00:00<00:00, 5876.53 examples/s]
Map:  93%|#########2| 6653/7161 [00:01<00:00, 5640.65 examples/s]
Map: 100%|##########| 7161/7161 [00:01<00:00, 5755.67 examples/s]

Map:   0%|          | 0/7161 [00:00<?, ? examples/s]
Map:  11%|#1        | 788/7161 [00:00<00:00, 7821.05 examples/s]
Map:  27%|##6       | 1922/7161 [00:00<00:00, 7638.87 examples/s]
Map:  42%|####1     | 3000/7161 [00:00<00:00, 7232.54 examples/s]
Map:  53%|#####2    | 3790/7161 [00:00<00:00, 7440.51 examples/s]
Map:  69%|######8   | 4920/7161 [00:00<00:00, 7476.03 examples/s]
Map:  84%|########3 | 6000/7161 [00:00<00:00, 7181.31 examples/s]
Map:  95%|#########4| 6794/7161 [00:00<00:00, 7370.87 examples/s]
Map: 100%|##########| 7161/7161 [00:00<00:00, 7287.66 examples/s]

Map:   0%|          | 0/7161 [00:00<?, ? examples/s]
Map:  42%|####1     | 3000/7161 [00:00<00:00, 28717.82 examples/s]
Map:  84%|########3 | 6000/7161 [00:00<00:00, 29300.79 examples/s]
Map: 100%|##########| 7161/7161 [00:00<00:00, 29297.38 examples/s]

Filter:   0%|          | 0/7161 [00:00<?, ? examples/s]
Filter:  84%|########3 | 6000/7161 [00:00<00:00, 51054.79 examples/s]
Filter: 100%|##########| 7161/7161 [00:00<00:00, 50695.16 examples/s]

Map:   0%|          | 0/807 [00:00<?, ? examples/s]
Map: 100%|##########| 807/807 [00:00<00:00, 11785.57 examples/s]

Creating json from Arrow format:   0%|          | 0/16 [00:00<?, ?ba/s]
Creating json from Arrow format:  12%|#2        | 2/16 [00:00<00:00, 18.22ba/s]
Creating json from Arrow format:  25%|##5       | 4/16 [00:00<00:00, 18.39ba/s]
Creating json from Arrow format:  38%|###7      | 6/16 [00:00<00:00, 18.47ba/s]
Creating json from Arrow format:  56%|#####6    | 9/16 [00:00<00:00, 21.51ba/s]
Creating json from Arrow format:  88%|########7 | 14/16 [00:00<00:00, 28.65ba/s]
Creating json from Arrow format: 100%|##########| 16/16 [00:00<00:00, 26.77ba/s]

Map:   0%|          | 0/257 [00:00<?, ? examples/s]
Map: 100%|##########| 257/257 [00:00<00:00, 6849.87 examples/s]

Map:   0%|          | 0/257 [00:00<?, ? examples/s]
Map: 100%|##########| 257/257 [00:00<00:00, 24458.53 examples/s]

Filter:   0%|          | 0/257 [00:00<?, ? examples/s]
Filter: 100%|##########| 257/257 [00:00<00:00, 37720.41 examples/s]

Map:   0%|          | 0/11 [00:00<?, ? examples/s]
Map: 100%|##########| 11/11 [00:00<00:00, 3794.50 examples/s]

Map:   0%|          | 0/11 [00:00<?, ? examples/s]
Map: 100%|##########| 11/11 [00:00<00:00, 4475.44 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 154.37ba/s]

Map:   0%|          | 0/257 [00:00<?, ? examples/s]
Map: 100%|##########| 257/257 [00:00<00:00, 6941.13 examples/s]

Map:   0%|          | 0/257 [00:00<?, ? examples/s]
Map: 100%|##########| 257/257 [00:00<00:00, 7635.89 examples/s]

Map:   0%|          | 0/257 [00:00<?, ? examples/s]
Map: 100%|##########| 257/257 [00:00<00:00, 7063.01 examples/s]

Map:   0%|          | 0/257 [00:00<?, ? examples/s]
Map: 100%|##########| 257/257 [00:00<00:00, 25166.02 examples/s]

Filter:   0%|          | 0/257 [00:00<?, ? examples/s]
Filter: 100%|##########| 257/257 [00:00<00:00, 37705.90 examples/s]

Map:   0%|          | 0/11 [00:00<?, ? examples/s]
Map: 100%|##########| 11/11 [00:00<00:00, 3736.42 examples/s]INFO 2025-05-20 11:01:31,914 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.13%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 48.44ba/s]

Map:   0%|          | 0/515 [00:00<?, ? examples/s]
Map: 100%|##########| 515/515 [00:00<00:00, 7422.35 examples/s]

Map:   0%|          | 0/515 [00:00<?, ? examples/s]
Map: 100%|##########| 515/515 [00:00<00:00, 27969.63 examples/s]

Filter:   0%|          | 0/515 [00:00<?, ? examples/s]
Filter: 100%|##########| 515/515 [00:00<00:00, 43561.15 examples/s]

Map:   0%|          | 0/23 [00:00<?, ? examples/s]
Map: 100%|##########| 23/23 [00:00<00:00, 5872.59 examples/s]

Map:   0%|          | 0/23 [00:00<?, ? examples/s]
Map: 100%|##########| 23/23 [00:00<00:00, 8093.04 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 94.09ba/s]

Map:   0%|          | 0/515 [00:00<?, ? examples/s]
Map: 100%|##########| 515/515 [00:00<00:00, 7404.51 examples/s]

Map:   0%|          | 0/515 [00:00<?, ? examples/s]
Map: 100%|##########| 515/515 [00:00<00:00, 8366.09 examples/s]

Map:   0%|          | 0/515 [00:00<?, ? examples/s]
Map: 100%|##########| 515/515 [00:00<00:00, 7552.07 examples/s]

Map:   0%|          | 0/515 [00:00<?, ? examples/s]
Map: 100%|##########| 515/515 [00:00<00:00, 28761.39 examples/s]

Filter:   0%|          | 0/515 [00:00<?, ? examples/s]
Filter: 100%|##########| 515/515 [00:00<00:00, 44783.07 examples/s]

Map:   0%|          | 0/23 [00:00<?, ? examples/s]
Map: 100%|##########| 23/23 [00:00<00:00, 5763.13 examples/s]INFO 2025-05-20 11:01:32,368 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.27%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 68.07ba/s]

Map:   0%|          | 0/66 [00:00<?, ? examples/s]
Map: 100%|##########| 66/66 [00:00<00:00, 5898.78 examples/s]

Map:   0%|          | 0/66 [00:00<?, ? examples/s]
Map: 100%|##########| 66/66 [00:00<00:00, 15852.03 examples/s]

Filter:   0%|          | 0/66 [00:00<?, ? examples/s]
Filter: 100%|##########| 66/66 [00:00<00:00, 21477.54 examples/s]

Map:   0%|          | 0/22 [00:00<?, ? examples/s]
Map: 100%|##########| 22/22 [00:00<00:00, 5597.49 examples/s]

Map:   0%|          | 0/22 [00:00<?, ? examples/s]
Map: 100%|##########| 22/22 [00:00<00:00, 8131.36 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 564.05ba/s]

Map:   0%|          | 0/66 [00:00<?, ? examples/s]
Map: 100%|##########| 66/66 [00:00<00:00, 5853.63 examples/s]

Map:   0%|          | 0/66 [00:00<?, ? examples/s]
Map: 100%|##########| 66/66 [00:00<00:00, 6509.22 examples/s]

Map:   0%|          | 0/66 [00:00<?, ? examples/s]
Map: 100%|##########| 66/66 [00:00<00:00, 5881.24 examples/s]

Map:   0%|          | 0/66 [00:00<?, ? examples/s]
Map: 100%|##########| 66/66 [00:00<00:00, 15661.01 examples/s]

Filter:   0%|          | 0/66 [00:00<?, ? examples/s]
Filter: 100%|##########| 66/66 [00:00<00:00, 21577.99 examples/s]

Map:   0%|          | 0/22 [00:00<?, ? examples/s]
Map: 100%|##########| 22/22 [00:00<00:00, 5668.67 examples/s]INFO 2025-05-20 11:01:32,533 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.04%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 306.33ba/s]

Map:   0%|          | 0/7327 [00:00<?, ? examples/s]
Map:  11%|#         | 777/7327 [00:00<00:00, 7705.92 examples/s]
Map:  26%|##5       | 1886/7327 [00:00<00:00, 7485.76 examples/s]
Map:  41%|####      | 3000/7327 [00:00<00:00, 7230.53 examples/s]
Map:  52%|#####1    | 3797/7327 [00:00<00:00, 7457.65 examples/s]
Map:  67%|######7   | 4922/7327 [00:00<00:00, 7471.06 examples/s]
Map:  82%|########1 | 6000/7327 [00:00<00:00, 7217.99 examples/s]
Map:  93%|#########2| 6785/7327 [00:00<00:00, 7374.69 examples/s]
Map: 100%|##########| 7327/7327 [00:01<00:00, 7321.34 examples/s]

Map:   0%|          | 0/7327 [00:00<?, ? examples/s]
Map:  41%|####      | 3000/7327 [00:00<00:00, 29470.25 examples/s]
Map:  82%|########1 | 6000/7327 [00:00<00:00, 29583.72 examples/s]
Map: 100%|##########| 7327/7327 [00:00<00:00, 29772.89 examples/s]

Filter:   0%|          | 0/7327 [00:00<?, ? examples/s]
Filter:  82%|########1 | 6000/7327 [00:00<00:00, 50668.90 examples/s]
Filter: 100%|##########| 7327/7327 [00:00<00:00, 50448.17 examples/s]

Map:   0%|          | 0/477 [00:00<?, ? examples/s]
Map: 100%|##########| 477/477 [00:00<00:00, 11314.74 examples/s]

Map:   0%|          | 0/477 [00:00<?, ? examples/s]
Map: 100%|##########| 477/477 [00:00<00:00, 27022.02 examples/s]

Creating json from Arrow format:   0%|          | 0/8 [00:00<?, ?ba/s]
Creating json from Arrow format:  50%|#####     | 4/8 [00:00<00:00, 37.19ba/s]
Creating json from Arrow format: 100%|##########| 8/8 [00:00<00:00, 38.63ba/s]
Creating json from Arrow format: 100%|##########| 8/8 [00:00<00:00, 38.37ba/s]

Map:   0%|          | 0/7327 [00:00<?, ? examples/s]
Map:  11%|#         | 785/7327 [00:00<00:00, 7786.57 examples/s]
Map:  26%|##6       | 1909/7327 [00:00<00:00, 7580.29 examples/s]
Map:  41%|####      | 3000/7327 [00:00<00:00, 7208.08 examples/s]
Map:  52%|#####1    | 3785/7327 [00:00<00:00, 7406.50 examples/s]
Map:  67%|######7   | 4912/7327 [00:00<00:00, 7445.00 examples/s]
Map:  82%|########1 | 6000/7327 [00:00<00:00, 7217.89 examples/s]
Map:  93%|#########2| 6782/7327 [00:00<00:00, 7366.40 examples/s]
Map: 100%|##########| 7327/7327 [00:01<00:00, 7317.02 examples/s]

Map:   0%|          | 0/7327 [00:00<?, ? examples/s]
Map:  10%|#         | 758/7327 [00:00<00:00, 7525.98 examples/s]
Map:  24%|##3       | 1758/7327 [00:00<00:00, 6898.85 examples/s]
Map:  38%|###7      | 2767/7327 [00:00<00:00, 6781.68 examples/s]
Map:  51%|#####1    | 3762/7327 [00:00<00:00, 6657.82 examples/s]
Map:  66%|######5   | 4809/7327 [00:00<00:00, 6773.46 examples/s]
Map:  79%|#######9  | 5815/7327 [00:00<00:00, 6748.46 examples/s]
Map:  92%|#########2| 6751/7327 [00:01<00:00, 6517.74 examples/s]
Map: 100%|##########| 7327/7327 [00:01<00:00, 6581.70 examples/s]

Map:   0%|          | 0/7327 [00:00<?, ? examples/s]
Map:  11%|#         | 786/7327 [00:00<00:00, 7801.60 examples/s]
Map:  26%|##5       | 1903/7327 [00:00<00:00, 7555.32 examples/s]
Map:  41%|####      | 3000/7327 [00:00<00:00, 7197.24 examples/s]
Map:  52%|#####1    | 3786/7327 [00:00<00:00, 7400.87 examples/s]
Map:  67%|######7   | 4920/7327 [00:00<00:00, 7461.74 examples/s]
Map:  82%|########1 | 6000/7327 [00:00<00:00, 7232.23 examples/s]
Map:  93%|#########2| 6783/7327 [00:00<00:00, 7380.93 examples/s]
Map: 100%|##########| 7327/7327 [00:01<00:00, 7326.48 examples/s]

Map:   0%|          | 0/7327 [00:00<?, ? examples/s]
Map:  41%|####      | 3000/7327 [00:00<00:00, 29297.74 examples/s]
Map:  82%|########1 | 6000/7327 [00:00<00:00, 29383.10 examples/s]
Map: 100%|##########| 7327/7327 [00:00<00:00, 29592.70 examples/s]

Filter:   0%|          | 0/7327 [00:00<?, ? examples/s]
Filter:  82%|########1 | 6000/7327 [00:00<00:00, 50355.31 examples/s]
Filter: 100%|##########| 7327/7327 [00:00<00:00, 50137.31 examples/s]

Map:   0%|          | 0/477 [00:00<?, ? examples/s]
Map: 100%|##########| 477/477 [00:00<00:00, 11540.29 examples/s]

Creating json from Arrow format:   0%|          | 0/16 [00:00<?, ?ba/s]
Creating json from Arrow format:  12%|#2        | 2/16 [00:00<00:00, 18.62ba/s]
Creating json from Arrow format:  25%|##5       | 4/16 [00:00<00:00, 18.28ba/s]
Creating json from Arrow format:  38%|###7      | 6/16 [00:00<00:00, 17.96ba/s]
Creating json from Arrow format:  56%|#####6    | 9/16 [00:00<00:00, 20.50ba/s]
Creating json from Arrow format:  81%|########1 | 13/16 [00:00<00:00, 26.17ba/s]
Creating json from Arrow format: 100%|##########| 16/16 [00:00<00:00, 25.63ba/s]

Map:   0%|          | 0/193 [00:00<?, ? examples/s]
Map: 100%|##########| 193/193 [00:00<00:00, 6382.97 examples/s]

Map:   0%|          | 0/193 [00:00<?, ? examples/s]
Map: 100%|##########| 193/193 [00:00<00:00, 22597.86 examples/s]

Filter:   0%|          | 0/193 [00:00<?, ? examples/s]
Filter: 100%|##########| 193/193 [00:00<00:00, 33451.82 examples/s]

Map:   0%|          | 0/29 [00:00<?, ? examples/s]
Map: 100%|##########| 29/29 [00:00<00:00, 6453.12 examples/s]

Map:   0%|          | 0/29 [00:00<?, ? examples/s]
Map: 100%|##########| 29/29 [00:00<00:00, 9628.34 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 125.65ba/s]

Map:   0%|          | 0/193 [00:00<?, ? examples/s]
Map: 100%|##########| 193/193 [00:00<00:00, 6437.59 examples/s]

Map:   0%|          | 0/193 [00:00<?, ? examples/s]
Map: 100%|##########| 193/193 [00:00<00:00, 6798.76 examples/s]

Map:   0%|          | 0/193 [00:00<?, ? examples/s]
Map: 100%|##########| 193/193 [00:00<00:00, 6526.23 examples/s]

Map:   0%|          | 0/193 [00:00<?, ? examples/s]
Map: 100%|##########| 193/193 [00:00<00:00, 22781.02 examples/s]

Filter:   0%|          | 0/193 [00:00<?, ? examples/s]
Filter: 100%|##########| 193/193 [00:00<00:00, 33796.79 examples/s]

Map:   0%|          | 0/29 [00:00<?, ? examples/s]
Map: 100%|##########| 29/29 [00:00<00:00, 6257.90 examples/s]INFO 2025-05-20 11:01:39,298 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.11%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 37.71ba/s]

Map:   0%|          | 0/203 [00:00<?, ? examples/s]
Map: 100%|##########| 203/203 [00:00<00:00, 6777.01 examples/s]

Map:   0%|          | 0/203 [00:00<?, ? examples/s]
Map: 100%|##########| 203/203 [00:00<00:00, 23721.72 examples/s]

Filter:   0%|          | 0/203 [00:00<?, ? examples/s]
Filter: 100%|##########| 203/203 [00:00<00:00, 35059.03 examples/s]

Map:   0%|          | 0/48 [00:00<?, ? examples/s]
Map: 100%|##########| 48/48 [00:00<00:00, 7939.37 examples/s]

Map:   0%|          | 0/48 [00:00<?, ? examples/s]
Map: 100%|##########| 48/48 [00:00<00:00, 13730.25 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 148.11ba/s]

Map:   0%|          | 0/203 [00:00<?, ? examples/s]
Map: 100%|##########| 203/203 [00:00<00:00, 6757.70 examples/s]

Map:   0%|          | 0/203 [00:00<?, ? examples/s]
Map: 100%|##########| 203/203 [00:00<00:00, 7196.60 examples/s]

Map:   0%|          | 0/203 [00:00<?, ? examples/s]
Map: 100%|##########| 203/203 [00:00<00:00, 6839.34 examples/s]

Map:   0%|          | 0/203 [00:00<?, ? examples/s]
Map: 100%|##########| 203/203 [00:00<00:00, 23670.28 examples/s]

Filter:   0%|          | 0/203 [00:00<?, ? examples/s]
Filter: 100%|##########| 203/203 [00:00<00:00, 35428.11 examples/s]

Map:   0%|          | 0/48 [00:00<?, ? examples/s]
Map: 100%|##########| 48/48 [00:00<00:00, 5048.31 examples/s]INFO 2025-05-20 11:01:39,570 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.12%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 60.62ba/s]

Map:   0%|          | 0/106 [00:00<?, ? examples/s]
Map: 100%|##########| 106/106 [00:00<00:00, 6129.40 examples/s]

Map:   0%|          | 0/106 [00:00<?, ? examples/s]
Map: 100%|##########| 106/106 [00:00<00:00, 18722.99 examples/s]

Filter:   0%|          | 0/106 [00:00<?, ? examples/s]
Filter: 100%|##########| 106/106 [00:00<00:00, 28021.95 examples/s]

Map:   0%|          | 0/9 [00:00<?, ? examples/s]
Map: 100%|##########| 9/9 [00:00<00:00, 3179.11 examples/s]

Map:   0%|          | 0/9 [00:00<?, ? examples/s]
Map: 100%|##########| 9/9 [00:00<00:00, 3814.54 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 307.12ba/s]

Map:   0%|          | 0/106 [00:00<?, ? examples/s]
Map: 100%|##########| 106/106 [00:00<00:00, 6045.06 examples/s]

Map:   0%|          | 0/106 [00:00<?, ? examples/s]
Map: 100%|##########| 106/106 [00:00<00:00, 6750.11 examples/s]

Map:   0%|          | 0/106 [00:00<?, ? examples/s]
Map: 100%|##########| 106/106 [00:00<00:00, 6272.43 examples/s]

Map:   0%|          | 0/106 [00:00<?, ? examples/s]
Map: 100%|##########| 106/106 [00:00<00:00, 19256.59 examples/s]

Filter:   0%|          | 0/106 [00:00<?, ? examples/s]
Filter: 100%|##########| 106/106 [00:00<00:00, 27107.87 examples/s]

Map:   0%|          | 0/9 [00:00<?, ? examples/s]
Map: 100%|##########| 9/9 [00:00<00:00, 3035.44 examples/s]INFO 2025-05-20 11:01:39,751 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.06%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 123.98ba/s]

Map:   0%|          | 0/1052 [00:00<?, ? examples/s]
Map:  75%|#######4  | 786/1052 [00:00<00:00, 7803.63 examples/s]
Map: 100%|##########| 1052/1052 [00:00<00:00, 7191.37 examples/s]

Map:   0%|          | 0/1052 [00:00<?, ? examples/s]
Map:  62%|######2   | 656/1052 [00:00<00:00, 2686.05 examples/s]
Map: 100%|##########| 1052/1052 [00:00<00:00, 3939.21 examples/s]

Filter:   0%|          | 0/1052 [00:00<?, ? examples/s]
Filter: 100%|##########| 1052/1052 [00:00<00:00, 42643.08 examples/s]

Map:   0%|          | 0/51 [00:00<?, ? examples/s]
Map: 100%|##########| 51/51 [00:00<00:00, 7968.91 examples/s]

Map:   0%|          | 0/51 [00:00<?, ? examples/s]
Map: 100%|##########| 51/51 [00:00<00:00, 13846.17 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 57.39ba/s]

Map:   0%|          | 0/1052 [00:00<?, ? examples/s]
Map:  74%|#######4  | 782/1052 [00:00<00:00, 7758.47 examples/s]
Map: 100%|##########| 1052/1052 [00:00<00:00, 7157.69 examples/s]

Map:   0%|          | 0/1052 [00:00<?, ? examples/s]
Map:  84%|########3 | 883/1052 [00:00<00:00, 8781.60 examples/s]
Map: 100%|##########| 1052/1052 [00:00<00:00, 7253.91 examples/s]

Map:   0%|          | 0/1052 [00:00<?, ? examples/s]
Map:  74%|#######4  | 781/1052 [00:00<00:00, 7748.14 examples/s]
Map: 100%|##########| 1052/1052 [00:00<00:00, 7171.59 examples/s]

Map:   0%|          | 0/1052 [00:00<?, ? examples/s]
Map: 100%|##########| 1052/1052 [00:00<00:00, 28143.77 examples/s]

Filter:   0%|          | 0/1052 [00:00<?, ? examples/s]
Filter: 100%|##########| 1052/1052 [00:00<00:00, 44086.60 examples/s]

Map:   0%|          | 0/51 [00:00<?, ? examples/s]
Map: 100%|##########| 51/51 [00:00<00:00, 8034.16 examples/s]INFO 2025-05-20 11:01:40,882 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.55%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 31.68ba/s]

Map:   0%|          | 0/1006 [00:00<?, ? examples/s]
Map:  78%|#######8  | 789/1006 [00:00<00:00, 7827.77 examples/s]
Map: 100%|##########| 1006/1006 [00:00<00:00, 7285.17 examples/s]

Map:   0%|          | 0/1006 [00:00<?, ? examples/s]
Map: 100%|##########| 1006/1006 [00:00<00:00, 27783.43 examples/s]

Filter:   0%|          | 0/1006 [00:00<?, ? examples/s]
Filter: 100%|##########| 1006/1006 [00:00<00:00, 46230.63 examples/s]

Map:   0%|          | 0/51 [00:00<?, ? examples/s]
Map: 100%|##########| 51/51 [00:00<00:00, 7767.79 examples/s]

Map:   0%|          | 0/51 [00:00<?, ? examples/s]
Map: 100%|##########| 51/51 [00:00<00:00, 13474.61 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 72.77ba/s]

Map:   0%|          | 0/1006 [00:00<?, ? examples/s]
Map:  78%|#######7  | 782/1006 [00:00<00:00, 7758.15 examples/s]
Map: 100%|##########| 1006/1006 [00:00<00:00, 7259.34 examples/s]

Map:   0%|          | 0/1006 [00:00<?, ? examples/s]
Map:  91%|######### | 914/1006 [00:00<00:00, 9088.07 examples/s]
Map: 100%|##########| 1006/1006 [00:00<00:00, 7691.40 examples/s]

Map:   0%|          | 0/1006 [00:00<?, ? examples/s]
Map:  79%|#######8  | 794/1006 [00:00<00:00, 7883.30 examples/s]
Map: 100%|##########| 1006/1006 [00:00<00:00, 7305.81 examples/s]

Map:   0%|          | 0/1006 [00:00<?, ? examples/s]
Map: 100%|##########| 1006/1006 [00:00<00:00, 28131.11 examples/s]

Filter:   0%|          | 0/1006 [00:00<?, ? examples/s]
Filter: 100%|##########| 1006/1006 [00:00<00:00, 45455.20 examples/s]

Map:   0%|          | 0/51 [00:00<?, ? examples/s]
Map: 100%|##########| 51/51 [00:00<00:00, 7998.41 examples/s]INFO 2025-05-20 11:01:41,796 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.52%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 37.27ba/s]

Map:   0%|          | 0/543 [00:00<?, ? examples/s]
Map: 100%|##########| 543/543 [00:00<00:00, 7356.31 examples/s]

Map:   0%|          | 0/543 [00:00<?, ? examples/s]
Map: 100%|##########| 543/543 [00:00<00:00, 28346.24 examples/s]

Filter:   0%|          | 0/543 [00:00<?, ? examples/s]
Filter: 100%|##########| 543/543 [00:00<00:00, 43798.21 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 5313.28 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 7447.93 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 91.17ba/s]

Map:   0%|          | 0/543 [00:00<?, ? examples/s]
Map: 100%|##########| 543/543 [00:00<00:00, 7410.87 examples/s]

Map:   0%|          | 0/543 [00:00<?, ? examples/s]
Map: 100%|##########| 543/543 [00:00<00:00, 8125.63 examples/s]

Map:   0%|          | 0/543 [00:00<?, ? examples/s]
Map: 100%|##########| 543/543 [00:00<00:00, 7520.65 examples/s]

Map:   0%|          | 0/543 [00:00<?, ? examples/s]
Map: 100%|##########| 543/543 [00:00<00:00, 28829.57 examples/s]

Filter:   0%|          | 0/543 [00:00<?, ? examples/s]
Filter: 100%|##########| 543/543 [00:00<00:00, 44513.86 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 5320.36 examples/s]INFO 2025-05-20 11:01:42,334 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.28%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 47.18ba/s]

Map:   0%|          | 0/3495 [00:00<?, ? examples/s]
Map:  23%|##2       | 789/3495 [00:00<00:00, 7819.81 examples/s]
Map:  54%|#####4    | 1898/3495 [00:00<00:00, 7520.02 examples/s]
Map:  86%|########5 | 3000/3495 [00:00<00:00, 7221.03 examples/s]
Map: 100%|##########| 3495/3495 [00:00<00:00, 7292.47 examples/s]

Map:   0%|          | 0/3495 [00:00<?, ? examples/s]
Map:  86%|########6 | 3016/3495 [00:00<00:00, 30026.59 examples/s]
Map: 100%|##########| 3495/3495 [00:00<00:00, 29902.17 examples/s]

Filter:   0%|          | 0/3495 [00:00<?, ? examples/s]
Filter: 100%|##########| 3495/3495 [00:00<00:00, 49814.77 examples/s]

Map:   0%|          | 0/197 [00:00<?, ? examples/s]
Map: 100%|##########| 197/197 [00:00<00:00, 10675.01 examples/s]

Map:   0%|          | 0/197 [00:00<?, ? examples/s]
Map: 100%|##########| 197/197 [00:00<00:00, 23032.78 examples/s]

Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 36.49ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 36.43ba/s]

Map:   0%|          | 0/3495 [00:00<?, ? examples/s]
Map:  23%|##2       | 791/3495 [00:00<00:00, 7852.18 examples/s]
Map:  55%|#####4    | 1907/3495 [00:00<00:00, 7566.80 examples/s]
Map:  86%|########5 | 3000/3495 [00:00<00:00, 7176.57 examples/s]
Map: 100%|##########| 3495/3495 [00:00<00:00, 7258.55 examples/s]

Map:   0%|          | 0/3495 [00:00<?, ? examples/s]
Map:  24%|##3       | 831/3495 [00:00<00:00, 8265.29 examples/s]
Map:  55%|#####5    | 1924/3495 [00:00<00:00, 7594.81 examples/s]
Map:  86%|########5 | 3000/3495 [00:00<00:00, 6815.45 examples/s]
Map: 100%|##########| 3495/3495 [00:00<00:00, 7020.85 examples/s]

Map:   0%|          | 0/3495 [00:00<?, ? examples/s]
Map:  23%|##2       | 787/3495 [00:00<00:00, 7808.15 examples/s]
Map:  55%|#####4    | 1917/3495 [00:00<00:00, 7612.07 examples/s]
Map:  86%|########5 | 3000/3495 [00:00<00:00, 7289.18 examples/s]
Map: 100%|##########| 3495/3495 [00:00<00:00, 7354.45 examples/s]

Map:   0%|          | 0/3495 [00:00<?, ? examples/s]
Map:  86%|########5 | 3000/3495 [00:00<00:00, 29713.89 examples/s]
Map: 100%|##########| 3495/3495 [00:00<00:00, 29633.55 examples/s]

Filter:   0%|          | 0/3495 [00:00<?, ? examples/s]
Filter: 100%|##########| 3495/3495 [00:00<00:00, 49040.02 examples/s]

Map:   0%|          | 0/197 [00:00<?, ? examples/s]
Map: 100%|##########| 197/197 [00:00<00:00, 10757.29 examples/s]INFO 2025-05-20 11:01:45,146 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.82%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/8 [00:00<?, ?ba/s]
Creating json from Arrow format:  25%|##5       | 2/8 [00:00<00:00, 16.02ba/s]
Creating json from Arrow format:  50%|#####     | 4/8 [00:00<00:00, 17.56ba/s]
Creating json from Arrow format: 100%|##########| 8/8 [00:00<00:00, 24.78ba/s]

Map:   0%|          | 0/814 [00:00<?, ? examples/s]
Map:  96%|#########6| 783/814 [00:00<00:00, 7768.30 examples/s]
Map: 100%|##########| 814/814 [00:00<00:00, 7216.99 examples/s]

Map:   0%|          | 0/814 [00:00<?, ? examples/s]
Map: 100%|##########| 814/814 [00:00<00:00, 27516.05 examples/s]

Filter:   0%|          | 0/814 [00:00<?, ? examples/s]
Filter: 100%|##########| 814/814 [00:00<00:00, 44703.22 examples/s]

Map:   0%|          | 0/161 [00:00<?, ? examples/s]
Map: 100%|##########| 161/161 [00:00<00:00, 10397.61 examples/s]

Map:   0%|          | 0/161 [00:00<?, ? examples/s]
Map: 100%|##########| 161/161 [00:00<00:00, 21934.03 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 36.96ba/s]

Map:   0%|          | 0/814 [00:00<?, ? examples/s]
Map:  94%|#########4| 767/814 [00:00<00:00, 7611.34 examples/s]
Map: 100%|##########| 814/814 [00:00<00:00, 7070.46 examples/s]

Map:   0%|          | 0/814 [00:00<?, ? examples/s]
Map: 100%|##########| 814/814 [00:00<00:00, 7214.12 examples/s]
Map: 100%|##########| 814/814 [00:00<00:00, 7150.83 examples/s]

Map:   0%|          | 0/814 [00:00<?, ? examples/s]
Map:  96%|#########6| 785/814 [00:00<00:00, 7794.24 examples/s]
Map: 100%|##########| 814/814 [00:00<00:00, 7291.50 examples/s]

Map:   0%|          | 0/814 [00:00<?, ? examples/s]
Map: 100%|##########| 814/814 [00:00<00:00, 27915.84 examples/s]

Filter:   0%|          | 0/814 [00:00<?, ? examples/s]
Filter: 100%|##########| 814/814 [00:00<00:00, 45066.11 examples/s]

Map:   0%|          | 0/161 [00:00<?, ? examples/s]
Map: 100%|##########| 161/161 [00:00<00:00, 10334.12 examples/s]INFO 2025-05-20 11:01:46,200 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.45%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 25.52ba/s]

Map:   0%|          | 0/1221 [00:00<?, ? examples/s]
Map:  65%|######4   | 791/1221 [00:00<00:00, 7843.94 examples/s]
Map: 100%|##########| 1221/1221 [00:00<00:00, 7293.78 examples/s]

Map:   0%|          | 0/1221 [00:00<?, ? examples/s]
Map: 100%|##########| 1221/1221 [00:00<00:00, 27471.99 examples/s]

Filter:   0%|          | 0/1221 [00:00<?, ? examples/s]
Filter: 100%|##########| 1221/1221 [00:00<00:00, 46714.76 examples/s]

Map:   0%|          | 0/59 [00:00<?, ? examples/s]
Map: 100%|##########| 59/59 [00:00<00:00, 8501.58 examples/s]

Map:   0%|          | 0/59 [00:00<?, ? examples/s]
Map: 100%|##########| 59/59 [00:00<00:00, 15026.05 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 50.67ba/s]

Map:   0%|          | 0/1221 [00:00<?, ? examples/s]
Map:  64%|######4   | 783/1221 [00:00<00:00, 7769.95 examples/s]
Map: 100%|##########| 1221/1221 [00:00<00:00, 7216.59 examples/s]

Map:   0%|          | 0/1221 [00:00<?, ? examples/s]
Map:  73%|#######2  | 891/1221 [00:00<00:00, 8868.76 examples/s]
Map: 100%|##########| 1221/1221 [00:00<00:00, 7040.84 examples/s]

Map:   0%|          | 0/1221 [00:00<?, ? examples/s]
Map:  65%|######5   | 798/1221 [00:00<00:00, 7920.50 examples/s]
Map: 100%|##########| 1221/1221 [00:00<00:00, 7250.50 examples/s]

Map:   0%|          | 0/1221 [00:00<?, ? examples/s]
Map: 100%|##########| 1221/1221 [00:00<00:00, 27601.99 examples/s]

Filter:   0%|          | 0/1221 [00:00<?, ? examples/s]
Filter: 100%|##########| 1221/1221 [00:00<00:00, 47138.29 examples/s]

Map:   0%|          | 0/59 [00:00<?, ? examples/s]
Map: 100%|##########| 59/59 [00:00<00:00, 8528.53 examples/s]INFO 2025-05-20 11:01:47,289 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.63%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format:  67%|######6   | 2/3 [00:00<00:00, 17.04ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 22.47ba/s]

Map:   0%|          | 0/394 [00:00<?, ? examples/s]
Map: 100%|##########| 394/394 [00:00<00:00, 7476.82 examples/s]

Map:   0%|          | 0/394 [00:00<?, ? examples/s]
Map: 100%|##########| 394/394 [00:00<00:00, 26790.67 examples/s]

Filter:   0%|          | 0/394 [00:00<?, ? examples/s]
Filter: 100%|##########| 394/394 [00:00<00:00, 42318.97 examples/s]

Map:   0%|          | 0/33 [00:00<?, ? examples/s]
Map: 100%|##########| 33/33 [00:00<00:00, 7085.70 examples/s]

Map:   0%|          | 0/33 [00:00<?, ? examples/s]
Map: 100%|##########| 33/33 [00:00<00:00, 10841.39 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 144.98ba/s]

Map:   0%|          | 0/394 [00:00<?, ? examples/s]
Map: 100%|##########| 394/394 [00:00<00:00, 7551.88 examples/s]

Map:   0%|          | 0/394 [00:00<?, ? examples/s]
Map: 100%|##########| 394/394 [00:00<00:00, 8541.27 examples/s]

Map:   0%|          | 0/394 [00:00<?, ? examples/s]
Map: 100%|##########| 394/394 [00:00<00:00, 7504.86 examples/s]

Map:   0%|          | 0/394 [00:00<?, ? examples/s]
Map: 100%|##########| 394/394 [00:00<00:00, 27546.73 examples/s]

Filter:   0%|          | 0/394 [00:00<?, ? examples/s]
Filter: 100%|##########| 394/394 [00:00<00:00, 42904.58 examples/s]

Map:   0%|          | 0/33 [00:00<?, ? examples/s]
Map: 100%|##########| 33/33 [00:00<00:00, 7007.85 examples/s]INFO 2025-05-20 11:01:47,776 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.21%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 50.29ba/s]

Map:   0%|          | 0/227 [00:00<?, ? examples/s]
Map: 100%|##########| 227/227 [00:00<00:00, 6997.29 examples/s]

Map:   0%|          | 0/227 [00:00<?, ? examples/s]
Map: 100%|##########| 227/227 [00:00<00:00, 24491.50 examples/s]

Filter:   0%|          | 0/227 [00:00<?, ? examples/s]
Filter: 100%|##########| 227/227 [00:00<00:00, 35866.31 examples/s]

Map:   0%|          | 0/46 [00:00<?, ? examples/s]
Map: 100%|##########| 46/46 [00:00<00:00, 7904.06 examples/s]

Map:   0%|          | 0/46 [00:00<?, ? examples/s]
Map: 100%|##########| 46/46 [00:00<00:00, 13026.67 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 164.94ba/s]

Map:   0%|          | 0/227 [00:00<?, ? examples/s]
Map: 100%|##########| 227/227 [00:00<00:00, 6977.90 examples/s]

Map:   0%|          | 0/227 [00:00<?, ? examples/s]
Map: 100%|##########| 227/227 [00:00<00:00, 7591.83 examples/s]

Map:   0%|          | 0/227 [00:00<?, ? examples/s]
Map: 100%|##########| 227/227 [00:00<00:00, 6982.46 examples/s]

Map:   0%|          | 0/227 [00:00<?, ? examples/s]
Map: 100%|##########| 227/227 [00:00<00:00, 24710.16 examples/s]

Filter:   0%|          | 0/227 [00:00<?, ? examples/s]
Filter: 100%|##########| 227/227 [00:00<00:00, 36395.53 examples/s]

Map:   0%|          | 0/46 [00:00<?, ? examples/s]
Map: 100%|##########| 46/46 [00:00<00:00, 7849.39 examples/s]INFO 2025-05-20 11:01:48,049 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.13%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 65.13ba/s]

Map:   0%|          | 0/3679 [00:00<?, ? examples/s]
Map:  21%|##1       | 790/3679 [00:00<00:00, 7831.30 examples/s]
Map:  52%|#####1    | 1903/3679 [00:00<00:00, 7544.48 examples/s]
Map:  82%|########1 | 3000/3679 [00:00<00:00, 7180.13 examples/s]
Map: 100%|##########| 3679/3679 [00:00<00:00, 7267.55 examples/s]

Map:   0%|          | 0/3679 [00:00<?, ? examples/s]
Map:  82%|########1 | 3000/3679 [00:00<00:00, 29412.45 examples/s]
Map: 100%|##########| 3679/3679 [00:00<00:00, 29611.61 examples/s]

Filter:   0%|          | 0/3679 [00:00<?, ? examples/s]
Filter: 100%|##########| 3679/3679 [00:00<00:00, 49851.86 examples/s]

Map:   0%|          | 0/201 [00:00<?, ? examples/s]
Map: 100%|##########| 201/201 [00:00<00:00, 10703.83 examples/s]

Map:   0%|          | 0/201 [00:00<?, ? examples/s]
Map: 100%|##########| 201/201 [00:00<00:00, 23420.80 examples/s]

Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 34.08ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 34.02ba/s]

Map:   0%|          | 0/3679 [00:00<?, ? examples/s]
Map:  22%|##1       | 792/3679 [00:00<00:00, 7855.08 examples/s]
Map:  52%|#####1    | 1906/3679 [00:00<00:00, 7556.20 examples/s]
Map:  82%|########1 | 3000/3679 [00:00<00:00, 7203.16 examples/s]
Map: 100%|##########| 3679/3679 [00:00<00:00, 7285.79 examples/s]

Map:   0%|          | 0/3679 [00:00<?, ? examples/s]
Map:  23%|##3       | 847/3679 [00:00<00:00, 8427.17 examples/s]
Map:  52%|#####1    | 1904/3679 [00:00<00:00, 7484.23 examples/s]
Map:  81%|########1 | 2995/3679 [00:00<00:00, 7375.29 examples/s]
Map: 100%|##########| 3679/3679 [00:00<00:00, 7134.57 examples/s]

Map:   0%|          | 0/3679 [00:00<?, ? examples/s]
Map:  22%|##1       | 795/3679 [00:00<00:00, 7889.96 examples/s]
Map:  53%|#####2    | 1933/3679 [00:00<00:00, 7678.41 examples/s]
Map:  82%|########1 | 3000/3679 [00:00<00:00, 7349.41 examples/s]
Map: 100%|##########| 3679/3679 [00:00<00:00, 7424.68 examples/s]

Map:   0%|          | 0/3679 [00:00<?, ? examples/s]
Map:  82%|########1 | 3000/3679 [00:00<00:00, 29794.45 examples/s]
Map: 100%|##########| 3679/3679 [00:00<00:00, 29845.86 examples/s]

Filter:   0%|          | 0/3679 [00:00<?, ? examples/s]
Filter: 100%|##########| 3679/3679 [00:00<00:00, 49750.12 examples/s]

Map:   0%|          | 0/201 [00:00<?, ? examples/s]
Map: 100%|##########| 201/201 [00:00<00:00, 10837.30 examples/s]INFO 2025-05-20 11:01:50,982 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.92%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/8 [00:00<?, ?ba/s]
Creating json from Arrow format:  25%|##5       | 2/8 [00:00<00:00, 17.62ba/s]
Creating json from Arrow format:  50%|#####     | 4/8 [00:00<00:00, 17.94ba/s]
Creating json from Arrow format: 100%|##########| 8/8 [00:00<00:00, 27.33ba/s]
Creating json from Arrow format: 100%|##########| 8/8 [00:00<00:00, 24.65ba/s]

Map:   0%|          | 0/234 [00:00<?, ? examples/s]
Map: 100%|##########| 234/234 [00:00<00:00, 6962.04 examples/s]

Map:   0%|          | 0/234 [00:00<?, ? examples/s]
Map: 100%|##########| 234/234 [00:00<00:00, 24161.57 examples/s]

Filter:   0%|          | 0/234 [00:00<?, ? examples/s]
Filter: 100%|##########| 234/234 [00:00<00:00, 36343.90 examples/s]

Map:   0%|          | 0/3 [00:00<?, ? examples/s]
Map: 100%|##########| 3/3 [00:00<00:00, 1260.81 examples/s]

Map:   0%|          | 0/3 [00:00<?, ? examples/s]
Map: 100%|##########| 3/3 [00:00<00:00, 1373.23 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 264.91ba/s]

Map:   0%|          | 0/234 [00:00<?, ? examples/s]
Map: 100%|##########| 234/234 [00:00<00:00, 7072.16 examples/s]

Map:   0%|          | 0/234 [00:00<?, ? examples/s]
Map: 100%|##########| 234/234 [00:00<00:00, 8177.67 examples/s]

Map:   0%|          | 0/234 [00:00<?, ? examples/s]
Map: 100%|##########| 234/234 [00:00<00:00, 7098.09 examples/s]

Map:   0%|          | 0/234 [00:00<?, ? examples/s]
Map: 100%|##########| 234/234 [00:00<00:00, 24587.08 examples/s]

Filter:   0%|          | 0/234 [00:00<?, ? examples/s]
Filter: 100%|##########| 234/234 [00:00<00:00, 37452.00 examples/s]

Map:   0%|          | 0/3 [00:00<?, ? examples/s]
Map: 100%|##########| 3/3 [00:00<00:00, 1301.50 examples/s]INFO 2025-05-20 11:01:51,554 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.12%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 111.10ba/s]

Map:   0%|          | 0/1906 [00:00<?, ? examples/s]
Map:  41%|####      | 776/1906 [00:00<00:00, 7692.02 examples/s]
Map: 100%|#########9| 1902/1906 [00:00<00:00, 7556.58 examples/s]
Map: 100%|##########| 1906/1906 [00:00<00:00, 7327.73 examples/s]

Map:   0%|          | 0/1906 [00:00<?, ? examples/s]
Map: 100%|##########| 1906/1906 [00:00<00:00, 28768.32 examples/s]

Filter:   0%|          | 0/1906 [00:00<?, ? examples/s]
Filter: 100%|##########| 1906/1906 [00:00<00:00, 48581.88 examples/s]

Map:   0%|          | 0/113 [00:00<?, ? examples/s]
Map: 100%|##########| 113/113 [00:00<00:00, 9952.88 examples/s]

Map:   0%|          | 0/113 [00:00<?, ? examples/s]
Map: 100%|##########| 113/113 [00:00<00:00, 19549.43 examples/s]

Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 53.13ba/s]

Map:   0%|          | 0/1906 [00:00<?, ? examples/s]
Map:  41%|####1     | 786/1906 [00:00<00:00, 7804.26 examples/s]
Map: 100%|##########| 1906/1906 [00:00<00:00, 7375.32 examples/s]
Map: 100%|##########| 1906/1906 [00:00<00:00, 7394.30 examples/s]

Map:   0%|          | 0/1906 [00:00<?, ? examples/s]
Map:  46%|####5     | 876/1906 [00:00<00:00, 8707.05 examples/s]
Map: 100%|##########| 1906/1906 [00:00<00:00, 7156.84 examples/s]
Map: 100%|##########| 1906/1906 [00:00<00:00, 7304.95 examples/s]

Map:   0%|          | 0/1906 [00:00<?, ? examples/s]
Map:  41%|####1     | 790/1906 [00:00<00:00, 7833.61 examples/s]
Map: 100%|#########9| 1900/1906 [00:00<00:00, 7534.39 examples/s]
Map: 100%|##########| 1906/1906 [00:00<00:00, 7275.12 examples/s]

Map:   0%|          | 0/1906 [00:00<?, ? examples/s]
Map: 100%|##########| 1906/1906 [00:00<00:00, 28445.57 examples/s]

Filter:   0%|          | 0/1906 [00:00<?, ? examples/s]
Filter: 100%|##########| 1906/1906 [00:00<00:00, 48244.72 examples/s]

Map:   0%|          | 0/113 [00:00<?, ? examples/s]
Map: 100%|##########| 113/113 [00:00<00:00, 9721.78 examples/s]INFO 2025-05-20 11:01:53,086 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.00%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format:  50%|#####     | 2/4 [00:00<00:00, 16.67ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 23.41ba/s]

Map:   0%|          | 0/426 [00:00<?, ? examples/s]
Map: 100%|##########| 426/426 [00:00<00:00, 7199.39 examples/s]

Map:   0%|          | 0/426 [00:00<?, ? examples/s]
Map: 100%|##########| 426/426 [00:00<00:00, 27119.58 examples/s]

Filter:   0%|          | 0/426 [00:00<?, ? examples/s]
Filter: 100%|##########| 426/426 [00:00<00:00, 41240.21 examples/s]

Map:   0%|          | 0/19 [00:00<?, ? examples/s]
Map: 100%|##########| 19/19 [00:00<00:00, 5057.23 examples/s]

Map:   0%|          | 0/19 [00:00<?, ? examples/s]
Map: 100%|##########| 19/19 [00:00<00:00, 7141.48 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 95.11ba/s]

Map:   0%|          | 0/426 [00:00<?, ? examples/s]
Map: 100%|##########| 426/426 [00:00<00:00, 7194.64 examples/s]

Map:   0%|          | 0/426 [00:00<?, ? examples/s]
Map: 100%|##########| 426/426 [00:00<00:00, 7935.64 examples/s]

Map:   0%|          | 0/426 [00:00<?, ? examples/s]
Map: 100%|##########| 426/426 [00:00<00:00, 7227.58 examples/s]

Map:   0%|          | 0/426 [00:00<?, ? examples/s]
Map: 100%|##########| 426/426 [00:00<00:00, 27265.66 examples/s]

Filter:   0%|          | 0/426 [00:00<?, ? examples/s]
Filter: 100%|##########| 426/426 [00:00<00:00, 42086.29 examples/s]

Map:   0%|          | 0/19 [00:00<?, ? examples/s]
Map: 100%|##########| 19/19 [00:00<00:00, 5270.27 examples/s]INFO 2025-05-20 11:01:53,648 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.22%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 27.30ba/s]

Map:   0%|          | 0/196 [00:00<?, ? examples/s]
Map: 100%|##########| 196/196 [00:00<00:00, 6898.76 examples/s]

Map:   0%|          | 0/196 [00:00<?, ? examples/s]
Map: 100%|##########| 196/196 [00:00<00:00, 23224.67 examples/s]

Filter:   0%|          | 0/196 [00:00<?, ? examples/s]
Filter: 100%|##########| 196/196 [00:00<00:00, 34168.06 examples/s]

Map:   0%|          | 0/28 [00:00<?, ? examples/s]
Map: 100%|##########| 28/28 [00:00<00:00, 6355.01 examples/s]

Map:   0%|          | 0/28 [00:00<?, ? examples/s]
Map: 100%|##########| 28/28 [00:00<00:00, 9422.38 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 256.02ba/s]

Map:   0%|          | 0/196 [00:00<?, ? examples/s]
Map: 100%|##########| 196/196 [00:00<00:00, 7038.69 examples/s]

Map:   0%|          | 0/196 [00:00<?, ? examples/s]
Map: 100%|##########| 196/196 [00:00<00:00, 7979.61 examples/s]

Map:   0%|          | 0/196 [00:00<?, ? examples/s]
Map: 100%|##########| 196/196 [00:00<00:00, 7031.52 examples/s]

Map:   0%|          | 0/196 [00:00<?, ? examples/s]
Map: 100%|##########| 196/196 [00:00<00:00, 24278.90 examples/s]

Filter:   0%|          | 0/196 [00:00<?, ? examples/s]
Filter: 100%|##########| 196/196 [00:00<00:00, 35375.17 examples/s]

Map:   0%|          | 0/28 [00:00<?, ? examples/s]
Map: 100%|##########| 28/28 [00:00<00:00, 6493.45 examples/s]INFO 2025-05-20 11:01:53,909 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.11%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 103.34ba/s]

Map:   0%|          | 0/175 [00:00<?, ? examples/s]
Map: 100%|##########| 175/175 [00:00<00:00, 6745.42 examples/s]

Map:   0%|          | 0/175 [00:00<?, ? examples/s]
Map: 100%|##########| 175/175 [00:00<00:00, 22964.15 examples/s]

Filter:   0%|          | 0/175 [00:00<?, ? examples/s]
Filter: 100%|##########| 175/175 [00:00<00:00, 33404.78 examples/s]

Map:   0%|          | 0/38 [00:00<?, ? examples/s]
Map: 100%|##########| 38/38 [00:00<00:00, 7247.01 examples/s]

Map:   0%|          | 0/38 [00:00<?, ? examples/s]
Map: 100%|##########| 38/38 [00:00<00:00, 11574.70 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 198.92ba/s]

Map:   0%|          | 0/175 [00:00<?, ? examples/s]
Map: 100%|##########| 175/175 [00:00<00:00, 6714.63 examples/s]

Map:   0%|          | 0/175 [00:00<?, ? examples/s]
Map: 100%|##########| 175/175 [00:00<00:00, 7396.77 examples/s]

Map:   0%|          | 0/175 [00:00<?, ? examples/s]
Map: 100%|##########| 175/175 [00:00<00:00, 6864.65 examples/s]

Map:   0%|          | 0/175 [00:00<?, ? examples/s]
Map: 100%|##########| 175/175 [00:00<00:00, 23170.76 examples/s]

Filter:   0%|          | 0/175 [00:00<?, ? examples/s]
Filter: 100%|##########| 175/175 [00:00<00:00, 34155.57 examples/s]

Map:   0%|          | 0/38 [00:00<?, ? examples/s]
Map: 100%|##########| 38/38 [00:00<00:00, 7391.19 examples/s]INFO 2025-05-20 11:01:54,133 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.10%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 82.80ba/s]

Map:   0%|          | 0/158 [00:00<?, ? examples/s]
Map: 100%|##########| 158/158 [00:00<00:00, 6321.54 examples/s]

Map:   0%|          | 0/158 [00:00<?, ? examples/s]
Map: 100%|##########| 158/158 [00:00<00:00, 21797.19 examples/s]

Filter:   0%|          | 0/158 [00:00<?, ? examples/s]
Filter: 100%|##########| 158/158 [00:00<00:00, 31300.78 examples/s]

Map:   0%|          | 0/25 [00:00<?, ? examples/s]
Map: 100%|##########| 25/25 [00:00<00:00, 5942.62 examples/s]

Map:   0%|          | 0/25 [00:00<?, ? examples/s]
Map: 100%|##########| 25/25 [00:00<00:00, 8635.23 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 184.33ba/s]

Map:   0%|          | 0/158 [00:00<?, ? examples/s]
Map: 100%|##########| 158/158 [00:00<00:00, 6403.83 examples/s]

Map:   0%|          | 0/158 [00:00<?, ? examples/s]
Map: 100%|##########| 158/158 [00:00<00:00, 6901.04 examples/s]

Map:   0%|          | 0/158 [00:00<?, ? examples/s]
Map: 100%|##########| 158/158 [00:00<00:00, 6561.78 examples/s]

Map:   0%|          | 0/158 [00:00<?, ? examples/s]
Map: 100%|##########| 158/158 [00:00<00:00, 22369.62 examples/s]

Filter:   0%|          | 0/158 [00:00<?, ? examples/s]
Filter: 100%|##########| 158/158 [00:00<00:00, 30994.81 examples/s]

Map:   0%|          | 0/25 [00:00<?, ? examples/s]
Map: 100%|##########| 25/25 [00:00<00:00, 5870.10 examples/s]INFO 2025-05-20 11:01:54,352 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.09%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 76.32ba/s]

Map:   0%|          | 0/1153 [00:00<?, ? examples/s]
Map:  70%|#######   | 809/1153 [00:00<00:00, 8025.90 examples/s]
Map: 100%|##########| 1153/1153 [00:00<00:00, 7417.16 examples/s]

Map:   0%|          | 0/1153 [00:00<?, ? examples/s]
Map: 100%|##########| 1153/1153 [00:00<00:00, 29243.00 examples/s]

Filter:   0%|          | 0/1153 [00:00<?, ? examples/s]
Filter: 100%|##########| 1153/1153 [00:00<00:00, 47398.61 examples/s]

Map:   0%|          | 0/50 [00:00<?, ? examples/s]
Map: 100%|##########| 50/50 [00:00<00:00, 8046.16 examples/s]

Map:   0%|          | 0/50 [00:00<?, ? examples/s]
Map: 100%|##########| 50/50 [00:00<00:00, 13560.63 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 66.10ba/s]

Map:   0%|          | 0/1153 [00:00<?, ? examples/s]
Map:  70%|######9   | 805/1153 [00:00<00:00, 7989.30 examples/s]
Map: 100%|##########| 1153/1153 [00:00<00:00, 7347.26 examples/s]

Map:   0%|          | 0/1153 [00:00<?, ? examples/s]
Map:  80%|########  | 925/1153 [00:00<00:00, 9198.75 examples/s]
Map: 100%|##########| 1153/1153 [00:00<00:00, 7901.73 examples/s]

Map:   0%|          | 0/1153 [00:00<?, ? examples/s]
Map:  70%|######9   | 806/1153 [00:00<00:00, 8002.71 examples/s]
Map: 100%|##########| 1153/1153 [00:00<00:00, 7433.94 examples/s]

Map:   0%|          | 0/1153 [00:00<?, ? examples/s]
Map: 100%|##########| 1153/1153 [00:00<00:00, 28644.90 examples/s]

Filter:   0%|          | 0/1153 [00:00<?, ? examples/s]
Filter: 100%|##########| 1153/1153 [00:00<00:00, 47419.52 examples/s]

Map:   0%|          | 0/50 [00:00<?, ? examples/s]
Map: 100%|##########| 50/50 [00:00<00:00, 8128.50 examples/s]INFO 2025-05-20 11:01:55,277 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.60%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 35.17ba/s]

Map:   0%|          | 0/13798 [00:00<?, ? examples/s]
Map:   6%|5         | 781/13798 [00:00<00:01, 7750.54 examples/s]
Map:  14%|#3        | 1890/13798 [00:00<00:01, 7504.02 examples/s]
Map:  22%|##1       | 2997/13798 [00:00<00:01, 7440.11 examples/s]
Map:  29%|##9       | 4017/13798 [00:00<00:01, 7171.93 examples/s]
Map:  35%|###4      | 4809/13798 [00:00<00:01, 7379.13 examples/s]
Map:  43%|####2     | 5893/13798 [00:00<00:01, 7317.85 examples/s]
Map:  51%|#####     | 7000/13798 [00:00<00:00, 7235.98 examples/s]
Map:  56%|#####6    | 7771/13798 [00:01<00:00, 7350.07 examples/s]
Map:  64%|######4   | 8898/13798 [00:01<00:00, 7402.57 examples/s]
Map:  72%|#######2  | 9995/13798 [00:01<00:00, 7369.77 examples/s]
Map:  80%|#######9  | 11000/13798 [00:01<00:00, 7056.08 examples/s]
Map:  86%|########5 | 11804/13798 [00:01<00:00, 7281.08 examples/s]
Map:  91%|#########1| 12557/13798 [00:01<00:00, 7340.94 examples/s]
Map:  96%|#########6| 13308/13798 [00:01<00:00, 7382.52 examples/s]
Map: 100%|##########| 13798/13798 [00:01<00:00, 7321.80 examples/s]

Map:   0%|          | 0/13798 [00:00<?, ? examples/s]
Map:   7%|7         | 1000/13798 [00:00<00:03, 3765.83 examples/s]
Map:  29%|##8       | 4000/13798 [00:00<00:00, 12759.04 examples/s]
Map:  51%|#####     | 7000/13798 [00:00<00:00, 18327.12 examples/s]
Map:  72%|#######2  | 10000/13798 [00:00<00:00, 21239.16 examples/s]
Map:  96%|#########5| 13206/13798 [00:00<00:00, 24537.31 examples/s]
Map: 100%|##########| 13798/13798 [00:00<00:00, 19569.66 examples/s]

Filter:   0%|          | 0/13798 [00:00<?, ? examples/s]
Filter:  43%|####3     | 6000/13798 [00:00<00:00, 49944.78 examples/s]
Filter:  87%|########6 | 12000/13798 [00:00<00:00, 50639.23 examples/s]
Filter: 100%|##########| 13798/13798 [00:00<00:00, 50497.85 examples/s]

Map:   0%|          | 0/1181 [00:00<?, ? examples/s]
Map: 100%|##########| 1181/1181 [00:00<00:00, 11738.14 examples/s]

Map:   0%|          | 0/1181 [00:00<?, ? examples/s]
Map: 100%|##########| 1181/1181 [00:00<00:00, 28472.65 examples/s]

Creating json from Arrow format:   0%|          | 0/15 [00:00<?, ?ba/s]
Creating json from Arrow format:  27%|##6       | 4/15 [00:00<00:00, 36.52ba/s]
Creating json from Arrow format:  53%|#####3    | 8/15 [00:00<00:00, 37.31ba/s]
Creating json from Arrow format:  80%|########  | 12/15 [00:00<00:00, 37.55ba/s]
Creating json from Arrow format: 100%|##########| 15/15 [00:00<00:00, 39.17ba/s]

Map:   0%|          | 0/13798 [00:00<?, ? examples/s]
Map:   6%|5         | 769/13798 [00:00<00:01, 7630.81 examples/s]
Map:  14%|#3        | 1888/13798 [00:00<00:01, 7509.91 examples/s]
Map:  22%|##1       | 2992/13798 [00:00<00:01, 7435.22 examples/s]
Map:  29%|##9       | 4004/13798 [00:00<00:01, 7148.43 examples/s]
Map:  35%|###4      | 4800/13798 [00:00<00:01, 7374.77 examples/s]
Map:  43%|####2     | 5874/13798 [00:00<00:01, 7291.92 examples/s]
Map:  51%|#####     | 7000/13798 [00:00<00:00, 7240.95 examples/s]
Map:  56%|#####6    | 7779/13798 [00:01<00:00, 7375.61 examples/s]
Map:  64%|######4   | 8896/13798 [00:01<00:00, 7395.02 examples/s]
Map:  72%|#######2  | 9991/13798 [00:01<00:00, 7361.50 examples/s]
Map:  80%|#######9  | 10974/13798 [00:01<00:00, 7097.87 examples/s]
Map:  85%|########4 | 11723/13798 [00:01<00:00, 7187.05 examples/s]
Map:  90%|######### | 12471/13798 [00:01<00:00, 7259.12 examples/s]
Map:  96%|#########5| 13220/13798 [00:01<00:00, 7319.46 examples/s]
Map: 100%|##########| 13798/13798 [00:01<00:00, 7302.20 examples/s]

Map:   0%|          | 0/13798 [00:00<?, ? examples/s]
Map:   4%|4         | 570/13798 [00:00<00:02, 5665.48 examples/s]
Map:   9%|9         | 1301/13798 [00:00<00:02, 5122.60 examples/s]
Map:  13%|#3        | 1861/13798 [00:00<00:02, 5308.78 examples/s]
Map:  19%|#8        | 2586/13798 [00:00<00:02, 5087.47 examples/s]
Map:  24%|##3       | 3297/13798 [00:00<00:02, 4946.21 examples/s]
Map:  28%|##8       | 3871/13798 [00:00<00:01, 5159.23 examples/s]
Map:  33%|###3      | 4618/13798 [00:00<00:01, 5091.62 examples/s]
Map:  39%|###8      | 5359/13798 [00:01<00:01, 5035.18 examples/s]
Map:  43%|####2     | 5920/13798 [00:01<00:01, 5176.18 examples/s]
Map:  48%|####7     | 6589/13798 [00:01<00:01, 4925.48 examples/s]
Map:  53%|#####3    | 7356/13798 [00:01<00:01, 4981.71 examples/s]
Map:  57%|#####7    | 7929/13798 [00:01<00:01, 5158.15 examples/s]
Map:  63%|######2   | 8683/13798 [00:01<00:01, 5111.22 examples/s]
Map:  68%|######8   | 9433/13798 [00:01<00:00, 5071.84 examples/s]
Map:  72%|#######2  | 9990/13798 [00:01<00:00, 5188.35 examples/s]
Map:  78%|#######7  | 10750/13798 [00:02<00:00, 5144.56 examples/s]
Map:  82%|########1 | 11288/13798 [00:02<00:00, 4796.13 examples/s]
Map:  86%|########6 | 11871/13798 [00:02<00:00, 5044.11 examples/s]
Map:  92%|#########1| 12655/13798 [00:02<00:00, 5103.86 examples/s]
Map:  97%|#########7| 13431/13798 [00:02<00:00, 5123.86 examples/s]
Map: 100%|##########| 13798/13798 [00:02<00:00, 5076.75 examples/s]

Map:   0%|          | 0/13798 [00:00<?, ? examples/s]
Map:   6%|5         | 775/13798 [00:00<00:01, 7691.29 examples/s]
Map:  14%|#3        | 1893/13798 [00:00<00:01, 7523.27 examples/s]
Map:  22%|##1       | 2997/13798 [00:00<00:01, 7438.19 examples/s]
Map:  29%|##9       | 4022/13798 [00:00<00:01, 7187.26 examples/s]
Map:  35%|###4      | 4812/13798 [00:00<00:01, 7385.72 examples/s]
Map:  43%|####2     | 5878/13798 [00:00<00:01, 7276.89 examples/s]
Map:  51%|#####     | 6998/13798 [00:00<00:00, 7341.68 examples/s]
Map:  58%|#####8    | 8060/13798 [00:01<00:00, 7249.54 examples/s]
Map:  64%|######4   | 8845/13798 [00:01<00:00, 7391.21 examples/s]
Map:  72%|#######2  | 9942/13798 [00:01<00:00, 7361.69 examples/s]
Map:  79%|#######9  | 10946/13798 [00:01<00:00, 7141.83 examples/s]
Map:  85%|########4 | 11690/13798 [00:01<00:00, 7210.38 examples/s]
Map:  90%|######### | 12439/13798 [00:01<00:00, 7279.31 examples/s]
Map:  96%|#########5| 13191/13798 [00:01<00:00, 7341.05 examples/s]
Map: 100%|##########| 13798/13798 [00:01<00:00, 7315.20 examples/s]

Map:   0%|          | 0/13798 [00:00<?, ? examples/s]
Map:  22%|##1       | 3000/13798 [00:00<00:00, 28394.44 examples/s]
Map:  43%|####3     | 6000/13798 [00:00<00:00, 29140.80 examples/s]
Map:  66%|######6   | 9161/13798 [00:00<00:00, 30247.06 examples/s]
Map: 100%|##########| 13798/13798 [00:00<00:00, 29686.79 examples/s]
Map: 100%|##########| 13798/13798 [00:00<00:00, 29545.40 examples/s]

Filter:   0%|          | 0/13798 [00:00<?, ? examples/s]
Filter:  43%|####3     | 6000/13798 [00:00<00:00, 49950.72 examples/s]
Filter:  87%|########6 | 12000/13798 [00:00<00:00, 50602.10 examples/s]
Filter: 100%|##########| 13798/13798 [00:00<00:00, 50523.72 examples/s]

Map:   0%|          | 0/1181 [00:00<?, ? examples/s]
Map: 100%|##########| 1181/1181 [00:00<00:00, 11686.50 examples/s]
Map: 100%|##########| 1181/1181 [00:00<00:00, 11576.54 examples/s]

Creating json from Arrow format:   0%|          | 0/29 [00:00<?, ?ba/s]
Creating json from Arrow format:   7%|6         | 2/29 [00:00<00:01, 18.55ba/s]
Creating json from Arrow format:  14%|#3        | 4/29 [00:00<00:01, 18.84ba/s]
Creating json from Arrow format:  21%|##        | 6/29 [00:00<00:01, 19.22ba/s]
Creating json from Arrow format:  28%|##7       | 8/29 [00:00<00:01, 19.01ba/s]
Creating json from Arrow format:  34%|###4      | 10/29 [00:00<00:01, 18.86ba/s]
Creating json from Arrow format:  45%|####4     | 13/29 [00:00<00:00, 19.41ba/s]
Creating json from Arrow format:  55%|#####5    | 16/29 [00:00<00:00, 22.28ba/s]
Creating json from Arrow format:  72%|#######2  | 21/29 [00:00<00:00, 28.42ba/s]
Creating json from Arrow format:  86%|########6 | 25/29 [00:01<00:00, 31.05ba/s]
Creating json from Arrow format: 100%|##########| 29/29 [00:01<00:00, 26.66ba/s]

Map:   0%|          | 0/1307 [00:00<?, ? examples/s]
Map:  61%|######1   | 800/1307 [00:00<00:00, 7946.50 examples/s]
Map: 100%|##########| 1307/1307 [00:00<00:00, 7529.88 examples/s]

Map:   0%|          | 0/1307 [00:00<?, ? examples/s]
Map: 100%|##########| 1307/1307 [00:00<00:00, 29714.11 examples/s]

Filter:   0%|          | 0/1307 [00:00<?, ? examples/s]
Filter: 100%|##########| 1307/1307 [00:00<00:00, 47110.41 examples/s]

Map:   0%|          | 0/75 [00:00<?, ? examples/s]
Map: 100%|##########| 75/75 [00:00<00:00, 9210.15 examples/s]

Map:   0%|          | 0/75 [00:00<?, ? examples/s]
Map: 100%|##########| 75/75 [00:00<00:00, 17027.87 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 66.23ba/s]

Map:   0%|          | 0/1307 [00:00<?, ? examples/s]
Map:  61%|######    | 796/1307 [00:00<00:00, 7900.62 examples/s]
Map: 100%|##########| 1307/1307 [00:00<00:00, 7366.58 examples/s]

Map:   0%|          | 0/1307 [00:00<?, ? examples/s]
Map:  70%|#######   | 917/1307 [00:00<00:00, 9115.39 examples/s]
Map: 100%|##########| 1307/1307 [00:00<00:00, 7846.07 examples/s]

Map:   0%|          | 0/1307 [00:00<?, ? examples/s]
Map:  61%|######1   | 802/1307 [00:00<00:00, 7961.68 examples/s]
Map: 100%|##########| 1307/1307 [00:00<00:00, 7566.35 examples/s]

Map:   0%|          | 0/1307 [00:00<?, ? examples/s]
Map: 100%|##########| 1307/1307 [00:00<00:00, 29295.22 examples/s]

Filter:   0%|          | 0/1307 [00:00<?, ? examples/s]
Filter: 100%|##########| 1307/1307 [00:00<00:00, 47876.50 examples/s]

Map:   0%|          | 0/75 [00:00<?, ? examples/s]
Map: 100%|##########| 75/75 [00:00<00:00, 9214.20 examples/s]INFO 2025-05-20 11:02:09,430 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.68%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 37.90ba/s]

Map:   0%|          | 0/1071 [00:00<?, ? examples/s]
Map:  74%|#######3  | 791/1071 [00:00<00:00, 7851.19 examples/s]
Map: 100%|##########| 1071/1071 [00:00<00:00, 7332.42 examples/s]

Map:   0%|          | 0/1071 [00:00<?, ? examples/s]
Map: 100%|##########| 1071/1071 [00:00<00:00, 29755.11 examples/s]

Filter:   0%|          | 0/1071 [00:00<?, ? examples/s]
Filter: 100%|##########| 1071/1071 [00:00<00:00, 46720.68 examples/s]

Map:   0%|          | 0/55 [00:00<?, ? examples/s]
Map: 100%|##########| 55/55 [00:00<00:00, 8542.69 examples/s]

Map:   0%|          | 0/55 [00:00<?, ? examples/s]
Map: 100%|##########| 55/55 [00:00<00:00, 14271.64 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 87.99ba/s]

Map:   0%|          | 0/1071 [00:00<?, ? examples/s]
Map:  75%|#######5  | 805/1071 [00:00<00:00, 7993.44 examples/s]
Map: 100%|##########| 1071/1071 [00:00<00:00, 7457.35 examples/s]

Map:   0%|          | 0/1071 [00:00<?, ? examples/s]
Map:  88%|########7 | 942/1071 [00:00<00:00, 9376.42 examples/s]
Map: 100%|##########| 1071/1071 [00:00<00:00, 8056.57 examples/s]

Map:   0%|          | 0/1071 [00:00<?, ? examples/s]
Map:  75%|#######5  | 806/1071 [00:00<00:00, 8006.12 examples/s]
Map: 100%|##########| 1071/1071 [00:00<00:00, 2870.41 examples/s]

Map:   0%|          | 0/1071 [00:00<?, ? examples/s]
Map: 100%|##########| 1071/1071 [00:00<00:00, 29125.60 examples/s]

Filter:   0%|          | 0/1071 [00:00<?, ? examples/s]
Filter: 100%|##########| 1071/1071 [00:00<00:00, 47153.23 examples/s]

Map:   0%|          | 0/55 [00:00<?, ? examples/s]
Map: 100%|##########| 55/55 [00:00<00:00, 8391.05 examples/s]INFO 2025-05-20 11:02:10,582 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.56%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 45.53ba/s]

Map:   0%|          | 0/2617 [00:00<?, ? examples/s]
Map:  30%|##9       | 784/2617 [00:00<00:00, 7774.67 examples/s]
Map:  73%|#######2  | 1902/2617 [00:00<00:00, 7550.72 examples/s]
Map: 100%|##########| 2617/2617 [00:00<00:00, 7204.28 examples/s]

Map:   0%|          | 0/2617 [00:00<?, ? examples/s]
Map: 100%|##########| 2617/2617 [00:00<00:00, 28777.59 examples/s]

Filter:   0%|          | 0/2617 [00:00<?, ? examples/s]
Filter: 100%|##########| 2617/2617 [00:00<00:00, 47984.25 examples/s]

Map:   0%|          | 0/108 [00:00<?, ? examples/s]
Map: 100%|##########| 108/108 [00:00<00:00, 9584.95 examples/s]

Map:   0%|          | 0/108 [00:00<?, ? examples/s]
Map: 100%|##########| 108/108 [00:00<00:00, 19324.47 examples/s]

Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 40.34ba/s]

Map:   0%|          | 0/2617 [00:00<?, ? examples/s]
Map:  30%|###       | 792/2617 [00:00<00:00, 7858.40 examples/s]
Map:  73%|#######3  | 1912/2617 [00:00<00:00, 7586.30 examples/s]
Map: 100%|##########| 2617/2617 [00:00<00:00, 7223.22 examples/s]

Map:   0%|          | 0/2617 [00:00<?, ? examples/s]
Map:  33%|###3      | 864/2617 [00:00<00:00, 8586.78 examples/s]
Map:  76%|#######6  | 1996/2617 [00:00<00:00, 7874.09 examples/s]
Map: 100%|##########| 2617/2617 [00:00<00:00, 7217.10 examples/s]

Map:   0%|          | 0/2617 [00:00<?, ? examples/s]
Map:  30%|##9       | 783/2617 [00:00<00:00, 7769.14 examples/s]
Map:  73%|#######2  | 1903/2617 [00:00<00:00, 7555.89 examples/s]
Map: 100%|##########| 2617/2617 [00:00<00:00, 7253.76 examples/s]

Map:   0%|          | 0/2617 [00:00<?, ? examples/s]
Map: 100%|##########| 2617/2617 [00:00<00:00, 28813.85 examples/s]

Filter:   0%|          | 0/2617 [00:00<?, ? examples/s]
Filter: 100%|##########| 2617/2617 [00:00<00:00, 48010.70 examples/s]

Map:   0%|          | 0/108 [00:00<?, ? examples/s]
Map: 100%|##########| 108/108 [00:00<00:00, 9640.23 examples/s]INFO 2025-05-20 11:02:12,736 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.36%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/6 [00:00<?, ?ba/s]
Creating json from Arrow format:  33%|###3      | 2/6 [00:00<00:00, 19.43ba/s]
Creating json from Arrow format: 100%|##########| 6/6 [00:00<00:00, 30.52ba/s]
Creating json from Arrow format: 100%|##########| 6/6 [00:00<00:00, 28.84ba/s]

Map:   0%|          | 0/760 [00:00<?, ? examples/s]
Map: 100%|##########| 760/760 [00:00<00:00, 7184.00 examples/s]
Map: 100%|##########| 760/760 [00:00<00:00, 7112.57 examples/s]

Map:   0%|          | 0/760 [00:00<?, ? examples/s]
Map: 100%|##########| 760/760 [00:00<00:00, 28116.67 examples/s]

Filter:   0%|          | 0/760 [00:00<?, ? examples/s]
Filter: 100%|##########| 760/760 [00:00<00:00, 44702.85 examples/s]

Map:   0%|          | 0/163 [00:00<?, ? examples/s]
Map: 100%|##########| 163/163 [00:00<00:00, 10439.17 examples/s]

Map:   0%|          | 0/163 [00:00<?, ? examples/s]
Map: 100%|##########| 163/163 [00:00<00:00, 22515.86 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 37.54ba/s]

Map:   0%|          | 0/760 [00:00<?, ? examples/s]
Map: 100%|##########| 760/760 [00:00<00:00, 7313.97 examples/s]
Map: 100%|##########| 760/760 [00:00<00:00, 7241.70 examples/s]

Map:   0%|          | 0/760 [00:00<?, ? examples/s]
Map: 100%|##########| 760/760 [00:00<00:00, 6995.85 examples/s]
Map: 100%|##########| 760/760 [00:00<00:00, 6932.43 examples/s]

Map:   0%|          | 0/760 [00:00<?, ? examples/s]
Map: 100%|##########| 760/760 [00:00<00:00, 7149.87 examples/s]
Map: 100%|##########| 760/760 [00:00<00:00, 7086.25 examples/s]

Map:   0%|          | 0/760 [00:00<?, ? examples/s]
Map: 100%|##########| 760/760 [00:00<00:00, 28430.13 examples/s]

Filter:   0%|          | 0/760 [00:00<?, ? examples/s]
Filter: 100%|##########| 760/760 [00:00<00:00, 44368.11 examples/s]

Map:   0%|          | 0/163 [00:00<?, ? examples/s]
Map: 100%|##########| 163/163 [00:00<00:00, 10674.70 examples/s]INFO 2025-05-20 11:02:13,638 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.43%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 28.54ba/s]

Map:   0%|          | 0/1195 [00:00<?, ? examples/s]
Map:  67%|######7   | 804/1195 [00:00<00:00, 7971.98 examples/s]
Map: 100%|##########| 1195/1195 [00:00<00:00, 7468.26 examples/s]

Map:   0%|          | 0/1195 [00:00<?, ? examples/s]
Map: 100%|##########| 1195/1195 [00:00<00:00, 29141.68 examples/s]

Filter:   0%|          | 0/1195 [00:00<?, ? examples/s]
Filter: 100%|##########| 1195/1195 [00:00<00:00, 46892.89 examples/s]

Map:   0%|          | 0/69 [00:00<?, ? examples/s]
Map: 100%|##########| 69/69 [00:00<00:00, 8968.30 examples/s]

Map:   0%|          | 0/69 [00:00<?, ? examples/s]
Map: 100%|##########| 69/69 [00:00<00:00, 15973.45 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 72.58ba/s]

Map:   0%|          | 0/1195 [00:00<?, ? examples/s]
Map:  67%|######6   | 796/1195 [00:00<00:00, 7902.67 examples/s]
Map: 100%|##########| 1195/1195 [00:00<00:00, 7415.78 examples/s]

Map:   0%|          | 0/1195 [00:00<?, ? examples/s]
Map:  77%|#######7  | 921/1195 [00:00<00:00, 9160.08 examples/s]
Map: 100%|##########| 1195/1195 [00:00<00:00, 7877.08 examples/s]

Map:   0%|          | 0/1195 [00:00<?, ? examples/s]
Map:  67%|######7   | 803/1195 [00:00<00:00, 7971.75 examples/s]
Map: 100%|##########| 1195/1195 [00:00<00:00, 7508.95 examples/s]

Map:   0%|          | 0/1195 [00:00<?, ? examples/s]
Map: 100%|##########| 1195/1195 [00:00<00:00, 30159.23 examples/s]

Filter:   0%|          | 0/1195 [00:00<?, ? examples/s]
Filter: 100%|##########| 1195/1195 [00:00<00:00, 47301.80 examples/s]

Map:   0%|          | 0/69 [00:00<?, ? examples/s]
Map: 100%|##########| 69/69 [00:00<00:00, 9118.91 examples/s]INFO 2025-05-20 11:02:14,651 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.62%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 36.22ba/s]

Map:   0%|          | 0/576 [00:00<?, ? examples/s]
Map: 100%|##########| 576/576 [00:00<00:00, 7156.26 examples/s]

Map:   0%|          | 0/576 [00:00<?, ? examples/s]
Map: 100%|##########| 576/576 [00:00<00:00, 27795.38 examples/s]

Filter:   0%|          | 0/576 [00:00<?, ? examples/s]
Filter: 100%|##########| 576/576 [00:00<00:00, 42311.06 examples/s]

Map:   0%|          | 0/37 [00:00<?, ? examples/s]
Map: 100%|##########| 37/37 [00:00<00:00, 7227.18 examples/s]

Map:   0%|          | 0/37 [00:00<?, ? examples/s]
Map: 100%|##########| 37/37 [00:00<00:00, 11402.59 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 56.41ba/s]

Map:   0%|          | 0/576 [00:00<?, ? examples/s]
Map: 100%|##########| 576/576 [00:00<00:00, 7167.16 examples/s]

Map:   0%|          | 0/576 [00:00<?, ? examples/s]
Map: 100%|##########| 576/576 [00:00<00:00, 7510.01 examples/s]

Map:   0%|          | 0/576 [00:00<?, ? examples/s]
Map: 100%|##########| 576/576 [00:00<00:00, 7254.90 examples/s]

Map:   0%|          | 0/576 [00:00<?, ? examples/s]
Map: 100%|##########| 576/576 [00:00<00:00, 28028.20 examples/s]

Filter:   0%|          | 0/576 [00:00<?, ? examples/s]
Filter: 100%|##########| 576/576 [00:00<00:00, 42099.45 examples/s]

Map:   0%|          | 0/37 [00:00<?, ? examples/s]
Map: 100%|##########| 37/37 [00:00<00:00, 7207.71 examples/s]INFO 2025-05-20 11:02:15,252 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.30%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 34.37ba/s]

Map:   0%|          | 0/8462 [00:00<?, ? examples/s]
Map:   9%|9         | 776/8462 [00:00<00:00, 7689.99 examples/s]
Map:  22%|##2       | 1891/8462 [00:00<00:00, 7510.60 examples/s]
Map:  35%|###5      | 3000/8462 [00:00<00:00, 7187.62 examples/s]
Map:  45%|####4     | 3796/8462 [00:00<00:00, 7423.09 examples/s]
Map:  58%|#####8    | 4920/8462 [00:00<00:00, 7447.15 examples/s]
Map:  71%|#######   | 5997/8462 [00:00<00:00, 7345.35 examples/s]
Map:  83%|########3 | 7056/8462 [00:00<00:00, 7244.49 examples/s]
Map:  93%|#########2| 7841/8462 [00:01<00:00, 7391.44 examples/s]
Map: 100%|##########| 8462/8462 [00:01<00:00, 7320.32 examples/s]

Map:   0%|          | 0/8462 [00:00<?, ? examples/s]
Map:  35%|###5      | 3000/8462 [00:00<00:00, 29091.35 examples/s]
Map:  71%|#######   | 6000/8462 [00:00<00:00, 28963.22 examples/s]
Map: 100%|##########| 8462/8462 [00:00<00:00, 29498.42 examples/s]

Filter:   0%|          | 0/8462 [00:00<?, ? examples/s]
Filter:  71%|#######   | 6000/8462 [00:00<00:00, 50421.60 examples/s]
Filter: 100%|##########| 8462/8462 [00:00<00:00, 50524.65 examples/s]

Map:   0%|          | 0/483 [00:00<?, ? examples/s]
Map: 100%|##########| 483/483 [00:00<00:00, 11504.95 examples/s]

Map:   0%|          | 0/483 [00:00<?, ? examples/s]
Map: 100%|##########| 483/483 [00:00<00:00, 27171.45 examples/s]

Creating json from Arrow format:   0%|          | 0/9 [00:00<?, ?ba/s]
Creating json from Arrow format:  44%|####4     | 4/9 [00:00<00:00, 34.13ba/s]
Creating json from Arrow format:  89%|########8 | 8/9 [00:00<00:00, 33.09ba/s]
Creating json from Arrow format: 100%|##########| 9/9 [00:00<00:00, 33.71ba/s]

Map:   0%|          | 0/8462 [00:00<?, ? examples/s]
Map:   9%|9         | 788/8462 [00:00<00:00, 7822.21 examples/s]
Map:  23%|##2       | 1909/8462 [00:00<00:00, 7578.66 examples/s]
Map:  35%|###5      | 3000/8462 [00:00<00:00, 7211.75 examples/s]
Map:  45%|####4     | 3797/8462 [00:00<00:00, 7447.04 examples/s]
Map:  58%|#####8    | 4921/8462 [00:00<00:00, 7462.57 examples/s]
Map:  71%|#######   | 5997/8462 [00:00<00:00, 7353.18 examples/s]
Map:  83%|########3 | 7056/8462 [00:00<00:00, 7248.86 examples/s]
Map:  93%|#########2| 7840/8462 [00:01<00:00, 7391.67 examples/s]
Map: 100%|##########| 8462/8462 [00:01<00:00, 7332.22 examples/s]

Map:   0%|          | 0/8462 [00:00<?, ? examples/s]
Map:   9%|8         | 725/8462 [00:00<00:01, 7206.28 examples/s]
Map:  21%|##        | 1743/8462 [00:00<00:01, 6628.41 examples/s]
Map:  33%|###2      | 2755/8462 [00:00<00:00, 6536.69 examples/s]
Map:  44%|####4     | 3751/8462 [00:00<00:00, 6508.65 examples/s]
Map:  56%|#####6    | 4751/8462 [00:00<00:00, 6536.41 examples/s]
Map:  68%|######8   | 5769/8462 [00:00<00:00, 6572.52 examples/s]
Map:  80%|#######9  | 6728/8462 [00:01<00:00, 6286.05 examples/s]
Map:  87%|########7 | 7363/8462 [00:01<00:00, 6215.12 examples/s]
Map:  95%|#########4| 8000/8462 [00:01<00:00, 6135.91 examples/s]
Map: 100%|##########| 8462/8462 [00:01<00:00, 6378.66 examples/s]

Map:   0%|          | 0/8462 [00:00<?, ? examples/s]
Map:   9%|9         | 776/8462 [00:00<00:00, 7696.21 examples/s]
Map:  22%|##1       | 1855/8462 [00:00<00:00, 7348.89 examples/s]
Map:  35%|###5      | 2967/8462 [00:00<00:00, 7375.65 examples/s]
Map:  47%|####7     | 4006/8462 [00:00<00:00, 7186.64 examples/s]
Map:  57%|#####6    | 4795/8462 [00:00<00:00, 7381.41 examples/s]
Map:  69%|######9   | 5862/8462 [00:00<00:00, 7278.11 examples/s]
Map:  82%|########2 | 6973/8462 [00:00<00:00, 7319.33 examples/s]
Map:  95%|#########4| 8029/8462 [00:01<00:00, 7223.55 examples/s]
Map: 100%|##########| 8462/8462 [00:01<00:00, 7279.17 examples/s]

Map:   0%|          | 0/8462 [00:00<?, ? examples/s]
Map:  35%|###5      | 3000/8462 [00:00<00:00, 28790.42 examples/s]
Map:  71%|#######   | 6000/8462 [00:00<00:00, 29061.80 examples/s]
Map: 100%|##########| 8462/8462 [00:00<00:00, 29571.66 examples/s]

Filter:   0%|          | 0/8462 [00:00<?, ? examples/s]
Filter:  71%|#######   | 6000/8462 [00:00<00:00, 50054.94 examples/s]
Filter: 100%|##########| 8462/8462 [00:00<00:00, 50209.44 examples/s]

Map:   0%|          | 0/483 [00:00<?, ? examples/s]
Map: 100%|##########| 483/483 [00:00<00:00, 11577.80 examples/s]

Creating json from Arrow format:   0%|          | 0/18 [00:00<?, ?ba/s]
Creating json from Arrow format:  11%|#1        | 2/18 [00:00<00:00, 16.25ba/s]
Creating json from Arrow format:  22%|##2       | 4/18 [00:00<00:00, 17.82ba/s]
Creating json from Arrow format:  33%|###3      | 6/18 [00:00<00:00, 17.63ba/s]
Creating json from Arrow format:  44%|####4     | 8/18 [00:00<00:00, 17.98ba/s]
Creating json from Arrow format:  67%|######6   | 12/18 [00:00<00:00, 23.87ba/s]
Creating json from Arrow format:  89%|########8 | 16/18 [00:00<00:00, 28.40ba/s]
Creating json from Arrow format: 100%|##########| 18/18 [00:00<00:00, 25.27ba/s]

Map:   0%|          | 0/402 [00:00<?, ? examples/s]
Map: 100%|##########| 402/402 [00:00<00:00, 7160.34 examples/s]

Map:   0%|          | 0/402 [00:00<?, ? examples/s]
Map: 100%|##########| 402/402 [00:00<00:00, 26681.91 examples/s]

Filter:   0%|          | 0/402 [00:00<?, ? examples/s]
Filter: 100%|##########| 402/402 [00:00<00:00, 40946.87 examples/s]

Map:   0%|          | 0/24 [00:00<?, ? examples/s]
Map: 100%|##########| 24/24 [00:00<00:00, 5821.38 examples/s]

Map:   0%|          | 0/24 [00:00<?, ? examples/s]
Map: 100%|##########| 24/24 [00:00<00:00, 8370.47 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 109.59ba/s]

Map:   0%|          | 0/402 [00:00<?, ? examples/s]
Map: 100%|##########| 402/402 [00:00<00:00, 7224.62 examples/s]

Map:   0%|          | 0/402 [00:00<?, ? examples/s]
Map: 100%|##########| 402/402 [00:00<00:00, 7932.66 examples/s]

Map:   0%|          | 0/402 [00:00<?, ? examples/s]
Map: 100%|##########| 402/402 [00:00<00:00, 7309.53 examples/s]

Map:   0%|          | 0/402 [00:00<?, ? examples/s]
Map: 100%|##########| 402/402 [00:00<00:00, 27647.04 examples/s]

Filter:   0%|          | 0/402 [00:00<?, ? examples/s]
Filter: 100%|##########| 402/402 [00:00<00:00, 41916.97 examples/s]

Map:   0%|          | 0/24 [00:00<?, ? examples/s]
Map: 100%|##########| 24/24 [00:00<00:00, 5517.91 examples/s]INFO 2025-05-20 11:02:23,214 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.21%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 31.97ba/s]

Map:   0%|          | 0/98 [00:00<?, ? examples/s]
Map: 100%|##########| 98/98 [00:00<00:00, 6036.03 examples/s]

Map:   0%|          | 0/98 [00:00<?, ? examples/s]
Map: 100%|##########| 98/98 [00:00<00:00, 18381.26 examples/s]

Filter:   0%|          | 0/98 [00:00<?, ? examples/s]
Filter: 100%|##########| 98/98 [00:00<00:00, 26237.83 examples/s]

Map:   0%|          | 0/27 [00:00<?, ? examples/s]
Map: 100%|##########| 27/27 [00:00<00:00, 6295.65 examples/s]

Map:   0%|          | 0/27 [00:00<?, ? examples/s]
Map: 100%|##########| 27/27 [00:00<00:00, 9053.18 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 325.37ba/s]

Map:   0%|          | 0/98 [00:00<?, ? examples/s]
Map: 100%|##########| 98/98 [00:00<00:00, 6119.52 examples/s]

Map:   0%|          | 0/98 [00:00<?, ? examples/s]
Map: 100%|##########| 98/98 [00:00<00:00, 6797.11 examples/s]

Map:   0%|          | 0/98 [00:00<?, ? examples/s]
Map: 100%|##########| 98/98 [00:00<00:00, 6211.25 examples/s]

Map:   0%|          | 0/98 [00:00<?, ? examples/s]
Map: 100%|##########| 98/98 [00:00<00:00, 18628.68 examples/s]

Filter:   0%|          | 0/98 [00:00<?, ? examples/s]
Filter: 100%|##########| 98/98 [00:00<00:00, 26448.86 examples/s]

Map:   0%|          | 0/27 [00:00<?, ? examples/s]
Map: 100%|##########| 27/27 [00:00<00:00, 6150.34 examples/s]INFO 2025-05-20 11:02:23,408 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.06%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 137.28ba/s]

Map:   0%|          | 0/1297 [00:00<?, ? examples/s]
Map:  60%|#####9    | 778/1297 [00:00<00:00, 7723.04 examples/s]
Map: 100%|##########| 1297/1297 [00:00<00:00, 7125.14 examples/s]

Map:   0%|          | 0/1297 [00:00<?, ? examples/s]
Map: 100%|##########| 1297/1297 [00:00<00:00, 27834.12 examples/s]

Filter:   0%|          | 0/1297 [00:00<?, ? examples/s]
Filter: 100%|##########| 1297/1297 [00:00<00:00, 46153.04 examples/s]

Map:   0%|          | 0/100 [00:00<?, ? examples/s]
Map: 100%|##########| 100/100 [00:00<00:00, 9412.50 examples/s]

Map:   0%|          | 0/100 [00:00<?, ? examples/s]
Map: 100%|##########| 100/100 [00:00<00:00, 19172.21 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 49.35ba/s]

Map:   0%|          | 0/1297 [00:00<?, ? examples/s]
Map:  60%|#####9    | 777/1297 [00:00<00:00, 7707.38 examples/s]
Map: 100%|##########| 1297/1297 [00:00<00:00, 7132.11 examples/s]

Map:   0%|          | 0/1297 [00:00<?, ? examples/s]
Map:  67%|######7   | 870/1297 [00:00<00:00, 8648.87 examples/s]
Map: 100%|##########| 1297/1297 [00:00<00:00, 7150.98 examples/s]

Map:   0%|          | 0/1297 [00:00<?, ? examples/s]
Map:  60%|######    | 780/1297 [00:00<00:00, 7743.09 examples/s]
Map: 100%|##########| 1297/1297 [00:00<00:00, 7141.74 examples/s]

Map:   0%|          | 0/1297 [00:00<?, ? examples/s]
Map: 100%|##########| 1297/1297 [00:00<00:00, 27713.17 examples/s]

Filter:   0%|          | 0/1297 [00:00<?, ? examples/s]
Filter: 100%|##########| 1297/1297 [00:00<00:00, 45820.28 examples/s]

Map:   0%|          | 0/100 [00:00<?, ? examples/s]
Map: 100%|##########| 100/100 [00:00<00:00, 9597.95 examples/s]INFO 2025-05-20 11:02:24,511 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.68%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format:  67%|######6   | 2/3 [00:00<00:00, 19.79ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 24.87ba/s]

Map:   0%|          | 0/763 [00:00<?, ? examples/s]
Map:  99%|#########9| 756/763 [00:00<00:00, 7495.44 examples/s]
Map: 100%|##########| 763/763 [00:00<00:00, 6950.47 examples/s]

Map:   0%|          | 0/763 [00:00<?, ? examples/s]
Map: 100%|##########| 763/763 [00:00<00:00, 27995.29 examples/s]

Filter:   0%|          | 0/763 [00:00<?, ? examples/s]
Filter: 100%|##########| 763/763 [00:00<00:00, 43986.72 examples/s]

Map:   0%|          | 0/137 [00:00<?, ? examples/s]
Map: 100%|##########| 137/137 [00:00<00:00, 10047.91 examples/s]

Map:   0%|          | 0/137 [00:00<?, ? examples/s]
Map: 100%|##########| 137/137 [00:00<00:00, 20577.25 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 33.53ba/s]

Map:   0%|          | 0/763 [00:00<?, ? examples/s]
Map: 100%|##########| 763/763 [00:00<00:00, 7162.93 examples/s]
Map: 100%|##########| 763/763 [00:00<00:00, 7094.07 examples/s]

Map:   0%|          | 0/763 [00:00<?, ? examples/s]
Map: 100%|##########| 763/763 [00:00<00:00, 6981.22 examples/s]
Map: 100%|##########| 763/763 [00:00<00:00, 6917.75 examples/s]

Map:   0%|          | 0/763 [00:00<?, ? examples/s]
Map: 100%|##########| 763/763 [00:00<00:00, 7166.51 examples/s]
Map: 100%|##########| 763/763 [00:00<00:00, 7101.20 examples/s]

Map:   0%|          | 0/763 [00:00<?, ? examples/s]
Map: 100%|##########| 763/763 [00:00<00:00, 28535.73 examples/s]

Filter:   0%|          | 0/763 [00:00<?, ? examples/s]
Filter: 100%|##########| 763/763 [00:00<00:00, 43913.09 examples/s]

Map:   0%|          | 0/137 [00:00<?, ? examples/s]
Map: 100%|##########| 137/137 [00:00<00:00, 10154.98 examples/s]INFO 2025-05-20 11:02:25,335 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.42%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 23.56ba/s]

Map:   0%|          | 0/362 [00:00<?, ? examples/s]
Map: 100%|##########| 362/362 [00:00<00:00, 6882.64 examples/s]

Map:   0%|          | 0/362 [00:00<?, ? examples/s]
Map: 100%|##########| 362/362 [00:00<00:00, 25726.69 examples/s]

Filter:   0%|          | 0/362 [00:00<?, ? examples/s]
Filter: 100%|##########| 362/362 [00:00<00:00, 39471.18 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 5275.22 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 7486.65 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 83.40ba/s]

Map:   0%|          | 0/362 [00:00<?, ? examples/s]
Map: 100%|##########| 362/362 [00:00<00:00, 6927.58 examples/s]

Map:   0%|          | 0/362 [00:00<?, ? examples/s]
Map: 100%|##########| 362/362 [00:00<00:00, 7218.73 examples/s]

Map:   0%|          | 0/362 [00:00<?, ? examples/s]
Map: 100%|##########| 362/362 [00:00<00:00, 6986.68 examples/s]

Map:   0%|          | 0/362 [00:00<?, ? examples/s]
Map: 100%|##########| 362/362 [00:00<00:00, 26845.21 examples/s]

Filter:   0%|          | 0/362 [00:00<?, ? examples/s]
Filter: 100%|##########| 362/362 [00:00<00:00, 40901.30 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 5323.36 examples/s]INFO 2025-05-20 11:02:25,780 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.19%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 23.27ba/s]

Map:   0%|          | 0/270 [00:00<?, ? examples/s]
Map: 100%|##########| 270/270 [00:00<00:00, 6546.02 examples/s]

Map:   0%|          | 0/270 [00:00<?, ? examples/s]
Map: 100%|##########| 270/270 [00:00<00:00, 24501.03 examples/s]

Filter:   0%|          | 0/270 [00:00<?, ? examples/s]
Filter: 100%|##########| 270/270 [00:00<00:00, 35894.20 examples/s]

Map:   0%|          | 0/36 [00:00<?, ? examples/s]
Map: 100%|##########| 36/36 [00:00<00:00, 6695.12 examples/s]

Map:   0%|          | 0/36 [00:00<?, ? examples/s]
Map: 100%|##########| 36/36 [00:00<00:00, 11177.36 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 86.35ba/s]

Map:   0%|          | 0/270 [00:00<?, ? examples/s]
Map: 100%|##########| 270/270 [00:00<00:00, 6586.15 examples/s]

Map:   0%|          | 0/270 [00:00<?, ? examples/s]
Map: 100%|##########| 270/270 [00:00<00:00, 6709.34 examples/s]

Map:   0%|          | 0/270 [00:00<?, ? examples/s]
Map: 100%|##########| 270/270 [00:00<00:00, 6573.84 examples/s]

Map:   0%|          | 0/270 [00:00<?, ? examples/s]
Map: 100%|##########| 270/270 [00:00<00:00, 25270.84 examples/s]

Filter:   0%|          | 0/270 [00:00<?, ? examples/s]
Filter: 100%|##########| 270/270 [00:00<00:00, 36025.52 examples/s]

Map:   0%|          | 0/36 [00:00<?, ? examples/s]
Map: 100%|##########| 36/36 [00:00<00:00, 6644.73 examples/s]INFO 2025-05-20 11:02:26,132 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.15%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 27.13ba/s]

Map:   0%|          | 0/164 [00:00<?, ? examples/s]
Map: 100%|##########| 164/164 [00:00<00:00, 6469.10 examples/s]

Map:   0%|          | 0/164 [00:00<?, ? examples/s]
Map: 100%|##########| 164/164 [00:00<00:00, 21592.30 examples/s]

Filter:   0%|          | 0/164 [00:00<?, ? examples/s]
Filter: 100%|##########| 164/164 [00:00<00:00, 32289.62 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 7465.95 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 12603.43 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 169.52ba/s]

Map:   0%|          | 0/164 [00:00<?, ? examples/s]
Map: 100%|##########| 164/164 [00:00<00:00, 6462.29 examples/s]

Map:   0%|          | 0/164 [00:00<?, ? examples/s]
Map: 100%|##########| 164/164 [00:00<00:00, 7004.03 examples/s]

Map:   0%|          | 0/164 [00:00<?, ? examples/s]
Map: 100%|##########| 164/164 [00:00<00:00, 6552.60 examples/s]

Map:   0%|          | 0/164 [00:00<?, ? examples/s]
Map: 100%|##########| 164/164 [00:00<00:00, 22398.03 examples/s]

Filter:   0%|          | 0/164 [00:00<?, ? examples/s]
Filter: 100%|##########| 164/164 [00:00<00:00, 32059.37 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 7430.58 examples/s]INFO 2025-05-20 11:02:26,387 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.09%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 62.10ba/s]

Map:   0%|          | 0/483 [00:00<?, ? examples/s]
Map: 100%|##########| 483/483 [00:00<00:00, 6785.72 examples/s]

Map:   0%|          | 0/483 [00:00<?, ? examples/s]
Map: 100%|##########| 483/483 [00:00<00:00, 26550.40 examples/s]

Filter:   0%|          | 0/483 [00:00<?, ? examples/s]
Filter: 100%|##########| 483/483 [00:00<00:00, 39517.19 examples/s]

Map:   0%|          | 0/68 [00:00<?, ? examples/s]
Map: 100%|##########| 68/68 [00:00<00:00, 8472.59 examples/s]

Map:   0%|          | 0/68 [00:00<?, ? examples/s]
Map: 100%|##########| 68/68 [00:00<00:00, 15726.33 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 45.79ba/s]

Map:   0%|          | 0/483 [00:00<?, ? examples/s]
Map: 100%|##########| 483/483 [00:00<00:00, 6732.86 examples/s]

Map:   0%|          | 0/483 [00:00<?, ? examples/s]
Map: 100%|##########| 483/483 [00:00<00:00, 6817.67 examples/s]

Map:   0%|          | 0/483 [00:00<?, ? examples/s]
Map: 100%|##########| 483/483 [00:00<00:00, 1606.69 examples/s]
Map: 100%|##########| 483/483 [00:00<00:00, 1600.44 examples/s]

Map:   0%|          | 0/483 [00:00<?, ? examples/s]
Map: 100%|##########| 483/483 [00:00<00:00, 25011.10 examples/s]

Filter:   0%|          | 0/483 [00:00<?, ? examples/s]
Filter: 100%|##########| 483/483 [00:00<00:00, 38109.95 examples/s]

Map:   0%|          | 0/68 [00:00<?, ? examples/s]
Map: 100%|##########| 68/68 [00:00<00:00, 8082.66 examples/s]INFO 2025-05-20 11:02:27,123 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.26%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 32.29ba/s]

Map:   0%|          | 0/2008 [00:00<?, ? examples/s]
Map:  38%|###8      | 772/2008 [00:00<00:00, 7658.13 examples/s]
Map:  92%|#########2| 1856/2008 [00:00<00:00, 7357.43 examples/s]
Map: 100%|##########| 2008/2008 [00:00<00:00, 7063.69 examples/s]

Map:   0%|          | 0/2008 [00:00<?, ? examples/s]
Map: 100%|##########| 2008/2008 [00:00<00:00, 28587.21 examples/s]

Filter:   0%|          | 0/2008 [00:00<?, ? examples/s]
Filter: 100%|##########| 2008/2008 [00:00<00:00, 45715.73 examples/s]

Map:   0%|          | 0/118 [00:00<?, ? examples/s]
Map: 100%|##########| 118/118 [00:00<00:00, 9689.46 examples/s]

Map:   0%|          | 0/118 [00:00<?, ? examples/s]
Map: 100%|##########| 118/118 [00:00<00:00, 19813.76 examples/s]

Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 46.72ba/s]

Map:   0%|          | 0/2008 [00:00<?, ? examples/s]
Map:  38%|###8      | 772/2008 [00:00<00:00, 7657.88 examples/s]
Map:  94%|#########3| 1879/2008 [00:00<00:00, 7463.12 examples/s]
Map: 100%|##########| 2008/2008 [00:00<00:00, 7152.79 examples/s]

Map:   0%|          | 0/2008 [00:00<?, ? examples/s]
Map:  42%|####1     | 842/2008 [00:00<00:00, 8376.77 examples/s]
Map:  95%|#########5| 1911/2008 [00:00<00:00, 7520.30 examples/s]
Map: 100%|##########| 2008/2008 [00:00<00:00, 6889.99 examples/s]

Map:   0%|          | 0/2008 [00:00<?, ? examples/s]
Map:  38%|###8      | 771/2008 [00:00<00:00, 7657.57 examples/s]
Map:  93%|#########3| 1874/2008 [00:00<00:00, 7446.49 examples/s]
Map: 100%|##########| 2008/2008 [00:00<00:00, 7127.27 examples/s]

Map:   0%|          | 0/2008 [00:00<?, ? examples/s]
Map: 100%|##########| 2008/2008 [00:00<00:00, 29083.26 examples/s]

Filter:   0%|          | 0/2008 [00:00<?, ? examples/s]
Filter: 100%|##########| 2008/2008 [00:00<00:00, 46111.69 examples/s]

Map:   0%|          | 0/118 [00:00<?, ? examples/s]
Map: 100%|##########| 118/118 [00:00<00:00, 9829.94 examples/s]INFO 2025-05-20 11:02:28,856 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.05%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/5 [00:00<?, ?ba/s]
Creating json from Arrow format:  40%|####      | 2/5 [00:00<00:00, 16.08ba/s]
Creating json from Arrow format: 100%|##########| 5/5 [00:00<00:00, 26.06ba/s]

Map:   0%|          | 0/1131 [00:00<?, ? examples/s]
Map:  69%|######9   | 781/1131 [00:00<00:00, 7751.35 examples/s]
Map: 100%|##########| 1131/1131 [00:00<00:00, 7177.89 examples/s]

Map:   0%|          | 0/1131 [00:00<?, ? examples/s]
Map: 100%|##########| 1131/1131 [00:00<00:00, 28117.16 examples/s]

Filter:   0%|          | 0/1131 [00:00<?, ? examples/s]
Filter: 100%|##########| 1131/1131 [00:00<00:00, 45578.43 examples/s]

Map:   0%|          | 0/58 [00:00<?, ? examples/s]
Map: 100%|##########| 58/58 [00:00<00:00, 8575.49 examples/s]

Map:   0%|          | 0/58 [00:00<?, ? examples/s]
Map: 100%|##########| 58/58 [00:00<00:00, 14381.04 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 54.81ba/s]

Map:   0%|          | 0/1131 [00:00<?, ? examples/s]
Map:  69%|######8   | 778/1131 [00:00<00:00, 7723.27 examples/s]
Map: 100%|##########| 1131/1131 [00:00<00:00, 7183.28 examples/s]

Map:   0%|          | 0/1131 [00:00<?, ? examples/s]
Map:  79%|#######8  | 893/1131 [00:00<00:00, 8885.98 examples/s]
Map: 100%|##########| 1131/1131 [00:00<00:00, 7291.76 examples/s]

Map:   0%|          | 0/1131 [00:00<?, ? examples/s]
Map:  70%|######9   | 790/1131 [00:00<00:00, 7842.72 examples/s]
Map: 100%|##########| 1131/1131 [00:00<00:00, 7325.01 examples/s]

Map:   0%|          | 0/1131 [00:00<?, ? examples/s]
Map: 100%|##########| 1131/1131 [00:00<00:00, 27956.59 examples/s]

Filter:   0%|          | 0/1131 [00:00<?, ? examples/s]
Filter: 100%|##########| 1131/1131 [00:00<00:00, 45422.20 examples/s]

Map:   0%|          | 0/58 [00:00<?, ? examples/s]
Map: 100%|##########| 58/58 [00:00<00:00, 8412.98 examples/s]INFO 2025-05-20 11:02:29,995 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.59%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format:  67%|######6   | 2/3 [00:00<00:00, 19.61ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 26.99ba/s]

Map:   0%|          | 0/565 [00:00<?, ? examples/s]
Map: 100%|##########| 565/565 [00:00<00:00, 7510.12 examples/s]

Map:   0%|          | 0/565 [00:00<?, ? examples/s]
Map: 100%|##########| 565/565 [00:00<00:00, 28341.25 examples/s]

Filter:   0%|          | 0/565 [00:00<?, ? examples/s]
Filter: 100%|##########| 565/565 [00:00<00:00, 43621.50 examples/s]

Map:   0%|          | 0/19 [00:00<?, ? examples/s]
Map: 100%|##########| 19/19 [00:00<00:00, 5220.21 examples/s]

Map:   0%|          | 0/19 [00:00<?, ? examples/s]
Map: 100%|##########| 19/19 [00:00<00:00, 6961.80 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 112.05ba/s]

Map:   0%|          | 0/565 [00:00<?, ? examples/s]
Map: 100%|##########| 565/565 [00:00<00:00, 7402.26 examples/s]

Map:   0%|          | 0/565 [00:00<?, ? examples/s]
Map: 100%|##########| 565/565 [00:00<00:00, 8527.80 examples/s]

Map:   0%|          | 0/565 [00:00<?, ? examples/s]
Map: 100%|##########| 565/565 [00:00<00:00, 7648.16 examples/s]

Map:   0%|          | 0/565 [00:00<?, ? examples/s]
Map: 100%|##########| 565/565 [00:00<00:00, 28976.81 examples/s]

Filter:   0%|          | 0/565 [00:00<?, ? examples/s]
Filter: 100%|##########| 565/565 [00:00<00:00, 44202.45 examples/s]

Map:   0%|          | 0/19 [00:00<?, ? examples/s]
Map: 100%|##########| 19/19 [00:00<00:00, 5313.85 examples/s]INFO 2025-05-20 11:02:30,571 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.29%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 65.48ba/s]

Map:   0%|          | 0/2995 [00:00<?, ? examples/s]
Map:  26%|##6       | 793/2995 [00:00<00:00, 7866.89 examples/s]
Map:  64%|######3   | 1916/2995 [00:00<00:00, 7601.02 examples/s]
Map: 100%|##########| 2995/2995 [00:00<00:00, 7217.89 examples/s]
Map: 100%|##########| 2995/2995 [00:00<00:00, 7308.38 examples/s]

Map:   0%|          | 0/2995 [00:00<?, ? examples/s]
Map: 100%|##########| 2995/2995 [00:00<00:00, 29571.91 examples/s]
Map: 100%|##########| 2995/2995 [00:00<00:00, 29285.74 examples/s]

Filter:   0%|          | 0/2995 [00:00<?, ? examples/s]
Filter: 100%|##########| 2995/2995 [00:00<00:00, 47179.76 examples/s]

Map:   0%|          | 0/156 [00:00<?, ? examples/s]
Map: 100%|##########| 156/156 [00:00<00:00, 10486.10 examples/s]

Map:   0%|          | 0/156 [00:00<?, ? examples/s]
Map: 100%|##########| 156/156 [00:00<00:00, 22000.32 examples/s]

Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 46.23ba/s]

Map:   0%|          | 0/2995 [00:00<?, ? examples/s]
Map:  26%|##6       | 791/2995 [00:00<00:00, 7853.44 examples/s]
Map:  64%|######4   | 1928/2995 [00:00<00:00, 7661.58 examples/s]
Map: 100%|##########| 2995/2995 [00:00<00:00, 7271.85 examples/s]
Map: 100%|##########| 2995/2995 [00:00<00:00, 7358.31 examples/s]

Map:   0%|          | 0/2995 [00:00<?, ? examples/s]
Map:  29%|##9       | 871/2995 [00:00<00:00, 8660.25 examples/s]
Map:  66%|######5   | 1967/2995 [00:00<00:00, 7736.84 examples/s]
Map: 100%|##########| 2995/2995 [00:00<00:00, 6941.66 examples/s]
Map: 100%|##########| 2995/2995 [00:00<00:00, 7171.86 examples/s]

Map:   0%|          | 0/2995 [00:00<?, ? examples/s]
Map:  26%|##6       | 788/2995 [00:00<00:00, 7820.94 examples/s]
Map:  64%|######4   | 1923/2995 [00:00<00:00, 7644.13 examples/s]
Map: 100%|##########| 2995/2995 [00:00<00:00, 7311.77 examples/s]
Map: 100%|##########| 2995/2995 [00:00<00:00, 7384.51 examples/s]

Map:   0%|          | 0/2995 [00:00<?, ? examples/s]
Map: 100%|##########| 2995/2995 [00:00<00:00, 29840.25 examples/s]
Map: 100%|##########| 2995/2995 [00:00<00:00, 29543.19 examples/s]

Filter:   0%|          | 0/2995 [00:00<?, ? examples/s]
Filter: 100%|##########| 2995/2995 [00:00<00:00, 47335.31 examples/s]

Map:   0%|          | 0/156 [00:00<?, ? examples/s]
Map: 100%|##########| 156/156 [00:00<00:00, 10432.59 examples/s]INFO 2025-05-20 11:02:33,015 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.56%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/7 [00:00<?, ?ba/s]
Creating json from Arrow format:  29%|##8       | 2/7 [00:00<00:00, 18.71ba/s]
Creating json from Arrow format:  71%|#######1  | 5/7 [00:00<00:00, 24.10ba/s]
Creating json from Arrow format: 100%|##########| 7/7 [00:00<00:00, 28.53ba/s]

Map:   0%|          | 0/394 [00:00<?, ? examples/s]
Map: 100%|##########| 394/394 [00:00<00:00, 7256.96 examples/s]

Map:   0%|          | 0/394 [00:00<?, ? examples/s]
Map: 100%|##########| 394/394 [00:00<00:00, 26760.30 examples/s]

Filter:   0%|          | 0/394 [00:00<?, ? examples/s]
Filter: 100%|##########| 394/394 [00:00<00:00, 41904.75 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 5328.47 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 7527.47 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 124.54ba/s]

Map:   0%|          | 0/394 [00:00<?, ? examples/s]
Map: 100%|##########| 394/394 [00:00<00:00, 7366.73 examples/s]

Map:   0%|          | 0/394 [00:00<?, ? examples/s]
Map: 100%|##########| 394/394 [00:00<00:00, 8299.13 examples/s]

Map:   0%|          | 0/394 [00:00<?, ? examples/s]
Map: 100%|##########| 394/394 [00:00<00:00, 7408.67 examples/s]

Map:   0%|          | 0/394 [00:00<?, ? examples/s]
Map: 100%|##########| 394/394 [00:00<00:00, 28143.46 examples/s]

Filter:   0%|          | 0/394 [00:00<?, ? examples/s]
Filter: 100%|##########| 394/394 [00:00<00:00, 42664.22 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 5424.60 examples/s]INFO 2025-05-20 11:02:33,618 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.21%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 43.80ba/s]

Map:   0%|          | 0/431 [00:00<?, ? examples/s]
Map: 100%|##########| 431/431 [00:00<00:00, 7264.98 examples/s]

Map:   0%|          | 0/431 [00:00<?, ? examples/s]
Map: 100%|##########| 431/431 [00:00<00:00, 27455.80 examples/s]

Filter:   0%|          | 0/431 [00:00<?, ? examples/s]
Filter: 100%|##########| 431/431 [00:00<00:00, 42679.79 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 5501.93 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 7748.10 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 106.23ba/s]

Map:   0%|          | 0/431 [00:00<?, ? examples/s]
Map: 100%|##########| 431/431 [00:00<00:00, 7364.81 examples/s]

Map:   0%|          | 0/431 [00:00<?, ? examples/s]
Map: 100%|##########| 431/431 [00:00<00:00, 8055.19 examples/s]

Map:   0%|          | 0/431 [00:00<?, ? examples/s]
Map: 100%|##########| 431/431 [00:00<00:00, 7397.79 examples/s]

Map:   0%|          | 0/431 [00:00<?, ? examples/s]
Map: 100%|##########| 431/431 [00:00<00:00, 27619.40 examples/s]

Filter:   0%|          | 0/431 [00:00<?, ? examples/s]
Filter: 100%|##########| 431/431 [00:00<00:00, 42641.53 examples/s]

Map:   0%|          | 0/21 [00:00<?, ? examples/s]
Map: 100%|##########| 21/21 [00:00<00:00, 5443.78 examples/s]INFO 2025-05-20 11:02:34,024 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.22%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 31.65ba/s]

Map:   0%|          | 0/639 [00:00<?, ? examples/s]
Map: 100%|##########| 639/639 [00:00<00:00, 7397.79 examples/s]

Map:   0%|          | 0/639 [00:00<?, ? examples/s]
Map: 100%|##########| 639/639 [00:00<00:00, 28452.71 examples/s]

Filter:   0%|          | 0/639 [00:00<?, ? examples/s]
Filter: 100%|##########| 639/639 [00:00<00:00, 44130.94 examples/s]

Map:   0%|          | 0/37 [00:00<?, ? examples/s]
Map: 100%|##########| 37/37 [00:00<00:00, 7475.76 examples/s]

Map:   0%|          | 0/37 [00:00<?, ? examples/s]
Map: 100%|##########| 37/37 [00:00<00:00, 11224.45 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 73.27ba/s]

Map:   0%|          | 0/639 [00:00<?, ? examples/s]
Map: 100%|##########| 639/639 [00:00<00:00, 7402.26 examples/s]

Map:   0%|          | 0/639 [00:00<?, ? examples/s]
Map: 100%|##########| 639/639 [00:00<00:00, 8227.10 examples/s]

Map:   0%|          | 0/639 [00:00<?, ? examples/s]
Map: 100%|##########| 639/639 [00:00<00:00, 7490.02 examples/s]

Map:   0%|          | 0/639 [00:00<?, ? examples/s]
Map: 100%|##########| 639/639 [00:00<00:00, 28670.95 examples/s]

Filter:   0%|          | 0/639 [00:00<?, ? examples/s]
Filter: 100%|##########| 639/639 [00:00<00:00, 44440.47 examples/s]

Map:   0%|          | 0/37 [00:00<?, ? examples/s]
Map: 100%|##########| 37/37 [00:00<00:00, 7380.12 examples/s]INFO 2025-05-20 11:02:34,584 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.33%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 41.58ba/s]

Map:   0%|          | 0/1148 [00:00<?, ? examples/s]
Map:  69%|######9   | 797/1148 [00:00<00:00, 7901.85 examples/s]
Map: 100%|##########| 1148/1148 [00:00<00:00, 7350.32 examples/s]

Map:   0%|          | 0/1148 [00:00<?, ? examples/s]
Map: 100%|##########| 1148/1148 [00:00<00:00, 28239.01 examples/s]

Filter:   0%|          | 0/1148 [00:00<?, ? examples/s]
Filter: 100%|##########| 1148/1148 [00:00<00:00, 47227.78 examples/s]

Map:   0%|          | 0/63 [00:00<?, ? examples/s]
Map: 100%|##########| 63/63 [00:00<00:00, 8778.19 examples/s]

Map:   0%|          | 0/63 [00:00<?, ? examples/s]
Map: 100%|##########| 63/63 [00:00<00:00, 15719.28 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 59.84ba/s]

Map:   0%|          | 0/1148 [00:00<?, ? examples/s]
Map:  69%|######8   | 792/1148 [00:00<00:00, 7857.18 examples/s]
Map: 100%|##########| 1148/1148 [00:00<00:00, 7331.57 examples/s]

Map:   0%|          | 0/1148 [00:00<?, ? examples/s]
Map:  79%|#######8  | 904/1148 [00:00<00:00, 8991.39 examples/s]
Map: 100%|##########| 1148/1148 [00:00<00:00, 7408.23 examples/s]

Map:   0%|          | 0/1148 [00:00<?, ? examples/s]
Map:  69%|######9   | 795/1148 [00:00<00:00, 7894.30 examples/s]
Map: 100%|##########| 1148/1148 [00:00<00:00, 7387.22 examples/s]

Map:   0%|          | 0/1148 [00:00<?, ? examples/s]
Map: 100%|##########| 1148/1148 [00:00<00:00, 28394.71 examples/s]

Filter:   0%|          | 0/1148 [00:00<?, ? examples/s]
Filter: 100%|##########| 1148/1148 [00:00<00:00, 46719.14 examples/s]

Map:   0%|          | 0/63 [00:00<?, ? examples/s]
Map: 100%|##########| 63/63 [00:00<00:00, 8720.54 examples/s]INFO 2025-05-20 11:02:35,564 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.60%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 28.96ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 28.91ba/s]

Map:   0%|          | 0/1398 [00:00<?, ? examples/s]
Map:  57%|#####7    | 801/1398 [00:00<00:00, 7940.45 examples/s]
Map: 100%|##########| 1398/1398 [00:00<00:00, 7526.91 examples/s]

Map:   0%|          | 0/1398 [00:00<?, ? examples/s]
Map: 100%|##########| 1398/1398 [00:00<00:00, 29225.69 examples/s]

Filter:   0%|          | 0/1398 [00:00<?, ? examples/s]
Filter: 100%|##########| 1398/1398 [00:00<00:00, 47433.14 examples/s]

Map:   0%|          | 0/71 [00:00<?, ? examples/s]
Map: 100%|##########| 71/71 [00:00<00:00, 9097.16 examples/s]

Map:   0%|          | 0/71 [00:00<?, ? examples/s]
Map: 100%|##########| 71/71 [00:00<00:00, 16558.92 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 61.51ba/s]

Map:   0%|          | 0/1398 [00:00<?, ? examples/s]
Map:  58%|#####7    | 810/1398 [00:00<00:00, 8045.36 examples/s]
Map: 100%|##########| 1398/1398 [00:00<00:00, 7577.31 examples/s]

Map:   0%|          | 0/1398 [00:00<?, ? examples/s]
Map:  66%|######5   | 919/1398 [00:00<00:00, 9135.88 examples/s]
Map: 100%|##########| 1398/1398 [00:00<00:00, 8084.01 examples/s]

Map:   0%|          | 0/1398 [00:00<?, ? examples/s]
Map:  58%|#####8    | 813/1398 [00:00<00:00, 8079.06 examples/s]
Map: 100%|##########| 1398/1398 [00:00<00:00, 7591.74 examples/s]

Map:   0%|          | 0/1398 [00:00<?, ? examples/s]
Map: 100%|##########| 1398/1398 [00:00<00:00, 29652.97 examples/s]

Filter:   0%|          | 0/1398 [00:00<?, ? examples/s]
Filter: 100%|##########| 1398/1398 [00:00<00:00, 46680.55 examples/s]

Map:   0%|          | 0/71 [00:00<?, ? examples/s]
Map: 100%|##########| 71/71 [00:00<00:00, 8984.09 examples/s]INFO 2025-05-20 11:02:36,753 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.73%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 33.99ba/s]

Map:   0%|          | 0/192 [00:00<?, ? examples/s]
Map: 100%|##########| 192/192 [00:00<00:00, 6683.21 examples/s]

Map:   0%|          | 0/192 [00:00<?, ? examples/s]
Map: 100%|##########| 192/192 [00:00<00:00, 22968.07 examples/s]

Filter:   0%|          | 0/192 [00:00<?, ? examples/s]
Filter: 100%|##########| 192/192 [00:00<00:00, 33623.08 examples/s]

Map:   0%|          | 0/45 [00:00<?, ? examples/s]
Map: 100%|##########| 45/45 [00:00<00:00, 7802.87 examples/s]

Map:   0%|          | 0/45 [00:00<?, ? examples/s]
Map: 100%|##########| 45/45 [00:00<00:00, 12963.16 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 186.49ba/s]

Map:   0%|          | 0/192 [00:00<?, ? examples/s]
Map: 100%|##########| 192/192 [00:00<00:00, 6713.07 examples/s]

Map:   0%|          | 0/192 [00:00<?, ? examples/s]
Map: 100%|##########| 192/192 [00:00<00:00, 7458.82 examples/s]

Map:   0%|          | 0/192 [00:00<?, ? examples/s]
Map: 100%|##########| 192/192 [00:00<00:00, 6961.26 examples/s]

Map:   0%|          | 0/192 [00:00<?, ? examples/s]
Map: 100%|##########| 192/192 [00:00<00:00, 24168.13 examples/s]

Filter:   0%|          | 0/192 [00:00<?, ? examples/s]
Filter: 100%|##########| 192/192 [00:00<00:00, 34458.98 examples/s]

Map:   0%|          | 0/45 [00:00<?, ? examples/s]
Map: 100%|##########| 45/45 [00:00<00:00, 7830.06 examples/s]INFO 2025-05-20 11:02:37,076 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.11%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 66.21ba/s]

Map:   0%|          | 0/1092 [00:00<?, ? examples/s]
Map:  72%|#######1  | 782/1092 [00:00<00:00, 7762.89 examples/s]
Map: 100%|##########| 1092/1092 [00:00<00:00, 6999.15 examples/s]

Map:   0%|          | 0/1092 [00:00<?, ? examples/s]
Map: 100%|##########| 1092/1092 [00:00<00:00, 27879.82 examples/s]

Filter:   0%|          | 0/1092 [00:00<?, ? examples/s]
Filter: 100%|##########| 1092/1092 [00:00<00:00, 46472.40 examples/s]

Map:   0%|          | 0/64 [00:00<?, ? examples/s]
Map: 100%|##########| 64/64 [00:00<00:00, 8730.46 examples/s]

Map:   0%|          | 0/64 [00:00<?, ? examples/s]
Map: 100%|##########| 64/64 [00:00<00:00, 15564.18 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 51.56ba/s]

Map:   0%|          | 0/1092 [00:00<?, ? examples/s]
Map:  72%|#######1  | 782/1092 [00:00<00:00, 7763.48 examples/s]
Map: 100%|##########| 1092/1092 [00:00<00:00, 7145.54 examples/s]

Map:   0%|          | 0/1092 [00:00<?, ? examples/s]
Map:  82%|########1 | 891/1092 [00:00<00:00, 8857.89 examples/s]
Map: 100%|##########| 1092/1092 [00:00<00:00, 7279.10 examples/s]

Map:   0%|          | 0/1092 [00:00<?, ? examples/s]
Map:  73%|#######2  | 793/1092 [00:00<00:00, 7872.70 examples/s]
Map: 100%|##########| 1092/1092 [00:00<00:00, 7205.78 examples/s]

Map:   0%|          | 0/1092 [00:00<?, ? examples/s]
Map: 100%|##########| 1092/1092 [00:00<00:00, 28512.61 examples/s]

Filter:   0%|          | 0/1092 [00:00<?, ? examples/s]
Filter: 100%|##########| 1092/1092 [00:00<00:00, 46344.02 examples/s]

Map:   0%|          | 0/64 [00:00<?, ? examples/s]
Map: 100%|##########| 64/64 [00:00<00:00, 8788.77 examples/s]INFO 2025-05-20 11:02:38,011 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.57%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 28.24ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 28.19ba/s]

Map:   0%|          | 0/1246 [00:00<?, ? examples/s]
Map:  63%|######3   | 791/1246 [00:00<00:00, 7842.45 examples/s]
Map: 100%|##########| 1246/1246 [00:00<00:00, 7137.49 examples/s]

Map:   0%|          | 0/1246 [00:00<?, ? examples/s]
Map: 100%|##########| 1246/1246 [00:00<00:00, 28206.82 examples/s]

Filter:   0%|          | 0/1246 [00:00<?, ? examples/s]
Filter: 100%|##########| 1246/1246 [00:00<00:00, 46451.36 examples/s]

Map:   0%|          | 0/58 [00:00<?, ? examples/s]
Map: 100%|##########| 58/58 [00:00<00:00, 8283.78 examples/s]

Map:   0%|          | 0/58 [00:00<?, ? examples/s]
Map: 100%|##########| 58/58 [00:00<00:00, 14796.52 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 49.70ba/s]

Map:   0%|          | 0/1246 [00:00<?, ? examples/s]
Map:  63%|######2   | 782/1246 [00:00<00:00, 7758.56 examples/s]
Map: 100%|##########| 1246/1246 [00:00<00:00, 7127.78 examples/s]

Map:   0%|          | 0/1246 [00:00<?, ? examples/s]
Map:  72%|#######2  | 902/1246 [00:00<00:00, 8973.69 examples/s]
Map: 100%|##########| 1246/1246 [00:00<00:00, 7197.57 examples/s]

Map:   0%|          | 0/1246 [00:00<?, ? examples/s]
Map:  63%|######3   | 786/1246 [00:00<00:00, 7804.00 examples/s]
Map: 100%|##########| 1246/1246 [00:00<00:00, 7062.02 examples/s]

Map:   0%|          | 0/1246 [00:00<?, ? examples/s]
Map: 100%|##########| 1246/1246 [00:00<00:00, 27324.17 examples/s]

Filter:   0%|          | 0/1246 [00:00<?, ? examples/s]
Filter: 100%|##########| 1246/1246 [00:00<00:00, 46936.55 examples/s]

Map:   0%|          | 0/58 [00:00<?, ? examples/s]
Map: 100%|##########| 58/58 [00:00<00:00, 8464.79 examples/s]INFO 2025-05-20 11:02:39,153 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.65%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format:  67%|######6   | 2/3 [00:00<00:00, 18.87ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 24.76ba/s]

Map:   0%|          | 0/288 [00:00<?, ? examples/s]
Map: 100%|##########| 288/288 [00:00<00:00, 6681.89 examples/s]

Map:   0%|          | 0/288 [00:00<?, ? examples/s]
Map: 100%|##########| 288/288 [00:00<00:00, 25318.79 examples/s]

Filter:   0%|          | 0/288 [00:00<?, ? examples/s]
Filter: 100%|##########| 288/288 [00:00<00:00, 37467.73 examples/s]

Map:   0%|          | 0/12 [00:00<?, ? examples/s]
Map: 100%|##########| 12/12 [00:00<00:00, 3725.51 examples/s]

Map:   0%|          | 0/12 [00:00<?, ? examples/s]
Map: 100%|##########| 12/12 [00:00<00:00, 4887.52 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 92.80ba/s]

Map:   0%|          | 0/288 [00:00<?, ? examples/s]
Map: 100%|##########| 288/288 [00:00<00:00, 6774.49 examples/s]

Map:   0%|          | 0/288 [00:00<?, ? examples/s]
Map: 100%|##########| 288/288 [00:00<00:00, 6950.73 examples/s]

Map:   0%|          | 0/288 [00:00<?, ? examples/s]
Map: 100%|##########| 288/288 [00:00<00:00, 6783.28 examples/s]

Map:   0%|          | 0/288 [00:00<?, ? examples/s]
Map: 100%|##########| 288/288 [00:00<00:00, 24207.61 examples/s]

Filter:   0%|          | 0/288 [00:00<?, ? examples/s]
Filter: 100%|##########| 288/288 [00:00<00:00, 38419.88 examples/s]

Map:   0%|          | 0/12 [00:00<?, ? examples/s]
Map: 100%|##########| 12/12 [00:00<00:00, 3852.70 examples/s]INFO 2025-05-20 11:02:39,583 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.15%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 24.88ba/s]

Map:   0%|          | 0/156 [00:00<?, ? examples/s]
Map: 100%|##########| 156/156 [00:00<00:00, 6455.58 examples/s]

Map:   0%|          | 0/156 [00:00<?, ? examples/s]
Map: 100%|##########| 156/156 [00:00<00:00, 22240.36 examples/s]

Filter:   0%|          | 0/156 [00:00<?, ? examples/s]
Filter: 100%|##########| 156/156 [00:00<00:00, 31004.14 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 5238.95 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 7450.58 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 186.07ba/s]

Map:   0%|          | 0/156 [00:00<?, ? examples/s]
Map: 100%|##########| 156/156 [00:00<00:00, 6476.73 examples/s]

Map:   0%|          | 0/156 [00:00<?, ? examples/s]
Map: 100%|##########| 156/156 [00:00<00:00, 6867.10 examples/s]

Map:   0%|          | 0/156 [00:00<?, ? examples/s]
Map: 100%|##########| 156/156 [00:00<00:00, 6470.96 examples/s]

Map:   0%|          | 0/156 [00:00<?, ? examples/s]
Map: 100%|##########| 156/156 [00:00<00:00, 21989.23 examples/s]

Filter:   0%|          | 0/156 [00:00<?, ? examples/s]
Filter: 100%|##########| 156/156 [00:00<00:00, 31642.88 examples/s]

Map:   0%|          | 0/20 [00:00<?, ? examples/s]
Map: 100%|##########| 20/20 [00:00<00:00, 5306.89 examples/s]INFO 2025-05-20 11:02:39,830 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.08%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 56.60ba/s]

Map:   0%|          | 0/160 [00:00<?, ? examples/s]
Map: 100%|##########| 160/160 [00:00<00:00, 6559.04 examples/s]

Map:   0%|          | 0/160 [00:00<?, ? examples/s]
Map: 100%|##########| 160/160 [00:00<00:00, 22073.83 examples/s]

Filter:   0%|          | 0/160 [00:00<?, ? examples/s]
Filter: 100%|##########| 160/160 [00:00<00:00, 32366.58 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 7716.72 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 12725.26 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 191.87ba/s]

Map:   0%|          | 0/160 [00:00<?, ? examples/s]
Map: 100%|##########| 160/160 [00:00<00:00, 6609.17 examples/s]

Map:   0%|          | 0/160 [00:00<?, ? examples/s]
Map: 100%|##########| 160/160 [00:00<00:00, 7019.39 examples/s]

Map:   0%|          | 0/160 [00:00<?, ? examples/s]
Map: 100%|##########| 160/160 [00:00<00:00, 6636.95 examples/s]

Map:   0%|          | 0/160 [00:00<?, ? examples/s]
Map: 100%|##########| 160/160 [00:00<00:00, 22574.29 examples/s]

Filter:   0%|          | 0/160 [00:00<?, ? examples/s]
Filter: 100%|##########| 160/160 [00:00<00:00, 31964.21 examples/s]

Map:   0%|          | 0/43 [00:00<?, ? examples/s]
Map: 100%|##########| 43/43 [00:00<00:00, 7729.61 examples/s]INFO 2025-05-20 11:02:40,057 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.09%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 65.73ba/s]

Map:   0%|          | 0/3081 [00:00<?, ? examples/s]
Map:  25%|##5       | 772/3081 [00:00<00:00, 7649.43 examples/s]
Map:  61%|######    | 1867/3081 [00:00<00:00, 7400.75 examples/s]
Map:  95%|#########5| 2941/3081 [00:00<00:00, 7277.53 examples/s]
Map: 100%|##########| 3081/3081 [00:00<00:00, 7032.26 examples/s]

Map:   0%|          | 0/3081 [00:00<?, ? examples/s]
Map:  97%|#########7| 3000/3081 [00:00<00:00, 29218.49 examples/s]
Map: 100%|##########| 3081/3081 [00:00<00:00, 28881.37 examples/s]

Filter:   0%|          | 0/3081 [00:00<?, ? examples/s]
Filter: 100%|##########| 3081/3081 [00:00<00:00, 48027.63 examples/s]

Map:   0%|          | 0/239 [00:00<?, ? examples/s]
Map: 100%|##########| 239/239 [00:00<00:00, 10720.24 examples/s]

Map:   0%|          | 0/239 [00:00<?, ? examples/s]
Map: 100%|##########| 239/239 [00:00<00:00, 24524.49 examples/s]

Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format:  75%|#######5  | 3/4 [00:00<00:00, 27.75ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 33.93ba/s]

Map:   0%|          | 0/3081 [00:00<?, ? examples/s]
Map:  25%|##4       | 761/3081 [00:00<00:00, 7541.95 examples/s]
Map:  60%|######    | 1856/3081 [00:00<00:00, 7370.91 examples/s]
Map:  96%|#########5| 2950/3081 [00:00<00:00, 7329.09 examples/s]
Map: 100%|##########| 3081/3081 [00:00<00:00, 7076.98 examples/s]

Map:   0%|          | 0/3081 [00:00<?, ? examples/s]
Map:  26%|##5       | 800/3081 [00:00<00:00, 7942.93 examples/s]
Map:  59%|#####9    | 1824/3081 [00:00<00:00, 7180.81 examples/s]
Map:  91%|#########1| 2810/3081 [00:00<00:00, 6876.57 examples/s]
Map: 100%|##########| 3081/3081 [00:00<00:00, 6600.68 examples/s]

Map:   0%|          | 0/3081 [00:00<?, ? examples/s]
Map:  25%|##5       | 779/3081 [00:00<00:00, 7739.87 examples/s]
Map:  61%|######    | 1874/3081 [00:00<00:00, 7436.51 examples/s]
Map:  96%|#########5| 2956/3081 [00:00<00:00, 7319.21 examples/s]
Map: 100%|##########| 3081/3081 [00:00<00:00, 7086.68 examples/s]

Map:   0%|          | 0/3081 [00:00<?, ? examples/s]
Map:  97%|#########7| 3000/3081 [00:00<00:00, 29074.88 examples/s]
Map: 100%|##########| 3081/3081 [00:00<00:00, 28763.09 examples/s]

Filter:   0%|          | 0/3081 [00:00<?, ? examples/s]
Filter: 100%|##########| 3081/3081 [00:00<00:00, 47627.06 examples/s]

Map:   0%|          | 0/239 [00:00<?, ? examples/s]
Map: 100%|##########| 239/239 [00:00<00:00, 10763.06 examples/s]INFO 2025-05-20 11:02:42,687 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.62%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/7 [00:00<?, ?ba/s]
Creating json from Arrow format:  29%|##8       | 2/7 [00:00<00:00, 15.51ba/s]
Creating json from Arrow format:  57%|#####7    | 4/7 [00:00<00:00, 16.22ba/s]
Creating json from Arrow format: 100%|##########| 7/7 [00:00<00:00, 21.18ba/s]

Map:   0%|          | 0/1044 [00:00<?, ? examples/s]
Map:  78%|#######7  | 811/1044 [00:00<00:00, 8043.46 examples/s]
Map: 100%|##########| 1044/1044 [00:00<00:00, 7457.26 examples/s]

Map:   0%|          | 0/1044 [00:00<?, ? examples/s]
Map: 100%|##########| 1044/1044 [00:00<00:00, 27761.53 examples/s]

Filter:   0%|          | 0/1044 [00:00<?, ? examples/s]
Filter: 100%|##########| 1044/1044 [00:00<00:00, 45748.40 examples/s]

Map:   0%|          | 0/57 [00:00<?, ? examples/s]
Map: 100%|##########| 57/57 [00:00<00:00, 8251.09 examples/s]

Map:   0%|          | 0/57 [00:00<?, ? examples/s]
Map: 100%|##########| 57/57 [00:00<00:00, 15208.35 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 83.21ba/s]

Map:   0%|          | 0/1044 [00:00<?, ? examples/s]
Map:  78%|#######8  | 815/1044 [00:00<00:00, 8087.90 examples/s]
Map: 100%|##########| 1044/1044 [00:00<00:00, 7504.67 examples/s]

Map:   0%|          | 0/1044 [00:00<?, ? examples/s]
Map:  88%|########7 | 915/1044 [00:00<00:00, 2804.56 examples/s]
Map: 100%|##########| 1044/1044 [00:00<00:00, 2862.61 examples/s]

Map:   0%|          | 0/1044 [00:00<?, ? examples/s]
Map:  78%|#######8  | 816/1044 [00:00<00:00, 8104.63 examples/s]
Map: 100%|##########| 1044/1044 [00:00<00:00, 7568.97 examples/s]

Map:   0%|          | 0/1044 [00:00<?, ? examples/s]
Map: 100%|##########| 1044/1044 [00:00<00:00, 28984.63 examples/s]

Filter:   0%|          | 0/1044 [00:00<?, ? examples/s]
Filter: 100%|##########| 1044/1044 [00:00<00:00, 45869.16 examples/s]

Map:   0%|          | 0/57 [00:00<?, ? examples/s]
Map: 100%|##########| 57/57 [00:00<00:00, 8606.02 examples/s]INFO 2025-05-20 11:02:44,094 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.54%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 39.80ba/s]

Map:   0%|          | 0/2879 [00:00<?, ? examples/s]
Map:  27%|##7       | 791/2879 [00:00<00:00, 7849.43 examples/s]
Map:  66%|######5   | 1900/2879 [00:00<00:00, 7531.80 examples/s]
Map: 100%|##########| 2879/2879 [00:00<00:00, 7152.40 examples/s]
Map: 100%|##########| 2879/2879 [00:00<00:00, 7248.58 examples/s]

Map:   0%|          | 0/2879 [00:00<?, ? examples/s]
Map: 100%|##########| 2879/2879 [00:00<00:00, 28959.81 examples/s]

Filter:   0%|          | 0/2879 [00:00<?, ? examples/s]
Filter: 100%|##########| 2879/2879 [00:00<00:00, 50153.68 examples/s]

Map:   0%|          | 0/153 [00:00<?, ? examples/s]
Map: 100%|##########| 153/153 [00:00<00:00, 10602.35 examples/s]

Map:   0%|          | 0/153 [00:00<?, ? examples/s]
Map: 100%|##########| 153/153 [00:00<00:00, 22492.32 examples/s]

Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 52.81ba/s]

Map:   0%|          | 0/2879 [00:00<?, ? examples/s]
Map:  27%|##7       | 791/2879 [00:00<00:00, 7843.34 examples/s]
Map:  67%|######7   | 1932/2879 [00:00<00:00, 7673.03 examples/s]
Map: 100%|##########| 2879/2879 [00:00<00:00, 7288.62 examples/s]
Map: 100%|##########| 2879/2879 [00:00<00:00, 7375.03 examples/s]

Map:   0%|          | 0/2879 [00:00<?, ? examples/s]
Map:  30%|###       | 877/2879 [00:00<00:00, 8714.20 examples/s]
Map:  69%|######9   | 2000/2879 [00:00<00:00, 7380.48 examples/s]
Map:  99%|#########9| 2862/2879 [00:00<00:00, 7845.59 examples/s]
Map: 100%|##########| 2879/2879 [00:00<00:00, 7529.53 examples/s]

Map:   0%|          | 0/2879 [00:00<?, ? examples/s]
Map:  28%|##7       | 805/2879 [00:00<00:00, 7982.99 examples/s]
Map:  68%|######8   | 1959/2879 [00:00<00:00, 7777.53 examples/s]
Map: 100%|##########| 2879/2879 [00:00<00:00, 7371.62 examples/s]
Map: 100%|##########| 2879/2879 [00:00<00:00, 7468.94 examples/s]

Map:   0%|          | 0/2879 [00:00<?, ? examples/s]
Map: 100%|##########| 2879/2879 [00:00<00:00, 29793.22 examples/s]

Filter:   0%|          | 0/2879 [00:00<?, ? examples/s]
Filter: 100%|##########| 2879/2879 [00:00<00:00, 50128.07 examples/s]

Map:   0%|          | 0/153 [00:00<?, ? examples/s]
Map: 100%|##########| 153/153 [00:00<00:00, 10365.67 examples/s]INFO 2025-05-20 11:02:46,395 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.50%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/6 [00:00<?, ?ba/s]
Creating json from Arrow format:  33%|###3      | 2/6 [00:00<00:00, 18.84ba/s]
Creating json from Arrow format:  83%|########3 | 5/6 [00:00<00:00, 24.79ba/s]
Creating json from Arrow format: 100%|##########| 6/6 [00:00<00:00, 25.31ba/s]

Map:   0%|          | 0/193 [00:00<?, ? examples/s]
Map: 100%|##########| 193/193 [00:00<00:00, 6649.59 examples/s]

Map:   0%|          | 0/193 [00:00<?, ? examples/s]
Map: 100%|##########| 193/193 [00:00<00:00, 23403.41 examples/s]

Filter:   0%|          | 0/193 [00:00<?, ? examples/s]
Filter: 100%|##########| 193/193 [00:00<00:00, 33942.75 examples/s]

Map:   0%|          | 0/12 [00:00<?, ? examples/s]
Map: 100%|##########| 12/12 [00:00<00:00, 3846.51 examples/s]

Map:   0%|          | 0/12 [00:00<?, ? examples/s]
Map: 100%|##########| 12/12 [00:00<00:00, 4765.80 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 185.80ba/s]

Map:   0%|          | 0/193 [00:00<?, ? examples/s]
Map: 100%|##########| 193/193 [00:00<00:00, 6652.81 examples/s]

Map:   0%|          | 0/193 [00:00<?, ? examples/s]
Map: 100%|##########| 193/193 [00:00<00:00, 7445.46 examples/s]

Map:   0%|          | 0/193 [00:00<?, ? examples/s]
Map: 100%|##########| 193/193 [00:00<00:00, 7053.98 examples/s]

Map:   0%|          | 0/193 [00:00<?, ? examples/s]
Map: 100%|##########| 193/193 [00:00<00:00, 23536.10 examples/s]

Filter:   0%|          | 0/193 [00:00<?, ? examples/s]
Filter: 100%|##########| 193/193 [00:00<00:00, 34871.23 examples/s]

Map:   0%|          | 0/12 [00:00<?, ? examples/s]
Map: 100%|##########| 12/12 [00:00<00:00, 3897.45 examples/s]INFO 2025-05-20 11:02:46,861 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.10%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 58.72ba/s]

Map:   0%|          | 0/3191 [00:00<?, ? examples/s]
Map:  25%|##4       | 790/3191 [00:00<00:00, 7838.37 examples/s]
Map:  59%|#####9    | 1891/3191 [00:00<00:00, 7496.36 examples/s]
Map:  94%|#########4| 3000/3191 [00:00<00:00, 7164.21 examples/s]
Map: 100%|##########| 3191/3191 [00:00<00:00, 7226.67 examples/s]

Map:   0%|          | 0/3191 [00:00<?, ? examples/s]
Map:  94%|#########4| 3000/3191 [00:00<00:00, 28882.61 examples/s]
Map: 100%|##########| 3191/3191 [00:00<00:00, 28589.55 examples/s]

Filter:   0%|          | 0/3191 [00:00<?, ? examples/s]
Filter: 100%|##########| 3191/3191 [00:00<00:00, 48995.58 examples/s]

Map:   0%|          | 0/192 [00:00<?, ? examples/s]
Map: 100%|##########| 192/192 [00:00<00:00, 10733.84 examples/s]

Map:   0%|          | 0/192 [00:00<?, ? examples/s]
Map: 100%|##########| 192/192 [00:00<00:00, 23601.49 examples/s]

Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 36.28ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 36.21ba/s]

Map:   0%|          | 0/3191 [00:00<?, ? examples/s]
Map:  25%|##4       | 782/3191 [00:00<00:00, 7755.77 examples/s]
Map:  59%|#####8    | 1878/3191 [00:00<00:00, 7439.14 examples/s]
Map:  93%|#########3| 2973/3191 [00:00<00:00, 7364.43 examples/s]
Map: 100%|##########| 3191/3191 [00:00<00:00, 7113.58 examples/s]

Map:   0%|          | 0/3191 [00:00<?, ? examples/s]
Map:  27%|##6       | 852/3191 [00:00<00:00, 8473.72 examples/s]
Map:  58%|#####8    | 1866/3191 [00:00<00:00, 7300.78 examples/s]
Map:  91%|#########1| 2918/3191 [00:00<00:00, 7154.40 examples/s]
Map: 100%|##########| 3191/3191 [00:00<00:00, 6768.70 examples/s]

Map:   0%|          | 0/3191 [00:00<?, ? examples/s]
Map:  25%|##4       | 792/3191 [00:00<00:00, 7858.81 examples/s]
Map:  59%|#####8    | 1879/3191 [00:00<00:00, 7431.14 examples/s]
Map:  93%|#########2| 2964/3191 [00:00<00:00, 7329.43 examples/s]
Map: 100%|##########| 3191/3191 [00:00<00:00, 7134.01 examples/s]

Map:   0%|          | 0/3191 [00:00<?, ? examples/s]
Map:  94%|#########4| 3000/3191 [00:00<00:00, 29584.02 examples/s]
Map: 100%|##########| 3191/3191 [00:00<00:00, 29270.05 examples/s]

Filter:   0%|          | 0/3191 [00:00<?, ? examples/s]
Filter: 100%|##########| 3191/3191 [00:00<00:00, 48878.56 examples/s]

Map:   0%|          | 0/192 [00:00<?, ? examples/s]
Map: 100%|##########| 192/192 [00:00<00:00, 10898.72 examples/s]INFO 2025-05-20 11:02:49,506 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.67%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/7 [00:00<?, ?ba/s]
Creating json from Arrow format:  29%|##8       | 2/7 [00:00<00:00, 14.39ba/s]
Creating json from Arrow format:  57%|#####7    | 4/7 [00:00<00:00, 15.78ba/s]
Creating json from Arrow format: 100%|##########| 7/7 [00:00<00:00, 20.58ba/s]

Map:   0%|          | 0/427 [00:00<?, ? examples/s]
Map: 100%|##########| 427/427 [00:00<00:00, 7238.57 examples/s]

Map:   0%|          | 0/427 [00:00<?, ? examples/s]
Map: 100%|##########| 427/427 [00:00<00:00, 27183.24 examples/s]

Filter:   0%|          | 0/427 [00:00<?, ? examples/s]
Filter: 100%|##########| 427/427 [00:00<00:00, 41597.21 examples/s]

Map:   0%|          | 0/102 [00:00<?, ? examples/s]
Map: 100%|##########| 102/102 [00:00<00:00, 10033.51 examples/s]

Map:   0%|          | 0/102 [00:00<?, ? examples/s]
Map: 100%|##########| 102/102 [00:00<00:00, 19232.14 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 70.70ba/s]

Map:   0%|          | 0/427 [00:00<?, ? examples/s]
Map: 100%|##########| 427/427 [00:00<00:00, 7288.74 examples/s]

Map:   0%|          | 0/427 [00:00<?, ? examples/s]
Map: 100%|##########| 427/427 [00:00<00:00, 7392.63 examples/s]

Map:   0%|          | 0/427 [00:00<?, ? examples/s]
Map: 100%|##########| 427/427 [00:00<00:00, 7180.85 examples/s]

Map:   0%|          | 0/427 [00:00<?, ? examples/s]
Map: 100%|##########| 427/427 [00:00<00:00, 27017.16 examples/s]

Filter:   0%|          | 0/427 [00:00<?, ? examples/s]
Filter: 100%|##########| 427/427 [00:00<00:00, 41611.71 examples/s]

Map:   0%|          | 0/102 [00:00<?, ? examples/s]
Map: 100%|##########| 102/102 [00:00<00:00, 10069.88 examples/s]INFO 2025-05-20 11:02:50,268 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.24%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 23.81ba/s]

Map:   0%|          | 0/1216 [00:00<?, ? examples/s]
Map:  66%|######5   | 797/1216 [00:00<00:00, 7903.70 examples/s]
Map: 100%|##########| 1216/1216 [00:00<00:00, 7272.01 examples/s]

Map:   0%|          | 0/1216 [00:00<?, ? examples/s]
Map: 100%|##########| 1216/1216 [00:00<00:00, 28493.15 examples/s]

Filter:   0%|          | 0/1216 [00:00<?, ? examples/s]
Filter: 100%|##########| 1216/1216 [00:00<00:00, 47046.59 examples/s]

Map:   0%|          | 0/70 [00:00<?, ? examples/s]
Map: 100%|##########| 70/70 [00:00<00:00, 9014.19 examples/s]

Map:   0%|          | 0/70 [00:00<?, ? examples/s]
Map: 100%|##########| 70/70 [00:00<00:00, 15719.10 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 52.71ba/s]

Map:   0%|          | 0/1216 [00:00<?, ? examples/s]
Map:  65%|######5   | 793/1216 [00:00<00:00, 7872.48 examples/s]
Map: 100%|##########| 1216/1216 [00:00<00:00, 7203.42 examples/s]

Map:   0%|          | 0/1216 [00:00<?, ? examples/s]
Map:  74%|#######4  | 904/1216 [00:00<00:00, 8987.15 examples/s]
Map: 100%|##########| 1216/1216 [00:00<00:00, 7313.18 examples/s]

Map:   0%|          | 0/1216 [00:00<?, ? examples/s]
Map:  65%|######5   | 792/1216 [00:00<00:00, 7868.85 examples/s]
Map: 100%|##########| 1216/1216 [00:00<00:00, 7211.00 examples/s]

Map:   0%|          | 0/1216 [00:00<?, ? examples/s]
Map: 100%|##########| 1216/1216 [00:00<00:00, 27583.36 examples/s]

Filter:   0%|          | 0/1216 [00:00<?, ? examples/s]
Filter: 100%|##########| 1216/1216 [00:00<00:00, 46966.44 examples/s]

Map:   0%|          | 0/70 [00:00<?, ? examples/s]
Map: 100%|##########| 70/70 [00:00<00:00, 8970.95 examples/s]INFO 2025-05-20 11:02:51,315 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.63%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format:  67%|######6   | 2/3 [00:00<00:00, 19.63ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 26.07ba/s]

Map:   0%|          | 0/411 [00:00<?, ? examples/s]
Map: 100%|##########| 411/411 [00:00<00:00, 7111.92 examples/s]

Map:   0%|          | 0/411 [00:00<?, ? examples/s]
Map: 100%|##########| 411/411 [00:00<00:00, 27078.73 examples/s]

Filter:   0%|          | 0/411 [00:00<?, ? examples/s]
Filter: 100%|##########| 411/411 [00:00<00:00, 41383.21 examples/s]

Map:   0%|          | 0/103 [00:00<?, ? examples/s]
Map: 100%|##########| 103/103 [00:00<00:00, 9938.42 examples/s]

Map:   0%|          | 0/103 [00:00<?, ? examples/s]
Map: 100%|##########| 103/103 [00:00<00:00, 19092.82 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 79.84ba/s]

Map:   0%|          | 0/411 [00:00<?, ? examples/s]
Map: 100%|##########| 411/411 [00:00<00:00, 7245.39 examples/s]

Map:   0%|          | 0/411 [00:00<?, ? examples/s]
Map: 100%|##########| 411/411 [00:00<00:00, 7660.51 examples/s]

Map:   0%|          | 0/411 [00:00<?, ? examples/s]
Map: 100%|##########| 411/411 [00:00<00:00, 7199.57 examples/s]

Map:   0%|          | 0/411 [00:00<?, ? examples/s]
Map: 100%|##########| 411/411 [00:00<00:00, 27501.66 examples/s]

Filter:   0%|          | 0/411 [00:00<?, ? examples/s]
Filter: 100%|##########| 411/411 [00:00<00:00, 41418.01 examples/s]

Map:   0%|          | 0/103 [00:00<?, ? examples/s]
Map: 100%|##########| 103/103 [00:00<00:00, 10078.93 examples/s]INFO 2025-05-20 11:02:51,831 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.23%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 25.66ba/s]

Map:   0%|          | 0/68 [00:00<?, ? examples/s]
Map: 100%|##########| 68/68 [00:00<00:00, 5799.83 examples/s]

Map:   0%|          | 0/68 [00:00<?, ? examples/s]
Map: 100%|##########| 68/68 [00:00<00:00, 15983.67 examples/s]

Filter:   0%|          | 0/68 [00:00<?, ? examples/s]
Filter: 100%|##########| 68/68 [00:00<00:00, 21880.53 examples/s]

Map:   0%|          | 0/16 [00:00<?, ? examples/s]
Map: 100%|##########| 16/16 [00:00<00:00, 4701.81 examples/s]

Map:   0%|          | 0/16 [00:00<?, ? examples/s]
Map: 100%|##########| 16/16 [00:00<00:00, 6197.72 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 553.78ba/s]

Map:   0%|          | 0/68 [00:00<?, ? examples/s]
Map: 100%|##########| 68/68 [00:00<00:00, 5756.29 examples/s]

Map:   0%|          | 0/68 [00:00<?, ? examples/s]
Map: 100%|##########| 68/68 [00:00<00:00, 6634.24 examples/s]

Map:   0%|          | 0/68 [00:00<?, ? examples/s]
Map: 100%|##########| 68/68 [00:00<00:00, 6043.67 examples/s]

Map:   0%|          | 0/68 [00:00<?, ? examples/s]
Map: 100%|##########| 68/68 [00:00<00:00, 16050.23 examples/s]

Filter:   0%|          | 0/68 [00:00<?, ? examples/s]
Filter: 100%|##########| 68/68 [00:00<00:00, 21883.88 examples/s]

Map:   0%|          | 0/16 [00:00<?, ? examples/s]
Map: 100%|##########| 16/16 [00:00<00:00, 4650.97 examples/s]INFO 2025-05-20 11:02:52,006 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.04%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 250.36ba/s]

Map:   0%|          | 0/1415 [00:00<?, ? examples/s]
Map:  56%|#####5    | 789/1415 [00:00<00:00, 7833.86 examples/s]
Map: 100%|##########| 1415/1415 [00:00<00:00, 7257.19 examples/s]

Map:   0%|          | 0/1415 [00:00<?, ? examples/s]
Map: 100%|##########| 1415/1415 [00:00<00:00, 27963.61 examples/s]

Filter:   0%|          | 0/1415 [00:00<?, ? examples/s]
Filter: 100%|##########| 1415/1415 [00:00<00:00, 46029.05 examples/s]

Map:   0%|          | 0/58 [00:00<?, ? examples/s]
Map: 100%|##########| 58/58 [00:00<00:00, 8385.72 examples/s]

Map:   0%|          | 0/58 [00:00<?, ? examples/s]
Map: 100%|##########| 58/58 [00:00<00:00, 14602.02 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 44.17ba/s]

Map:   0%|          | 0/1415 [00:00<?, ? examples/s]
Map:  55%|#####5    | 783/1415 [00:00<00:00, 7771.46 examples/s]
Map: 100%|##########| 1415/1415 [00:00<00:00, 7291.66 examples/s]

Map:   0%|          | 0/1415 [00:00<?, ? examples/s]
Map:  63%|######3   | 895/1415 [00:00<00:00, 8909.61 examples/s]
Map: 100%|##########| 1415/1415 [00:00<00:00, 7253.86 examples/s]

Map:   0%|          | 0/1415 [00:00<?, ? examples/s]
Map:  56%|#####6    | 793/1415 [00:00<00:00, 7866.17 examples/s]
Map: 100%|##########| 1415/1415 [00:00<00:00, 7312.27 examples/s]

Map:   0%|          | 0/1415 [00:00<?, ? examples/s]
Map: 100%|##########| 1415/1415 [00:00<00:00, 28802.56 examples/s]

Filter:   0%|          | 0/1415 [00:00<?, ? examples/s]
Filter: 100%|##########| 1415/1415 [00:00<00:00, 46373.24 examples/s]

Map:   0%|          | 0/58 [00:00<?, ? examples/s]
Map: 100%|##########| 58/58 [00:00<00:00, 8411.23 examples/s]INFO 2025-05-20 11:02:53,168 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.73%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format:  67%|######6   | 2/3 [00:00<00:00, 18.05ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 22.15ba/s]

Map:   0%|          | 0/220 [00:00<?, ? examples/s]
Map: 100%|##########| 220/220 [00:00<00:00, 7075.25 examples/s]

Map:   0%|          | 0/220 [00:00<?, ? examples/s]
Map: 100%|##########| 220/220 [00:00<00:00, 24113.38 examples/s]

Filter:   0%|          | 0/220 [00:00<?, ? examples/s]
Filter: 100%|##########| 220/220 [00:00<00:00, 37207.54 examples/s]

Map:   0%|          | 0/4 [00:00<?, ? examples/s]
Map: 100%|##########| 4/4 [00:00<00:00, 1666.72 examples/s]

Map:   0%|          | 0/4 [00:00<?, ? examples/s]
Map: 100%|##########| 4/4 [00:00<00:00, 1869.33 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 214.71ba/s]

Map:   0%|          | 0/220 [00:00<?, ? examples/s]
Map: 100%|##########| 220/220 [00:00<00:00, 7117.21 examples/s]

Map:   0%|          | 0/220 [00:00<?, ? examples/s]
Map: 100%|##########| 220/220 [00:00<00:00, 7976.44 examples/s]

Map:   0%|          | 0/220 [00:00<?, ? examples/s]
Map: 100%|##########| 220/220 [00:00<00:00, 7074.81 examples/s]

Map:   0%|          | 0/220 [00:00<?, ? examples/s]
Map: 100%|##########| 220/220 [00:00<00:00, 24772.39 examples/s]

Filter:   0%|          | 0/220 [00:00<?, ? examples/s]
Filter: 100%|##########| 220/220 [00:00<00:00, 38215.31 examples/s]

Map:   0%|          | 0/4 [00:00<?, ? examples/s]
Map: 100%|##########| 4/4 [00:00<00:00, 1667.72 examples/s]INFO 2025-05-20 11:02:53,539 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.11%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 65.09ba/s]

Map:   0%|          | 0/1445 [00:00<?, ? examples/s]
Map:  54%|#####3    | 776/1445 [00:00<00:00, 7695.12 examples/s]
Map: 100%|##########| 1445/1445 [00:00<00:00, 7220.01 examples/s]

Map:   0%|          | 0/1445 [00:00<?, ? examples/s]
Map: 100%|##########| 1445/1445 [00:00<00:00, 28952.21 examples/s]

Filter:   0%|          | 0/1445 [00:00<?, ? examples/s]
Filter: 100%|##########| 1445/1445 [00:00<00:00, 47278.10 examples/s]

Map:   0%|          | 0/104 [00:00<?, ? examples/s]
Map: 100%|##########| 104/104 [00:00<00:00, 9626.97 examples/s]

Map:   0%|          | 0/104 [00:00<?, ? examples/s]
Map: 100%|##########| 104/104 [00:00<00:00, 18553.34 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 45.70ba/s]

Map:   0%|          | 0/1445 [00:00<?, ? examples/s]
Map:  55%|#####4    | 788/1445 [00:00<00:00, 7822.77 examples/s]
Map: 100%|##########| 1445/1445 [00:00<00:00, 7312.90 examples/s]

Map:   0%|          | 0/1445 [00:00<?, ? examples/s]
Map:  60%|######    | 874/1445 [00:00<00:00, 8691.39 examples/s]
Map: 100%|##########| 1445/1445 [00:00<00:00, 7294.89 examples/s]

Map:   0%|          | 0/1445 [00:00<?, ? examples/s]
Map:  55%|#####4    | 791/1445 [00:00<00:00, 7845.47 examples/s]
Map: 100%|##########| 1445/1445 [00:00<00:00, 7314.74 examples/s]

Map:   0%|          | 0/1445 [00:00<?, ? examples/s]
Map: 100%|##########| 1445/1445 [00:00<00:00, 29188.83 examples/s]

Filter:   0%|          | 0/1445 [00:00<?, ? examples/s]
Filter: 100%|##########| 1445/1445 [00:00<00:00, 46645.34 examples/s]

Map:   0%|          | 0/104 [00:00<?, ? examples/s]
Map: 100%|##########| 104/104 [00:00<00:00, 9763.80 examples/s]INFO 2025-05-20 11:02:54,749 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.76%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 24.67ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 24.63ba/s]

Map:   0%|          | 0/2147 [00:00<?, ? examples/s]
Map:  37%|###7      | 799/2147 [00:00<00:00, 7929.87 examples/s]
Map:  89%|########9 | 1914/2147 [00:00<00:00, 7584.34 examples/s]
Map: 100%|##########| 2147/2147 [00:00<00:00, 7375.27 examples/s]

Map:   0%|          | 0/2147 [00:00<?, ? examples/s]
Map: 100%|##########| 2147/2147 [00:00<00:00, 29076.98 examples/s]

Filter:   0%|          | 0/2147 [00:00<?, ? examples/s]
Filter: 100%|##########| 2147/2147 [00:00<00:00, 49053.38 examples/s]

Map:   0%|          | 0/116 [00:00<?, ? examples/s]
Map: 100%|##########| 116/116 [00:00<00:00, 10097.74 examples/s]

Map:   0%|          | 0/116 [00:00<?, ? examples/s]
Map: 100%|##########| 116/116 [00:00<00:00, 20190.03 examples/s]

Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 50.42ba/s]

Map:   0%|          | 0/2147 [00:00<?, ? examples/s]
Map:  37%|###6      | 791/2147 [00:00<00:00, 7857.01 examples/s]
Map:  89%|########9 | 1916/2147 [00:00<00:00, 7607.45 examples/s]
Map: 100%|##########| 2147/2147 [00:00<00:00, 7375.86 examples/s]

Map:   0%|          | 0/2147 [00:00<?, ? examples/s]
Map:  42%|####1     | 895/2147 [00:00<00:00, 8900.40 examples/s]
Map:  93%|#########3| 2000/2147 [00:00<00:00, 6992.27 examples/s]
Map: 100%|##########| 2147/2147 [00:00<00:00, 7206.98 examples/s]

Map:   0%|          | 0/2147 [00:00<?, ? examples/s]
Map:  37%|###7      | 804/2147 [00:00<00:00, 7974.49 examples/s]
Map:  90%|######### | 1943/2147 [00:00<00:00, 7706.62 examples/s]
Map: 100%|##########| 2147/2147 [00:00<00:00, 7427.66 examples/s]

Map:   0%|          | 0/2147 [00:00<?, ? examples/s]
Map: 100%|##########| 2147/2147 [00:00<00:00, 29547.82 examples/s]

Filter:   0%|          | 0/2147 [00:00<?, ? examples/s]
Filter: 100%|##########| 2147/2147 [00:00<00:00, 48202.13 examples/s]

Map:   0%|          | 0/116 [00:00<?, ? examples/s]
Map: 100%|##########| 116/116 [00:00<00:00, 9077.56 examples/s]INFO 2025-05-20 11:02:56,583 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.12%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/5 [00:00<?, ?ba/s]
Creating json from Arrow format:  40%|####      | 2/5 [00:00<00:00, 17.47ba/s]
Creating json from Arrow format: 100%|##########| 5/5 [00:00<00:00, 27.02ba/s]

Map:   0%|          | 0/1585 [00:00<?, ? examples/s]
Map:  51%|#####     | 802/1585 [00:00<00:00, 7962.92 examples/s]
Map: 100%|##########| 1585/1585 [00:00<00:00, 7239.39 examples/s]

Map:   0%|          | 0/1585 [00:00<?, ? examples/s]
Map: 100%|##########| 1585/1585 [00:00<00:00, 29057.60 examples/s]

Filter:   0%|          | 0/1585 [00:00<?, ? examples/s]
Filter: 100%|##########| 1585/1585 [00:00<00:00, 47140.38 examples/s]

Map:   0%|          | 0/71 [00:00<?, ? examples/s]
Map: 100%|##########| 71/71 [00:00<00:00, 9074.15 examples/s]

Map:   0%|          | 0/71 [00:00<?, ? examples/s]
Map: 100%|##########| 71/71 [00:00<00:00, 16444.62 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 47.13ba/s]

Map:   0%|          | 0/1585 [00:00<?, ? examples/s]
Map:  51%|#####1    | 810/1585 [00:00<00:00, 8038.54 examples/s]
Map: 100%|##########| 1585/1585 [00:00<00:00, 7343.80 examples/s]

Map:   0%|          | 0/1585 [00:00<?, ? examples/s]
Map:  59%|#####8    | 930/1585 [00:00<00:00, 9250.69 examples/s]
Map: 100%|##########| 1585/1585 [00:00<00:00, 7371.12 examples/s]

Map:   0%|          | 0/1585 [00:00<?, ? examples/s]
Map:  51%|#####     | 802/1585 [00:00<00:00, 7961.37 examples/s]
Map: 100%|##########| 1585/1585 [00:00<00:00, 7220.17 examples/s]

Map:   0%|          | 0/1585 [00:00<?, ? examples/s]
Map:  52%|#####2    | 830/1585 [00:00<00:00, 3248.50 examples/s]
Map: 100%|##########| 1585/1585 [00:00<00:00, 5422.65 examples/s]

Filter:   0%|          | 0/1585 [00:00<?, ? examples/s]
Filter: 100%|##########| 1585/1585 [00:00<00:00, 47395.46 examples/s]

Map:   0%|          | 0/71 [00:00<?, ? examples/s]
Map: 100%|##########| 71/71 [00:00<00:00, 8998.20 examples/s]INFO 2025-05-20 11:02:58,288 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.82%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format:  75%|#######5  | 3/4 [00:00<00:00, 25.61ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 32.52ba/s]

Map:   0%|          | 0/287 [00:00<?, ? examples/s]
Map: 100%|##########| 287/287 [00:00<00:00, 7098.80 examples/s]

Map:   0%|          | 0/287 [00:00<?, ? examples/s]
Map: 100%|##########| 287/287 [00:00<00:00, 25633.30 examples/s]

Filter:   0%|          | 0/287 [00:00<?, ? examples/s]
Filter: 100%|##########| 287/287 [00:00<00:00, 37868.54 examples/s]

Map:   0%|          | 0/11 [00:00<?, ? examples/s]
Map: 100%|##########| 11/11 [00:00<00:00, 3635.44 examples/s]

Map:   0%|          | 0/11 [00:00<?, ? examples/s]
Map: 100%|##########| 11/11 [00:00<00:00, 4524.16 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 155.96ba/s]

Map:   0%|          | 0/287 [00:00<?, ? examples/s]
Map: 100%|##########| 287/287 [00:00<00:00, 7143.07 examples/s]

Map:   0%|          | 0/287 [00:00<?, ? examples/s]
Map: 100%|##########| 287/287 [00:00<00:00, 7806.11 examples/s]

Map:   0%|          | 0/287 [00:00<?, ? examples/s]
Map: 100%|##########| 287/287 [00:00<00:00, 7251.51 examples/s]

Map:   0%|          | 0/287 [00:00<?, ? examples/s]
Map: 100%|##########| 287/287 [00:00<00:00, 26363.67 examples/s]

Filter:   0%|          | 0/287 [00:00<?, ? examples/s]
Filter: 100%|##########| 287/287 [00:00<00:00, 39635.35 examples/s]

Map:   0%|          | 0/11 [00:00<?, ? examples/s]
Map: 100%|##########| 11/11 [00:00<00:00, 3723.46 examples/s]INFO 2025-05-20 11:02:58,698 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.15%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 48.67ba/s]

Map:   0%|          | 0/407 [00:00<?, ? examples/s]
Map: 100%|##########| 407/407 [00:00<00:00, 6557.78 examples/s]

Map:   0%|          | 0/407 [00:00<?, ? examples/s]
Map: 100%|##########| 407/407 [00:00<00:00, 25473.51 examples/s]

Filter:   0%|          | 0/407 [00:00<?, ? examples/s]
Filter: 100%|##########| 407/407 [00:00<00:00, 39056.51 examples/s]

Map:   0%|          | 0/64 [00:00<?, ? examples/s]
Map: 100%|##########| 64/64 [00:00<00:00, 8607.01 examples/s]

Map:   0%|          | 0/64 [00:00<?, ? examples/s]
Map: 100%|##########| 64/64 [00:00<00:00, 15900.69 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 53.97ba/s]

Map:   0%|          | 0/407 [00:00<?, ? examples/s]
Map: 100%|##########| 407/407 [00:00<00:00, 6785.12 examples/s]

Map:   0%|          | 0/407 [00:00<?, ? examples/s]
Map: 100%|##########| 407/407 [00:00<00:00, 6943.39 examples/s]

Map:   0%|          | 0/407 [00:00<?, ? examples/s]
Map: 100%|##########| 407/407 [00:00<00:00, 6854.81 examples/s]

Map:   0%|          | 0/407 [00:00<?, ? examples/s]
Map: 100%|##########| 407/407 [00:00<00:00, 26696.51 examples/s]

Filter:   0%|          | 0/407 [00:00<?, ? examples/s]
Filter: 100%|##########| 407/407 [00:00<00:00, 39819.03 examples/s]

Map:   0%|          | 0/64 [00:00<?, ? examples/s]
Map: 100%|##########| 64/64 [00:00<00:00, 8399.11 examples/s]INFO 2025-05-20 11:02:59,146 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.22%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 18.01ba/s]

Map:   0%|          | 0/283 [00:00<?, ? examples/s]
Map: 100%|##########| 283/283 [00:00<00:00, 7186.29 examples/s]

Map:   0%|          | 0/283 [00:00<?, ? examples/s]
Map: 100%|##########| 283/283 [00:00<00:00, 25576.68 examples/s]

Filter:   0%|          | 0/283 [00:00<?, ? examples/s]
Filter: 100%|##########| 283/283 [00:00<00:00, 39526.74 examples/s]

Map:   0%|          | 0/4 [00:00<?, ? examples/s]
Map: 100%|##########| 4/4 [00:00<00:00, 1630.44 examples/s]

Map:   0%|          | 0/4 [00:00<?, ? examples/s]
Map: 100%|##########| 4/4 [00:00<00:00, 1818.67 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 190.02ba/s]

Map:   0%|          | 0/283 [00:00<?, ? examples/s]
Map: 100%|##########| 283/283 [00:00<00:00, 7333.06 examples/s]

Map:   0%|          | 0/283 [00:00<?, ? examples/s]
Map: 100%|##########| 283/283 [00:00<00:00, 8287.40 examples/s]

Map:   0%|          | 0/283 [00:00<?, ? examples/s]
Map: 100%|##########| 283/283 [00:00<00:00, 7337.69 examples/s]

Map:   0%|          | 0/283 [00:00<?, ? examples/s]
Map: 100%|##########| 283/283 [00:00<00:00, 26362.87 examples/s]

Filter:   0%|          | 0/283 [00:00<?, ? examples/s]
Filter: 100%|##########| 283/283 [00:00<00:00, 40038.72 examples/s]

Map:   0%|          | 0/4 [00:00<?, ? examples/s]
Map: 100%|##########| 4/4 [00:00<00:00, 1687.34 examples/s]INFO 2025-05-20 11:02:59,475 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.14%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 67.40ba/s]

Map:   0%|          | 0/2488 [00:00<?, ? examples/s]
Map:  32%|###1      | 790/2488 [00:00<00:00, 7837.35 examples/s]
Map:  76%|#######6  | 1895/2488 [00:00<00:00, 7511.24 examples/s]
Map: 100%|##########| 2488/2488 [00:00<00:00, 7175.91 examples/s]

Map:   0%|          | 0/2488 [00:00<?, ? examples/s]
Map: 100%|##########| 2488/2488 [00:00<00:00, 28004.20 examples/s]

Filter:   0%|          | 0/2488 [00:00<?, ? examples/s]
Filter: 100%|##########| 2488/2488 [00:00<00:00, 48203.49 examples/s]

Map:   0%|          | 0/167 [00:00<?, ? examples/s]
Map: 100%|##########| 167/167 [00:00<00:00, 10555.92 examples/s]

Map:   0%|          | 0/167 [00:00<?, ? examples/s]
Map: 100%|##########| 167/167 [00:00<00:00, 21829.67 examples/s]

Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 36.95ba/s]

Map:   0%|          | 0/2488 [00:00<?, ? examples/s]
Map:  32%|###1      | 792/2488 [00:00<00:00, 7856.68 examples/s]
Map:  77%|#######6  | 1912/2488 [00:00<00:00, 7583.90 examples/s]
Map: 100%|##########| 2488/2488 [00:00<00:00, 7238.38 examples/s]

Map:   0%|          | 0/2488 [00:00<?, ? examples/s]
Map:  34%|###4      | 853/2488 [00:00<00:00, 8472.82 examples/s]
Map:  79%|#######8  | 1955/2488 [00:00<00:00, 7699.98 examples/s]
Map: 100%|##########| 2488/2488 [00:00<00:00, 6968.90 examples/s]

Map:   0%|          | 0/2488 [00:00<?, ? examples/s]
Map:  31%|###1      | 781/2488 [00:00<00:00, 7755.35 examples/s]
Map:  76%|#######5  | 1890/2488 [00:00<00:00, 7501.56 examples/s]
Map: 100%|##########| 2488/2488 [00:00<00:00, 7175.48 examples/s]

Map:   0%|          | 0/2488 [00:00<?, ? examples/s]
Map: 100%|##########| 2488/2488 [00:00<00:00, 28952.08 examples/s]

Filter:   0%|          | 0/2488 [00:00<?, ? examples/s]
Filter: 100%|##########| 2488/2488 [00:00<00:00, 49139.12 examples/s]

Map:   0%|          | 0/167 [00:00<?, ? examples/s]
Map: 100%|##########| 167/167 [00:00<00:00, 10735.67 examples/s]INFO 2025-05-20 11:03:01,536 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.30%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/6 [00:00<?, ?ba/s]
Creating json from Arrow format:  33%|###3      | 2/6 [00:00<00:00, 16.23ba/s]
Creating json from Arrow format:  83%|########3 | 5/6 [00:00<00:00, 22.43ba/s]
Creating json from Arrow format: 100%|##########| 6/6 [00:00<00:00, 25.28ba/s]

Map:   0%|          | 0/3348 [00:00<?, ? examples/s]
Map:  24%|##3       | 791/3348 [00:00<00:00, 7848.20 examples/s]
Map:  56%|#####5    | 1870/3348 [00:00<00:00, 7395.24 examples/s]
Map:  89%|########8 | 2969/3348 [00:00<00:00, 7357.90 examples/s]
Map: 100%|##########| 3348/3348 [00:00<00:00, 7165.31 examples/s]

Map:   0%|          | 0/3348 [00:00<?, ? examples/s]
Map:  90%|########9 | 3000/3348 [00:00<00:00, 29190.29 examples/s]
Map: 100%|##########| 3348/3348 [00:00<00:00, 29075.43 examples/s]

Filter:   0%|          | 0/3348 [00:00<?, ? examples/s]
Filter: 100%|##########| 3348/3348 [00:00<00:00, 49572.95 examples/s]

Map:   0%|          | 0/180 [00:00<?, ? examples/s]
Map: 100%|##########| 180/180 [00:00<00:00, 10566.18 examples/s]

Map:   0%|          | 0/180 [00:00<?, ? examples/s]
Map: 100%|##########| 180/180 [00:00<00:00, 22892.59 examples/s]

Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 36.88ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 36.82ba/s]

Map:   0%|          | 0/3348 [00:00<?, ? examples/s]
Map:  23%|##3       | 786/3348 [00:00<00:00, 7797.97 examples/s]
Map:  57%|#####6    | 1900/3348 [00:00<00:00, 7543.82 examples/s]
Map:  90%|########9 | 3000/3348 [00:00<00:00, 7199.74 examples/s]
Map: 100%|##########| 3348/3348 [00:00<00:00, 7262.92 examples/s]

Map:   0%|          | 0/3348 [00:00<?, ? examples/s]
Map:  26%|##5       | 863/3348 [00:00<00:00, 8578.85 examples/s]
Map:  58%|#####7    | 1936/3348 [00:00<00:00, 7602.58 examples/s]
Map:  90%|########9 | 2997/3348 [00:00<00:00, 7335.36 examples/s]
Map: 100%|##########| 3348/3348 [00:00<00:00, 6892.92 examples/s]

Map:   0%|          | 0/3348 [00:00<?, ? examples/s]
Map:  24%|##3       | 794/3348 [00:00<00:00, 7877.48 examples/s]
Map:  57%|#####6    | 1894/3348 [00:00<00:00, 7500.77 examples/s]
Map:  89%|########9 | 2990/3348 [00:00<00:00, 7400.67 examples/s]
Map: 100%|##########| 3348/3348 [00:00<00:00, 7212.54 examples/s]

Map:   0%|          | 0/3348 [00:00<?, ? examples/s]
Map:  90%|########9 | 3000/3348 [00:00<00:00, 29190.22 examples/s]
Map: 100%|##########| 3348/3348 [00:00<00:00, 29072.12 examples/s]

Filter:   0%|          | 0/3348 [00:00<?, ? examples/s]
Filter: 100%|##########| 3348/3348 [00:00<00:00, 48448.24 examples/s]

Map:   0%|          | 0/180 [00:00<?, ? examples/s]
Map: 100%|##########| 180/180 [00:00<00:00, 10650.40 examples/s]INFO 2025-05-20 11:03:04,486 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (1.74%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/7 [00:00<?, ?ba/s]
Creating json from Arrow format:  29%|##8       | 2/7 [00:00<00:00, 15.93ba/s]
Creating json from Arrow format:  57%|#####7    | 4/7 [00:00<00:00, 17.88ba/s]
Creating json from Arrow format: 100%|##########| 7/7 [00:00<00:00, 22.38ba/s]

Map:   0%|          | 0/757 [00:00<?, ? examples/s]
Map: 100%|##########| 757/757 [00:00<00:00, 7558.64 examples/s]

Map:   0%|          | 0/757 [00:00<?, ? examples/s]
Map: 100%|##########| 757/757 [00:00<00:00, 29308.60 examples/s]

Filter:   0%|          | 0/757 [00:00<?, ? examples/s]
Filter: 100%|##########| 757/757 [00:00<00:00, 44538.26 examples/s]

Map:   0%|          | 0/30 [00:00<?, ? examples/s]
Map: 100%|##########| 30/30 [00:00<00:00, 6758.47 examples/s]

Map:   0%|          | 0/30 [00:00<?, ? examples/s]
Map: 100%|##########| 30/30 [00:00<00:00, 9833.47 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 68.43ba/s]

Map:   0%|          | 0/757 [00:00<?, ? examples/s]
Map: 100%|##########| 757/757 [00:00<00:00, 7529.87 examples/s]

Map:   0%|          | 0/757 [00:00<?, ? examples/s]
Map: 100%|##########| 757/757 [00:00<00:00, 8126.58 examples/s]

Map:   0%|          | 0/757 [00:00<?, ? examples/s]
Map: 100%|##########| 757/757 [00:00<00:00, 7691.27 examples/s]

Map:   0%|          | 0/757 [00:00<?, ? examples/s]
Map: 100%|##########| 757/757 [00:00<00:00, 29971.76 examples/s]

Filter:   0%|          | 0/757 [00:00<?, ? examples/s]
Filter: 100%|##########| 757/757 [00:00<00:00, 45048.21 examples/s]

Map:   0%|          | 0/30 [00:00<?, ? examples/s]
Map: 100%|##########| 30/30 [00:00<00:00, 6688.77 examples/s]INFO 2025-05-20 11:03:05,409 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.39%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 43.47ba/s]

Map:   0%|          | 0/220 [00:00<?, ? examples/s]
Map: 100%|##########| 220/220 [00:00<00:00, 6770.76 examples/s]

Map:   0%|          | 0/220 [00:00<?, ? examples/s]
Map: 100%|##########| 220/220 [00:00<00:00, 23640.78 examples/s]

Filter:   0%|          | 0/220 [00:00<?, ? examples/s]
Filter: 100%|##########| 220/220 [00:00<00:00, 36004.01 examples/s]

Map:   0%|          | 0/38 [00:00<?, ? examples/s]
Map: 100%|##########| 38/38 [00:00<00:00, 7339.45 examples/s]

Map:   0%|          | 0/38 [00:00<?, ? examples/s]
Map: 100%|##########| 38/38 [00:00<00:00, 11586.48 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 126.25ba/s]

Map:   0%|          | 0/220 [00:00<?, ? examples/s]
Map: 100%|##########| 220/220 [00:00<00:00, 6759.21 examples/s]

Map:   0%|          | 0/220 [00:00<?, ? examples/s]
Map: 100%|##########| 220/220 [00:00<00:00, 7233.37 examples/s]

Map:   0%|          | 0/220 [00:00<?, ? examples/s]
Map: 100%|##########| 220/220 [00:00<00:00, 6888.02 examples/s]

Map:   0%|          | 0/220 [00:00<?, ? examples/s]
Map: 100%|##########| 220/220 [00:00<00:00, 24574.47 examples/s]

Filter:   0%|          | 0/220 [00:00<?, ? examples/s]
Filter: 100%|##########| 220/220 [00:00<00:00, 35555.91 examples/s]

Map:   0%|          | 0/38 [00:00<?, ? examples/s]
Map: 100%|##########| 38/38 [00:00<00:00, 7269.49 examples/s]INFO 2025-05-20 11:03:05,710 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.12%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 46.01ba/s]

Map:   0%|          | 0/475 [00:00<?, ? examples/s]
Map: 100%|##########| 475/475 [00:00<00:00, 7151.32 examples/s]

Map:   0%|          | 0/475 [00:00<?, ? examples/s]
Map: 100%|##########| 475/475 [00:00<00:00, 27208.59 examples/s]

Filter:   0%|          | 0/475 [00:00<?, ? examples/s]
Filter: 100%|##########| 475/475 [00:00<00:00, 42062.59 examples/s]

Map:   0%|          | 0/93 [00:00<?, ? examples/s]
Map: 100%|##########| 93/93 [00:00<00:00, 9895.74 examples/s]

Map:   0%|          | 0/93 [00:00<?, ? examples/s]
Map: 100%|##########| 93/93 [00:00<00:00, 18516.58 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 63.98ba/s]

Map:   0%|          | 0/475 [00:00<?, ? examples/s]
Map: 100%|##########| 475/475 [00:00<00:00, 7134.80 examples/s]

Map:   0%|          | 0/475 [00:00<?, ? examples/s]
Map: 100%|##########| 475/475 [00:00<00:00, 7278.53 examples/s]

Map:   0%|          | 0/475 [00:00<?, ? examples/s]
Map: 100%|##########| 475/475 [00:00<00:00, 7306.56 examples/s]

Map:   0%|          | 0/475 [00:00<?, ? examples/s]
Map: 100%|##########| 475/475 [00:00<00:00, 27774.52 examples/s]

Filter:   0%|          | 0/475 [00:00<?, ? examples/s]
Filter: 100%|##########| 475/475 [00:00<00:00, 41485.39 examples/s]

Map:   0%|          | 0/93 [00:00<?, ? examples/s]
Map: 100%|##########| 93/93 [00:00<00:00, 9782.33 examples/s]INFO 2025-05-20 11:03:06,180 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.26%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 45.66ba/s]

Map:   0%|          | 0/1886 [00:00<?, ? examples/s]
Map:  42%|####2     | 796/1886 [00:00<00:00, 7896.45 examples/s]
Map: 100%|##########| 1886/1886 [00:00<00:00, 7272.18 examples/s]
Map: 100%|##########| 1886/1886 [00:00<00:00, 7315.38 examples/s]

Map:   0%|          | 0/1886 [00:00<?, ? examples/s]
Map: 100%|##########| 1886/1886 [00:00<00:00, 28237.01 examples/s]

Filter:   0%|          | 0/1886 [00:00<?, ? examples/s]
Filter: 100%|##########| 1886/1886 [00:00<00:00, 48466.78 examples/s]

Map:   0%|          | 0/86 [00:00<?, ? examples/s]
Map: 100%|##########| 86/86 [00:00<00:00, 9471.93 examples/s]

Map:   0%|          | 0/86 [00:00<?, ? examples/s]
Map: 100%|##########| 86/86 [00:00<00:00, 17943.99 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 36.83ba/s]

Map:   0%|          | 0/1886 [00:00<?, ? examples/s]
Map:  42%|####1     | 792/1886 [00:00<00:00, 7850.84 examples/s]
Map: 100%|##########| 1886/1886 [00:00<00:00, 7377.97 examples/s]
Map: 100%|##########| 1886/1886 [00:00<00:00, 7402.62 examples/s]

Map:   0%|          | 0/1886 [00:00<?, ? examples/s]
Map:  48%|####8     | 908/1886 [00:00<00:00, 9028.95 examples/s]
Map: 100%|##########| 1886/1886 [00:00<00:00, 7144.19 examples/s]
Map: 100%|##########| 1886/1886 [00:00<00:00, 7335.07 examples/s]

Map:   0%|          | 0/1886 [00:00<?, ? examples/s]
Map:  42%|####2     | 800/1886 [00:00<00:00, 7934.35 examples/s]
Map: 100%|##########| 1886/1886 [00:00<00:00, 7303.16 examples/s]
Map: 100%|##########| 1886/1886 [00:00<00:00, 7349.51 examples/s]

Map:   0%|          | 0/1886 [00:00<?, ? examples/s]
Map: 100%|##########| 1886/1886 [00:00<00:00, 29074.33 examples/s]

Filter:   0%|          | 0/1886 [00:00<?, ? examples/s]
Filter: 100%|##########| 1886/1886 [00:00<00:00, 48427.62 examples/s]

Map:   0%|          | 0/86 [00:00<?, ? examples/s]
Map: 100%|##########| 86/86 [00:00<00:00, 9378.60 examples/s]INFO 2025-05-20 11:03:07,729 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.98%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/4 [00:00<?, ?ba/s]
Creating json from Arrow format:  50%|#####     | 2/4 [00:00<00:00, 19.00ba/s]
Creating json from Arrow format: 100%|##########| 4/4 [00:00<00:00, 25.53ba/s]

Map:   0%|          | 0/1064 [00:00<?, ? examples/s]
Map:  75%|#######4  | 794/1064 [00:00<00:00, 7876.91 examples/s]
Map: 100%|##########| 1064/1064 [00:00<00:00, 7130.56 examples/s]

Map:   0%|          | 0/1064 [00:00<?, ? examples/s]
Map: 100%|##########| 1064/1064 [00:00<00:00, 28227.32 examples/s]

Filter:   0%|          | 0/1064 [00:00<?, ? examples/s]
Filter: 100%|##########| 1064/1064 [00:00<00:00, 45489.88 examples/s]

Map:   0%|          | 0/69 [00:00<?, ? examples/s]
Map: 100%|##########| 69/69 [00:00<00:00, 8931.21 examples/s]

Map:   0%|          | 0/69 [00:00<?, ? examples/s]
Map: 100%|##########| 69/69 [00:00<00:00, 16414.67 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 52.14ba/s]

Map:   0%|          | 0/1064 [00:00<?, ? examples/s]
Map:  74%|#######4  | 791/1064 [00:00<00:00, 7850.88 examples/s]
Map: 100%|##########| 1064/1064 [00:00<00:00, 7292.14 examples/s]

Map:   0%|          | 0/1064 [00:00<?, ? examples/s]
Map:  84%|########4 | 899/1064 [00:00<00:00, 8935.26 examples/s]
Map: 100%|##########| 1064/1064 [00:00<00:00, 7303.39 examples/s]

Map:   0%|          | 0/1064 [00:00<?, ? examples/s]
Map:  74%|#######4  | 788/1064 [00:00<00:00, 7817.07 examples/s]
Map: 100%|##########| 1064/1064 [00:00<00:00, 7157.42 examples/s]

Map:   0%|          | 0/1064 [00:00<?, ? examples/s]
Map: 100%|##########| 1064/1064 [00:00<00:00, 28175.99 examples/s]

Filter:   0%|          | 0/1064 [00:00<?, ? examples/s]
Filter: 100%|##########| 1064/1064 [00:00<00:00, 45920.99 examples/s]

Map:   0%|          | 0/69 [00:00<?, ? examples/s]
Map: 100%|##########| 69/69 [00:00<00:00, 9044.25 examples/s]INFO 2025-05-20 11:03:08,791 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.56%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 31.07ba/s]

Map:   0%|          | 0/1148 [00:00<?, ? examples/s]
Map:  69%|######9   | 795/1148 [00:00<00:00, 7893.32 examples/s]
Map: 100%|##########| 1148/1148 [00:00<00:00, 7349.56 examples/s]

Map:   0%|          | 0/1148 [00:00<?, ? examples/s]
Map: 100%|##########| 1148/1148 [00:00<00:00, 27940.61 examples/s]

Filter:   0%|          | 0/1148 [00:00<?, ? examples/s]
Filter: 100%|##########| 1148/1148 [00:00<00:00, 45080.62 examples/s]

Map:   0%|          | 0/52 [00:00<?, ? examples/s]
Map: 100%|##########| 52/52 [00:00<00:00, 8111.57 examples/s]

Map:   0%|          | 0/52 [00:00<?, ? examples/s]
Map: 100%|##########| 52/52 [00:00<00:00, 13748.35 examples/s]

Creating json from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 2/2 [00:00<00:00, 63.46ba/s]

Map:   0%|          | 0/1148 [00:00<?, ? examples/s]
Map:  68%|######8   | 781/1148 [00:00<00:00, 7750.78 examples/s]
Map: 100%|##########| 1148/1148 [00:00<00:00, 7240.23 examples/s]

Map:   0%|          | 0/1148 [00:00<?, ? examples/s]
Map:  80%|#######9  | 917/1148 [00:00<00:00, 9123.61 examples/s]
Map: 100%|##########| 1148/1148 [00:00<00:00, 7791.01 examples/s]

Map:   0%|          | 0/1148 [00:00<?, ? examples/s]
Map:  70%|######9   | 798/1148 [00:00<00:00, 7912.15 examples/s]
Map: 100%|##########| 1148/1148 [00:00<00:00, 7443.57 examples/s]

Map:   0%|          | 0/1148 [00:00<?, ? examples/s]
Map: 100%|##########| 1148/1148 [00:00<00:00, 29557.48 examples/s]

Filter:   0%|          | 0/1148 [00:00<?, ? examples/s]
Filter: 100%|##########| 1148/1148 [00:00<00:00, 46597.51 examples/s]

Map:   0%|          | 0/52 [00:00<?, ? examples/s]
Map: 100%|##########| 52/52 [00:00<00:00, 8133.35 examples/s]INFO 2025-05-20 11:03:09,822 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.60%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 3/3 [00:00<00:00, 33.69ba/s]

Map:   0%|          | 0/8322 [00:00<?, ? examples/s]
Map:   9%|9         | 773/8322 [00:00<00:00, 7664.37 examples/s]
Map:  23%|##2       | 1885/8322 [00:00<00:00, 7487.63 examples/s]
Map:  36%|###5      | 2994/8322 [00:00<00:00, 7439.19 examples/s]
Map:  48%|####8     | 4020/8322 [00:00<00:00, 7187.04 examples/s]
Map:  58%|#####7    | 4822/8322 [00:00<00:00, 7420.77 examples/s]
Map:  71%|#######   | 5897/8322 [00:00<00:00, 7322.05 examples/s]
Map:  80%|#######9  | 6638/8322 [00:00<00:00, 7342.80 examples/s]
Map:  89%|########8 | 7377/8322 [00:01<00:00, 7258.97 examples/s]
Map: 100%|##########| 8322/8322 [00:01<00:00, 7229.77 examples/s]
Map: 100%|##########| 8322/8322 [00:01<00:00, 7303.55 examples/s]

Map:   0%|          | 0/8322 [00:00<?, ? examples/s]
Map:  36%|###6      | 3000/8322 [00:00<00:00, 28927.70 examples/s]
Map:  72%|#######2  | 6000/8322 [00:00<00:00, 29129.19 examples/s]
Map: 100%|##########| 8322/8322 [00:00<00:00, 29545.73 examples/s]

Filter:   0%|          | 0/8322 [00:00<?, ? examples/s]
Filter:  72%|#######2  | 6000/8322 [00:00<00:00, 51211.88 examples/s]
Filter: 100%|##########| 8322/8322 [00:00<00:00, 51085.44 examples/s]

Map:   0%|          | 0/887 [00:00<?, ? examples/s]
Map: 100%|##########| 887/887 [00:00<00:00, 11831.92 examples/s]

Map:   0%|          | 0/887 [00:00<?, ? examples/s]
Map: 100%|##########| 887/887 [00:00<00:00, 28975.12 examples/s]

Creating json from Arrow format:   0%|          | 0/10 [00:00<?, ?ba/s]
Creating json from Arrow format:  40%|####      | 4/10 [00:00<00:00, 38.68ba/s]
Creating json from Arrow format:  80%|########  | 8/10 [00:00<00:00, 37.60ba/s]
Creating json from Arrow format: 100%|##########| 10/10 [00:00<00:00, 41.03ba/s]

Map:   0%|          | 0/8322 [00:00<?, ? examples/s]
Map:   9%|9         | 789/8322 [00:00<00:00, 7827.04 examples/s]
Map:  23%|##2       | 1900/8322 [00:00<00:00, 7535.75 examples/s]
Map:  36%|###6      | 3000/8322 [00:00<00:00, 7137.32 examples/s]
Map:  46%|####5     | 3799/8322 [00:00<00:00, 7397.36 examples/s]
Map:  59%|#####9    | 4933/8322 [00:00<00:00, 7457.42 examples/s]
Map:  72%|#######2  | 5993/8322 [00:00<00:00, 7310.51 examples/s]
Map:  85%|########4 | 7042/8322 [00:00<00:00, 7199.47 examples/s]
Map:  94%|#########4| 7824/8322 [00:01<00:00, 7349.16 examples/s]
Map: 100%|##########| 8322/8322 [00:01<00:00, 7287.26 examples/s]

Map:   0%|          | 0/8322 [00:00<?, ? examples/s]
Map:   8%|7         | 650/8322 [00:00<00:01, 6457.57 examples/s]
Map:  18%|#7        | 1496/8322 [00:00<00:01, 5891.68 examples/s]
Map:  28%|##7       | 2326/8322 [00:00<00:01, 3141.75 examples/s]
Map:  36%|###5      | 2991/8322 [00:00<00:01, 3859.72 examples/s]
Map:  46%|####5     | 3797/8322 [00:00<00:01, 4319.72 examples/s]
Map:  52%|#####2    | 4349/8322 [00:01<00:00, 4587.62 examples/s]
Map:  60%|######    | 5000/8322 [00:01<00:00, 4886.22 examples/s]
Map:  68%|######8   | 5668/8322 [00:01<00:00, 5335.22 examples/s]
Map:  77%|#######7  | 6422/8322 [00:01<00:00, 5223.49 examples/s]
Map:  84%|########4 | 7000/8322 [00:01<00:00, 5259.06 examples/s]
Map:  92%|#########1| 7621/8322 [00:01<00:00, 5503.82 examples/s]
Map: 100%|##########| 8322/8322 [00:01<00:00, 5389.44 examples/s]
Map: 100%|##########| 8322/8322 [00:01<00:00, 4864.69 examples/s]

Map:   0%|          | 0/8322 [00:00<?, ? examples/s]
Map:   9%|9         | 779/8322 [00:00<00:00, 7727.39 examples/s]
Map:  23%|##2       | 1900/8322 [00:00<00:00, 7549.91 examples/s]
Map:  36%|###6      | 3000/8322 [00:00<00:00, 7174.98 examples/s]
Map:  46%|####5     | 3789/8322 [00:00<00:00, 7394.22 examples/s]
Map:  59%|#####9    | 4918/8322 [00:00<00:00, 7446.39 examples/s]
Map:  72%|#######2  | 6000/8322 [00:00<00:00, 7240.33 examples/s]
Map:  81%|########1 | 6772/8322 [00:00<00:00, 7361.36 examples/s]
Map:  95%|#########4| 7873/8322 [00:01<00:00, 7350.69 examples/s]
Map: 100%|##########| 8322/8322 [00:01<00:00, 7301.21 examples/s]

Map:   0%|          | 0/8322 [00:00<?, ? examples/s]
Map:  36%|###6      | 3000/8322 [00:00<00:00, 28693.59 examples/s]
Map:  72%|#######2  | 6000/8322 [00:00<00:00, 29094.99 examples/s]
Map: 100%|##########| 8322/8322 [00:00<00:00, 29509.13 examples/s]

Filter:   0%|          | 0/8322 [00:00<?, ? examples/s]
Filter:  72%|#######2  | 6000/8322 [00:00<00:00, 51232.62 examples/s]
Filter: 100%|##########| 8322/8322 [00:00<00:00, 51189.58 examples/s]

Map:   0%|          | 0/887 [00:00<?, ? examples/s]
Map: 100%|##########| 887/887 [00:00<00:00, 11538.82 examples/s]

Creating json from Arrow format:   0%|          | 0/18 [00:00<?, ?ba/s]
Creating json from Arrow format:  11%|#1        | 2/18 [00:00<00:00, 18.14ba/s]
Creating json from Arrow format:  22%|##2       | 4/18 [00:00<00:00, 18.10ba/s]
Creating json from Arrow format:  39%|###8      | 7/18 [00:00<00:00, 18.75ba/s]
Creating json from Arrow format:  56%|#####5    | 10/18 [00:00<00:00, 20.58ba/s]
Creating json from Arrow format:  83%|########3 | 15/18 [00:00<00:00, 27.75ba/s]
Creating json from Arrow format: 100%|##########| 18/18 [00:00<00:00, 25.63ba/s]

Map:   0%|          | 0/463 [00:00<?, ? examples/s]
Map: 100%|##########| 463/463 [00:00<00:00, 7059.63 examples/s]

Map:   0%|          | 0/463 [00:00<?, ? examples/s]
Map: 100%|##########| 463/463 [00:00<00:00, 27327.34 examples/s]

Filter:   0%|          | 0/463 [00:00<?, ? examples/s]
Filter: 100%|##########| 463/463 [00:00<00:00, 41131.08 examples/s]

Map:   0%|          | 0/23 [00:00<?, ? examples/s]
Map: 100%|##########| 23/23 [00:00<00:00, 5493.37 examples/s]

Map:   0%|          | 0/23 [00:00<?, ? examples/s]
Map: 100%|##########| 23/23 [00:00<00:00, 8061.25 examples/s]

Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 79.29ba/s]

Map:   0%|          | 0/463 [00:00<?, ? examples/s]
Map: 100%|##########| 463/463 [00:00<00:00, 7215.89 examples/s]

Map:   0%|          | 0/463 [00:00<?, ? examples/s]
Map: 100%|##########| 463/463 [00:00<00:00, 7733.42 examples/s]

Map:   0%|          | 0/463 [00:00<?, ? examples/s]
Map: 100%|##########| 463/463 [00:00<00:00, 7147.27 examples/s]

Map:   0%|          | 0/463 [00:00<?, ? examples/s]
Map: 100%|##########| 463/463 [00:00<00:00, 27862.93 examples/s]

Filter:   0%|          | 0/463 [00:00<?, ? examples/s]
Filter: 100%|##########| 463/463 [00:00<00:00, 42242.29 examples/s]

Map:   0%|          | 0/23 [00:00<?, ? examples/s]
Map: 100%|##########| 23/23 [00:00<00:00, 5686.35 examples/s]INFO 2025-05-20 11:03:18,226 instructlab.sdg.datamixing:774: Knowledge detected to be less than 3.00% of skills (0.24%), upsampling to: 11824


Creating json from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Creating json from Arrow format: 100%|##########| 1/1 [00:00<00:00, 23.47ba/s]INFO 2025-05-20 11:03:48,953 instructlab.sdg.datamixing:158: Loading dataset from /usr/share/instructlab/sdg/datasets/skills.jsonl ...
INFO 2025-05-20 11:04:21,138 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id']
INFO 2025-05-20 11:04:21,138 instructlab.sdg.datamixing:161: Dataset loaded with 394141 samples
INFO 2025-05-20 11:05:12,578 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_6_2_1__About_workload_updates_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1272 examples [00:00, 79214.19 examples/s]INFO 2025-05-20 11:05:12,635 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:05:12,635 instructlab.sdg.datamixing:161: Dataset loaded with 1272 samples
INFO 2025-05-20 11:05:12,635 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   3%|3         | 375/11824 [00:01<00:47, 241.14 examples/s]
Map (num_proc=8):   8%|8         | 1000/11824 [00:01<00:15, 709.06 examples/s]
Map (num_proc=8):  15%|#5        | 1804/11824 [00:02<00:13, 738.21 examples/s]
Map (num_proc=8):  21%|##        | 2451/11824 [00:02<00:08, 1104.08 examples/s]
Map (num_proc=8):  28%|##7       | 3274/11824 [00:03<00:09, 940.59 examples/s] 
Map (num_proc=8):  33%|###3      | 3933/11824 [00:04<00:06, 1267.14 examples/s]
Map (num_proc=8):  40%|####      | 4753/11824 [00:05<00:06, 1036.05 examples/s]
Map (num_proc=8):  46%|####5     | 5414/11824 [00:05<00:04, 1340.20 examples/s]
Map (num_proc=8):  52%|#####2    | 6190/11824 [00:06<00:05, 1060.58 examples/s]
Map (num_proc=8):  58%|#####7    | 6849/11824 [00:06<00:03, 1338.01 examples/s]
Map (num_proc=8):  65%|######4   | 7669/11824 [00:07<00:03, 1087.73 examples/s]
Map (num_proc=8):  70%|#######   | 8330/11824 [00:07<00:02, 1368.97 examples/s]
Map (num_proc=8):  77%|#######7  | 9146/11824 [00:08<00:02, 1100.12 examples/s]
Map (num_proc=8):  83%|########2 | 9809/11824 [00:08<00:01, 1379.58 examples/s]
Map (num_proc=8):  89%|########9 | 10536/11824 [00:09<00:01, 1093.54 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1790.54 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1156.46 examples/s]INFO 2025-05-20 11:05:23,863 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1_6__About_RunStrategies_for_virtual_machines_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 4516 examples [00:00, 61770.65 examples/s]INFO 2025-05-20 11:05:23,977 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:05:23,977 instructlab.sdg.datamixing:161: Dataset loaded with 4516 samples
INFO 2025-05-20 11:05:23,977 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 678/11824 [00:01<00:23, 473.60 examples/s]
Map (num_proc=8):  18%|#8        | 2154/11824 [00:02<00:10, 920.64 examples/s]
Map (num_proc=8):  30%|###       | 3580/11824 [00:03<00:07, 1086.16 examples/s]
Map (num_proc=8):  43%|####2     | 5058/11824 [00:04<00:05, 1179.99 examples/s]
Map (num_proc=8):  56%|#####5    | 6593/11824 [00:05<00:04, 1254.24 examples/s]
Map (num_proc=8):  68%|######7   | 8028/11824 [00:06<00:02, 1271.84 examples/s]
Map (num_proc=8):  80%|########  | 9507/11824 [00:08<00:01, 1302.01 examples/s]
Map (num_proc=8):  94%|#########3| 11056/11824 [00:09<00:00, 1351.92 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:09<00:00, 1263.12 examples/s]INFO 2025-05-20 11:05:34,299 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_3_1__About_virtual_machine_events_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 216 examples [00:00, 49954.22 examples/s]INFO 2025-05-20 11:05:34,344 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:05:34,344 instructlab.sdg.datamixing:161: Dataset loaded with 216 samples
INFO 2025-05-20 11:05:34,344 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 704/11824 [00:00<00:13, 794.59 examples/s]
Map (num_proc=8):  19%|#8        | 2242/11824 [00:01<00:06, 1588.70 examples/s]
Map (num_proc=8):  32%|###1      | 3726/11824 [00:02<00:04, 1873.86 examples/s]
Map (num_proc=8):  43%|####3     | 5138/11824 [00:02<00:03, 1989.39 examples/s]
Map (num_proc=8):  56%|#####5    | 6612/11824 [00:03<00:02, 2086.97 examples/s]
Map (num_proc=8):  69%|######8   | 8143/11824 [00:04<00:01, 2173.25 examples/s]
Map (num_proc=8):  82%|########1 | 9647/11824 [00:04<00:00, 2215.83 examples/s]
Map (num_proc=8):  94%|#########4| 11144/11824 [00:05<00:00, 2274.47 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:05<00:00, 2098.23 examples/s]INFO 2025-05-20 11:05:40,715 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_8_5__Libguestfs_tools_and_virtctl_guestfs_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 534 examples [00:00, 54873.17 examples/s]INFO 2025-05-20 11:05:40,770 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:05:40,770 instructlab.sdg.datamixing:161: Dataset loaded with 534 samples
INFO 2025-05-20 11:05:40,770 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 633/11824 [00:01<00:26, 425.66 examples/s]
Map (num_proc=8):  18%|#7        | 2110/11824 [00:02<00:10, 893.04 examples/s]
Map (num_proc=8):  30%|###       | 3589/11824 [00:03<00:07, 1069.13 examples/s]
Map (num_proc=8):  43%|####2     | 5075/11824 [00:04<00:05, 1171.68 examples/s]
Map (num_proc=8):  55%|#####5    | 6550/11824 [00:05<00:04, 1218.98 examples/s]
Map (num_proc=8):  68%|######7   | 8022/11824 [00:07<00:03, 1243.11 examples/s]
Map (num_proc=8):  80%|########  | 9502/11824 [00:08<00:01, 1269.94 examples/s]
Map (num_proc=8):  93%|#########3| 11012/11824 [00:09<00:00, 1320.49 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:09<00:00, 1235.03 examples/s]INFO 2025-05-20 11:05:51,350 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_12_2__Removing_a_secret__config_map__or_service_account_from_a_virtual_machine_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1431 examples [00:00, 72296.42 examples/s]INFO 2025-05-20 11:05:51,399 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:05:51,399 instructlab.sdg.datamixing:161: Dataset loaded with 1431 samples
INFO 2025-05-20 11:05:51,399 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 690/11824 [00:01<00:23, 476.70 examples/s]
Map (num_proc=8):   9%|9         | 1084/11824 [00:01<00:14, 765.82 examples/s]
Map (num_proc=8):  18%|#8        | 2153/11824 [00:02<00:10, 955.31 examples/s]
Map (num_proc=8):  21%|##1       | 2538/11824 [00:02<00:08, 1145.98 examples/s]
Map (num_proc=8):  31%|###       | 3622/11824 [00:03<00:07, 1141.83 examples/s]
Map (num_proc=8):  34%|###4      | 4033/11824 [00:03<00:05, 1318.48 examples/s]
Map (num_proc=8):  43%|####3     | 5095/11824 [00:04<00:05, 1218.47 examples/s]
Map (num_proc=8):  46%|####6     | 5453/11824 [00:04<00:04, 1348.91 examples/s]
Map (num_proc=8):  55%|#####5    | 6559/11824 [00:05<00:04, 1257.92 examples/s]
Map (num_proc=8):  58%|#####8    | 6913/11824 [00:05<00:03, 1392.08 examples/s]
Map (num_proc=8):  68%|######7   | 8028/11824 [00:06<00:02, 1272.75 examples/s]
Map (num_proc=8):  71%|#######   | 8390/11824 [00:07<00:02, 1396.87 examples/s]
Map (num_proc=8):  80%|########  | 9503/11824 [00:07<00:01, 1286.82 examples/s]
Map (num_proc=8):  83%|########3 | 9868/11824 [00:08<00:01, 1415.13 examples/s]
Map (num_proc=8):  93%|#########2| 10987/11824 [00:09<00:00, 1330.59 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:09<00:00, 1271.40 examples/s]INFO 2025-05-20 11:06:01,660 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_16_10__Configuring_vGPU_passthrough_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 8421 examples [00:00, 64315.90 examples/s]
Generating train split: 19765 examples [00:00, 89912.26 examples/s]
Generating train split: 19765 examples [00:00, 85358.84 examples/s]INFO 2025-05-20 11:06:01,924 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:06:01,924 instructlab.sdg.datamixing:161: Dataset loaded with 19765 samples


Map (num_proc=8):   0%|          | 0/19765 [00:00<?, ? examples/s]
Map (num_proc=8):   1%|          | 163/19765 [00:00<00:23, 822.73 examples/s]
Map (num_proc=8):  12%|#1        | 2301/19765 [00:00<00:01, 9207.78 examples/s]
Map (num_proc=8):  22%|##1       | 4272/19765 [00:00<00:01, 12768.64 examples/s]
Map (num_proc=8):  30%|##9       | 5866/19765 [00:00<00:01, 13606.39 examples/s]
Map (num_proc=8):  37%|###7      | 7392/19765 [00:00<00:00, 12693.64 examples/s]
Map (num_proc=8):  45%|####4     | 8860/19765 [00:00<00:01, 10805.21 examples/s]
Map (num_proc=8):  53%|#####2    | 10393/19765 [00:00<00:00, 11617.51 examples/s]
Map (num_proc=8):  61%|######1   | 12093/19765 [00:01<00:00, 13017.89 examples/s]
Map (num_proc=8):  69%|######8   | 13540/19765 [00:01<00:00, 13410.52 examples/s]
Map (num_proc=8):  76%|#######5  | 14950/19765 [00:01<00:00, 13068.12 examples/s]
Map (num_proc=8):  83%|########2 | 16394/19765 [00:01<00:00, 12565.32 examples/s]
Map (num_proc=8):  90%|########9 | 17787/19765 [00:01<00:00, 12338.83 examples/s]
Map (num_proc=8):  97%|#########7| 19236/19765 [00:01<00:00, 12676.14 examples/s]
Map (num_proc=8): 100%|##########| 19765/19765 [00:01<00:00, 11386.15 examples/s]INFO 2025-05-20 11:06:04,060 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_3_2__Storage_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 582 examples [00:00, 103238.95 examples/s]INFO 2025-05-20 11:06:04,083 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:06:04,083 instructlab.sdg.datamixing:161: Dataset loaded with 582 samples
INFO 2025-05-20 11:06:04,083 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 766/11824 [00:00<00:10, 1056.62 examples/s]
Map (num_proc=8):  19%|#8        | 2210/11824 [00:01<00:05, 1859.89 examples/s]
Map (num_proc=8):  31%|###1      | 3690/11824 [00:01<00:03, 2268.57 examples/s]
Map (num_proc=8):  44%|####3     | 5169/11824 [00:02<00:02, 2448.14 examples/s]
Map (num_proc=8):  56%|#####6    | 6647/11824 [00:02<00:02, 2558.29 examples/s]
Map (num_proc=8):  69%|######8   | 8111/11824 [00:03<00:01, 2618.42 examples/s]
Map (num_proc=8):  81%|########1 | 9579/11824 [00:03<00:00, 2654.79 examples/s]
Map (num_proc=8):  94%|#########3| 11060/11824 [00:04<00:00, 2725.34 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:04<00:00, 2535.69 examples/s]INFO 2025-05-20 11:06:09,409 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_4_1__Prerequisites_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2670 examples [00:00, 69530.50 examples/s]INFO 2025-05-20 11:06:09,466 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:06:09,466 instructlab.sdg.datamixing:161: Dataset loaded with 2670 samples
INFO 2025-05-20 11:06:09,466 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 694/11824 [00:01<00:23, 465.02 examples/s]
Map (num_proc=8):  18%|#8        | 2180/11824 [00:02<00:10, 915.02 examples/s]
Map (num_proc=8):  31%|###       | 3659/11824 [00:03<00:07, 1097.02 examples/s]
Map (num_proc=8):  43%|####3     | 5133/11824 [00:04<00:05, 1188.01 examples/s]
Map (num_proc=8):  56%|#####6    | 6628/11824 [00:05<00:04, 1246.99 examples/s]
Map (num_proc=8):  69%|######8   | 8108/11824 [00:07<00:02, 1277.47 examples/s]
Map (num_proc=8):  81%|########  | 9575/11824 [00:08<00:01, 1292.67 examples/s]
Map (num_proc=8):  94%|#########3| 11071/11824 [00:09<00:00, 1336.11 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:09<00:00, 1253.83 examples/s]INFO 2025-05-20 11:06:19,954 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_6_2__Restarting_a_virtual_machine_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 476 examples [00:00, 75013.67 examples/s]INFO 2025-05-20 11:06:19,994 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:06:19,995 instructlab.sdg.datamixing:161: Dataset loaded with 476 samples
INFO 2025-05-20 11:06:19,995 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 717/11824 [00:01<00:18, 586.87 examples/s]
Map (num_proc=8):  18%|#8        | 2173/11824 [00:02<00:08, 1102.92 examples/s]
Map (num_proc=8):  31%|###       | 3631/11824 [00:03<00:06, 1313.27 examples/s]
Map (num_proc=8):  43%|####3     | 5121/11824 [00:03<00:04, 1433.07 examples/s]
Map (num_proc=8):  56%|#####5    | 6589/11824 [00:04<00:03, 1493.99 examples/s]
Map (num_proc=8):  68%|######8   | 8057/11824 [00:05<00:02, 1531.26 examples/s]
Map (num_proc=8):  81%|########  | 9524/11824 [00:06<00:01, 1551.71 examples/s]
Map (num_proc=8):  93%|#########3| 11016/11824 [00:07<00:00, 1600.35 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1506.46 examples/s]INFO 2025-05-20 11:06:28,742 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_2_2__Example_manifests_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 5153 examples [00:00, 72446.48 examples/s]INFO 2025-05-20 11:06:28,848 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:06:28,848 instructlab.sdg.datamixing:161: Dataset loaded with 5153 samples
INFO 2025-05-20 11:06:28,848 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 675/11824 [00:01<00:23, 474.90 examples/s]
Map (num_proc=8):  18%|#8        | 2151/11824 [00:02<00:10, 955.89 examples/s]
Map (num_proc=8):  31%|###       | 3636/11824 [00:03<00:07, 1154.03 examples/s]
Map (num_proc=8):  43%|####3     | 5106/11824 [00:04<00:05, 1251.84 examples/s]
Map (num_proc=8):  56%|#####5    | 6585/11824 [00:05<00:04, 1308.77 examples/s]
Map (num_proc=8):  68%|######8   | 8074/11824 [00:06<00:02, 1344.91 examples/s]
Map (num_proc=8):  81%|########  | 9556/11824 [00:07<00:01, 1368.80 examples/s]
Map (num_proc=8):  93%|#########3| 11054/11824 [00:08<00:00, 1413.54 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1320.66 examples/s]INFO 2025-05-20 11:06:38,837 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_2_2__Viewing_virtual_machine_logs_in_the_web_console_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 494 examples [00:00, 43025.65 examples/s]INFO 2025-05-20 11:06:38,901 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:06:38,901 instructlab.sdg.datamixing:161: Dataset loaded with 494 samples
INFO 2025-05-20 11:06:38,901 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|4         | 559/11824 [00:02<00:49, 227.05 examples/s]
Map (num_proc=8):   8%|8         | 1000/11824 [00:02<00:23, 451.95 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:02<00:14, 721.19 examples/s]
Map (num_proc=8):  17%|#6        | 2000/11824 [00:04<00:20, 484.33 examples/s]
Map (num_proc=8):  21%|##        | 2478/11824 [00:04<00:14, 640.05 examples/s]
Map (num_proc=8):  30%|##9       | 3508/11824 [00:06<00:12, 646.94 examples/s]
Map (num_proc=8):  33%|###3      | 3956/11824 [00:06<00:09, 811.18 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:06<00:07, 996.83 examples/s]
Map (num_proc=8):  42%|####2     | 4993/11824 [00:08<00:10, 637.50 examples/s]
Map (num_proc=8):  46%|####5     | 5434/11824 [00:08<00:07, 812.82 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:08<00:05, 1011.75 examples/s]
Map (num_proc=8):  55%|#####4    | 6458/11824 [00:09<00:08, 627.39 examples/s] 
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:10<00:06, 814.43 examples/s]
Map (num_proc=8):  62%|######2   | 7390/11824 [00:10<00:04, 1019.01 examples/s]
Map (num_proc=8):  67%|######7   | 7952/11824 [00:11<00:06, 635.90 examples/s] 
Map (num_proc=8):  71%|#######   | 8390/11824 [00:11<00:04, 817.47 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:12<00:02, 1014.80 examples/s]
Map (num_proc=8):  80%|#######9  | 9413/11824 [00:13<00:03, 626.44 examples/s] 
Map (num_proc=8):  83%|########3 | 9868/11824 [00:13<00:02, 817.17 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:14<00:01, 1012.87 examples/s]
Map (num_proc=8):  92%|#########2| 10926/11824 [00:15<00:01, 666.16 examples/s] 
Map (num_proc=8): 100%|##########| 11824/11824 [00:15<00:00, 1091.45 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:15<00:00, 743.02 examples/s] INFO 2025-05-20 11:06:56,285 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_12__Managing_config_maps__secrets__and_service_accounts_in_virtual_machines_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2020 examples [00:00, 83970.05 examples/s]INFO 2025-05-20 11:06:56,348 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:06:56,348 instructlab.sdg.datamixing:161: Dataset loaded with 2020 samples
INFO 2025-05-20 11:06:56,348 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 693/11824 [00:01<00:23, 477.36 examples/s]
Map (num_proc=8):   9%|9         | 1085/11824 [00:01<00:14, 759.46 examples/s]
Map (num_proc=8):  18%|#8        | 2169/11824 [00:02<00:09, 980.50 examples/s]
Map (num_proc=8):  22%|##1       | 2581/11824 [00:02<00:07, 1185.21 examples/s]
Map (num_proc=8):  30%|###       | 3596/11824 [00:03<00:07, 1144.43 examples/s]
Map (num_proc=8):  34%|###4      | 4021/11824 [00:03<00:05, 1329.39 examples/s]
Map (num_proc=8):  43%|####2     | 5062/11824 [00:04<00:05, 1235.82 examples/s]
Map (num_proc=8):  46%|####5     | 5434/11824 [00:04<00:04, 1370.36 examples/s]
Map (num_proc=8):  55%|#####5    | 6508/11824 [00:05<00:04, 1273.57 examples/s]
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:05<00:03, 1423.66 examples/s]
Map (num_proc=8):  68%|######7   | 7990/11824 [00:06<00:02, 1308.76 examples/s]
Map (num_proc=8):  71%|#######   | 8390/11824 [00:06<00:02, 1452.09 examples/s]
Map (num_proc=8):  80%|########  | 9461/11824 [00:07<00:01, 1320.09 examples/s]
Map (num_proc=8):  83%|########3 | 9868/11824 [00:07<00:01, 1485.33 examples/s]
Map (num_proc=8):  93%|#########2| 10954/11824 [00:08<00:00, 1401.17 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:09<00:00, 1307.03 examples/s]INFO 2025-05-20 11:07:06,470 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_5_6__Deleting_a_standalone_virtual_machine_instance_using_the_web_console_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2887 examples [00:00, 85114.30 examples/s]INFO 2025-05-20 11:07:06,538 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:07:06,538 instructlab.sdg.datamixing:161: Dataset loaded with 2887 samples
INFO 2025-05-20 11:07:06,538 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 739/11824 [00:01<00:15, 731.34 examples/s]
Map (num_proc=8):  19%|#8        | 2214/11824 [00:01<00:06, 1400.50 examples/s]
Map (num_proc=8):  31%|###1      | 3688/11824 [00:02<00:04, 1685.98 examples/s]
Map (num_proc=8):  44%|####3     | 5156/11824 [00:03<00:03, 1832.86 examples/s]
Map (num_proc=8):  56%|#####6    | 6634/11824 [00:03<00:02, 1911.37 examples/s]
Map (num_proc=8):  68%|######8   | 8083/11824 [00:04<00:01, 1958.43 examples/s]
Map (num_proc=8):  81%|########  | 9566/11824 [00:05<00:01, 2003.14 examples/s]
Map (num_proc=8):  93%|#########3| 11047/11824 [00:05<00:00, 2074.69 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:06<00:00, 1915.30 examples/s]INFO 2025-05-20 11:07:13,614 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_OpenShift_Virtualization_installation__usage__and_release_notes_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 288 examples [00:00, 54047.41 examples/s]INFO 2025-05-20 11:07:13,658 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:07:13,658 instructlab.sdg.datamixing:161: Dataset loaded with 288 samples
INFO 2025-05-20 11:07:13,658 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 704/11824 [00:01<00:21, 515.24 examples/s]
Map (num_proc=8):  18%|#8        | 2175/11824 [00:02<00:09, 1016.31 examples/s]
Map (num_proc=8):  31%|###       | 3649/11824 [00:03<00:06, 1226.89 examples/s]
Map (num_proc=8):  43%|####3     | 5128/11824 [00:04<00:05, 1338.22 examples/s]
Map (num_proc=8):  56%|#####5    | 6600/11824 [00:05<00:03, 1400.07 examples/s]
Map (num_proc=8):  68%|######8   | 8078/11824 [00:06<00:02, 1442.16 examples/s]
Map (num_proc=8):  81%|########  | 9559/11824 [00:07<00:01, 1469.59 examples/s]
Map (num_proc=8):  94%|#########3| 11061/11824 [00:08<00:00, 1544.58 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1419.29 examples/s]INFO 2025-05-20 11:07:23,032 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_1__Making_open_source_more_inclusive_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 496 examples [00:00, 89636.55 examples/s]INFO 2025-05-20 11:07:23,076 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:07:23,076 instructlab.sdg.datamixing:161: Dataset loaded with 496 samples
INFO 2025-05-20 11:07:23,076 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   7%|6         | 788/11824 [00:00<00:09, 1110.90 examples/s]
Map (num_proc=8):  19%|#9        | 2253/11824 [00:01<00:04, 2084.70 examples/s]
Map (num_proc=8):  32%|###1      | 3733/11824 [00:01<00:03, 2530.17 examples/s]
Map (num_proc=8):  44%|####4     | 5218/11824 [00:02<00:02, 2775.91 examples/s]
Map (num_proc=8):  57%|#####6    | 6691/11824 [00:02<00:01, 2908.23 examples/s]
Map (num_proc=8):  69%|######9   | 8174/11824 [00:03<00:01, 2996.80 examples/s]
Map (num_proc=8):  82%|########1 | 9644/11824 [00:03<00:00, 3046.47 examples/s]
Map (num_proc=8):  94%|#########4| 11162/11824 [00:03<00:00, 3166.34 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:04<00:00, 2839.38 examples/s]INFO 2025-05-20 11:07:27,998 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_20_20__Expanding_a_virtual_machine_disk_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 6680 examples [00:00, 62996.65 examples/s]
Generating train split: 15721 examples [00:00, 69613.84 examples/s]
Generating train split: 31352 examples [00:00, 98091.94 examples/s]
Generating train split: 31352 examples [00:00, 89883.86 examples/s]INFO 2025-05-20 11:07:28,366 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:07:28,366 instructlab.sdg.datamixing:161: Dataset loaded with 31352 samples


Map (num_proc=8):   0%|          | 0/31352 [00:00<?, ? examples/s]
Map (num_proc=8):   1%|          | 176/31352 [00:00<00:31, 977.00 examples/s]
Map (num_proc=8):   6%|6         | 1924/31352 [00:00<00:03, 7875.74 examples/s]
Map (num_proc=8):  12%|#2        | 3861/31352 [00:00<00:02, 12009.62 examples/s]
Map (num_proc=8):  18%|#7        | 5573/31352 [00:00<00:01, 12973.08 examples/s]
Map (num_proc=8):  23%|##2       | 7209/31352 [00:00<00:01, 12938.64 examples/s]
Map (num_proc=8):  27%|##7       | 8594/31352 [00:00<00:02, 10804.53 examples/s]
Map (num_proc=8):  31%|###1      | 9863/31352 [00:00<00:01, 11154.44 examples/s]
Map (num_proc=8):  37%|###6      | 11451/31352 [00:01<00:01, 12252.10 examples/s]
Map (num_proc=8):  42%|####1     | 13131/31352 [00:01<00:01, 13486.18 examples/s]
Map (num_proc=8):  47%|####7     | 14869/31352 [00:01<00:01, 14073.45 examples/s]
Map (num_proc=8):  52%|#####2    | 16334/31352 [00:01<00:01, 11633.13 examples/s]
Map (num_proc=8):  57%|#####7    | 17953/31352 [00:01<00:01, 12664.99 examples/s]
Map (num_proc=8):  62%|######2   | 19580/31352 [00:01<00:00, 13386.03 examples/s]
Map (num_proc=8):  68%|######7   | 21282/31352 [00:01<00:00, 14171.11 examples/s]
Map (num_proc=8):  73%|#######2  | 22885/31352 [00:01<00:00, 14211.92 examples/s]
Map (num_proc=8):  78%|#######8  | 24466/31352 [00:02<00:00, 12455.37 examples/s]
Map (num_proc=8):  83%|########2 | 25961/31352 [00:02<00:00, 12925.86 examples/s]
Map (num_proc=8):  88%|########7 | 27557/31352 [00:02<00:00, 13713.11 examples/s]
Map (num_proc=8):  94%|#########3| 29336/31352 [00:02<00:00, 14320.58 examples/s]
Map (num_proc=8):  98%|#########8| 30854/31352 [00:02<00:00, 14269.66 examples/s]
Map (num_proc=8): 100%|##########| 31352/31352 [00:02<00:00, 11966.77 examples/s]INFO 2025-05-20 11:07:31,487 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_8_3__Generalizing_a_Windows_VM_using_sysprep_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2632 examples [00:00, 76556.23 examples/s]INFO 2025-05-20 11:07:31,540 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:07:31,540 instructlab.sdg.datamixing:161: Dataset loaded with 2632 samples
INFO 2025-05-20 11:07:31,541 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 722/11824 [00:01<00:15, 721.92 examples/s]
Map (num_proc=8):  19%|#8        | 2201/11824 [00:01<00:06, 1416.19 examples/s]
Map (num_proc=8):  31%|###1      | 3680/11824 [00:02<00:04, 1711.04 examples/s]
Map (num_proc=8):  44%|####3     | 5161/11824 [00:03<00:03, 1869.22 examples/s]
Map (num_proc=8):  56%|#####6    | 6649/11824 [00:03<00:02, 1959.44 examples/s]
Map (num_proc=8):  69%|######8   | 8125/11824 [00:04<00:01, 2015.85 examples/s]
Map (num_proc=8):  81%|########1 | 9603/11824 [00:05<00:01, 2047.06 examples/s]
Map (num_proc=8):  94%|#########3| 11099/11824 [00:05<00:00, 2115.14 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:06<00:00, 1938.63 examples/s]INFO 2025-05-20 11:07:38,594 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_19_7__Viewing_the_IP_address_of_NICs_on_a_virtual_machine_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 6676 examples [00:00, 60834.82 examples/s]
Generating train split: 16100 examples [00:00, 82939.34 examples/s]INFO 2025-05-20 11:07:38,829 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:07:38,829 instructlab.sdg.datamixing:161: Dataset loaded with 16100 samples


Map (num_proc=8):   0%|          | 0/16100 [00:00<?, ? examples/s]
Map (num_proc=8):   2%|2         | 349/16100 [00:00<00:09, 1700.88 examples/s]
Map (num_proc=8):  15%|#4        | 2346/16100 [00:00<00:01, 8797.17 examples/s]
Map (num_proc=8):  25%|##5       | 4084/16100 [00:00<00:01, 11544.73 examples/s]
Map (num_proc=8):  36%|###6      | 5834/16100 [00:00<00:00, 12961.80 examples/s]
Map (num_proc=8):  45%|####5     | 7303/16100 [00:00<00:00, 12450.20 examples/s]
Map (num_proc=8):  54%|#####3    | 8679/16100 [00:00<00:00, 11536.83 examples/s]
Map (num_proc=8):  62%|######1   | 9981/16100 [00:00<00:00, 11541.06 examples/s]
Map (num_proc=8):  72%|#######1  | 11561/16100 [00:01<00:00, 12542.13 examples/s]
Map (num_proc=8):  82%|########2 | 13213/16100 [00:01<00:00, 13069.93 examples/s]
Map (num_proc=8):  93%|#########2| 14918/16100 [00:01<00:00, 13533.27 examples/s]
Map (num_proc=8): 100%|##########| 16100/16100 [00:01<00:00, 10852.28 examples/s]INFO 2025-05-20 11:07:40,778 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_1_4__Restricted_network_environments_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 4409 examples [00:00, 66601.91 examples/s]INFO 2025-05-20 11:07:40,863 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:07:40,863 instructlab.sdg.datamixing:161: Dataset loaded with 4409 samples
INFO 2025-05-20 11:07:40,863 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 689/11824 [00:01<00:22, 484.19 examples/s]
Map (num_proc=8):  18%|#8        | 2166/11824 [00:02<00:09, 978.42 examples/s]
Map (num_proc=8):  31%|###       | 3646/11824 [00:03<00:06, 1177.85 examples/s]
Map (num_proc=8):  43%|####3     | 5128/11824 [00:04<00:05, 1281.96 examples/s]
Map (num_proc=8):  56%|#####5    | 6609/11824 [00:05<00:03, 1340.48 examples/s]
Map (num_proc=8):  68%|######8   | 8085/11824 [00:06<00:02, 1376.34 examples/s]
Map (num_proc=8):  81%|########  | 9556/11824 [00:07<00:01, 1397.88 examples/s]
Map (num_proc=8):  93%|#########3| 11051/11824 [00:08<00:00, 1456.15 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1352.27 examples/s]INFO 2025-05-20 11:07:50,749 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_2__Editing_virtual_machine_templates_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2664 examples [00:00, 87422.35 examples/s]INFO 2025-05-20 11:07:50,811 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:07:50,811 instructlab.sdg.datamixing:161: Dataset loaded with 2664 samples
INFO 2025-05-20 11:07:50,811 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 719/11824 [00:01<00:20, 553.76 examples/s]
Map (num_proc=8):  19%|#8        | 2199/11824 [00:02<00:09, 1061.55 examples/s]
Map (num_proc=8):  31%|###1      | 3680/11824 [00:03<00:06, 1270.63 examples/s]
Map (num_proc=8):  44%|####3     | 5168/11824 [00:04<00:04, 1382.88 examples/s]
Map (num_proc=8):  56%|#####6    | 6642/11824 [00:05<00:03, 1446.65 examples/s]
Map (num_proc=8):  69%|######8   | 8124/11824 [00:06<00:02, 1472.77 examples/s]
Map (num_proc=8):  81%|########1 | 9596/11824 [00:07<00:01, 1503.72 examples/s]
Map (num_proc=8):  94%|#########3| 11102/11824 [00:07<00:00, 1546.38 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1443.79 examples/s]INFO 2025-05-20 11:08:00,022 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_11_2__About_the_QEMU_guest_agent_information_in_the_web_console_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1463 examples [00:00, 123610.39 examples/s]INFO 2025-05-20 11:08:00,060 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:08:00,060 instructlab.sdg.datamixing:161: Dataset loaded with 1463 samples
INFO 2025-05-20 11:08:00,060 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 743/11824 [00:00<00:13, 808.63 examples/s]
Map (num_proc=8):  19%|#8        | 2211/11824 [00:01<00:06, 1589.56 examples/s]
Map (num_proc=8):  31%|###1      | 3685/11824 [00:02<00:04, 1930.85 examples/s]
Map (num_proc=8):  44%|####3     | 5157/11824 [00:02<00:03, 2111.12 examples/s]
Map (num_proc=8):  56%|#####6    | 6627/11824 [00:03<00:02, 2218.56 examples/s]
Map (num_proc=8):  69%|######8   | 8101/11824 [00:03<00:01, 2291.77 examples/s]
Map (num_proc=8):  81%|########1 | 9584/11824 [00:04<00:00, 2344.38 examples/s]
Map (num_proc=8):  94%|#########3| 11092/11824 [00:05<00:00, 2451.49 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:05<00:00, 2208.15 examples/s]INFO 2025-05-20 11:08:06,351 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_8_1__About_reviewing_top_consumers_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1408 examples [00:00, 114104.26 examples/s]INFO 2025-05-20 11:08:06,397 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:08:06,397 instructlab.sdg.datamixing:161: Dataset loaded with 1408 samples
INFO 2025-05-20 11:08:06,397 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 746/11824 [00:00<00:13, 810.11 examples/s]
Map (num_proc=8):  19%|#8        | 2215/11824 [00:01<00:06, 1587.56 examples/s]
Map (num_proc=8):  31%|###1      | 3688/11824 [00:02<00:04, 1929.54 examples/s]
Map (num_proc=8):  44%|####3     | 5166/11824 [00:02<00:03, 2113.88 examples/s]
Map (num_proc=8):  56%|#####6    | 6649/11824 [00:03<00:02, 2216.23 examples/s]
Map (num_proc=8):  69%|######8   | 8122/11824 [00:03<00:01, 2281.59 examples/s]
Map (num_proc=8):  81%|########1 | 9604/11824 [00:04<00:00, 2324.58 examples/s]
Map (num_proc=8):  94%|#########3| 11106/11824 [00:05<00:00, 2425.50 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:05<00:00, 2194.87 examples/s]INFO 2025-05-20 11:08:12,734 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_3__Installing_OpenShift_Virtualization_using_the_web_console_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1413 examples [00:00, 77274.29 examples/s]INFO 2025-05-20 11:08:12,792 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:08:12,792 instructlab.sdg.datamixing:161: Dataset loaded with 1413 samples
INFO 2025-05-20 11:08:12,792 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 697/11824 [00:01<00:25, 432.03 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:01<00:10, 1026.07 examples/s]
Map (num_proc=8):  18%|#8        | 2170/11824 [00:02<00:11, 815.79 examples/s] 
Map (num_proc=8):  25%|##5       | 2956/11824 [00:02<00:06, 1293.64 examples/s]
Map (num_proc=8):  31%|###       | 3657/11824 [00:03<00:08, 963.21 examples/s] 
Map (num_proc=8):  43%|####3     | 5122/11824 [00:05<00:06, 1089.17 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:05<00:04, 1432.53 examples/s]
Map (num_proc=8):  56%|#####5    | 6614/11824 [00:06<00:04, 1084.26 examples/s]
Map (num_proc=8):  68%|######8   | 8099/11824 [00:07<00:03, 1155.95 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:07<00:02, 1458.65 examples/s]
Map (num_proc=8):  81%|########  | 9574/11824 [00:08<00:02, 1121.94 examples/s]
Map (num_proc=8):  94%|#########3| 11076/11824 [00:09<00:00, 1215.17 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1168.49 examples/s]INFO 2025-05-20 11:08:24,106 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_3_4__Viewing_resource_events_in_the_CLI_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 360 examples [00:00, 74042.54 examples/s]INFO 2025-05-20 11:08:24,145 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:08:24,145 instructlab.sdg.datamixing:161: Dataset loaded with 360 samples
INFO 2025-05-20 11:08:24,145 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 741/11824 [00:00<00:14, 777.14 examples/s]
Map (num_proc=8):  19%|#8        | 2222/11824 [00:01<00:06, 1496.18 examples/s]
Map (num_proc=8):  31%|###1      | 3676/11824 [00:02<00:04, 1782.01 examples/s]
Map (num_proc=8):  44%|####3     | 5149/11824 [00:02<00:03, 1945.02 examples/s]
Map (num_proc=8):  56%|#####5    | 6612/11824 [00:03<00:02, 2031.87 examples/s]
Map (num_proc=8):  68%|######8   | 8073/11824 [00:04<00:01, 2085.28 examples/s]
Map (num_proc=8):  81%|########  | 9560/11824 [00:04<00:01, 2131.94 examples/s]
Map (num_proc=8):  94%|#########3| 11056/11824 [00:05<00:00, 2217.82 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:05<00:00, 2039.23 examples/s]INFO 2025-05-20 11:08:30,844 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_1_2__Resources_monitored_actively_in_the_Virtualization_Overview_page_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1713 examples [00:00, 116131.81 examples/s]INFO 2025-05-20 11:08:30,898 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:08:30,898 instructlab.sdg.datamixing:161: Dataset loaded with 1713 samples
INFO 2025-05-20 11:08:30,898 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 726/11824 [00:01<00:18, 615.55 examples/s]
Map (num_proc=8):  19%|#8        | 2216/11824 [00:02<00:07, 1210.80 examples/s]
Map (num_proc=8):  31%|###1      | 3692/11824 [00:02<00:05, 1458.16 examples/s]
Map (num_proc=8):  44%|####3     | 5173/11824 [00:03<00:04, 1590.98 examples/s]
Map (num_proc=8):  56%|#####6    | 6645/11824 [00:04<00:03, 1661.81 examples/s]
Map (num_proc=8):  69%|######8   | 8126/11824 [00:05<00:02, 1710.66 examples/s]
Map (num_proc=8):  81%|########1 | 9602/11824 [00:06<00:01, 1740.21 examples/s]
Map (num_proc=8):  94%|#########3| 11098/11824 [00:06<00:00, 1819.99 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1667.87 examples/s]INFO 2025-05-20 11:08:39,019 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_1_1__What_you_can_do_with_OpenShift_Virtualization_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 302 examples [00:00, 43434.48 examples/s]INFO 2025-05-20 11:08:39,059 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:08:39,059 instructlab.sdg.datamixing:161: Dataset loaded with 302 samples
INFO 2025-05-20 11:08:39,060 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 595/11824 [00:01<00:37, 300.73 examples/s]
Map (num_proc=8):  17%|#7        | 2061/11824 [00:03<00:14, 677.95 examples/s]
Map (num_proc=8):  30%|##9       | 3530/11824 [00:04<00:09, 837.22 examples/s]
Map (num_proc=8):  42%|####2     | 5006/11824 [00:06<00:07, 921.19 examples/s]
Map (num_proc=8):  55%|#####4    | 6480/11824 [00:07<00:05, 970.21 examples/s]
Map (num_proc=8):  67%|######7   | 7954/11824 [00:08<00:03, 1003.22 examples/s]
Map (num_proc=8):  80%|#######9  | 9430/11824 [00:10<00:02, 1021.25 examples/s]
Map (num_proc=8):  92%|#########2| 10925/11824 [00:11<00:00, 1075.94 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:11<00:00, 1331.04 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:11<00:00, 990.26 examples/s] INFO 2025-05-20 11:08:52,473 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_2_5__Adding_a_network_interface_to_a_virtual_machine_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 4365 examples [00:00, 72854.58 examples/s]INFO 2025-05-20 11:08:52,566 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:08:52,566 instructlab.sdg.datamixing:161: Dataset loaded with 4365 samples
INFO 2025-05-20 11:08:52,566 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 693/11824 [00:01<00:21, 518.22 examples/s]
Map (num_proc=8):  18%|#8        | 2172/11824 [00:02<00:09, 1051.07 examples/s]
Map (num_proc=8):  31%|###       | 3643/11824 [00:03<00:06, 1272.24 examples/s]
Map (num_proc=8):  43%|####3     | 5138/11824 [00:04<00:04, 1398.32 examples/s]
Map (num_proc=8):  56%|#####5    | 6610/11824 [00:05<00:03, 1460.55 examples/s]
Map (num_proc=8):  68%|######8   | 8078/11824 [00:05<00:02, 1497.39 examples/s]
Map (num_proc=8):  81%|########  | 9569/11824 [00:06<00:01, 1528.56 examples/s]
Map (num_proc=8):  94%|#########3| 11075/11824 [00:07<00:00, 1604.54 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1468.01 examples/s]INFO 2025-05-20 11:09:01,793 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_2_2__Supported_guest_operating_systems_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 512 examples [00:00, 80436.12 examples/s]INFO 2025-05-20 11:09:01,836 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:09:01,836 instructlab.sdg.datamixing:161: Dataset loaded with 512 samples
INFO 2025-05-20 11:09:01,836 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   7%|6         | 771/11824 [00:00<00:11, 922.18 examples/s]
Map (num_proc=8):  19%|#9        | 2250/11824 [00:01<00:05, 1786.51 examples/s]
Map (num_proc=8):  32%|###1      | 3728/11824 [00:01<00:03, 2171.15 examples/s]
Map (num_proc=8):  44%|####4     | 5204/11824 [00:02<00:02, 2374.80 examples/s]
Map (num_proc=8):  56%|#####6    | 6676/11824 [00:03<00:02, 2493.98 examples/s]
Map (num_proc=8):  69%|######8   | 8154/11824 [00:03<00:01, 2568.18 examples/s]
Map (num_proc=8):  82%|########1 | 9644/11824 [00:04<00:00, 2628.93 examples/s]
Map (num_proc=8):  94%|#########4| 11146/11824 [00:04<00:00, 2741.70 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:04<00:00, 2458.18 examples/s]INFO 2025-05-20 11:09:07,601 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_6_6__Additional_resources_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2929 examples [00:00, 71266.82 examples/s]INFO 2025-05-20 11:09:07,660 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:09:07,660 instructlab.sdg.datamixing:161: Dataset loaded with 2929 samples
INFO 2025-05-20 11:09:07,660 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 666/11824 [00:01<00:23, 482.87 examples/s]
Map (num_proc=8):   9%|9         | 1083/11824 [00:01<00:13, 799.27 examples/s]
Map (num_proc=8):  18%|#8        | 2133/11824 [00:02<00:09, 1039.63 examples/s]
Map (num_proc=8):  21%|##1       | 2525/11824 [00:02<00:07, 1243.51 examples/s]
Map (num_proc=8):  31%|###       | 3640/11824 [00:03<00:06, 1281.27 examples/s]
Map (num_proc=8):  34%|###3      | 4019/11824 [00:03<00:05, 1419.45 examples/s]
Map (num_proc=8):  43%|####2     | 5051/11824 [00:04<00:05, 1340.75 examples/s]
Map (num_proc=8):  46%|####5     | 5434/11824 [00:04<00:04, 1471.49 examples/s]
Map (num_proc=8):  55%|#####5    | 6533/11824 [00:05<00:03, 1406.80 examples/s]
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:05<00:03, 1544.34 examples/s]
Map (num_proc=8):  68%|######8   | 8082/11824 [00:06<00:02, 1469.03 examples/s]
Map (num_proc=8):  72%|#######1  | 8460/11824 [00:06<00:02, 1566.37 examples/s]
Map (num_proc=8):  81%|########  | 9554/11824 [00:07<00:01, 1467.04 examples/s]
Map (num_proc=8):  84%|########3 | 9918/11824 [00:07<00:01, 1568.00 examples/s]
Map (num_proc=8):  93%|#########2| 10978/11824 [00:08<00:00, 1490.37 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1401.91 examples/s]INFO 2025-05-20 11:09:17,307 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_14_3_3__Additional_resources_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 3497 examples [00:00, 78648.12 examples/s]INFO 2025-05-20 11:09:17,380 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:09:17,380 instructlab.sdg.datamixing:161: Dataset loaded with 3497 samples
INFO 2025-05-20 11:09:17,380 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 658/11824 [00:01<00:23, 475.23 examples/s]
Map (num_proc=8):  18%|#7        | 2121/11824 [00:02<00:09, 988.35 examples/s]
Map (num_proc=8):  30%|###       | 3602/11824 [00:03<00:06, 1213.59 examples/s]
Map (num_proc=8):  43%|####2     | 5077/11824 [00:04<00:05, 1328.11 examples/s]
Map (num_proc=8):  55%|#####5    | 6547/11824 [00:05<00:03, 1392.39 examples/s]
Map (num_proc=8):  68%|######7   | 8003/11824 [00:06<00:02, 1427.39 examples/s]
Map (num_proc=8):  80%|########  | 9487/11824 [00:07<00:01, 1459.19 examples/s]
Map (num_proc=8):  93%|#########2| 10978/11824 [00:08<00:00, 1527.41 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1409.44 examples/s]INFO 2025-05-20 11:09:26,962 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_11_4__Additional_resources_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 3172 examples [00:00, 90053.56 examples/s]INFO 2025-05-20 11:09:27,054 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:09:27,054 instructlab.sdg.datamixing:161: Dataset loaded with 3172 samples
INFO 2025-05-20 11:09:27,054 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 697/11824 [00:01<00:19, 576.14 examples/s]
Map (num_proc=8):  18%|#8        | 2182/11824 [00:02<00:08, 1143.78 examples/s]
Map (num_proc=8):  31%|###       | 3655/11824 [00:02<00:05, 1384.31 examples/s]
Map (num_proc=8):  44%|####3     | 5148/11824 [00:03<00:04, 1511.92 examples/s]
Map (num_proc=8):  56%|#####5    | 6608/11824 [00:04<00:03, 1574.90 examples/s]
Map (num_proc=8):  68%|######8   | 8092/11824 [00:05<00:02, 1626.03 examples/s]
Map (num_proc=8):  81%|########  | 9574/11824 [00:06<00:01, 1655.05 examples/s]
Map (num_proc=8):  94%|#########3| 11090/11824 [00:07<00:00, 1728.56 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1587.44 examples/s]INFO 2025-05-20 11:09:35,619 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_5__Deleting_virtual_machine_templates_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1316 examples [00:00, 138968.86 examples/s]INFO 2025-05-20 11:09:35,664 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:09:35,664 instructlab.sdg.datamixing:161: Dataset loaded with 1316 samples
INFO 2025-05-20 11:09:35,664 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 761/11824 [00:00<00:13, 816.24 examples/s]
Map (num_proc=8):  19%|#9        | 2254/11824 [00:01<00:06, 1573.34 examples/s]
Map (num_proc=8):  32%|###1      | 3737/11824 [00:02<00:04, 1911.15 examples/s]
Map (num_proc=8):  44%|####4     | 5209/11824 [00:02<00:03, 2068.93 examples/s]
Map (num_proc=8):  57%|#####6    | 6689/11824 [00:03<00:02, 2165.76 examples/s]
Map (num_proc=8):  69%|######9   | 8165/11824 [00:04<00:01, 2237.38 examples/s]
Map (num_proc=8):  82%|########1 | 9651/11824 [00:04<00:00, 2271.41 examples/s]
Map (num_proc=8):  94%|#########4| 11151/11824 [00:05<00:00, 2381.60 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:05<00:00, 2152.80 examples/s]INFO 2025-05-20 11:09:42,120 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_7__About_the_tekton_tasks_operator_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 233 examples [00:00, 30390.67 examples/s]INFO 2025-05-20 11:09:42,165 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:09:42,165 instructlab.sdg.datamixing:161: Dataset loaded with 233 samples
INFO 2025-05-20 11:09:42,165 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|4         | 565/11824 [00:03<01:03, 176.04 examples/s]
Map (num_proc=8):   8%|8         | 1000/11824 [00:03<00:31, 346.28 examples/s]
Map (num_proc=8):  17%|#7        | 2033/11824 [00:05<00:24, 404.54 examples/s]
Map (num_proc=8):  21%|##        | 2478/11824 [00:05<00:17, 531.73 examples/s]
Map (num_proc=8):  30%|##9       | 3513/11824 [00:08<00:16, 490.00 examples/s]
Map (num_proc=8):  33%|###3      | 3956/11824 [00:08<00:13, 602.39 examples/s]
Map (num_proc=8):  42%|####2     | 4997/11824 [00:10<00:12, 528.00 examples/s]
Map (num_proc=8):  46%|####5     | 5434/11824 [00:10<00:10, 635.10 examples/s]
Map (num_proc=8):  55%|#####4    | 6447/11824 [00:13<00:10, 536.84 examples/s]
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:13<00:07, 646.46 examples/s]
Map (num_proc=8):  67%|######7   | 7947/11824 [00:15<00:07, 550.96 examples/s]
Map (num_proc=8):  71%|#######   | 8390/11824 [00:15<00:05, 658.60 examples/s]
Map (num_proc=8):  80%|#######9  | 9432/11824 [00:17<00:04, 555.62 examples/s]
Map (num_proc=8):  83%|########3 | 9868/11824 [00:18<00:02, 654.68 examples/s]
Map (num_proc=8):  92%|#########2| 10923/11824 [00:20<00:01, 580.35 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:20<00:00, 833.09 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:20<00:00, 570.13 examples/s]INFO 2025-05-20 11:10:04,810 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_2_2__Editing_a_virtual_machine_YAML_configuration_using_the_web_console_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 4978 examples [00:00, 80123.28 examples/s]INFO 2025-05-20 11:10:04,906 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:10:04,906 instructlab.sdg.datamixing:161: Dataset loaded with 4978 samples
INFO 2025-05-20 11:10:04,906 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 693/11824 [00:01<00:21, 509.83 examples/s]
Map (num_proc=8):  18%|#8        | 2173/11824 [00:02<00:09, 1037.26 examples/s]
Map (num_proc=8):  31%|###       | 3644/11824 [00:03<00:06, 1254.60 examples/s]
Map (num_proc=8):  43%|####3     | 5129/11824 [00:04<00:04, 1371.08 examples/s]
Map (num_proc=8):  56%|#####5    | 6597/11824 [00:05<00:03, 1432.03 examples/s]
Map (num_proc=8):  68%|######8   | 8084/11824 [00:06<00:02, 1477.97 examples/s]
Map (num_proc=8):  81%|########  | 9561/11824 [00:07<00:01, 1502.97 examples/s]
Map (num_proc=8):  94%|#########3| 11072/11824 [00:07<00:00, 1581.17 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1446.46 examples/s]INFO 2025-05-20 11:10:14,300 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_5__Viewing_information_about_virtual_machine_workloads_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 771 examples [00:00, 84121.75 examples/s]INFO 2025-05-20 11:10:14,345 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:10:14,345 instructlab.sdg.datamixing:161: Dataset loaded with 771 samples
INFO 2025-05-20 11:10:14,345 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 702/11824 [00:01<00:20, 553.63 examples/s]
Map (num_proc=8):  18%|#8        | 2181/11824 [00:02<00:08, 1111.17 examples/s]
Map (num_proc=8):  31%|###       | 3661/11824 [00:03<00:06, 1348.67 examples/s]
Map (num_proc=8):  43%|####3     | 5133/11824 [00:03<00:04, 1474.27 examples/s]
Map (num_proc=8):  56%|#####6    | 6624/11824 [00:04<00:03, 1548.25 examples/s]
Map (num_proc=8):  68%|######8   | 8073/11824 [00:05<00:02, 1579.53 examples/s]
Map (num_proc=8):  81%|########1 | 9581/11824 [00:06<00:01, 1619.73 examples/s]
Map (num_proc=8):  94%|#########3| 11087/11824 [00:07<00:00, 1695.97 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1547.75 examples/s]INFO 2025-05-20 11:10:23,122 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_10_1__About_the_OpenShift_Container_Platform_cluster_checkup_framework_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 3025 examples [00:00, 72262.04 examples/s]INFO 2025-05-20 11:10:23,195 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:10:23,196 instructlab.sdg.datamixing:161: Dataset loaded with 3025 samples
INFO 2025-05-20 11:10:23,196 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 633/11824 [00:01<00:28, 392.94 examples/s]
Map (num_proc=8):   9%|8         | 1011/11824 [00:01<00:17, 619.45 examples/s]
Map (num_proc=8):  18%|#7        | 2104/11824 [00:02<00:11, 869.53 examples/s]
Map (num_proc=8):  21%|##1       | 2496/11824 [00:02<00:09, 1021.58 examples/s]
Map (num_proc=8):  31%|###       | 3611/11824 [00:03<00:07, 1079.27 examples/s]
Map (num_proc=8):  34%|###3      | 3987/11824 [00:04<00:06, 1179.55 examples/s]
Map (num_proc=8):  43%|####3     | 5091/11824 [00:04<00:05, 1252.60 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:05<00:03, 1618.58 examples/s]
Map (num_proc=8):  56%|#####5    | 6574/11824 [00:06<00:04, 1173.52 examples/s]
Map (num_proc=8):  62%|######2   | 7390/11824 [00:06<00:02, 1526.60 examples/s]
Map (num_proc=8):  68%|######8   | 8055/11824 [00:07<00:03, 1232.49 examples/s]
Map (num_proc=8):  71%|#######1  | 8410/11824 [00:07<00:02, 1312.22 examples/s]
Map (num_proc=8):  80%|########  | 9511/11824 [00:08<00:01, 1318.49 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:08<00:00, 1687.24 examples/s]
Map (num_proc=8):  93%|#########3| 11016/11824 [00:09<00:00, 1361.95 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:09<00:00, 1253.89 examples/s]INFO 2025-05-20 11:10:33,931 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_9_3__About_Telemetry_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 3234 examples [00:00, 88231.06 examples/s]INFO 2025-05-20 11:10:34,003 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:10:34,003 instructlab.sdg.datamixing:161: Dataset loaded with 3234 samples
INFO 2025-05-20 11:10:34,003 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 710/11824 [00:01<00:19, 566.37 examples/s]
Map (num_proc=8):  18%|#8        | 2186/11824 [00:02<00:08, 1148.02 examples/s]
Map (num_proc=8):  25%|##5       | 2956/11824 [00:02<00:05, 1649.18 examples/s]
Map (num_proc=8):  31%|###       | 3651/11824 [00:02<00:06, 1283.63 examples/s]
Map (num_proc=8):  43%|####3     | 5103/11824 [00:03<00:04, 1447.32 examples/s]
Map (num_proc=8):  56%|#####5    | 6575/11824 [00:04<00:03, 1540.21 examples/s]
Map (num_proc=8):  68%|######7   | 8040/11824 [00:05<00:02, 1590.60 examples/s]
Map (num_proc=8):  80%|########  | 9505/11824 [00:06<00:01, 1621.89 examples/s]
Map (num_proc=8):  93%|#########3| 11007/11824 [00:07<00:00, 1719.93 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1570.37 examples/s]INFO 2025-05-20 11:10:42,720 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_4_1__Deprecated_features_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 670 examples [00:00, 89667.63 examples/s]INFO 2025-05-20 11:10:42,773 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:10:42,773 instructlab.sdg.datamixing:161: Dataset loaded with 670 samples
INFO 2025-05-20 11:10:42,773 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 724/11824 [00:01<00:16, 665.16 examples/s]
Map (num_proc=8):  10%|#         | 1229/11824 [00:01<00:09, 1142.72 examples/s]
Map (num_proc=8):  19%|#8        | 2213/11824 [00:01<00:06, 1390.42 examples/s]
Map (num_proc=8):  23%|##3       | 2722/11824 [00:01<00:05, 1739.35 examples/s]
Map (num_proc=8):  31%|###1      | 3681/11824 [00:02<00:04, 1650.38 examples/s]
Map (num_proc=8):  36%|###5      | 4199/11824 [00:02<00:03, 1985.89 examples/s]
Map (num_proc=8):  44%|####3     | 5171/11824 [00:03<00:03, 1788.31 examples/s]
Map (num_proc=8):  48%|####8     | 5687/11824 [00:03<00:02, 2070.93 examples/s]
Map (num_proc=8):  56%|#####6    | 6652/11824 [00:04<00:02, 1857.89 examples/s]
Map (num_proc=8):  61%|######    | 7172/11824 [00:04<00:02, 2154.19 examples/s]
Map (num_proc=8):  69%|######8   | 8125/11824 [00:04<00:01, 1878.97 examples/s]
Map (num_proc=8):  73%|#######2  | 8608/11824 [00:04<00:01, 2111.75 examples/s]
Map (num_proc=8):  81%|########1 | 9602/11824 [00:05<00:01, 1902.40 examples/s]
Map (num_proc=8):  85%|########5 | 10072/11824 [00:05<00:00, 2124.21 examples/s]
Map (num_proc=8):  94%|#########3| 11103/11824 [00:06<00:00, 2017.52 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:06<00:00, 1849.47 examples/s]INFO 2025-05-20 11:10:50,286 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_3_1__Adding_items_to_a_boot_order_list_in_the_web_console_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2276 examples [00:00, 103658.65 examples/s]INFO 2025-05-20 11:10:50,341 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:10:50,342 instructlab.sdg.datamixing:161: Dataset loaded with 2276 samples
INFO 2025-05-20 11:10:50,342 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 707/11824 [00:01<00:21, 519.56 examples/s]
Map (num_proc=8):  18%|#8        | 2180/11824 [00:02<00:09, 1021.30 examples/s]
Map (num_proc=8):  31%|###       | 3664/11824 [00:03<00:06, 1234.94 examples/s]
Map (num_proc=8):  43%|####3     | 5141/11824 [00:04<00:04, 1345.53 examples/s]
Map (num_proc=8):  56%|#####5    | 6616/11824 [00:05<00:03, 1406.74 examples/s]
Map (num_proc=8):  69%|######8   | 8101/11824 [00:06<00:02, 1447.22 examples/s]
Map (num_proc=8):  81%|########1 | 9583/11824 [00:07<00:01, 1471.64 examples/s]
Map (num_proc=8):  94%|#########3| 11084/11824 [00:08<00:00, 1530.30 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1414.06 examples/s]INFO 2025-05-20 11:10:59,898 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_17_1__TLS_certificates_for_data_volume_imports_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1870 examples [00:00, 91549.83 examples/s]INFO 2025-05-20 11:10:59,949 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:10:59,949 instructlab.sdg.datamixing:161: Dataset loaded with 1870 samples
INFO 2025-05-20 11:10:59,949 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 662/11824 [00:01<00:22, 505.21 examples/s]
Map (num_proc=8):   9%|9         | 1070/11824 [00:01<00:13, 807.19 examples/s]
Map (num_proc=8):  18%|#8        | 2140/11824 [00:02<00:08, 1144.14 examples/s]
Map (num_proc=8):  25%|##5       | 2956/11824 [00:02<00:05, 1666.12 examples/s]
Map (num_proc=8):  31%|###       | 3625/11824 [00:03<00:06, 1314.72 examples/s]
Map (num_proc=8):  34%|###3      | 4014/11824 [00:03<00:05, 1464.50 examples/s]
Map (num_proc=8):  43%|####3     | 5105/11824 [00:03<00:04, 1476.33 examples/s]
Map (num_proc=8):  47%|####6     | 5503/11824 [00:04<00:03, 1600.21 examples/s]
Map (num_proc=8):  56%|#####5    | 6573/11824 [00:04<00:03, 1547.34 examples/s]
Map (num_proc=8):  59%|#####8    | 6952/11824 [00:05<00:02, 1657.82 examples/s]
Map (num_proc=8):  68%|######8   | 8054/11824 [00:05<00:02, 1589.55 examples/s]
Map (num_proc=8):  71%|#######1  | 8413/11824 [00:05<00:02, 1677.25 examples/s]
Map (num_proc=8):  81%|########  | 9536/11824 [00:06<00:01, 1614.78 examples/s]
Map (num_proc=8):  84%|########3 | 9895/11824 [00:06<00:01, 1700.83 examples/s]
Map (num_proc=8):  93%|#########3| 11044/11824 [00:07<00:00, 1727.17 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1533.08 examples/s]INFO 2025-05-20 11:11:08,909 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_11_3__Migrating_a_virtual_machine_instance_to_another_node_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 262 examples [00:00, 52890.58 examples/s]INFO 2025-05-20 11:11:08,939 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:11:08,939 instructlab.sdg.datamixing:161: Dataset loaded with 262 samples
INFO 2025-05-20 11:11:08,939 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 714/11824 [00:01<00:17, 619.57 examples/s]
Map (num_proc=8):  19%|#8        | 2192/11824 [00:01<00:07, 1221.22 examples/s]
Map (num_proc=8):  31%|###       | 3648/11824 [00:02<00:05, 1485.86 examples/s]
Map (num_proc=8):  43%|####3     | 5121/11824 [00:03<00:04, 1617.82 examples/s]
Map (num_proc=8):  56%|#####5    | 6604/11824 [00:04<00:03, 1701.19 examples/s]
Map (num_proc=8):  68%|######8   | 8069/11824 [00:05<00:02, 1748.41 examples/s]
Map (num_proc=8):  81%|########  | 9543/11824 [00:05<00:01, 1763.89 examples/s]
Map (num_proc=8):  93%|#########3| 11036/11824 [00:06<00:00, 1865.09 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:06<00:00, 1706.29 examples/s]INFO 2025-05-20 11:11:17,009 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_2__About_Red_Hat_OpenShift_Virtualization_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 687 examples [00:00, 87941.37 examples/s]INFO 2025-05-20 11:11:17,053 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:11:17,053 instructlab.sdg.datamixing:161: Dataset loaded with 687 samples
INFO 2025-05-20 11:11:17,053 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 744/11824 [00:00<00:14, 751.74 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:01<00:06, 1517.19 examples/s]
Map (num_proc=8):  19%|#8        | 2221/11824 [00:01<00:06, 1485.59 examples/s]
Map (num_proc=8):  25%|##5       | 2956/11824 [00:01<00:04, 2075.04 examples/s]
Map (num_proc=8):  31%|###1      | 3700/11824 [00:02<00:04, 1788.51 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:02<00:03, 2286.76 examples/s]
Map (num_proc=8):  44%|####3     | 5166/11824 [00:02<00:03, 1921.62 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:03<00:02, 2403.62 examples/s]
Map (num_proc=8):  56%|#####6    | 6656/11824 [00:03<00:02, 1998.90 examples/s]
Map (num_proc=8):  62%|######2   | 7390/11824 [00:03<00:01, 2475.12 examples/s]
Map (num_proc=8):  69%|######8   | 8121/11824 [00:04<00:01, 1990.34 examples/s]
Map (num_proc=8):  73%|#######3  | 8639/11824 [00:04<00:01, 2271.66 examples/s]
Map (num_proc=8):  81%|########1 | 9606/11824 [00:04<00:01, 2076.16 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:05<00:00, 2542.34 examples/s]
Map (num_proc=8):  94%|#########3| 11112/11824 [00:05<00:00, 2179.14 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:05<00:00, 2048.58 examples/s]INFO 2025-05-20 11:11:23,907 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_8_2__OpenShift_Container_Platform_client_commands_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 803 examples [00:00, 64799.64 examples/s]INFO 2025-05-20 11:11:23,956 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:11:23,956 instructlab.sdg.datamixing:161: Dataset loaded with 803 samples
INFO 2025-05-20 11:11:23,956 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 650/11824 [00:01<00:26, 419.17 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:01<00:09, 1074.40 examples/s]
Map (num_proc=8):  18%|#7        | 2126/11824 [00:02<00:11, 864.10 examples/s] 
Map (num_proc=8):  21%|##        | 2479/11824 [00:02<00:08, 1055.18 examples/s]
Map (num_proc=8):  31%|###       | 3617/11824 [00:03<00:07, 1124.46 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:03<00:04, 1606.46 examples/s]
Map (num_proc=8):  43%|####3     | 5089/11824 [00:04<00:05, 1162.36 examples/s]
Map (num_proc=8):  46%|####5     | 5434/11824 [00:04<00:04, 1310.84 examples/s]
Map (num_proc=8):  56%|#####5    | 6573/11824 [00:05<00:04, 1257.18 examples/s]
Map (num_proc=8):  62%|######2   | 7390/11824 [00:05<00:02, 1711.09 examples/s]
Map (num_proc=8):  68%|######8   | 8049/11824 [00:06<00:03, 1226.31 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:06<00:01, 1681.77 examples/s]
Map (num_proc=8):  81%|########  | 9535/11824 [00:07<00:01, 1216.77 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:08<00:00, 1666.09 examples/s]
Map (num_proc=8):  93%|#########3| 11035/11824 [00:08<00:00, 1271.96 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:09<00:00, 1282.26 examples/s]INFO 2025-05-20 11:11:34,628 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_12_1__Adding_a_secret__config_map__or_service_account_to_a_virtual_machine_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1545 examples [00:00, 85377.00 examples/s]INFO 2025-05-20 11:11:34,673 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:11:34,673 instructlab.sdg.datamixing:161: Dataset loaded with 1545 samples
INFO 2025-05-20 11:11:34,673 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 660/11824 [00:01<00:25, 443.67 examples/s]
Map (num_proc=8):  18%|#8        | 2138/11824 [00:02<00:10, 941.92 examples/s]
Map (num_proc=8):  31%|###       | 3623/11824 [00:03<00:07, 1138.65 examples/s]
Map (num_proc=8):  43%|####3     | 5100/11824 [00:04<00:05, 1246.99 examples/s]
Map (num_proc=8):  56%|#####5    | 6582/11824 [00:05<00:03, 1310.78 examples/s]
Map (num_proc=8):  68%|######8   | 8067/11824 [00:06<00:02, 1352.53 examples/s]
Map (num_proc=8):  81%|########  | 9554/11824 [00:07<00:01, 1388.39 examples/s]
Map (num_proc=8):  93%|#########3| 11051/11824 [00:08<00:00, 1443.51 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1323.31 examples/s]INFO 2025-05-20 11:11:44,927 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_11_3__Virtualization_metrics_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2897 examples [00:00, 101808.10 examples/s]INFO 2025-05-20 11:11:44,993 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:11:44,993 instructlab.sdg.datamixing:161: Dataset loaded with 2897 samples
INFO 2025-05-20 11:11:44,993 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 718/11824 [00:01<00:18, 603.19 examples/s]
Map (num_proc=8):  19%|#8        | 2193/11824 [00:02<00:08, 1196.16 examples/s]
Map (num_proc=8):  31%|###       | 3642/11824 [00:02<00:05, 1449.03 examples/s]
Map (num_proc=8):  43%|####3     | 5114/11824 [00:03<00:04, 1585.10 examples/s]
Map (num_proc=8):  56%|#####5    | 6621/11824 [00:04<00:03, 1679.73 examples/s]
Map (num_proc=8):  68%|######8   | 8088/11824 [00:05<00:02, 1717.18 examples/s]
Map (num_proc=8):  81%|########  | 9574/11824 [00:06<00:01, 1747.50 examples/s]
Map (num_proc=8):  94%|#########3| 11085/11824 [00:06<00:00, 1820.09 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1666.94 examples/s]INFO 2025-05-20 11:11:53,289 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_12_3__Managing_node_labeling_for_obsolete_CPU_models_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 277 examples [00:00, 60464.34 examples/s]INFO 2025-05-20 11:11:53,326 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:11:53,326 instructlab.sdg.datamixing:161: Dataset loaded with 277 samples
INFO 2025-05-20 11:11:53,327 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 693/11824 [00:01<00:17, 629.07 examples/s]
Map (num_proc=8):  18%|#8        | 2151/11824 [00:01<00:07, 1273.71 examples/s]
Map (num_proc=8):  31%|###       | 3633/11824 [00:02<00:05, 1568.04 examples/s]
Map (num_proc=8):  43%|####3     | 5105/11824 [00:03<00:03, 1717.22 examples/s]
Map (num_proc=8):  56%|#####5    | 6584/11824 [00:04<00:02, 1807.94 examples/s]
Map (num_proc=8):  68%|######8   | 8061/11824 [00:04<00:02, 1862.17 examples/s]
Map (num_proc=8):  81%|########  | 9542/11824 [00:05<00:01, 1902.61 examples/s]
Map (num_proc=8):  93%|#########3| 11035/11824 [00:06<00:00, 1995.04 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:06<00:00, 1812.83 examples/s]INFO 2025-05-20 11:12:01,001 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_12_5__Additional_resources_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1562 examples [00:00, 67157.71 examples/s]INFO 2025-05-20 11:12:01,057 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:12:01,057 instructlab.sdg.datamixing:161: Dataset loaded with 1562 samples
INFO 2025-05-20 11:12:01,057 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 593/11824 [00:01<00:32, 345.86 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:01<00:10, 995.01 examples/s]
Map (num_proc=8):  18%|#7        | 2071/11824 [00:02<00:13, 732.46 examples/s]
Map (num_proc=8):  25%|##5       | 2956/11824 [00:03<00:07, 1257.02 examples/s]
Map (num_proc=8):  30%|##9       | 3542/11824 [00:04<00:09, 881.47 examples/s] 
Map (num_proc=8):  33%|###3      | 3956/11824 [00:04<00:07, 1076.91 examples/s]
Map (num_proc=8):  42%|####2     | 5016/11824 [00:05<00:06, 1012.81 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:05<00:03, 1478.72 examples/s]
Map (num_proc=8):  55%|#####4    | 6500/11824 [00:06<00:05, 1014.43 examples/s]
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:06<00:04, 1191.25 examples/s]
Map (num_proc=8):  67%|######7   | 7968/11824 [00:07<00:03, 1070.94 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:07<00:01, 1523.98 examples/s]
Map (num_proc=8):  80%|#######9  | 9445/11824 [00:09<00:02, 1040.11 examples/s]
Map (num_proc=8):  83%|########3 | 9868/11824 [00:09<00:01, 1221.05 examples/s]
Map (num_proc=8):  93%|#########2| 10943/11824 [00:10<00:00, 1147.86 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1607.75 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1118.47 examples/s]INFO 2025-05-20 11:12:13,084 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_12_4_1__Using_skip_node_annotation_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 282 examples [00:00, 71077.08 examples/s]INFO 2025-05-20 11:12:13,116 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:12:13,116 instructlab.sdg.datamixing:161: Dataset loaded with 282 samples
INFO 2025-05-20 11:12:13,116 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 761/11824 [00:00<00:13, 839.19 examples/s]
Map (num_proc=8):  19%|#8        | 2240/11824 [00:01<00:05, 1669.71 examples/s]
Map (num_proc=8):  31%|###1      | 3720/11824 [00:02<00:04, 1991.23 examples/s]
Map (num_proc=8):  44%|####3     | 5199/11824 [00:02<00:02, 2217.30 examples/s]
Map (num_proc=8):  56%|#####6    | 6679/11824 [00:03<00:02, 2300.75 examples/s]
Map (num_proc=8):  69%|######9   | 8159/11824 [00:03<00:01, 2407.79 examples/s]
Map (num_proc=8):  82%|########1 | 9641/11824 [00:04<00:00, 2419.85 examples/s]
Map (num_proc=8):  94%|#########4| 11147/11824 [00:04<00:00, 2582.52 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:05<00:00, 2291.95 examples/s]INFO 2025-05-20 11:12:19,346 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_6__Monitoring_virtual_machine_health_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 3184 examples [00:00, 84644.80 examples/s]INFO 2025-05-20 11:12:19,419 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:12:19,419 instructlab.sdg.datamixing:161: Dataset loaded with 3184 samples
INFO 2025-05-20 11:12:19,419 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 687/11824 [00:01<00:21, 517.84 examples/s]
Map (num_proc=8):  18%|#8        | 2165/11824 [00:02<00:08, 1073.53 examples/s]
Map (num_proc=8):  31%|###       | 3635/11824 [00:03<00:06, 1310.61 examples/s]
Map (num_proc=8):  43%|####3     | 5116/11824 [00:04<00:04, 1441.23 examples/s]
Map (num_proc=8):  56%|#####5    | 6593/11824 [00:04<00:03, 1514.41 examples/s]
Map (num_proc=8):  68%|######8   | 8078/11824 [00:05<00:02, 1563.78 examples/s]
Map (num_proc=8):  81%|########  | 9554/11824 [00:06<00:01, 1590.64 examples/s]
Map (num_proc=8):  93%|#########3| 11054/11824 [00:07<00:00, 1662.23 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1513.64 examples/s]INFO 2025-05-20 11:12:28,589 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_1_2__About_virtual_machines_and_boot_sources_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 763 examples [00:00, 55406.06 examples/s]INFO 2025-05-20 11:12:28,628 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:12:28,628 instructlab.sdg.datamixing:161: Dataset loaded with 763 samples
INFO 2025-05-20 11:12:28,628 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|4         | 587/11824 [00:02<00:40, 278.64 examples/s]
Map (num_proc=8):   8%|8         | 1000/11824 [00:02<00:20, 521.46 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:02<00:12, 841.29 examples/s]
Map (num_proc=8):  18%|#7        | 2087/11824 [00:03<00:15, 619.61 examples/s]
Map (num_proc=8):  21%|##        | 2478/11824 [00:03<00:11, 819.39 examples/s]
Map (num_proc=8):  25%|##5       | 2956/11824 [00:03<00:08, 1097.82 examples/s]
Map (num_proc=8):  30%|###       | 3554/11824 [00:05<00:11, 724.22 examples/s] 
Map (num_proc=8):  33%|###3      | 3956/11824 [00:05<00:08, 922.58 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:05<00:06, 1186.68 examples/s]
Map (num_proc=8):  43%|####2     | 5037/11824 [00:06<00:08, 762.11 examples/s] 
Map (num_proc=8):  50%|#####     | 5912/11824 [00:07<00:05, 1143.25 examples/s]
Map (num_proc=8):  55%|#####5    | 6536/11824 [00:08<00:06, 806.85 examples/s] 
Map (num_proc=8):  62%|######2   | 7390/11824 [00:08<00:03, 1124.62 examples/s]
Map (num_proc=8):  68%|######7   | 8012/11824 [00:09<00:04, 818.39 examples/s] 
Map (num_proc=8):  71%|#######   | 8390/11824 [00:09<00:03, 962.67 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:10<00:02, 1178.46 examples/s]
Map (num_proc=8):  80%|########  | 9493/11824 [00:11<00:02, 801.17 examples/s] 
Map (num_proc=8):  88%|########7 | 10346/11824 [00:11<00:01, 1139.29 examples/s]
Map (num_proc=8):  93%|#########2| 10987/11824 [00:12<00:00, 861.37 examples/s] 
Map (num_proc=8): 100%|##########| 11824/11824 [00:13<00:00, 895.97 examples/s]INFO 2025-05-20 11:12:43,470 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_2__Viewing_OpenShift_Virtualization_logs_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 561 examples [00:00, 54475.26 examples/s]INFO 2025-05-20 11:12:43,512 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:12:43,512 instructlab.sdg.datamixing:161: Dataset loaded with 561 samples
INFO 2025-05-20 11:12:43,512 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 613/11824 [00:02<00:37, 302.59 examples/s]
Map (num_proc=8):   8%|8         | 1000/11824 [00:02<00:20, 536.65 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:02<00:12, 855.57 examples/s]
Map (num_proc=8):  18%|#7        | 2097/11824 [00:03<00:15, 645.19 examples/s]
Map (num_proc=8):  25%|##5       | 2956/11824 [00:03<00:08, 1040.98 examples/s]
Map (num_proc=8):  30%|###       | 3557/11824 [00:05<00:10, 760.30 examples/s] 
Map (num_proc=8):  33%|###3      | 3956/11824 [00:05<00:08, 936.20 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:05<00:06, 1164.24 examples/s]
Map (num_proc=8):  43%|####2     | 5057/11824 [00:06<00:08, 797.84 examples/s] 
Map (num_proc=8):  46%|####5     | 5434/11824 [00:06<00:06, 974.04 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:06<00:04, 1213.44 examples/s]
Map (num_proc=8):  55%|#####5    | 6514/11824 [00:08<00:06, 796.46 examples/s] 
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:08<00:04, 990.15 examples/s]
Map (num_proc=8):  62%|######2   | 7390/11824 [00:08<00:03, 1235.01 examples/s]
Map (num_proc=8):  68%|######7   | 7984/11824 [00:09<00:04, 796.44 examples/s] 
Map (num_proc=8):  71%|#######   | 8390/11824 [00:09<00:03, 996.82 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:09<00:02, 1243.22 examples/s]
Map (num_proc=8):  80%|########  | 9479/11824 [00:11<00:02, 808.75 examples/s] 
Map (num_proc=8):  83%|########3 | 9868/11824 [00:11<00:02, 911.86 examples/s]
Map (num_proc=8):  93%|#########2| 10961/11824 [00:12<00:00, 930.89 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:12<00:00, 1363.78 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:12<00:00, 920.61 examples/s] INFO 2025-05-20 11:12:57,930 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_8__About_the_virt_operator_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 778 examples [00:00, 77511.78 examples/s]INFO 2025-05-20 11:12:57,973 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:12:57,973 instructlab.sdg.datamixing:161: Dataset loaded with 778 samples
INFO 2025-05-20 11:12:57,974 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 703/11824 [00:01<00:20, 542.05 examples/s]
Map (num_proc=8):  18%|#8        | 2177/11824 [00:02<00:08, 1095.90 examples/s]
Map (num_proc=8):  31%|###       | 3646/11824 [00:03<00:06, 1331.99 examples/s]
Map (num_proc=8):  43%|####3     | 5095/11824 [00:03<00:04, 1448.46 examples/s]
Map (num_proc=8):  55%|#####5    | 6562/11824 [00:04<00:03, 1523.25 examples/s]
Map (num_proc=8):  68%|######7   | 8029/11824 [00:05<00:02, 1570.03 examples/s]
Map (num_proc=8):  80%|########  | 9500/11824 [00:06<00:01, 1601.69 examples/s]
Map (num_proc=8):  93%|#########2| 10992/11824 [00:07<00:00, 1687.67 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1541.04 examples/s]INFO 2025-05-20 11:13:06,967 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_14_4__Adding_VirtIO_drivers_container_disk_to_a_virtual_machine_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2799 examples [00:00, 75869.75 examples/s]INFO 2025-05-20 11:13:07,046 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:13:07,046 instructlab.sdg.datamixing:161: Dataset loaded with 2799 samples
INFO 2025-05-20 11:13:07,046 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 660/11824 [00:01<00:24, 457.40 examples/s]
Map (num_proc=8):   9%|8         | 1063/11824 [00:01<00:14, 730.20 examples/s]
Map (num_proc=8):  18%|#8        | 2145/11824 [00:02<00:09, 1018.71 examples/s]
Map (num_proc=8):  22%|##1       | 2543/11824 [00:02<00:07, 1184.91 examples/s]
Map (num_proc=8):  31%|###       | 3623/11824 [00:03<00:06, 1243.80 examples/s]
Map (num_proc=8):  34%|###3      | 4000/11824 [00:03<00:05, 1356.08 examples/s]
Map (num_proc=8):  43%|####3     | 5096/11824 [00:04<00:05, 1342.44 examples/s]
Map (num_proc=8):  46%|####6     | 5466/11824 [00:04<00:04, 1434.72 examples/s]
Map (num_proc=8):  56%|#####5    | 6576/11824 [00:05<00:03, 1391.65 examples/s]
Map (num_proc=8):  59%|#####8    | 6924/11824 [00:05<00:03, 1458.79 examples/s]
Map (num_proc=8):  68%|######8   | 8073/11824 [00:06<00:02, 1428.23 examples/s]
Map (num_proc=8):  71%|#######1  | 8431/11824 [00:06<00:02, 1502.82 examples/s]
Map (num_proc=8):  81%|########  | 9557/11824 [00:07<00:01, 1437.66 examples/s]
Map (num_proc=8):  84%|########3 | 9899/11824 [00:07<00:01, 1500.02 examples/s]
Map (num_proc=8):  93%|#########3| 11037/11824 [00:08<00:00, 1504.52 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1357.12 examples/s]INFO 2025-05-20 11:13:17,187 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_5__Managing_virtual_machine_instances_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 3023 examples [00:00, 104276.37 examples/s]INFO 2025-05-20 11:13:17,258 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:13:17,258 instructlab.sdg.datamixing:161: Dataset loaded with 3023 samples
INFO 2025-05-20 11:13:17,258 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 725/11824 [00:01<00:16, 683.19 examples/s]
Map (num_proc=8):  19%|#8        | 2201/11824 [00:01<00:07, 1334.19 examples/s]
Map (num_proc=8):  31%|###1      | 3682/11824 [00:02<00:05, 1615.52 examples/s]
Map (num_proc=8):  44%|####3     | 5165/11824 [00:03<00:03, 1764.56 examples/s]
Map (num_proc=8):  56%|#####6    | 6643/11824 [00:04<00:02, 1847.48 examples/s]
Map (num_proc=8):  69%|######8   | 8119/11824 [00:04<00:01, 1909.97 examples/s]
Map (num_proc=8):  81%|########1 | 9597/11824 [00:05<00:01, 1923.51 examples/s]
Map (num_proc=8):  94%|#########3| 11113/11824 [00:06<00:00, 1996.40 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:06<00:00, 1831.10 examples/s]INFO 2025-05-20 11:13:24,899 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_11_1__Prerequisites_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 4757 examples [00:00, 97986.98 examples/s]INFO 2025-05-20 11:13:24,986 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:13:24,986 instructlab.sdg.datamixing:161: Dataset loaded with 4757 samples
INFO 2025-05-20 11:13:24,986 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 710/11824 [00:01<00:21, 512.20 examples/s]
Map (num_proc=8):  19%|#8        | 2188/11824 [00:02<00:09, 1009.14 examples/s]
Map (num_proc=8):  31%|###       | 3664/11824 [00:03<00:06, 1218.09 examples/s]
Map (num_proc=8):  43%|####3     | 5143/11824 [00:04<00:05, 1328.72 examples/s]
Map (num_proc=8):  56%|#####5    | 6620/11824 [00:05<00:03, 1393.01 examples/s]
Map (num_proc=8):  69%|######8   | 8106/11824 [00:06<00:02, 1433.09 examples/s]
Map (num_proc=8):  81%|########  | 9574/11824 [00:07<00:01, 1455.52 examples/s]
Map (num_proc=8):  94%|#########3| 11090/11824 [00:08<00:00, 1516.92 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1396.97 examples/s]INFO 2025-05-20 11:13:34,789 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_1_3__Object_maximums_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 4547 examples [00:00, 79013.22 examples/s]INFO 2025-05-20 11:13:34,895 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:13:34,895 instructlab.sdg.datamixing:161: Dataset loaded with 4547 samples
INFO 2025-05-20 11:13:34,895 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 676/11824 [00:01<00:23, 468.33 examples/s]
Map (num_proc=8):  18%|#7        | 2071/11824 [00:02<00:10, 918.17 examples/s]
Map (num_proc=8):  30%|###       | 3570/11824 [00:03<00:07, 1163.47 examples/s]
Map (num_proc=8):  43%|####2     | 5063/11824 [00:04<00:05, 1286.51 examples/s]
Map (num_proc=8):  55%|#####5    | 6532/11824 [00:05<00:03, 1348.97 examples/s]
Map (num_proc=8):  68%|######7   | 8023/11824 [00:06<00:02, 1394.08 examples/s]
Map (num_proc=8):  80%|########  | 9502/11824 [00:07<00:01, 1420.17 examples/s]
Map (num_proc=8):  93%|#########2| 10992/11824 [00:08<00:00, 1479.34 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1358.61 examples/s]INFO 2025-05-20 11:13:45,032 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_14_5__Installing_VirtIO_drivers_during_Windows_installation_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2978 examples [00:00, 81276.39 examples/s]INFO 2025-05-20 11:13:45,107 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:13:45,107 instructlab.sdg.datamixing:161: Dataset loaded with 2978 samples
INFO 2025-05-20 11:13:45,107 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 684/11824 [00:01<00:24, 448.74 examples/s]
Map (num_proc=8):  18%|#8        | 2160/11824 [00:02<00:10, 909.12 examples/s]
Map (num_proc=8):  31%|###       | 3638/11824 [00:03<00:07, 1110.27 examples/s]
Map (num_proc=8):  43%|####3     | 5113/11824 [00:04<00:05, 1200.43 examples/s]
Map (num_proc=8):  56%|#####5    | 6595/11824 [00:05<00:04, 1269.79 examples/s]
Map (num_proc=8):  68%|######8   | 8096/11824 [00:06<00:02, 1311.84 examples/s]
Map (num_proc=8):  81%|########  | 9571/11824 [00:07<00:01, 1321.89 examples/s]
Map (num_proc=8):  94%|#########3| 11072/11824 [00:08<00:00, 1378.06 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:09<00:00, 1271.96 examples/s]INFO 2025-05-20 11:13:55,821 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_16_12__Configuring_a_watchdog_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 12288 examples [00:00, 90828.61 examples/s]
Generating train split: 15129 examples [00:00, 97487.11 examples/s]INFO 2025-05-20 11:13:56,010 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:13:56,010 instructlab.sdg.datamixing:161: Dataset loaded with 15129 samples


Map (num_proc=8):   0%|          | 0/15129 [00:00<?, ? examples/s]
Map (num_proc=8):   1%|1         | 161/15129 [00:00<00:19, 754.69 examples/s]
Map (num_proc=8):  12%|#2        | 1827/15129 [00:00<00:01, 7122.37 examples/s]
Map (num_proc=8):  25%|##4       | 3779/15129 [00:00<00:00, 11429.33 examples/s]
Map (num_proc=8):  35%|###4      | 5223/15129 [00:00<00:00, 12401.81 examples/s]
Map (num_proc=8):  46%|####5     | 6903/15129 [00:00<00:00, 13318.81 examples/s]
Map (num_proc=8):  56%|#####5    | 8424/15129 [00:00<00:00, 10024.78 examples/s]
Map (num_proc=8):  64%|######3   | 9639/15129 [00:00<00:00, 10293.01 examples/s]
Map (num_proc=8):  76%|#######5  | 11494/15129 [00:01<00:00, 11837.72 examples/s]
Map (num_proc=8):  88%|########8 | 13386/15129 [00:01<00:00, 13375.30 examples/s]
Map (num_proc=8):  98%|#########8| 14854/15129 [00:01<00:00, 12961.11 examples/s]
Map (num_proc=8): 100%|##########| 15129/15129 [00:01<00:00, 10071.04 examples/s]INFO 2025-05-20 11:13:58,283 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_4__About_the_cluster_network_addons_operator_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 525 examples [00:00, 53074.54 examples/s]INFO 2025-05-20 11:13:58,342 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:13:58,342 instructlab.sdg.datamixing:161: Dataset loaded with 525 samples
INFO 2025-05-20 11:13:58,342 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 651/11824 [00:01<00:28, 394.90 examples/s]
Map (num_proc=8):   9%|8         | 1027/11824 [00:01<00:17, 625.87 examples/s]
Map (num_proc=8):  18%|#7        | 2080/11824 [00:02<00:11, 833.50 examples/s]
Map (num_proc=8):  21%|##        | 2478/11824 [00:02<00:09, 1018.09 examples/s]
Map (num_proc=8):  30%|##9       | 3547/11824 [00:04<00:08, 1012.76 examples/s]
Map (num_proc=8):  33%|###3      | 3956/11824 [00:04<00:06, 1152.86 examples/s]
Map (num_proc=8):  43%|####2     | 5053/11824 [00:05<00:06, 1112.89 examples/s]
Map (num_proc=8):  46%|####5     | 5438/11824 [00:05<00:05, 1246.64 examples/s]
Map (num_proc=8):  55%|#####5    | 6527/11824 [00:06<00:04, 1141.65 examples/s]
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:06<00:03, 1248.06 examples/s]
Map (num_proc=8):  68%|######7   | 8005/11824 [00:07<00:03, 1161.19 examples/s]
Map (num_proc=8):  71%|#######1  | 8425/11824 [00:07<00:02, 1309.47 examples/s]
Map (num_proc=8):  80%|#######9  | 9451/11824 [00:08<00:02, 1138.45 examples/s]
Map (num_proc=8):  83%|########3 | 9869/11824 [00:09<00:01, 1276.76 examples/s]
Map (num_proc=8):  93%|#########2| 10994/11824 [00:09<00:00, 1238.40 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1146.79 examples/s]INFO 2025-05-20 11:14:10,082 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_15_1__About_vTPM_devices_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1053 examples [00:00, 100438.95 examples/s]INFO 2025-05-20 11:14:10,126 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:14:10,126 instructlab.sdg.datamixing:161: Dataset loaded with 1053 samples
INFO 2025-05-20 11:14:10,126 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 720/11824 [00:01<00:17, 644.89 examples/s]
Map (num_proc=8):  10%|#         | 1219/11824 [00:01<00:09, 1105.17 examples/s]
Map (num_proc=8):  19%|#8        | 2191/11824 [00:01<00:07, 1301.05 examples/s]
Map (num_proc=8):  23%|##2       | 2704/11824 [00:02<00:05, 1641.90 examples/s]
Map (num_proc=8):  31%|###1      | 3681/11824 [00:02<00:05, 1581.04 examples/s]
Map (num_proc=8):  35%|###5      | 4196/11824 [00:02<00:04, 1857.00 examples/s]
Map (num_proc=8):  44%|####3     | 5156/11824 [00:03<00:03, 1699.23 examples/s]
Map (num_proc=8):  48%|####8     | 5682/11824 [00:03<00:03, 1981.15 examples/s]
Map (num_proc=8):  56%|#####6    | 6635/11824 [00:04<00:02, 1749.86 examples/s]
Map (num_proc=8):  60%|######    | 7135/11824 [00:04<00:02, 1998.67 examples/s]
Map (num_proc=8):  69%|######8   | 8120/11824 [00:04<00:02, 1782.55 examples/s]
Map (num_proc=8):  73%|#######2  | 8611/11824 [00:05<00:01, 2018.99 examples/s]
Map (num_proc=8):  81%|########1 | 9595/11824 [00:05<00:01, 1793.61 examples/s]
Map (num_proc=8):  85%|########5 | 10075/11824 [00:05<00:00, 2015.79 examples/s]
Map (num_proc=8):  94%|#########3| 11092/11824 [00:06<00:00, 1904.12 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:06<00:00, 1757.11 examples/s]INFO 2025-05-20 11:14:18,085 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_12_4__Preventing_node_reconciliation_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 154 examples [00:00, 48756.25 examples/s]INFO 2025-05-20 11:14:18,120 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:14:18,120 instructlab.sdg.datamixing:161: Dataset loaded with 154 samples
INFO 2025-05-20 11:14:18,120 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   7%|6         | 773/11824 [00:00<00:10, 1026.24 examples/s]
Map (num_proc=8):  19%|#8        | 2204/11824 [00:01<00:05, 1852.09 examples/s]
Map (num_proc=8):  31%|###1      | 3691/11824 [00:01<00:03, 2286.93 examples/s]
Map (num_proc=8):  44%|####3     | 5156/11824 [00:02<00:02, 2519.50 examples/s]
Map (num_proc=8):  56%|#####6    | 6635/11824 [00:02<00:01, 2659.97 examples/s]
Map (num_proc=8):  69%|######8   | 8118/11824 [00:03<00:01, 2748.75 examples/s]
Map (num_proc=8):  81%|########1 | 9596/11824 [00:03<00:00, 2799.38 examples/s]
Map (num_proc=8):  94%|#########3| 11105/11824 [00:04<00:00, 2938.23 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:04<00:00, 2623.56 examples/s]INFO 2025-05-20 11:14:23,725 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_19__Virtual_machine_networking_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 8832 examples [00:00, 72577.98 examples/s]
Generating train split: 15131 examples [00:00, 89391.47 examples/s]INFO 2025-05-20 11:14:23,912 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:14:23,913 instructlab.sdg.datamixing:161: Dataset loaded with 15131 samples


Map (num_proc=8):   0%|          | 0/15131 [00:00<?, ? examples/s]
Map (num_proc=8):   1%|1         | 155/15131 [00:00<00:20, 713.38 examples/s]
Map (num_proc=8):  12%|#1        | 1795/15131 [00:00<00:01, 6846.54 examples/s]
Map (num_proc=8):  26%|##6       | 3961/15131 [00:00<00:00, 11862.33 examples/s]
Map (num_proc=8):  37%|###6      | 5583/15131 [00:00<00:00, 12805.18 examples/s]
Map (num_proc=8):  47%|####6     | 7111/15131 [00:00<00:00, 12811.19 examples/s]
Map (num_proc=8):  57%|#####6    | 8557/15131 [00:00<00:00, 10372.00 examples/s]
Map (num_proc=8):  66%|######5   | 9921/15131 [00:00<00:00, 11102.37 examples/s]
Map (num_proc=8):  76%|#######6  | 11542/15131 [00:01<00:00, 12246.79 examples/s]
Map (num_proc=8):  88%|########8 | 13338/15131 [00:01<00:00, 13316.95 examples/s]
Map (num_proc=8):  98%|#########7| 14765/15131 [00:01<00:00, 12463.72 examples/s]
Map (num_proc=8): 100%|##########| 15131/15131 [00:01<00:00, 10166.10 examples/s]INFO 2025-05-20 11:14:26,176 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_1_1_1__OpenShift_Virtualization_supported_cluster_version_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 415 examples [00:00, 41594.25 examples/s]INFO 2025-05-20 11:14:26,204 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:14:26,204 instructlab.sdg.datamixing:161: Dataset loaded with 415 samples
INFO 2025-05-20 11:14:26,204 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 628/11824 [00:01<00:35, 319.65 examples/s]
Map (num_proc=8):   8%|8         | 1000/11824 [00:02<00:19, 549.73 examples/s]
Map (num_proc=8):  18%|#7        | 2114/11824 [00:03<00:13, 739.21 examples/s]
Map (num_proc=8):  21%|##        | 2478/11824 [00:03<00:10, 909.94 examples/s]
Map (num_proc=8):  30%|###       | 3599/11824 [00:04<00:09, 908.64 examples/s]
Map (num_proc=8):  33%|###3      | 3956/11824 [00:04<00:07, 1056.21 examples/s]
Map (num_proc=8):  43%|####2     | 5074/11824 [00:05<00:06, 979.62 examples/s] 
Map (num_proc=8):  50%|#####     | 5912/11824 [00:06<00:04, 1380.02 examples/s]
Map (num_proc=8):  55%|#####5    | 6552/11824 [00:07<00:05, 958.16 examples/s] 
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:07<00:04, 1098.33 examples/s]
Map (num_proc=8):  68%|######8   | 8042/11824 [00:08<00:03, 1008.17 examples/s]
Map (num_proc=8):  71%|#######   | 8390/11824 [00:08<00:03, 1138.96 examples/s]
Map (num_proc=8):  80%|########  | 9507/11824 [00:10<00:02, 1024.48 examples/s]
Map (num_proc=8):  83%|########3 | 9868/11824 [00:10<00:01, 1160.83 examples/s]
Map (num_proc=8):  93%|#########3| 11018/11824 [00:11<00:00, 1098.56 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:11<00:00, 1018.73 examples/s]INFO 2025-05-20 11:14:39,501 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_4_1__Creating_a_custom_namespace_for_templates_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 454 examples [00:00, 62420.97 examples/s]INFO 2025-05-20 11:14:39,533 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:14:39,534 instructlab.sdg.datamixing:161: Dataset loaded with 454 samples
INFO 2025-05-20 11:14:39,534 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 677/11824 [00:01<00:24, 455.64 examples/s]
Map (num_proc=8):  18%|#8        | 2150/11824 [00:02<00:10, 928.61 examples/s]
Map (num_proc=8):  30%|###       | 3589/11824 [00:03<00:07, 1110.99 examples/s]
Map (num_proc=8):  43%|####2     | 5063/11824 [00:04<00:05, 1219.69 examples/s]
Map (num_proc=8):  55%|#####5    | 6531/11824 [00:05<00:04, 1279.60 examples/s]
Map (num_proc=8):  68%|######7   | 7997/11824 [00:06<00:02, 1314.83 examples/s]
Map (num_proc=8):  80%|########  | 9474/11824 [00:07<00:01, 1342.66 examples/s]
Map (num_proc=8):  93%|#########2| 10980/11824 [00:08<00:00, 1407.17 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:09<00:00, 1300.64 examples/s]INFO 2025-05-20 11:14:50,030 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_5__Technology_Preview_features_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 221 examples [00:00, 45027.75 examples/s]INFO 2025-05-20 11:14:50,071 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:14:50,071 instructlab.sdg.datamixing:161: Dataset loaded with 221 samples
INFO 2025-05-20 11:14:50,071 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 669/11824 [00:01<00:24, 450.00 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:01<00:10, 1024.14 examples/s]
Map (num_proc=8):  18%|#7        | 2092/11824 [00:02<00:10, 885.00 examples/s] 
Map (num_proc=8):  21%|##1       | 2497/11824 [00:02<00:08, 1060.03 examples/s]
Map (num_proc=8):  30%|###       | 3573/11824 [00:03<00:07, 1166.75 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:03<00:04, 1587.85 examples/s]
Map (num_proc=8):  43%|####2     | 5058/11824 [00:04<00:05, 1214.15 examples/s]
Map (num_proc=8):  46%|####5     | 5435/11824 [00:04<00:04, 1314.32 examples/s]
Map (num_proc=8):  55%|#####5    | 6537/11824 [00:05<00:04, 1314.78 examples/s]
Map (num_proc=8):  62%|######2   | 7390/11824 [00:05<00:02, 1694.48 examples/s]
Map (num_proc=8):  68%|######7   | 8010/11824 [00:06<00:02, 1285.35 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:06<00:01, 1669.46 examples/s]
Map (num_proc=8):  80%|########  | 9503/11824 [00:07<00:01, 1279.15 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:07<00:00, 1653.50 examples/s]
Map (num_proc=8):  93%|#########3| 10999/11824 [00:08<00:00, 1334.99 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1322.52 examples/s]INFO 2025-05-20 11:15:00,481 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_10_1__Installing_QEMU_guest_agent_on_a_Linux_virtual_machine_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2155 examples [00:00, 65097.52 examples/s]INFO 2025-05-20 11:15:00,548 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:15:00,549 instructlab.sdg.datamixing:161: Dataset loaded with 2155 samples
INFO 2025-05-20 11:15:00,549 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 661/11824 [00:01<00:29, 382.47 examples/s]
Map (num_proc=8):  18%|#8        | 2145/11824 [00:02<00:12, 796.47 examples/s]
Map (num_proc=8):  25%|##5       | 2956/11824 [00:03<00:07, 1137.22 examples/s]
Map (num_proc=8):  31%|###       | 3617/11824 [00:04<00:08, 914.55 examples/s] 
Map (num_proc=8):  38%|###7      | 4434/11824 [00:04<00:05, 1244.74 examples/s]
Map (num_proc=8):  43%|####3     | 5096/11824 [00:05<00:06, 977.15 examples/s] 
Map (num_proc=8):  50%|#####     | 5912/11824 [00:05<00:04, 1301.88 examples/s]
Map (num_proc=8):  56%|#####5    | 6577/11824 [00:06<00:05, 1008.55 examples/s]
Map (num_proc=8):  62%|######2   | 7390/11824 [00:06<00:03, 1329.68 examples/s]
Map (num_proc=8):  68%|######8   | 8055/11824 [00:07<00:03, 1021.91 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:08<00:02, 1342.24 examples/s]
Map (num_proc=8):  81%|########  | 9545/11824 [00:09<00:02, 1034.30 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:09<00:01, 1346.73 examples/s]
Map (num_proc=8):  93%|#########3| 11044/11824 [00:10<00:00, 1088.83 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1103.82 examples/s]INFO 2025-05-20 11:15:12,797 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_9_2__Deleting_nodes_from_a_bare_metal_cluster_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2063 examples [00:00, 109220.05 examples/s]INFO 2025-05-20 11:15:12,854 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:15:12,854 instructlab.sdg.datamixing:161: Dataset loaded with 2063 samples
INFO 2025-05-20 11:15:12,854 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 698/11824 [00:01<00:19, 570.30 examples/s]
Map (num_proc=8):  18%|#8        | 2167/11824 [00:02<00:08, 1184.35 examples/s]
Map (num_proc=8):  31%|###       | 3647/11824 [00:02<00:05, 1462.99 examples/s]
Map (num_proc=8):  43%|####2     | 5083/11824 [00:03<00:04, 1591.19 examples/s]
Map (num_proc=8):  56%|#####5    | 6585/11824 [00:04<00:03, 1694.07 examples/s]
Map (num_proc=8):  68%|######8   | 8056/11824 [00:05<00:02, 1745.28 examples/s]
Map (num_proc=8):  81%|########  | 9556/11824 [00:05<00:01, 1791.13 examples/s]
Map (num_proc=8):  94%|#########3| 11073/11824 [00:06<00:00, 1889.35 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1687.81 examples/s]INFO 2025-05-20 11:15:21,301 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_11_2__Live_migration_limits_and_timeouts_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1106 examples [00:00, 103699.65 examples/s]INFO 2025-05-20 11:15:21,348 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:15:21,348 instructlab.sdg.datamixing:161: Dataset loaded with 1106 samples
INFO 2025-05-20 11:15:21,348 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 608/11824 [00:01<00:19, 572.46 examples/s]
Map (num_proc=8):   9%|9         | 1086/11824 [00:01<00:10, 1041.28 examples/s]
Map (num_proc=8):  19%|#8        | 2196/11824 [00:01<00:06, 1422.48 examples/s]
Map (num_proc=8):  23%|##2       | 2670/11824 [00:01<00:05, 1709.63 examples/s]
Map (num_proc=8):  31%|###1      | 3680/11824 [00:02<00:04, 1717.83 examples/s]
Map (num_proc=8):  35%|###5      | 4187/11824 [00:02<00:03, 2013.19 examples/s]
Map (num_proc=8):  44%|####3     | 5151/11824 [00:03<00:03, 1831.50 examples/s]
Map (num_proc=8):  48%|####8     | 5680/11824 [00:03<00:02, 2103.62 examples/s]
Map (num_proc=8):  56%|#####6    | 6638/11824 [00:03<00:02, 1916.62 examples/s]
Map (num_proc=8):  61%|######    | 7163/11824 [00:04<00:02, 2141.57 examples/s]
Map (num_proc=8):  69%|######8   | 8110/11824 [00:04<00:01, 1910.65 examples/s]
Map (num_proc=8):  72%|#######2  | 8561/11824 [00:04<00:01, 2128.56 examples/s]
Map (num_proc=8):  81%|########1 | 9589/11824 [00:05<00:01, 1957.82 examples/s]
Map (num_proc=8):  85%|########4 | 10050/11824 [00:05<00:00, 2117.25 examples/s]
Map (num_proc=8):  94%|#########3| 11098/11824 [00:06<00:00, 2039.28 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:06<00:00, 1858.88 examples/s]INFO 2025-05-20 11:15:29,008 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1_7__Additional_resources_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 7187 examples [00:00, 80124.99 examples/s]INFO 2025-05-20 11:15:29,134 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:15:29,134 instructlab.sdg.datamixing:161: Dataset loaded with 7187 samples
INFO 2025-05-20 11:15:29,134 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 672/11824 [00:01<00:24, 460.73 examples/s]
Map (num_proc=8):   9%|9         | 1081/11824 [00:01<00:13, 768.27 examples/s]
Map (num_proc=8):  18%|#7        | 2126/11824 [00:02<00:09, 998.64 examples/s]
Map (num_proc=8):  21%|##1       | 2512/11824 [00:02<00:07, 1171.08 examples/s]
Map (num_proc=8):  31%|###       | 3609/11824 [00:03<00:06, 1261.06 examples/s]
Map (num_proc=8):  34%|###3      | 3972/11824 [00:03<00:05, 1375.24 examples/s]
Map (num_proc=8):  43%|####2     | 5080/11824 [00:04<00:04, 1352.37 examples/s]
Map (num_proc=8):  46%|####6     | 5452/11824 [00:04<00:04, 1463.57 examples/s]
Map (num_proc=8):  55%|#####5    | 6554/11824 [00:05<00:03, 1416.33 examples/s]
Map (num_proc=8):  58%|#####8    | 6913/11824 [00:05<00:03, 1526.14 examples/s]
Map (num_proc=8):  68%|######7   | 8025/11824 [00:06<00:02, 1438.10 examples/s]
Map (num_proc=8):  71%|#######   | 8393/11824 [00:06<00:02, 1554.97 examples/s]
Map (num_proc=8):  80%|########  | 9509/11824 [00:07<00:01, 1442.75 examples/s]
Map (num_proc=8):  83%|########3 | 9868/11824 [00:07<00:01, 1555.77 examples/s]
Map (num_proc=8):  93%|#########3| 11002/11824 [00:08<00:00, 1542.38 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1382.73 examples/s]INFO 2025-05-20 11:15:39,267 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_14_1_1__Installing_the_OADP_Operator_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1789 examples [00:00, 86847.34 examples/s]INFO 2025-05-20 11:15:39,317 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:15:39,318 instructlab.sdg.datamixing:161: Dataset loaded with 1789 samples
INFO 2025-05-20 11:15:39,318 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 668/11824 [00:01<00:24, 449.97 examples/s]
Map (num_proc=8):  18%|#8        | 2153/11824 [00:02<00:10, 940.69 examples/s]
Map (num_proc=8):  31%|###       | 3621/11824 [00:03<00:07, 1139.34 examples/s]
Map (num_proc=8):  43%|####3     | 5111/11824 [00:04<00:05, 1252.00 examples/s]
Map (num_proc=8):  56%|#####5    | 6576/11824 [00:05<00:04, 1308.77 examples/s]
Map (num_proc=8):  68%|######8   | 8062/11824 [00:06<00:02, 1351.81 examples/s]
Map (num_proc=8):  81%|########  | 9552/11824 [00:07<00:01, 1371.60 examples/s]
Map (num_proc=8):  93%|#########3| 11050/11824 [00:08<00:00, 1439.87 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1318.77 examples/s]INFO 2025-05-20 11:15:49,809 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_4_2__Subscribing_to_the_OpenShift_Virtualization_catalog_by_using_the_CLI_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2501 examples [00:00, 68350.49 examples/s]INFO 2025-05-20 11:15:49,884 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:15:49,884 instructlab.sdg.datamixing:161: Dataset loaded with 2501 samples
INFO 2025-05-20 11:15:49,884 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 658/11824 [00:01<00:29, 379.60 examples/s]
Map (num_proc=8):   9%|9         | 1078/11824 [00:01<00:16, 663.77 examples/s]
Map (num_proc=8):  18%|#7        | 2114/11824 [00:02<00:12, 800.17 examples/s]
Map (num_proc=8):  21%|##1       | 2531/11824 [00:03<00:09, 1010.16 examples/s]
Map (num_proc=8):  31%|###       | 3618/11824 [00:04<00:08, 978.65 examples/s] 
Map (num_proc=8):  34%|###3      | 4017/11824 [00:04<00:06, 1157.92 examples/s]
Map (num_proc=8):  43%|####3     | 5112/11824 [00:05<00:06, 1061.40 examples/s]
Map (num_proc=8):  46%|####6     | 5477/11824 [00:05<00:05, 1203.14 examples/s]
Map (num_proc=8):  56%|#####5    | 6597/11824 [00:06<00:04, 1092.10 examples/s]
Map (num_proc=8):  59%|#####8    | 6946/11824 [00:06<00:03, 1228.93 examples/s]
Map (num_proc=8):  68%|######8   | 8052/11824 [00:07<00:03, 1098.53 examples/s]
Map (num_proc=8):  71%|#######1  | 8404/11824 [00:08<00:02, 1227.35 examples/s]
Map (num_proc=8):  81%|########  | 9532/11824 [00:09<00:02, 1112.07 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:09<00:00, 1497.81 examples/s]
Map (num_proc=8):  93%|#########3| 11030/11824 [00:10<00:00, 1116.03 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1099.06 examples/s]INFO 2025-05-20 11:16:02,251 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_3__New_and_changed_features_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 821 examples [00:00, 117642.84 examples/s]INFO 2025-05-20 11:16:02,294 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:16:02,295 instructlab.sdg.datamixing:161: Dataset loaded with 821 samples
INFO 2025-05-20 11:16:02,295 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   7%|6         | 771/11824 [00:00<00:13, 819.56 examples/s]
Map (num_proc=8):  19%|#8        | 2242/11824 [00:01<00:06, 1578.86 examples/s]
Map (num_proc=8):  32%|###1      | 3739/11824 [00:02<00:04, 1931.05 examples/s]
Map (num_proc=8):  44%|####4     | 5207/11824 [00:02<00:03, 2106.32 examples/s]
Map (num_proc=8):  57%|#####6    | 6690/11824 [00:03<00:02, 2217.55 examples/s]
Map (num_proc=8):  69%|######9   | 8172/11824 [00:03<00:01, 2285.83 examples/s]
Map (num_proc=8):  82%|########1 | 9655/11824 [00:04<00:00, 2334.14 examples/s]
Map (num_proc=8):  94%|#########4| 11153/11824 [00:05<00:00, 2459.24 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:05<00:00, 2187.89 examples/s]INFO 2025-05-20 11:16:08,960 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_6_4__Unpausing_a_virtual_machine_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 500 examples [00:00, 76686.73 examples/s]INFO 2025-05-20 11:16:08,994 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:16:08,994 instructlab.sdg.datamixing:161: Dataset loaded with 500 samples
INFO 2025-05-20 11:16:08,994 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 705/11824 [00:01<00:20, 546.64 examples/s]
Map (num_proc=8):  19%|#8        | 2192/11824 [00:02<00:08, 1107.14 examples/s]
Map (num_proc=8):  31%|###1      | 3669/11824 [00:03<00:06, 1341.63 examples/s]
Map (num_proc=8):  43%|####3     | 5143/11824 [00:03<00:04, 1474.17 examples/s]
Map (num_proc=8):  56%|#####5    | 6619/11824 [00:04<00:03, 1533.02 examples/s]
Map (num_proc=8):  69%|######8   | 8104/11824 [00:05<00:02, 1576.60 examples/s]
Map (num_proc=8):  81%|########1 | 9591/11824 [00:06<00:01, 1621.98 examples/s]
Map (num_proc=8):  94%|#########3| 11095/11824 [00:07<00:00, 1688.06 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1538.56 examples/s]INFO 2025-05-20 11:16:18,102 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1_1__Using_a_Quick_Start_to_create_a_virtual_machine_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 7559 examples [00:00, 81021.44 examples/s]INFO 2025-05-20 11:16:18,227 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:16:18,227 instructlab.sdg.datamixing:161: Dataset loaded with 7559 samples
INFO 2025-05-20 11:16:18,227 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 639/11824 [00:01<00:29, 384.48 examples/s]
Map (num_proc=8):   9%|8         | 1056/11824 [00:01<00:16, 652.75 examples/s]
Map (num_proc=8):  18%|#7        | 2083/11824 [00:02<00:11, 834.87 examples/s]
Map (num_proc=8):  21%|##1       | 2523/11824 [00:03<00:08, 1036.02 examples/s]
Map (num_proc=8):  30%|###       | 3559/11824 [00:04<00:07, 1033.43 examples/s]
Map (num_proc=8):  33%|###3      | 3956/11824 [00:04<00:06, 1173.61 examples/s]
Map (num_proc=8):  43%|####2     | 5027/11824 [00:05<00:06, 1106.42 examples/s]
Map (num_proc=8):  46%|####5     | 5434/11824 [00:05<00:05, 1242.89 examples/s]
Map (num_proc=8):  55%|#####5    | 6523/11824 [00:06<00:04, 1159.10 examples/s]
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:06<00:03, 1278.66 examples/s]
Map (num_proc=8):  68%|######7   | 8000/11824 [00:07<00:03, 1186.59 examples/s]
Map (num_proc=8):  71%|#######   | 8393/11824 [00:07<00:02, 1292.20 examples/s]
Map (num_proc=8):  80%|########  | 9479/11824 [00:08<00:01, 1185.91 examples/s]
Map (num_proc=8):  83%|########3 | 9868/11824 [00:08<00:01, 1302.26 examples/s]
Map (num_proc=8):  93%|#########2| 10984/11824 [00:09<00:00, 1258.09 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1159.37 examples/s]INFO 2025-05-20 11:16:30,038 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_6__About_the_ssp_operator_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 471 examples [00:00, 100005.93 examples/s]INFO 2025-05-20 11:16:30,077 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:16:30,078 instructlab.sdg.datamixing:161: Dataset loaded with 471 samples
INFO 2025-05-20 11:16:30,078 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   7%|6         | 770/11824 [00:00<00:12, 875.40 examples/s]
Map (num_proc=8):  19%|#9        | 2250/11824 [00:01<00:05, 1656.37 examples/s]
Map (num_proc=8):  32%|###1      | 3727/11824 [00:02<00:04, 1984.27 examples/s]
Map (num_proc=8):  44%|####4     | 5210/11824 [00:02<00:03, 2167.71 examples/s]
Map (num_proc=8):  57%|#####6    | 6691/11824 [00:03<00:02, 2269.13 examples/s]
Map (num_proc=8):  69%|######9   | 8178/11824 [00:03<00:01, 2337.73 examples/s]
Map (num_proc=8):  82%|########1 | 9647/11824 [00:04<00:00, 2383.90 examples/s]
Map (num_proc=8):  94%|#########4| 11159/11824 [00:05<00:00, 2464.89 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:05<00:00, 2235.17 examples/s]INFO 2025-05-20 11:16:36,565 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_2_3__Editing_a_virtual_machine_YAML_configuration_using_the_CLI_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 3925 examples [00:00, 80189.79 examples/s]INFO 2025-05-20 11:16:36,652 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:16:36,652 instructlab.sdg.datamixing:161: Dataset loaded with 3925 samples
INFO 2025-05-20 11:16:36,653 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 698/11824 [00:01<00:23, 477.59 examples/s]
Map (num_proc=8):  18%|#8        | 2160/11824 [00:02<00:10, 958.24 examples/s]
Map (num_proc=8):  31%|###       | 3649/11824 [00:03<00:06, 1172.43 examples/s]
Map (num_proc=8):  43%|####3     | 5114/11824 [00:04<00:05, 1274.90 examples/s]
Map (num_proc=8):  56%|#####5    | 6600/11824 [00:05<00:03, 1340.11 examples/s]
Map (num_proc=8):  68%|######8   | 8081/11824 [00:06<00:02, 1378.92 examples/s]
Map (num_proc=8):  81%|########  | 9553/11824 [00:07<00:01, 1401.68 examples/s]
Map (num_proc=8):  93%|#########3| 11050/11824 [00:08<00:00, 1465.90 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1346.11 examples/s]INFO 2025-05-20 11:16:47,012 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_7_3_1__Viewing_the_SCC_and_RBAC_definitions_for_the_kubevirt_controller_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 871 examples [00:00, 85172.96 examples/s]INFO 2025-05-20 11:16:47,057 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:16:47,058 instructlab.sdg.datamixing:161: Dataset loaded with 871 samples
INFO 2025-05-20 11:16:47,058 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 680/11824 [00:01<00:20, 542.65 examples/s]
Map (num_proc=8):  18%|#8        | 2156/11824 [00:02<00:08, 1157.68 examples/s]
Map (num_proc=8):  31%|###       | 3640/11824 [00:02<00:05, 1440.64 examples/s]
Map (num_proc=8):  43%|####3     | 5110/11824 [00:03<00:04, 1581.76 examples/s]
Map (num_proc=8):  56%|#####5    | 6595/11824 [00:04<00:03, 1674.61 examples/s]
Map (num_proc=8):  68%|######8   | 8072/11824 [00:05<00:02, 1727.91 examples/s]
Map (num_proc=8):  81%|########  | 9553/11824 [00:06<00:01, 1766.45 examples/s]
Map (num_proc=8):  94%|#########3| 11072/11824 [00:06<00:00, 1872.02 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1666.54 examples/s]INFO 2025-05-20 11:16:55,682 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_2_1__OpenShift_Virtualization_supported_cluster_version_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 420 examples [00:00, 83870.10 examples/s]INFO 2025-05-20 11:16:55,720 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:16:55,720 instructlab.sdg.datamixing:161: Dataset loaded with 420 samples
INFO 2025-05-20 11:16:55,720 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   7%|6         | 773/11824 [00:00<00:12, 893.13 examples/s]
Map (num_proc=8):  19%|#8        | 2237/11824 [00:01<00:05, 1745.94 examples/s]
Map (num_proc=8):  31%|###1      | 3716/11824 [00:01<00:03, 2154.31 examples/s]
Map (num_proc=8):  44%|####3     | 5195/11824 [00:02<00:02, 2379.43 examples/s]
Map (num_proc=8):  56%|#####6    | 6667/11824 [00:03<00:02, 2511.83 examples/s]
Map (num_proc=8):  69%|######8   | 8147/11824 [00:03<00:01, 2602.34 examples/s]
Map (num_proc=8):  81%|########1 | 9626/11824 [00:04<00:00, 2653.61 examples/s]
Map (num_proc=8):  94%|#########4| 11127/11824 [00:04<00:00, 2787.10 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:04<00:00, 2450.36 examples/s]INFO 2025-05-20 11:17:01,873 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_12_3_3__Configuring_obsolete_CPU_models_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 388 examples [00:00, 65824.94 examples/s]INFO 2025-05-20 11:17:01,915 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:17:01,915 instructlab.sdg.datamixing:161: Dataset loaded with 388 samples
INFO 2025-05-20 11:17:01,915 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 717/11824 [00:01<00:16, 667.52 examples/s]
Map (num_proc=8):  19%|#8        | 2200/11824 [00:01<00:06, 1378.32 examples/s]
Map (num_proc=8):  31%|###1      | 3677/11824 [00:02<00:04, 1665.06 examples/s]
Map (num_proc=8):  44%|####3     | 5156/11824 [00:03<00:03, 1851.14 examples/s]
Map (num_proc=8):  56%|#####6    | 6623/11824 [00:03<00:02, 1923.09 examples/s]
Map (num_proc=8):  69%|######8   | 8107/11824 [00:04<00:01, 1992.41 examples/s]
Map (num_proc=8):  81%|########1 | 9584/11824 [00:05<00:01, 2046.01 examples/s]
Map (num_proc=8):  94%|#########3| 11072/11824 [00:05<00:00, 2107.79 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:06<00:00, 1906.86 examples/s]INFO 2025-05-20 11:17:09,524 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_7__Using_the_OpenShift_Container_Platform_dashboard_to_get_cluster_information_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 341 examples [00:00, 53937.39 examples/s]INFO 2025-05-20 11:17:09,566 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:17:09,566 instructlab.sdg.datamixing:161: Dataset loaded with 341 samples
INFO 2025-05-20 11:17:09,566 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 642/11824 [00:01<00:24, 463.07 examples/s]
Map (num_proc=8):  18%|#7        | 2121/11824 [00:02<00:09, 1034.30 examples/s]
Map (num_proc=8):  30%|###       | 3579/11824 [00:03<00:06, 1267.75 examples/s]
Map (num_proc=8):  43%|####2     | 5062/11824 [00:04<00:04, 1404.59 examples/s]
Map (num_proc=8):  55%|#####5    | 6537/11824 [00:05<00:03, 1478.84 examples/s]
Map (num_proc=8):  68%|######7   | 7995/11824 [00:05<00:02, 1510.46 examples/s]
Map (num_proc=8):  80%|########  | 9477/11824 [00:06<00:01, 1545.45 examples/s]
Map (num_proc=8):  93%|#########2| 10973/11824 [00:07<00:00, 1636.77 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1481.80 examples/s]INFO 2025-05-20 11:17:19,151 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_9_1__Prerequisites_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2356 examples [00:00, 91554.76 examples/s]INFO 2025-05-20 11:17:19,220 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:17:19,220 instructlab.sdg.datamixing:161: Dataset loaded with 2356 samples
INFO 2025-05-20 11:17:19,220 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 704/11824 [00:01<00:22, 489.64 examples/s]
Map (num_proc=8):  18%|#8        | 2161/11824 [00:02<00:09, 969.68 examples/s]
Map (num_proc=8):  31%|###       | 3641/11824 [00:03<00:06, 1191.56 examples/s]
Map (num_proc=8):  43%|####3     | 5118/11824 [00:04<00:05, 1291.51 examples/s]
Map (num_proc=8):  56%|#####5    | 6589/11824 [00:05<00:03, 1361.94 examples/s]
Map (num_proc=8):  68%|######8   | 8045/11824 [00:06<00:02, 1392.96 examples/s]
Map (num_proc=8):  81%|########  | 9531/11824 [00:07<00:01, 1424.55 examples/s]
Map (num_proc=8):  93%|#########3| 11031/11824 [00:08<00:00, 1479.02 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1365.03 examples/s]INFO 2025-05-20 11:17:29,435 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_20_19__Re_using_persistent_volumes_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 6595 examples [00:00, 63203.60 examples/s]
Generating train split: 13540 examples [00:00, 65691.97 examples/s]
Generating train split: 28745 examples [00:00, 94334.35 examples/s]
Generating train split: 28777 examples [00:00, 86275.05 examples/s]INFO 2025-05-20 11:17:29,806 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:17:29,806 instructlab.sdg.datamixing:161: Dataset loaded with 28777 samples


Map (num_proc=8):   0%|          | 0/28777 [00:00<?, ? examples/s]
Map (num_proc=8):   0%|          | 134/28777 [00:00<00:47, 600.25 examples/s]
Map (num_proc=8):   6%|5         | 1651/28777 [00:00<00:04, 6261.40 examples/s]
Map (num_proc=8):  13%|#2        | 3682/28777 [00:00<00:02, 11210.47 examples/s]
Map (num_proc=8):  18%|#7        | 5145/28777 [00:00<00:01, 12077.73 examples/s]
Map (num_proc=8):  24%|##3       | 6817/28777 [00:00<00:01, 13015.04 examples/s]
Map (num_proc=8):  29%|##8       | 8278/28777 [00:00<00:02, 10231.56 examples/s]
Map (num_proc=8):  33%|###3      | 9559/28777 [00:00<00:01, 10816.44 examples/s]
Map (num_proc=8):  39%|###8      | 11111/28777 [00:01<00:01, 11886.69 examples/s]
Map (num_proc=8):  45%|####4     | 12836/28777 [00:01<00:01, 13220.56 examples/s]
Map (num_proc=8):  50%|#####     | 14519/28777 [00:01<00:01, 13298.79 examples/s]
Map (num_proc=8):  55%|#####5    | 15928/28777 [00:01<00:01, 12422.78 examples/s]
Map (num_proc=8):  60%|#####9    | 17264/28777 [00:01<00:00, 12119.54 examples/s]
Map (num_proc=8):  65%|######5   | 18774/28777 [00:01<00:00, 12736.89 examples/s]
Map (num_proc=8):  71%|#######   | 20326/28777 [00:01<00:00, 13308.37 examples/s]
Map (num_proc=8):  77%|#######6  | 22131/28777 [00:01<00:00, 14108.63 examples/s]
Map (num_proc=8):  82%|########1 | 23583/28777 [00:01<00:00, 13489.42 examples/s]
Map (num_proc=8):  87%|########6 | 24953/28777 [00:02<00:00, 12393.38 examples/s]
Map (num_proc=8):  91%|#########1| 26289/28777 [00:02<00:00, 12645.21 examples/s]
Map (num_proc=8):  97%|#########6| 27812/28777 [00:02<00:00, 12980.12 examples/s]
Map (num_proc=8): 100%|##########| 28777/28777 [00:02<00:00, 11301.62 examples/s]INFO 2025-05-20 11:17:33,313 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_5__Enabling_the_virtctl_client_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2689 examples [00:00, 105930.10 examples/s]INFO 2025-05-20 11:17:33,359 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:17:33,359 instructlab.sdg.datamixing:161: Dataset loaded with 2689 samples
INFO 2025-05-20 11:17:33,359 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 728/11824 [00:01<00:18, 614.35 examples/s]
Map (num_proc=8):  19%|#8        | 2208/11824 [00:01<00:07, 1225.14 examples/s]
Map (num_proc=8):  31%|###1      | 3686/11824 [00:02<00:05, 1488.85 examples/s]
Map (num_proc=8):  44%|####3     | 5165/11824 [00:03<00:04, 1626.79 examples/s]
Map (num_proc=8):  56%|#####6    | 6646/11824 [00:04<00:03, 1709.04 examples/s]
Map (num_proc=8):  69%|######8   | 8124/11824 [00:05<00:02, 1758.57 examples/s]
Map (num_proc=8):  81%|########1 | 9607/11824 [00:05<00:01, 1792.52 examples/s]
Map (num_proc=8):  94%|#########3| 11092/11824 [00:06<00:00, 1875.26 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:06<00:00, 1700.72 examples/s]INFO 2025-05-20 11:17:41,770 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_1_1__Prerequisites_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2197 examples [00:00, 118698.05 examples/s]INFO 2025-05-20 11:17:41,824 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:17:41,824 instructlab.sdg.datamixing:161: Dataset loaded with 2197 samples
INFO 2025-05-20 11:17:41,824 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 653/11824 [00:01<00:20, 558.26 examples/s]
Map (num_proc=8):  18%|#8        | 2137/11824 [00:01<00:07, 1214.96 examples/s]
Map (num_proc=8):  31%|###       | 3618/11824 [00:02<00:05, 1497.58 examples/s]
Map (num_proc=8):  43%|####3     | 5095/11824 [00:03<00:04, 1645.23 examples/s]
Map (num_proc=8):  56%|#####6    | 6645/11824 [00:04<00:02, 1762.77 examples/s]
Map (num_proc=8):  69%|######8   | 8125/11824 [00:05<00:02, 1805.83 examples/s]
Map (num_proc=8):  81%|########1 | 9603/11824 [00:05<00:01, 1835.25 examples/s]
Map (num_proc=8):  94%|#########4| 11122/11824 [00:06<00:00, 1928.25 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:06<00:00, 1728.32 examples/s]INFO 2025-05-20 11:17:50,129 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_1__Preparing_your_cluster_for_OpenShift_Virtualization_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 5342 examples [00:00, 78109.87 examples/s]INFO 2025-05-20 11:17:50,232 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:17:50,232 instructlab.sdg.datamixing:161: Dataset loaded with 5342 samples
INFO 2025-05-20 11:17:50,232 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 668/11824 [00:01<00:24, 454.47 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:01<00:09, 1042.87 examples/s]
Map (num_proc=8):  18%|#7        | 2123/11824 [00:02<00:10, 926.66 examples/s] 
Map (num_proc=8):  25%|##5       | 2956/11824 [00:02<00:06, 1406.84 examples/s]
Map (num_proc=8):  30%|###       | 3599/11824 [00:03<00:07, 1127.76 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:03<00:04, 1559.01 examples/s]
Map (num_proc=8):  43%|####2     | 5077/11824 [00:04<00:05, 1222.47 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:04<00:03, 1629.50 examples/s]
Map (num_proc=8):  55%|#####5    | 6544/11824 [00:05<00:04, 1245.20 examples/s]
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:05<00:03, 1344.47 examples/s]
Map (num_proc=8):  68%|######7   | 8024/11824 [00:06<00:02, 1351.34 examples/s]
Map (num_proc=8):  71%|#######   | 8390/11824 [00:06<00:02, 1437.86 examples/s]
Map (num_proc=8):  80%|########  | 9500/11824 [00:07<00:01, 1402.10 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:07<00:00, 1808.74 examples/s]
Map (num_proc=8):  93%|#########2| 10989/11824 [00:08<00:00, 1432.17 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1356.13 examples/s]INFO 2025-05-20 11:18:00,623 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_14_1__Installing_and_configuring_OADP_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1683 examples [00:00, 85564.84 examples/s]INFO 2025-05-20 11:18:00,671 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:18:00,672 instructlab.sdg.datamixing:161: Dataset loaded with 1683 samples
INFO 2025-05-20 11:18:00,672 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 636/11824 [00:01<00:27, 401.91 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:01<00:09, 1065.62 examples/s]
Map (num_proc=8):  18%|#8        | 2142/11824 [00:02<00:11, 839.43 examples/s] 
Map (num_proc=8):  31%|###       | 3635/11824 [00:03<00:07, 1070.91 examples/s]
Map (num_proc=8):  43%|####3     | 5101/11824 [00:04<00:05, 1168.42 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:05<00:03, 1503.65 examples/s]
Map (num_proc=8):  56%|#####5    | 6577/11824 [00:06<00:04, 1153.37 examples/s]
Map (num_proc=8):  68%|######8   | 8075/11824 [00:07<00:03, 1227.75 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:07<00:01, 1547.81 examples/s]
Map (num_proc=8):  81%|########  | 9558/11824 [00:08<00:01, 1190.28 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:08<00:00, 1553.66 examples/s]
Map (num_proc=8):  94%|#########3| 11066/11824 [00:09<00:00, 1218.36 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:09<00:00, 1224.31 examples/s]INFO 2025-05-20 11:18:12,017 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_9__OpenShift_Container_Platform_cluster_monitoring__logging__and_Telemetry_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2459 examples [00:00, 111762.66 examples/s]INFO 2025-05-20 11:18:12,065 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:18:12,065 instructlab.sdg.datamixing:161: Dataset loaded with 2459 samples
INFO 2025-05-20 11:18:12,065 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 715/11824 [00:01<00:21, 522.33 examples/s]
Map (num_proc=8):  18%|#8        | 2146/11824 [00:02<00:09, 1002.82 examples/s]
Map (num_proc=8):  31%|###1      | 3680/11824 [00:03<00:06, 1253.02 examples/s]
Map (num_proc=8):  44%|####3     | 5157/11824 [00:04<00:04, 1370.59 examples/s]
Map (num_proc=8):  56%|#####6    | 6628/11824 [00:05<00:03, 1423.67 examples/s]
Map (num_proc=8):  69%|######8   | 8121/11824 [00:06<00:02, 1462.98 examples/s]
Map (num_proc=8):  81%|########1 | 9597/11824 [00:07<00:01, 1487.09 examples/s]
Map (num_proc=8):  94%|#########3| 11101/11824 [00:08<00:00, 1538.08 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1417.99 examples/s]INFO 2025-05-20 11:18:21,927 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_3_1__Installing_the_OpenShift_Virtualization_Operator_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1189 examples [00:00, 73481.28 examples/s]INFO 2025-05-20 11:18:21,981 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:18:21,982 instructlab.sdg.datamixing:161: Dataset loaded with 1189 samples
INFO 2025-05-20 11:18:21,982 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 658/11824 [00:01<00:26, 426.16 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:01<00:09, 1075.94 examples/s]
Map (num_proc=8):  18%|#7        | 2121/11824 [00:02<00:11, 881.62 examples/s] 
Map (num_proc=8):  25%|##5       | 2956/11824 [00:02<00:06, 1435.99 examples/s]
Map (num_proc=8):  30%|###       | 3599/11824 [00:03<00:07, 1074.41 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:03<00:04, 1586.80 examples/s]
Map (num_proc=8):  43%|####2     | 5078/11824 [00:04<00:05, 1168.28 examples/s]
Map (num_proc=8):  46%|####5     | 5434/11824 [00:04<00:04, 1326.93 examples/s]
Map (num_proc=8):  55%|#####5    | 6542/11824 [00:05<00:04, 1278.86 examples/s]
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:05<00:03, 1436.80 examples/s]
Map (num_proc=8):  68%|######7   | 8017/11824 [00:06<00:02, 1324.13 examples/s]
Map (num_proc=8):  71%|#######   | 8390/11824 [00:06<00:02, 1480.39 examples/s]
Map (num_proc=8):  80%|########  | 9486/11824 [00:07<00:01, 1355.65 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:07<00:00, 1843.85 examples/s]
Map (num_proc=8):  93%|#########2| 10979/11824 [00:08<00:00, 1376.89 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1324.58 examples/s]INFO 2025-05-20 11:18:32,674 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_19_1__Configuring_the_virtual_machine_for_the_default_pod_network_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 6461 examples [00:00, 60653.55 examples/s]
Generating train split: 17407 examples [00:00, 86803.30 examples/s]
Generating train split: 17407 examples [00:00, 82584.61 examples/s]INFO 2025-05-20 11:18:32,920 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:18:32,920 instructlab.sdg.datamixing:161: Dataset loaded with 17407 samples


Map (num_proc=8):   0%|          | 0/17407 [00:00<?, ? examples/s]
Map (num_proc=8):   1%|          | 172/17407 [00:00<00:21, 797.68 examples/s]
Map (num_proc=8):  10%|9         | 1666/17407 [00:00<00:02, 6115.47 examples/s]
Map (num_proc=8):  22%|##1       | 3759/17407 [00:00<00:01, 11274.43 examples/s]
Map (num_proc=8):  31%|###       | 5380/17407 [00:00<00:00, 12710.12 examples/s]
Map (num_proc=8):  40%|###9      | 6885/17407 [00:00<00:00, 12222.62 examples/s]
Map (num_proc=8):  47%|####7     | 8264/17407 [00:00<00:00, 11149.59 examples/s]
Map (num_proc=8):  55%|#####4    | 9526/17407 [00:00<00:00, 10835.25 examples/s]
Map (num_proc=8):  65%|######4   | 11294/17407 [00:01<00:00, 12010.29 examples/s]
Map (num_proc=8):  75%|#######5  | 13066/17407 [00:01<00:00, 13303.43 examples/s]
Map (num_proc=8):  83%|########3 | 14491/17407 [00:01<00:00, 13428.29 examples/s]
Map (num_proc=8):  92%|#########1| 15960/17407 [00:01<00:00, 13313.18 examples/s]
Map (num_proc=8): 100%|#########9| 17348/17407 [00:01<00:00, 11693.70 examples/s]
Map (num_proc=8): 100%|##########| 17407/17407 [00:01<00:00, 10296.33 examples/s]INFO 2025-05-20 11:18:35,582 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_11_7__Configuring_virtual_machine_eviction_strategy_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 828 examples [00:00, 81214.25 examples/s]INFO 2025-05-20 11:18:35,608 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:18:35,608 instructlab.sdg.datamixing:161: Dataset loaded with 828 samples
INFO 2025-05-20 11:18:35,608 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 698/11824 [00:01<00:22, 501.84 examples/s]
Map (num_proc=8):  18%|#8        | 2167/11824 [00:02<00:09, 1000.31 examples/s]
Map (num_proc=8):  31%|###       | 3654/11824 [00:03<00:06, 1216.60 examples/s]
Map (num_proc=8):  43%|####3     | 5137/11824 [00:04<00:05, 1328.62 examples/s]
Map (num_proc=8):  56%|#####5    | 6619/11824 [00:05<00:03, 1393.42 examples/s]
Map (num_proc=8):  68%|######8   | 8085/11824 [00:06<00:02, 1428.02 examples/s]
Map (num_proc=8):  81%|########  | 9556/11824 [00:07<00:01, 1445.76 examples/s]
Map (num_proc=8):  94%|#########3| 11059/11824 [00:08<00:00, 1506.24 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1388.49 examples/s]INFO 2025-05-20 11:18:45,666 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_14_2_1__Additional_resources_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 223 examples [00:00, 50237.93 examples/s]INFO 2025-05-20 11:18:45,692 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:18:45,692 instructlab.sdg.datamixing:161: Dataset loaded with 223 samples
INFO 2025-05-20 11:18:45,692 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 713/11824 [00:01<00:20, 545.04 examples/s]
Map (num_proc=8):  19%|#8        | 2198/11824 [00:02<00:08, 1079.83 examples/s]
Map (num_proc=8):  31%|###1      | 3675/11824 [00:03<00:06, 1312.40 examples/s]
Map (num_proc=8):  44%|####3     | 5152/11824 [00:04<00:04, 1437.91 examples/s]
Map (num_proc=8):  56%|#####6    | 6636/11824 [00:04<00:03, 1511.39 examples/s]
Map (num_proc=8):  69%|######8   | 8112/11824 [00:05<00:02, 1555.42 examples/s]
Map (num_proc=8):  81%|########1 | 9595/11824 [00:06<00:01, 1585.81 examples/s]
Map (num_proc=8):  94%|#########3| 11094/11824 [00:07<00:00, 1657.99 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1506.64 examples/s]INFO 2025-05-20 11:18:55,085 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_10__Running_cluster_checkups_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2694 examples [00:00, 71602.09 examples/s]INFO 2025-05-20 11:18:55,157 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:18:55,157 instructlab.sdg.datamixing:161: Dataset loaded with 2694 samples
INFO 2025-05-20 11:18:55,157 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 660/11824 [00:01<00:28, 389.35 examples/s]
Map (num_proc=8):   9%|8         | 1048/11824 [00:01<00:17, 607.15 examples/s]
Map (num_proc=8):  18%|#8        | 2133/11824 [00:02<00:11, 850.49 examples/s]
Map (num_proc=8):  25%|##5       | 2956/11824 [00:03<00:07, 1250.26 examples/s]
Map (num_proc=8):  31%|###       | 3608/11824 [00:04<00:08, 973.19 examples/s] 
Map (num_proc=8):  38%|###7      | 4434/11824 [00:04<00:05, 1339.37 examples/s]
Map (num_proc=8):  43%|####3     | 5089/11824 [00:05<00:06, 1029.64 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:05<00:04, 1387.93 examples/s]
Map (num_proc=8):  56%|#####5    | 6585/11824 [00:06<00:04, 1061.97 examples/s]
Map (num_proc=8):  62%|######2   | 7390/11824 [00:06<00:03, 1388.51 examples/s]
Map (num_proc=8):  68%|######8   | 8055/11824 [00:07<00:03, 1067.27 examples/s]
Map (num_proc=8):  71%|#######1  | 8403/11824 [00:07<00:03, 1135.99 examples/s]
Map (num_proc=8):  81%|########  | 9523/11824 [00:08<00:02, 1133.92 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:09<00:01, 1459.16 examples/s]
Map (num_proc=8):  93%|#########3| 11029/11824 [00:10<00:00, 1176.18 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1141.20 examples/s]INFO 2025-05-20 11:19:07,227 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_12_2__Configuring_a_virtual_machine_with_the_node_exporter_service_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1663 examples [00:00, 73180.51 examples/s]INFO 2025-05-20 11:19:07,285 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:19:07,285 instructlab.sdg.datamixing:161: Dataset loaded with 1663 samples
INFO 2025-05-20 11:19:07,285 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 619/11824 [00:01<00:31, 350.52 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:01<00:10, 963.54 examples/s]
Map (num_proc=8):  18%|#7        | 2100/11824 [00:03<00:13, 728.65 examples/s]
Map (num_proc=8):  25%|##5       | 2956/11824 [00:03<00:07, 1223.50 examples/s]
Map (num_proc=8):  30%|###       | 3579/11824 [00:04<00:09, 871.12 examples/s] 
Map (num_proc=8):  33%|###3      | 3956/11824 [00:04<00:07, 1039.75 examples/s]
Map (num_proc=8):  43%|####2     | 5068/11824 [00:05<00:06, 997.76 examples/s] 
Map (num_proc=8):  50%|#####     | 5912/11824 [00:05<00:04, 1419.07 examples/s]
Map (num_proc=8):  55%|#####5    | 6552/11824 [00:06<00:05, 996.33 examples/s] 
Map (num_proc=8):  62%|######2   | 7390/11824 [00:06<00:03, 1406.91 examples/s]
Map (num_proc=8):  68%|######7   | 8029/11824 [00:08<00:03, 992.74 examples/s] 
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:08<00:02, 1401.81 examples/s]
Map (num_proc=8):  80%|########  | 9499/11824 [00:09<00:02, 989.71 examples/s] 
Map (num_proc=8):  88%|########7 | 10346/11824 [00:09<00:01, 1402.17 examples/s]
Map (num_proc=8):  93%|#########3| 11009/11824 [00:10<00:00, 1043.44 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1082.48 examples/s]INFO 2025-05-20 11:19:20,003 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_7_3__Additional_OpenShift_Container_Platform_security_context_constraints_and_Linux_capabilities_for_the_kubevirt_controller_service_account_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 745 examples [00:00, 69119.55 examples/s]INFO 2025-05-20 11:19:20,045 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:19:20,045 instructlab.sdg.datamixing:161: Dataset loaded with 745 samples
INFO 2025-05-20 11:19:20,045 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 639/11824 [00:01<00:32, 346.63 examples/s]
Map (num_proc=8):   9%|8         | 1052/11824 [00:02<00:18, 574.79 examples/s]
Map (num_proc=8):  18%|#7        | 2119/11824 [00:03<00:12, 762.46 examples/s]
Map (num_proc=8):  25%|##5       | 2956/11824 [00:03<00:07, 1138.18 examples/s]
Map (num_proc=8):  30%|###       | 3603/11824 [00:04<00:09, 874.60 examples/s] 
Map (num_proc=8):  33%|###3      | 3956/11824 [00:04<00:08, 961.33 examples/s]
Map (num_proc=8):  43%|####3     | 5091/11824 [00:05<00:06, 983.30 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:06<00:04, 1298.30 examples/s]
Map (num_proc=8):  56%|#####5    | 6574/11824 [00:07<00:05, 980.01 examples/s] 
Map (num_proc=8):  62%|######2   | 7390/11824 [00:07<00:03, 1289.44 examples/s]
Map (num_proc=8):  68%|######7   | 8025/11824 [00:08<00:03, 965.13 examples/s] 
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:08<00:02, 1288.74 examples/s]
Map (num_proc=8):  80%|########  | 9506/11824 [00:09<00:02, 964.40 examples/s] 
Map (num_proc=8):  88%|########7 | 10346/11824 [00:10<00:01, 1282.22 examples/s]
Map (num_proc=8):  93%|#########3| 11015/11824 [00:11<00:00, 1011.31 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:11<00:00, 1027.53 examples/s]INFO 2025-05-20 11:19:33,310 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_7__Known_issues_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 576 examples [00:00, 54984.73 examples/s]INFO 2025-05-20 11:19:33,361 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:19:33,361 instructlab.sdg.datamixing:161: Dataset loaded with 576 samples
INFO 2025-05-20 11:19:33,361 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|4         | 585/11824 [00:02<00:40, 274.57 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:02<00:12, 811.47 examples/s]
Map (num_proc=8):  17%|#7        | 2037/11824 [00:03<00:16, 583.30 examples/s]
Map (num_proc=8):  25%|##5       | 2956/11824 [00:03<00:08, 1019.06 examples/s]
Map (num_proc=8):  30%|##9       | 3503/11824 [00:05<00:11, 706.35 examples/s] 
Map (num_proc=8):  33%|###3      | 3956/11824 [00:05<00:08, 877.81 examples/s]
Map (num_proc=8):  42%|####2     | 4970/11824 [00:06<00:08, 813.33 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:06<00:04, 1205.71 examples/s]
Map (num_proc=8):  55%|#####4    | 6448/11824 [00:08<00:06, 814.07 examples/s] 
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:08<00:04, 990.45 examples/s]
Map (num_proc=8):  67%|######7   | 7932/11824 [00:09<00:04, 862.71 examples/s]
Map (num_proc=8):  71%|#######   | 8390/11824 [00:09<00:03, 1016.56 examples/s]
Map (num_proc=8):  80%|#######9  | 9403/11824 [00:11<00:02, 886.31 examples/s] 
Map (num_proc=8):  83%|########3 | 9868/11824 [00:11<00:01, 1039.35 examples/s]
Map (num_proc=8):  92%|#########2| 10902/11824 [00:12<00:00, 954.43 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:12<00:00, 1359.59 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:12<00:00, 910.61 examples/s] INFO 2025-05-20 11:19:48,386 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_11_4__Migrating_a_virtual_machine_over_a_dedicated_additional_network_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 371 examples [00:00, 53466.42 examples/s]INFO 2025-05-20 11:19:48,431 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:19:48,431 instructlab.sdg.datamixing:161: Dataset loaded with 371 samples
INFO 2025-05-20 11:19:48,431 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 656/11824 [00:01<00:28, 387.10 examples/s]
Map (num_proc=8):   8%|8         | 1000/11824 [00:01<00:17, 616.84 examples/s]
Map (num_proc=8):  18%|#8        | 2132/11824 [00:02<00:11, 849.19 examples/s]
Map (num_proc=8):  21%|##1       | 2485/11824 [00:03<00:09, 1017.88 examples/s]
Map (num_proc=8):  31%|###       | 3607/11824 [00:04<00:08, 1021.29 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:04<00:05, 1467.52 examples/s]
Map (num_proc=8):  43%|####3     | 5089/11824 [00:05<00:06, 1044.91 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:05<00:04, 1467.40 examples/s]
Map (num_proc=8):  56%|#####5    | 6589/11824 [00:06<00:04, 1057.15 examples/s]
Map (num_proc=8):  62%|######2   | 7390/11824 [00:06<00:03, 1465.26 examples/s]
Map (num_proc=8):  68%|######8   | 8072/11824 [00:07<00:03, 1065.23 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:07<00:02, 1460.35 examples/s]
Map (num_proc=8):  81%|########  | 9538/11824 [00:08<00:02, 1058.94 examples/s]
Map (num_proc=8):  83%|########3 | 9868/11824 [00:08<00:01, 1190.12 examples/s]
Map (num_proc=8):  93%|#########3| 11043/11824 [00:09<00:00, 1186.19 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1144.28 examples/s]INFO 2025-05-20 11:20:00,511 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_1_3__Creating_a_virtual_machine_template_in_the_web_console_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1034 examples [00:00, 65270.68 examples/s]INFO 2025-05-20 11:20:00,561 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:20:00,562 instructlab.sdg.datamixing:161: Dataset loaded with 1034 samples
INFO 2025-05-20 11:20:00,562 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 603/11824 [00:02<00:38, 291.93 examples/s]
Map (num_proc=8):   8%|8         | 1000/11824 [00:02<00:20, 527.87 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:02<00:12, 835.60 examples/s]
Map (num_proc=8):  18%|#7        | 2078/11824 [00:03<00:15, 630.23 examples/s]
Map (num_proc=8):  21%|##        | 2478/11824 [00:03<00:12, 758.09 examples/s]
Map (num_proc=8):  30%|##9       | 3535/11824 [00:05<00:10, 808.15 examples/s]
Map (num_proc=8):  33%|###3      | 3956/11824 [00:05<00:07, 991.11 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:05<00:06, 1213.43 examples/s]
Map (num_proc=8):  42%|####2     | 4996/11824 [00:06<00:08, 792.17 examples/s] 
Map (num_proc=8):  46%|####5     | 5434/11824 [00:06<00:06, 1002.18 examples/s]
Map (num_proc=8):  54%|#####3    | 6380/11824 [00:08<00:06, 831.48 examples/s] 
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:08<00:04, 1065.57 examples/s]
Map (num_proc=8):  62%|######2   | 7390/11824 [00:08<00:03, 1262.44 examples/s]
Map (num_proc=8):  67%|######7   | 7948/11824 [00:09<00:04, 824.27 examples/s] 
Map (num_proc=8):  71%|#######   | 8390/11824 [00:09<00:03, 1031.80 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:09<00:02, 1251.99 examples/s]
Map (num_proc=8):  80%|#######9  | 9423/11824 [00:11<00:02, 800.51 examples/s] 
Map (num_proc=8):  83%|########3 | 9868/11824 [00:11<00:01, 1021.37 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:11<00:01, 1243.16 examples/s]
Map (num_proc=8):  92%|#########1| 10824/11824 [00:12<00:01, 794.28 examples/s] 
Map (num_proc=8): 100%|##########| 11824/11824 [00:12<00:00, 1384.65 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:12<00:00, 913.86 examples/s] INFO 2025-05-20 11:20:15,416 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_1_1__Hardware_and_operating_system_requirements_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 4134 examples [00:00, 72125.18 examples/s]INFO 2025-05-20 11:20:15,518 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:20:15,518 instructlab.sdg.datamixing:161: Dataset loaded with 4134 samples
INFO 2025-05-20 11:20:15,518 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 648/11824 [00:01<00:31, 357.83 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:02<00:12, 861.84 examples/s]
Map (num_proc=8):  18%|#7        | 2100/11824 [00:03<00:13, 712.36 examples/s]
Map (num_proc=8):  21%|##1       | 2489/11824 [00:03<00:11, 846.67 examples/s]
Map (num_proc=8):  30%|###       | 3571/11824 [00:04<00:09, 910.79 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:04<00:05, 1278.58 examples/s]
Map (num_proc=8):  42%|####2     | 5023/11824 [00:05<00:07, 937.69 examples/s] 
Map (num_proc=8):  50%|#####     | 5912/11824 [00:05<00:04, 1306.51 examples/s]
Map (num_proc=8):  55%|#####4    | 6500/11824 [00:07<00:05, 952.97 examples/s] 
Map (num_proc=8):  62%|######2   | 7390/11824 [00:07<00:03, 1315.74 examples/s]
Map (num_proc=8):  67%|######7   | 7972/11824 [00:08<00:04, 954.65 examples/s] 
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:08<00:02, 1325.34 examples/s]
Map (num_proc=8):  80%|#######9  | 9455/11824 [00:09<00:02, 966.35 examples/s] 
Map (num_proc=8):  88%|########7 | 10346/11824 [00:09<00:01, 1317.03 examples/s]
Map (num_proc=8):  92%|#########1| 10839/11824 [00:10<00:01, 970.92 examples/s] 
Map (num_proc=8): 100%|##########| 11824/11824 [00:11<00:00, 1047.04 examples/s]INFO 2025-05-20 11:20:28,632 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_13_1__About_VirtIO_drivers_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2320 examples [00:00, 65985.74 examples/s]INFO 2025-05-20 11:20:28,703 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:20:28,703 instructlab.sdg.datamixing:161: Dataset loaded with 2320 samples
INFO 2025-05-20 11:20:28,703 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 661/11824 [00:01<00:28, 392.63 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:01<00:10, 991.80 examples/s]
Map (num_proc=8):  18%|#7        | 2113/11824 [00:02<00:12, 772.93 examples/s]
Map (num_proc=8):  25%|##5       | 2956/11824 [00:02<00:06, 1279.00 examples/s]
Map (num_proc=8):  30%|###       | 3593/11824 [00:04<00:08, 932.21 examples/s] 
Map (num_proc=8):  33%|###3      | 3956/11824 [00:04<00:07, 1094.94 examples/s]
Map (num_proc=8):  43%|####2     | 5054/11824 [00:05<00:06, 1058.29 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:05<00:03, 1507.72 examples/s]
Map (num_proc=8):  55%|#####4    | 6491/11824 [00:06<00:05, 1038.36 examples/s]
Map (num_proc=8):  67%|######7   | 7964/11824 [00:07<00:03, 1129.58 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:07<00:01, 1513.64 examples/s]
Map (num_proc=8):  80%|########  | 9479/11824 [00:08<00:02, 1106.70 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:08<00:00, 1508.04 examples/s]
Map (num_proc=8):  93%|#########2| 10938/11824 [00:09<00:00, 1122.20 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1155.18 examples/s]INFO 2025-05-20 11:20:40,728 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_5_1__Deleting_a_virtual_machine_template_in_the_web_console_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1149 examples [00:00, 135715.44 examples/s]INFO 2025-05-20 11:20:40,771 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:20:40,771 instructlab.sdg.datamixing:161: Dataset loaded with 1149 samples
INFO 2025-05-20 11:20:40,771 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 725/11824 [00:00<00:14, 771.67 examples/s]
Map (num_proc=8):  19%|#8        | 2203/11824 [00:01<00:06, 1585.84 examples/s]
Map (num_proc=8):  31%|###1      | 3679/11824 [00:02<00:04, 1958.26 examples/s]
Map (num_proc=8):  44%|####3     | 5155/11824 [00:02<00:03, 2140.28 examples/s]
Map (num_proc=8):  56%|#####6    | 6633/11824 [00:03<00:02, 2263.92 examples/s]
Map (num_proc=8):  69%|######8   | 8110/11824 [00:03<00:01, 2337.26 examples/s]
Map (num_proc=8):  81%|########1 | 9585/11824 [00:04<00:00, 2389.10 examples/s]
Map (num_proc=8):  94%|#########3| 11091/11824 [00:05<00:00, 2547.46 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:05<00:00, 2235.41 examples/s]INFO 2025-05-20 11:20:47,531 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_7_1__Accessing_virtual_machine_consoles_in_the_OpenShift_Container_Platform_web_console_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 6146 examples [00:00, 84698.06 examples/s]INFO 2025-05-20 11:20:47,639 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:20:47,639 instructlab.sdg.datamixing:161: Dataset loaded with 6146 samples
INFO 2025-05-20 11:20:47,639 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 692/11824 [00:01<00:19, 563.45 examples/s]
Map (num_proc=8):  18%|#8        | 2166/11824 [00:02<00:08, 1179.31 examples/s]
Map (num_proc=8):  31%|###       | 3646/11824 [00:02<00:05, 1458.19 examples/s]
Map (num_proc=8):  43%|####3     | 5126/11824 [00:03<00:04, 1585.48 examples/s]
Map (num_proc=8):  56%|#####5    | 6610/11824 [00:04<00:03, 1708.46 examples/s]
Map (num_proc=8):  68%|######8   | 8088/11824 [00:05<00:02, 1736.60 examples/s]
Map (num_proc=8):  81%|########  | 9564/11824 [00:05<00:01, 1801.86 examples/s]
Map (num_proc=8):  94%|#########3| 11070/11824 [00:06<00:00, 1873.17 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1677.65 examples/s]INFO 2025-05-20 11:20:56,380 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_15__Using_virtual_Trusted_Platform_Module_devices_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 808 examples [00:00, 99873.21 examples/s]INFO 2025-05-20 11:20:56,417 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:20:56,417 instructlab.sdg.datamixing:161: Dataset loaded with 808 samples
INFO 2025-05-20 11:20:56,417 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 732/11824 [00:01<00:18, 602.16 examples/s]
Map (num_proc=8):  10%|#         | 1241/11824 [00:01<00:10, 1017.61 examples/s]
Map (num_proc=8):  19%|#8        | 2206/11824 [00:02<00:07, 1203.26 examples/s]
Map (num_proc=8):  23%|##3       | 2739/11824 [00:02<00:05, 1514.20 examples/s]
Map (num_proc=8):  31%|###1      | 3692/11824 [00:02<00:05, 1440.86 examples/s]
Map (num_proc=8):  36%|###5      | 4228/11824 [00:03<00:04, 1725.48 examples/s]
Map (num_proc=8):  44%|####3     | 5168/11824 [00:03<00:04, 1546.25 examples/s]
Map (num_proc=8):  48%|####7     | 5669/11824 [00:03<00:03, 1778.75 examples/s]
Map (num_proc=8):  56%|#####6    | 6644/11824 [00:04<00:03, 1600.29 examples/s]
Map (num_proc=8):  60%|######    | 7126/11824 [00:04<00:02, 1811.77 examples/s]
Map (num_proc=8):  69%|######8   | 8130/11824 [00:05<00:02, 1639.41 examples/s]
Map (num_proc=8):  73%|#######2  | 8622/11824 [00:05<00:01, 1854.02 examples/s]
Map (num_proc=8):  81%|########1 | 9611/11824 [00:06<00:01, 1653.56 examples/s]
Map (num_proc=8):  85%|########5 | 10099/11824 [00:06<00:00, 1862.62 examples/s]
Map (num_proc=8):  94%|#########3| 11107/11824 [00:07<00:00, 1762.98 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1611.11 examples/s]INFO 2025-05-20 11:21:05,313 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_11_2_2__Cluster_wide_live_migration_limits_and_timeouts_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 883 examples [00:00, 87954.08 examples/s]INFO 2025-05-20 11:21:05,357 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:21:05,358 instructlab.sdg.datamixing:161: Dataset loaded with 883 samples
INFO 2025-05-20 11:21:05,358 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 710/11824 [00:01<00:19, 572.77 examples/s]
Map (num_proc=8):  19%|#8        | 2188/11824 [00:02<00:08, 1154.52 examples/s]
Map (num_proc=8):  31%|###1      | 3668/11824 [00:02<00:05, 1419.35 examples/s]
Map (num_proc=8):  43%|####3     | 5139/11824 [00:03<00:04, 1547.08 examples/s]
Map (num_proc=8):  56%|#####5    | 6604/11824 [00:04<00:03, 1619.75 examples/s]
Map (num_proc=8):  68%|######8   | 8082/11824 [00:05<00:02, 1667.69 examples/s]
Map (num_proc=8):  81%|########  | 9552/11824 [00:06<00:01, 1693.66 examples/s]
Map (num_proc=8):  93%|#########3| 11047/11824 [00:07<00:00, 1756.33 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1609.81 examples/s]INFO 2025-05-20 11:21:14,281 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_8_3__Additional_resources_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1315 examples [00:00, 111561.92 examples/s]INFO 2025-05-20 11:21:14,322 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:21:14,322 instructlab.sdg.datamixing:161: Dataset loaded with 1315 samples
INFO 2025-05-20 11:21:14,323 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 725/11824 [00:01<00:16, 693.17 examples/s]
Map (num_proc=8):  19%|#8        | 2204/11824 [00:01<00:06, 1441.10 examples/s]
Map (num_proc=8):  31%|###1      | 3681/11824 [00:02<00:04, 1786.09 examples/s]
Map (num_proc=8):  44%|####3     | 5165/11824 [00:02<00:03, 1977.69 examples/s]
Map (num_proc=8):  56%|#####6    | 6645/11824 [00:03<00:02, 2091.41 examples/s]
Map (num_proc=8):  69%|######8   | 8122/11824 [00:04<00:01, 2161.99 examples/s]
Map (num_proc=8):  81%|########1 | 9606/11824 [00:04<00:01, 2211.55 examples/s]
Map (num_proc=8):  94%|#########3| 11101/11824 [00:05<00:00, 2331.64 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:05<00:00, 2051.96 examples/s]INFO 2025-05-20 11:21:21,697 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_14_2__About_VirtIO_drivers_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2359 examples [00:00, 64939.41 examples/s]INFO 2025-05-20 11:21:21,773 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:21:21,773 instructlab.sdg.datamixing:161: Dataset loaded with 2359 samples
INFO 2025-05-20 11:21:21,773 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 690/11824 [00:01<00:26, 415.29 examples/s]
Map (num_proc=8):   8%|8         | 1000/11824 [00:01<00:18, 590.75 examples/s]
Map (num_proc=8):  18%|#8        | 2166/11824 [00:02<00:10, 884.04 examples/s]
Map (num_proc=8):  21%|##        | 2478/11824 [00:02<00:09, 1006.73 examples/s]
Map (num_proc=8):  31%|###       | 3646/11824 [00:04<00:07, 1064.24 examples/s]
Map (num_proc=8):  33%|###3      | 3956/11824 [00:04<00:06, 1137.23 examples/s]
Map (num_proc=8):  43%|####3     | 5135/11824 [00:05<00:05, 1154.40 examples/s]
Map (num_proc=8):  46%|####5     | 5434/11824 [00:05<00:05, 1216.83 examples/s]
Map (num_proc=8):  56%|#####5    | 6617/11824 [00:06<00:04, 1189.94 examples/s]
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:06<00:03, 1237.24 examples/s]
Map (num_proc=8):  68%|######8   | 8071/11824 [00:07<00:03, 1197.97 examples/s]
Map (num_proc=8):  71%|#######   | 8390/11824 [00:07<00:02, 1258.74 examples/s]
Map (num_proc=8):  81%|########  | 9548/11824 [00:08<00:01, 1208.95 examples/s]
Map (num_proc=8):  84%|########3 | 9914/11824 [00:08<00:01, 1295.43 examples/s]
Map (num_proc=8):  93%|#########3| 11050/11824 [00:09<00:00, 1262.81 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1154.66 examples/s]INFO 2025-05-20 11:21:33,807 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_5_4__Editing_a_standalone_virtual_machine_instance_using_the_web_console_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2867 examples [00:00, 70864.93 examples/s]INFO 2025-05-20 11:21:34,171 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:21:34,171 instructlab.sdg.datamixing:161: Dataset loaded with 2867 samples
INFO 2025-05-20 11:21:34,171 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 711/11824 [00:01<00:16, 688.02 examples/s]
Map (num_proc=8):  19%|#8        | 2199/11824 [00:01<00:06, 1456.71 examples/s]
Map (num_proc=8):  31%|###1      | 3679/11824 [00:02<00:04, 1807.20 examples/s]
Map (num_proc=8):  44%|####3     | 5158/11824 [00:02<00:03, 1997.75 examples/s]
Map (num_proc=8):  56%|#####6    | 6637/11824 [00:03<00:02, 2104.13 examples/s]
Map (num_proc=8):  69%|######8   | 8121/11824 [00:04<00:01, 2180.90 examples/s]
Map (num_proc=8):  81%|########1 | 9594/11824 [00:04<00:01, 2211.22 examples/s]
Map (num_proc=8):  94%|#########3| 11108/11824 [00:05<00:00, 2339.27 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:05<00:00, 2058.37 examples/s]INFO 2025-05-20 11:21:41,540 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_6_1__Starting_a_virtual_machine_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 429 examples [00:00, 67150.19 examples/s]INFO 2025-05-20 11:21:41,583 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:21:41,583 instructlab.sdg.datamixing:161: Dataset loaded with 429 samples
INFO 2025-05-20 11:21:41,583 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 703/11824 [00:01<00:17, 620.74 examples/s]
Map (num_proc=8):  18%|#8        | 2178/11824 [00:01<00:07, 1311.55 examples/s]
Map (num_proc=8):  31%|###       | 3658/11824 [00:02<00:05, 1626.60 examples/s]
Map (num_proc=8):  43%|####3     | 5137/11824 [00:03<00:03, 1797.06 examples/s]
Map (num_proc=8):  56%|#####6    | 6628/11824 [00:03<00:02, 1903.18 examples/s]
Map (num_proc=8):  69%|######8   | 8103/11824 [00:04<00:01, 1961.36 examples/s]
Map (num_proc=8):  80%|########  | 9488/11824 [00:05<00:01, 1960.11 examples/s]
Map (num_proc=8):  94%|#########3| 11097/11824 [00:06<00:00, 2136.12 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:06<00:00, 1874.10 examples/s]INFO 2025-05-20 11:21:49,555 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_13_5__Removing_the_VirtIO_container_disk_from_a_virtual_machine_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2248 examples [00:00, 65175.40 examples/s]INFO 2025-05-20 11:21:49,622 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:21:49,622 instructlab.sdg.datamixing:161: Dataset loaded with 2248 samples
INFO 2025-05-20 11:21:49,622 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 650/11824 [00:01<00:28, 390.73 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:01<00:10, 1007.52 examples/s]
Map (num_proc=8):  18%|#7        | 2128/11824 [00:02<00:12, 796.43 examples/s] 
Map (num_proc=8):  21%|##1       | 2506/11824 [00:02<00:09, 995.09 examples/s]
Map (num_proc=8):  30%|###       | 3605/11824 [00:03<00:08, 1020.67 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:04<00:04, 1479.69 examples/s]
Map (num_proc=8):  43%|####2     | 5080/11824 [00:05<00:06, 1056.06 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:05<00:03, 1500.80 examples/s]
Map (num_proc=8):  55%|#####5    | 6544/11824 [00:06<00:04, 1066.09 examples/s]
Map (num_proc=8):  62%|######2   | 7390/11824 [00:06<00:02, 1511.59 examples/s]
Map (num_proc=8):  68%|######7   | 8023/11824 [00:07<00:03, 1075.28 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:07<00:01, 1517.43 examples/s]
Map (num_proc=8):  80%|########  | 9493/11824 [00:08<00:02, 1076.37 examples/s]
Map (num_proc=8):  83%|########3 | 9868/11824 [00:08<00:01, 1232.42 examples/s]
Map (num_proc=8):  93%|#########3| 11000/11824 [00:09<00:00, 1209.61 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1177.29 examples/s]INFO 2025-05-20 11:22:01,509 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_13_4__Installing_VirtIO_drivers_on_an_existing_Windows_virtual_machine_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2550 examples [00:00, 65124.98 examples/s]INFO 2025-05-20 11:22:01,592 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:22:01,592 instructlab.sdg.datamixing:161: Dataset loaded with 2550 samples
INFO 2025-05-20 11:22:01,592 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 661/11824 [00:01<00:28, 394.87 examples/s]
Map (num_proc=8):   9%|9         | 1070/11824 [00:01<00:15, 679.08 examples/s]
Map (num_proc=8):  18%|#8        | 2149/11824 [00:02<00:11, 860.34 examples/s]
Map (num_proc=8):  25%|##5       | 2956/11824 [00:02<00:06, 1339.05 examples/s]
Map (num_proc=8):  31%|###       | 3618/11824 [00:04<00:08, 977.42 examples/s] 
Map (num_proc=8):  38%|###7      | 4434/11824 [00:04<00:05, 1424.60 examples/s]
Map (num_proc=8):  43%|####3     | 5113/11824 [00:05<00:06, 1040.59 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:05<00:04, 1459.68 examples/s]
Map (num_proc=8):  56%|#####5    | 6598/11824 [00:06<00:04, 1067.24 examples/s]
Map (num_proc=8):  62%|######2   | 7390/11824 [00:06<00:03, 1473.50 examples/s]
Map (num_proc=8):  68%|######8   | 8082/11824 [00:07<00:03, 1078.01 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:07<00:02, 1477.38 examples/s]
Map (num_proc=8):  81%|########  | 9550/11824 [00:08<00:02, 1080.95 examples/s]
Map (num_proc=8):  84%|########3 | 9928/11824 [00:08<00:01, 1235.82 examples/s]
Map (num_proc=8):  93%|#########3| 11050/11824 [00:09<00:00, 1199.49 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1164.34 examples/s]INFO 2025-05-20 11:22:13,619 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_1__How_OpenShift_Virtualization_architecture_works_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 588 examples [00:00, 49493.29 examples/s]INFO 2025-05-20 11:22:13,663 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:22:13,663 instructlab.sdg.datamixing:161: Dataset loaded with 588 samples
INFO 2025-05-20 11:22:13,663 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 620/11824 [00:02<00:39, 286.45 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:02<00:12, 799.44 examples/s]
Map (num_proc=8):  18%|#7        | 2082/11824 [00:03<00:16, 589.33 examples/s]
Map (num_proc=8):  21%|##        | 2478/11824 [00:03<00:12, 754.86 examples/s]
Map (num_proc=8):  30%|##9       | 3511/11824 [00:05<00:11, 750.47 examples/s]
Map (num_proc=8):  33%|###3      | 3956/11824 [00:05<00:08, 917.35 examples/s]
Map (num_proc=8):  43%|####2     | 5061/11824 [00:06<00:07, 858.37 examples/s]
Map (num_proc=8):  46%|####5     | 5434/11824 [00:06<00:06, 989.81 examples/s]
Map (num_proc=8):  55%|#####5    | 6544/11824 [00:08<00:05, 891.81 examples/s]
Map (num_proc=8):  62%|######2   | 7390/11824 [00:08<00:03, 1231.82 examples/s]
Map (num_proc=8):  68%|######7   | 8013/11824 [00:09<00:04, 856.23 examples/s] 
Map (num_proc=8):  71%|#######   | 8390/11824 [00:09<00:03, 986.79 examples/s]
Map (num_proc=8):  80%|########  | 9493/11824 [00:11<00:02, 888.25 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:11<00:01, 1228.21 examples/s]
Map (num_proc=8):  93%|#########2| 10995/11824 [00:12<00:00, 905.37 examples/s] 
Map (num_proc=8): 100%|##########| 11824/11824 [00:12<00:00, 1265.96 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:13<00:00, 902.59 examples/s] INFO 2025-05-20 11:22:28,900 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_7_1__About_the_OpenShift_Container_Platform_dashboards_page_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 332 examples [00:00, 56944.01 examples/s]INFO 2025-05-20 11:22:28,945 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:22:28,946 instructlab.sdg.datamixing:161: Dataset loaded with 332 samples
INFO 2025-05-20 11:22:28,946 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 643/11824 [00:01<00:30, 369.07 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:01<00:10, 959.56 examples/s]
Map (num_proc=8):  18%|#7        | 2099/11824 [00:02<00:13, 747.69 examples/s]
Map (num_proc=8):  25%|##5       | 2956/11824 [00:03<00:07, 1245.48 examples/s]
Map (num_proc=8):  30%|###       | 3569/11824 [00:04<00:09, 901.65 examples/s] 
Map (num_proc=8):  33%|###3      | 3956/11824 [00:04<00:07, 1082.10 examples/s]
Map (num_proc=8):  43%|####2     | 5030/11824 [00:05<00:06, 1028.08 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:05<00:04, 1477.39 examples/s]
Map (num_proc=8):  55%|#####4    | 6502/11824 [00:06<00:05, 1028.37 examples/s]
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:06<00:04, 1196.38 examples/s]
Map (num_proc=8):  67%|######7   | 7967/11824 [00:07<00:03, 1084.88 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:07<00:01, 1533.25 examples/s]
Map (num_proc=8):  80%|#######9  | 9453/11824 [00:09<00:02, 1061.39 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:09<00:00, 1506.61 examples/s]
Map (num_proc=8):  93%|#########2| 10948/11824 [00:10<00:00, 1100.00 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1126.98 examples/s]INFO 2025-05-20 11:22:41,331 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_4__Deploying_a_virtual_machine_template_to_a_custom_namespace_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 363 examples [00:00, 62696.93 examples/s]INFO 2025-05-20 11:22:41,376 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:22:41,377 instructlab.sdg.datamixing:161: Dataset loaded with 363 samples
INFO 2025-05-20 11:22:41,377 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 673/11824 [00:01<00:20, 538.97 examples/s]
Map (num_proc=8):  18%|#8        | 2159/11824 [00:02<00:08, 1150.78 examples/s]
Map (num_proc=8):  31%|###       | 3639/11824 [00:02<00:05, 1439.02 examples/s]
Map (num_proc=8):  43%|####3     | 5118/11824 [00:03<00:04, 1585.81 examples/s]
Map (num_proc=8):  56%|#####5    | 6590/11824 [00:04<00:03, 1654.70 examples/s]
Map (num_proc=8):  68%|######8   | 8068/11824 [00:05<00:02, 1712.32 examples/s]
Map (num_proc=8):  81%|########  | 9559/11824 [00:06<00:01, 1752.18 examples/s]
Map (num_proc=8):  94%|#########3| 11063/11824 [00:06<00:00, 1842.35 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1645.01 examples/s]INFO 2025-05-20 11:22:50,332 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_13_4__Additional_resources_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 6401 examples [00:00, 71578.70 examples/s]INFO 2025-05-20 11:22:50,455 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:22:50,455 instructlab.sdg.datamixing:161: Dataset loaded with 6401 samples
INFO 2025-05-20 11:22:50,455 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 624/11824 [00:01<00:32, 341.89 examples/s]
Map (num_proc=8):   9%|8         | 1028/11824 [00:02<00:19, 559.15 examples/s]
Map (num_proc=8):  18%|#7        | 2107/11824 [00:03<00:12, 778.71 examples/s]
Map (num_proc=8):  21%|##1       | 2509/11824 [00:03<00:10, 908.86 examples/s]
Map (num_proc=8):  30%|###       | 3580/11824 [00:04<00:08, 946.79 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:04<00:05, 1296.07 examples/s]
Map (num_proc=8):  43%|####2     | 5083/11824 [00:05<00:06, 970.51 examples/s] 
Map (num_proc=8):  50%|#####     | 5912/11824 [00:05<00:04, 1299.02 examples/s]
Map (num_proc=8):  56%|#####5    | 6569/11824 [00:07<00:05, 984.52 examples/s] 
Map (num_proc=8):  59%|#####8    | 6937/11824 [00:07<00:04, 1063.33 examples/s]
Map (num_proc=8):  68%|######7   | 7985/11824 [00:08<00:03, 1013.97 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:08<00:02, 1354.15 examples/s]
Map (num_proc=8):  81%|########  | 9527/11824 [00:09<00:02, 1019.60 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:09<00:01, 1324.02 examples/s]
Map (num_proc=8):  93%|#########2| 10956/11824 [00:10<00:00, 1022.37 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:11<00:00, 1045.66 examples/s]INFO 2025-05-20 11:23:03,718 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_1_4__Reviewing_top_consumers_for_projects__virtual_machines__and_nodes_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2145 examples [00:00, 115942.40 examples/s]INFO 2025-05-20 11:23:03,766 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:23:03,766 instructlab.sdg.datamixing:161: Dataset loaded with 2145 samples
INFO 2025-05-20 11:23:03,766 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 714/11824 [00:01<00:19, 576.85 examples/s]
Map (num_proc=8):  19%|#8        | 2197/11824 [00:02<00:08, 1170.73 examples/s]
Map (num_proc=8):  31%|###       | 3654/11824 [00:02<00:05, 1419.95 examples/s]
Map (num_proc=8):  43%|####3     | 5136/11824 [00:03<00:04, 1576.72 examples/s]
Map (num_proc=8):  56%|#####5    | 6613/11824 [00:04<00:03, 1654.51 examples/s]
Map (num_proc=8):  68%|######8   | 8074/11824 [00:05<00:02, 1699.05 examples/s]
Map (num_proc=8):  81%|########  | 9553/11824 [00:06<00:01, 1734.02 examples/s]
Map (num_proc=8):  93%|#########3| 11046/11824 [00:06<00:00, 1810.81 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1642.74 examples/s]INFO 2025-05-20 11:23:12,689 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_7__Accessing_virtual_machine_consoles_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 5911 examples [00:00, 85651.57 examples/s]INFO 2025-05-20 11:23:12,799 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:23:12,799 instructlab.sdg.datamixing:161: Dataset loaded with 5911 samples
INFO 2025-05-20 11:23:12,799 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 672/11824 [00:01<00:23, 484.46 examples/s]
Map (num_proc=8):  18%|#8        | 2166/11824 [00:02<00:09, 1038.04 examples/s]
Map (num_proc=8):  31%|###       | 3644/11824 [00:03<00:06, 1272.17 examples/s]
Map (num_proc=8):  43%|####3     | 5112/11824 [00:04<00:04, 1402.50 examples/s]
Map (num_proc=8):  56%|#####5    | 6602/11824 [00:05<00:03, 1479.76 examples/s]
Map (num_proc=8):  68%|######8   | 8080/11824 [00:05<00:02, 1521.84 examples/s]
Map (num_proc=8):  81%|########  | 9570/11824 [00:06<00:01, 1545.41 examples/s]
Map (num_proc=8):  94%|#########3| 11061/11824 [00:07<00:00, 1625.09 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1467.74 examples/s]INFO 2025-05-20 11:23:22,679 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_5__About_the_hostpath_provisioner_operator_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 398 examples [00:00, 63537.97 examples/s]INFO 2025-05-20 11:23:22,723 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:23:22,723 instructlab.sdg.datamixing:161: Dataset loaded with 398 samples
INFO 2025-05-20 11:23:22,723 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 702/11824 [00:01<00:25, 439.75 examples/s]
Map (num_proc=8):  18%|#8        | 2171/11824 [00:02<00:10, 879.05 examples/s]
Map (num_proc=8):  31%|###       | 3641/11824 [00:03<00:07, 1062.96 examples/s]
Map (num_proc=8):  43%|####3     | 5122/11824 [00:04<00:05, 1166.41 examples/s]
Map (num_proc=8):  56%|#####5    | 6565/11824 [00:06<00:04, 1210.88 examples/s]
Map (num_proc=8):  68%|######8   | 8056/11824 [00:07<00:02, 1256.87 examples/s]
Map (num_proc=8):  80%|########  | 9491/11824 [00:08<00:01, 1265.74 examples/s]
Map (num_proc=8):  93%|#########3| 11037/11824 [00:09<00:00, 1335.06 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:09<00:00, 1225.33 examples/s]INFO 2025-05-20 11:23:34,212 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1_4__Using_the_CLI_to_create_a_virtual_machine_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 6574 examples [00:00, 70353.11 examples/s]INFO 2025-05-20 11:23:34,341 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:23:34,341 instructlab.sdg.datamixing:161: Dataset loaded with 6574 samples
INFO 2025-05-20 11:23:34,341 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 638/11824 [00:01<00:31, 358.44 examples/s]
Map (num_proc=8):   9%|8         | 1022/11824 [00:01<00:17, 614.94 examples/s]
Map (num_proc=8):  18%|#8        | 2136/11824 [00:03<00:12, 805.51 examples/s]
Map (num_proc=8):  21%|##1       | 2539/11824 [00:03<00:09, 999.03 examples/s]
Map (num_proc=8):  31%|###       | 3614/11824 [00:04<00:08, 968.34 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:04<00:05, 1383.41 examples/s]
Map (num_proc=8):  43%|####2     | 5081/11824 [00:05<00:06, 982.76 examples/s] 
Map (num_proc=8):  50%|#####     | 5912/11824 [00:05<00:04, 1400.28 examples/s]
Map (num_proc=8):  55%|#####5    | 6561/11824 [00:06<00:05, 992.02 examples/s] 
Map (num_proc=8):  62%|######2   | 7390/11824 [00:06<00:03, 1391.40 examples/s]
Map (num_proc=8):  68%|######7   | 8037/11824 [00:08<00:03, 1000.45 examples/s]
Map (num_proc=8):  71%|#######   | 8390/11824 [00:08<00:03, 1118.11 examples/s]
Map (num_proc=8):  81%|########  | 9519/11824 [00:09<00:02, 1051.32 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:09<00:01, 1439.94 examples/s]
Map (num_proc=8):  93%|#########3| 11022/11824 [00:10<00:00, 1084.95 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1084.26 examples/s]INFO 2025-05-20 11:23:47,208 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_14_2__Collecting_data_about_virtual_machines_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 956 examples [00:00, 84296.98 examples/s]INFO 2025-05-20 11:23:47,257 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:23:47,257 instructlab.sdg.datamixing:161: Dataset loaded with 956 samples
INFO 2025-05-20 11:23:47,257 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 700/11824 [00:01<00:22, 485.67 examples/s]
Map (num_proc=8):  18%|#8        | 2178/11824 [00:02<00:09, 981.74 examples/s]
Map (num_proc=8):  31%|###       | 3657/11824 [00:03<00:06, 1193.37 examples/s]
Map (num_proc=8):  43%|####3     | 5136/11824 [00:04<00:05, 1301.44 examples/s]
Map (num_proc=8):  56%|#####6    | 6623/11824 [00:05<00:03, 1368.07 examples/s]
Map (num_proc=8):  69%|######8   | 8108/11824 [00:06<00:02, 1408.85 examples/s]
Map (num_proc=8):  81%|########1 | 9585/11824 [00:07<00:01, 1432.82 examples/s]
Map (num_proc=8):  94%|#########3| 11077/11824 [00:08<00:00, 1491.46 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1363.40 examples/s]INFO 2025-05-20 11:23:57,706 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_14_6__Removing_the_VirtIO_container_disk_from_a_virtual_machine_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2502 examples [00:00, 69978.72 examples/s]INFO 2025-05-20 11:23:57,777 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:23:57,778 instructlab.sdg.datamixing:161: Dataset loaded with 2502 samples
INFO 2025-05-20 11:23:57,778 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 672/11824 [00:01<00:23, 476.08 examples/s]
Map (num_proc=8):  18%|#8        | 2144/11824 [00:02<00:09, 1005.56 examples/s]
Map (num_proc=8):  31%|###       | 3610/11824 [00:03<00:06, 1255.04 examples/s]
Map (num_proc=8):  43%|####2     | 5083/11824 [00:04<00:04, 1382.95 examples/s]
Map (num_proc=8):  55%|#####5    | 6551/11824 [00:05<00:03, 1457.10 examples/s]
Map (num_proc=8):  68%|######7   | 8015/11824 [00:06<00:02, 1501.56 examples/s]
Map (num_proc=8):  80%|########  | 9495/11824 [00:06<00:01, 1536.20 examples/s]
Map (num_proc=8):  93%|#########3| 11000/11824 [00:07<00:00, 1610.89 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1454.80 examples/s]INFO 2025-05-20 11:24:07,800 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_14_1__Collecting_data_about_your_environment_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 925 examples [00:00, 92789.90 examples/s]INFO 2025-05-20 11:24:07,846 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:24:07,846 instructlab.sdg.datamixing:161: Dataset loaded with 925 samples
INFO 2025-05-20 11:24:07,846 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 683/11824 [00:01<00:24, 459.41 examples/s]
Map (num_proc=8):  18%|#8        | 2171/11824 [00:02<00:10, 944.63 examples/s]
Map (num_proc=8):  31%|###       | 3658/11824 [00:03<00:07, 1148.47 examples/s]
Map (num_proc=8):  43%|####3     | 5128/11824 [00:04<00:05, 1248.05 examples/s]
Map (num_proc=8):  56%|#####5    | 6620/11824 [00:05<00:03, 1317.12 examples/s]
Map (num_proc=8):  68%|######8   | 8094/11824 [00:06<00:02, 1354.24 examples/s]
Map (num_proc=8):  81%|########  | 9574/11824 [00:07<00:01, 1379.89 examples/s]
Map (num_proc=8):  94%|#########3| 11072/11824 [00:08<00:00, 1432.21 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1314.11 examples/s]INFO 2025-05-20 11:24:18,660 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_3_1__Before_you_begin_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 152 examples [00:00, 45279.42 examples/s]INFO 2025-05-20 11:24:18,699 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:24:18,699 instructlab.sdg.datamixing:161: Dataset loaded with 152 samples
INFO 2025-05-20 11:24:18,699 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   7%|6         | 785/11824 [00:00<00:13, 830.50 examples/s]
Map (num_proc=8):  19%|#9        | 2260/11824 [00:01<00:05, 1643.30 examples/s]
Map (num_proc=8):  32%|###1      | 3738/11824 [00:02<00:03, 2031.30 examples/s]
Map (num_proc=8):  44%|####4     | 5209/11824 [00:02<00:02, 2240.13 examples/s]
Map (num_proc=8):  57%|#####6    | 6689/11824 [00:03<00:02, 2369.86 examples/s]
Map (num_proc=8):  69%|######8   | 8153/11824 [00:03<00:01, 2446.46 examples/s]
Map (num_proc=8):  82%|########1 | 9648/11824 [00:04<00:00, 2513.72 examples/s]
Map (num_proc=8):  94%|#########4| 11141/11824 [00:04<00:00, 2653.07 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:05<00:00, 2307.94 examples/s]INFO 2025-05-20 11:24:25,478 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_4_3__Deploying_the_OpenShift_Virtualization_Operator_by_using_the_CLI_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2888 examples [00:00, 66701.27 examples/s]INFO 2025-05-20 11:24:25,557 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:24:25,557 instructlab.sdg.datamixing:161: Dataset loaded with 2888 samples
INFO 2025-05-20 11:24:25,557 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 682/11824 [00:01<00:27, 404.72 examples/s]
Map (num_proc=8):   9%|9         | 1079/11824 [00:01<00:15, 679.97 examples/s]
Map (num_proc=8):  18%|#8        | 2164/11824 [00:02<00:11, 875.01 examples/s]
Map (num_proc=8):  25%|##5       | 2956/11824 [00:02<00:06, 1337.81 examples/s]
Map (num_proc=8):  31%|###       | 3636/11824 [00:03<00:08, 1005.30 examples/s]
Map (num_proc=8):  34%|###3      | 4004/11824 [00:04<00:06, 1163.49 examples/s]
Map (num_proc=8):  43%|####3     | 5122/11824 [00:05<00:05, 1131.62 examples/s]
Map (num_proc=8):  46%|####6     | 5482/11824 [00:05<00:04, 1270.49 examples/s]
Map (num_proc=8):  56%|#####5    | 6603/11824 [00:06<00:04, 1188.41 examples/s]
Map (num_proc=8):  59%|#####8    | 6954/11824 [00:06<00:03, 1315.57 examples/s]
Map (num_proc=8):  68%|######8   | 8073/11824 [00:07<00:03, 1214.09 examples/s]
Map (num_proc=8):  71%|#######1  | 8428/11824 [00:07<00:02, 1341.61 examples/s]
Map (num_proc=8):  81%|########  | 9560/11824 [00:08<00:01, 1234.11 examples/s]
Map (num_proc=8):  84%|########3 | 9906/11824 [00:08<00:01, 1355.04 examples/s]
Map (num_proc=8):  94%|#########3| 11065/11824 [00:09<00:00, 1334.75 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:09<00:00, 1195.67 examples/s]INFO 2025-05-20 11:24:37,448 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_8_4__Creating_a_container_using_virtctl_guestfs_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 444 examples [00:00, 76759.86 examples/s]INFO 2025-05-20 11:24:37,496 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:24:37,496 instructlab.sdg.datamixing:161: Dataset loaded with 444 samples
INFO 2025-05-20 11:24:37,496 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 728/11824 [00:01<00:19, 571.19 examples/s]
Map (num_proc=8):  10%|#         | 1215/11824 [00:01<00:11, 950.94 examples/s]
Map (num_proc=8):  19%|#8        | 2189/11824 [00:02<00:08, 1129.38 examples/s]
Map (num_proc=8):  23%|##2       | 2715/11824 [00:02<00:06, 1444.25 examples/s]
Map (num_proc=8):  31%|###1      | 3685/11824 [00:03<00:05, 1375.93 examples/s]
Map (num_proc=8):  36%|###5      | 4240/11824 [00:03<00:04, 1688.98 examples/s]
Map (num_proc=8):  44%|####3     | 5167/11824 [00:03<00:04, 1483.24 examples/s]
Map (num_proc=8):  48%|####8     | 5709/11824 [00:04<00:03, 1728.56 examples/s]
Map (num_proc=8):  56%|#####6    | 6641/11824 [00:04<00:03, 1511.12 examples/s]
Map (num_proc=8):  60%|######    | 7145/11824 [00:04<00:02, 1757.86 examples/s]
Map (num_proc=8):  69%|######8   | 8132/11824 [00:05<00:02, 1562.49 examples/s]
Map (num_proc=8):  73%|#######2  | 8622/11824 [00:05<00:01, 1802.33 examples/s]
Map (num_proc=8):  81%|########1 | 9615/11824 [00:06<00:01, 1549.79 examples/s]
Map (num_proc=8):  86%|########5 | 10116/11824 [00:06<00:00, 1794.88 examples/s]
Map (num_proc=8):  94%|#########3| 11112/11824 [00:07<00:00, 1624.42 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1526.46 examples/s]INFO 2025-05-20 11:24:46,943 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_1_6__Snapshots_and_cloning_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2994 examples [00:00, 78253.11 examples/s]INFO 2025-05-20 11:24:47,021 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:24:47,021 instructlab.sdg.datamixing:161: Dataset loaded with 2994 samples
INFO 2025-05-20 11:24:47,021 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 681/11824 [00:01<00:24, 455.94 examples/s]
Map (num_proc=8):  18%|#8        | 2166/11824 [00:02<00:09, 969.98 examples/s]
Map (num_proc=8):  31%|###       | 3615/11824 [00:03<00:07, 1171.22 examples/s]
Map (num_proc=8):  43%|####3     | 5089/11824 [00:04<00:05, 1277.95 examples/s]
Map (num_proc=8):  56%|#####5    | 6589/11824 [00:05<00:03, 1365.89 examples/s]
Map (num_proc=8):  68%|######8   | 8081/11824 [00:06<00:02, 1409.28 examples/s]
Map (num_proc=8):  81%|########  | 9563/11824 [00:07<00:01, 1433.80 examples/s]
Map (num_proc=8):  93%|#########3| 11039/11824 [00:08<00:00, 1496.71 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1361.05 examples/s]INFO 2025-05-20 11:24:57,616 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_2_1__Editing_a_virtual_machine_in_the_web_console_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 4410 examples [00:00, 89948.75 examples/s]INFO 2025-05-20 11:24:57,706 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:24:57,706 instructlab.sdg.datamixing:161: Dataset loaded with 4410 samples
INFO 2025-05-20 11:24:57,706 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|4         | 584/11824 [00:01<00:27, 415.67 examples/s]
Map (num_proc=8):  17%|#7        | 2061/11824 [00:02<00:10, 970.94 examples/s]
Map (num_proc=8):  31%|###       | 3649/11824 [00:03<00:06, 1259.74 examples/s]
Map (num_proc=8):  43%|####3     | 5128/11824 [00:04<00:04, 1367.15 examples/s]
Map (num_proc=8):  56%|#####5    | 6604/11824 [00:05<00:03, 1414.11 examples/s]
Map (num_proc=8):  68%|######8   | 8096/11824 [00:06<00:02, 1479.72 examples/s]
Map (num_proc=8):  81%|########  | 9575/11824 [00:07<00:01, 1485.47 examples/s]
Map (num_proc=8):  94%|#########3| 11080/11824 [00:08<00:00, 1565.40 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1415.45 examples/s]INFO 2025-05-20 11:25:07,923 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_9_1__About_OpenShift_Container_Platform_monitoring_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 3241 examples [00:00, 88533.18 examples/s]INFO 2025-05-20 11:25:07,985 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:25:07,985 instructlab.sdg.datamixing:161: Dataset loaded with 3241 samples
INFO 2025-05-20 11:25:07,985 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 607/11824 [00:01<00:25, 441.02 examples/s]
Map (num_proc=8):  18%|#8        | 2179/11824 [00:02<00:08, 1073.32 examples/s]
Map (num_proc=8):  30%|###       | 3572/11824 [00:03<00:06, 1255.50 examples/s]
Map (num_proc=8):  43%|####2     | 5050/11824 [00:04<00:04, 1404.62 examples/s]
Map (num_proc=8):  56%|#####5    | 6613/11824 [00:04<00:03, 1514.68 examples/s]
Map (num_proc=8):  69%|######8   | 8106/11824 [00:05<00:02, 1548.46 examples/s]
Map (num_proc=8):  81%|########1 | 9586/11824 [00:06<00:01, 1575.60 examples/s]
Map (num_proc=8):  93%|#########2| 10996/11824 [00:07<00:00, 1630.04 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:07<00:00, 1485.67 examples/s]INFO 2025-05-20 11:25:17,801 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_7_1__About_workload_security_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 585 examples [00:00, 82351.66 examples/s]INFO 2025-05-20 11:25:17,848 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:25:17,848 instructlab.sdg.datamixing:161: Dataset loaded with 585 samples
INFO 2025-05-20 11:25:17,848 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 731/11824 [00:01<00:16, 654.34 examples/s]
Map (num_proc=8):  19%|#8        | 2209/11824 [00:01<00:07, 1342.86 examples/s]
Map (num_proc=8):  31%|###1      | 3682/11824 [00:02<00:04, 1660.63 examples/s]
Map (num_proc=8):  44%|####3     | 5164/11824 [00:03<00:03, 1838.59 examples/s]
Map (num_proc=8):  56%|#####6    | 6644/11824 [00:03<00:02, 1940.78 examples/s]
Map (num_proc=8):  69%|######8   | 8124/11824 [00:04<00:01, 2006.75 examples/s]
Map (num_proc=8):  81%|########1 | 9601/11824 [00:05<00:01, 2049.15 examples/s]
Map (num_proc=8):  93%|#########3| 11030/11824 [00:05<00:00, 2133.12 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:06<00:00, 1906.09 examples/s]INFO 2025-05-20 11:25:25,825 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_1_5__Additional_resources_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 878 examples [00:00, 60093.65 examples/s]INFO 2025-05-20 11:25:25,878 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:25:25,878 instructlab.sdg.datamixing:161: Dataset loaded with 878 samples
INFO 2025-05-20 11:25:25,878 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 608/11824 [00:02<00:39, 281.97 examples/s]
Map (num_proc=8):   8%|8         | 1000/11824 [00:02<00:21, 505.54 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:02<00:12, 819.32 examples/s]
Map (num_proc=8):  18%|#7        | 2083/11824 [00:03<00:16, 608.12 examples/s]
Map (num_proc=8):  21%|##        | 2478/11824 [00:03<00:12, 740.24 examples/s]
Map (num_proc=8):  30%|###       | 3558/11824 [00:05<00:10, 787.20 examples/s]
Map (num_proc=8):  33%|###3      | 3956/11824 [00:05<00:08, 879.25 examples/s]
Map (num_proc=8):  43%|####2     | 5043/11824 [00:06<00:07, 855.97 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:07<00:05, 1161.01 examples/s]
Map (num_proc=8):  55%|#####5    | 6519/11824 [00:08<00:06, 841.63 examples/s] 
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:08<00:05, 927.19 examples/s]
Map (num_proc=8):  68%|######7   | 7997/11824 [00:09<00:04, 880.48 examples/s]
Map (num_proc=8):  71%|#######   | 8390/11824 [00:10<00:03, 960.81 examples/s]
Map (num_proc=8):  80%|########  | 9473/11824 [00:11<00:02, 900.39 examples/s]
Map (num_proc=8):  83%|########3 | 9868/11824 [00:11<00:01, 1040.82 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:11<00:01, 1237.91 examples/s]
Map (num_proc=8):  93%|#########2| 10985/11824 [00:12<00:00, 899.14 examples/s] 
Map (num_proc=8): 100%|##########| 11824/11824 [00:13<00:00, 888.14 examples/s]INFO 2025-05-20 11:25:41,377 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_3__About_the_cdi_operator_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 570 examples [00:00, 96518.10 examples/s]INFO 2025-05-20 11:25:41,420 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:25:41,420 instructlab.sdg.datamixing:161: Dataset loaded with 570 samples
INFO 2025-05-20 11:25:41,420 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 740/11824 [00:01<00:15, 702.44 examples/s]
Map (num_proc=8):  19%|#8        | 2212/11824 [00:01<00:06, 1391.02 examples/s]
Map (num_proc=8):  31%|###1      | 3689/11824 [00:02<00:04, 1703.33 examples/s]
Map (num_proc=8):  44%|####3     | 5144/11824 [00:03<00:03, 1861.51 examples/s]
Map (num_proc=8):  56%|#####5    | 6619/11824 [00:03<00:02, 1967.91 examples/s]
Map (num_proc=8):  68%|######8   | 8093/11824 [00:04<00:01, 2027.83 examples/s]
Map (num_proc=8):  81%|########  | 9561/11824 [00:05<00:01, 2063.08 examples/s]
Map (num_proc=8):  93%|#########3| 11049/11824 [00:05<00:00, 2163.57 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:06<00:00, 1940.85 examples/s]INFO 2025-05-20 11:25:49,240 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1_5__Virtual_machine_storage_volume_types_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 5143 examples [00:00, 72227.93 examples/s]INFO 2025-05-20 11:25:49,346 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:25:49,346 instructlab.sdg.datamixing:161: Dataset loaded with 5143 samples
INFO 2025-05-20 11:25:49,346 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 651/11824 [00:01<00:28, 390.80 examples/s]
Map (num_proc=8):   9%|8         | 1018/11824 [00:01<00:16, 639.61 examples/s]
Map (num_proc=8):  18%|#8        | 2142/11824 [00:02<00:10, 884.72 examples/s]
Map (num_proc=8):  21%|##        | 2478/11824 [00:02<00:08, 1041.59 examples/s]
Map (num_proc=8):  31%|###       | 3607/11824 [00:03<00:07, 1070.68 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:04<00:04, 1530.34 examples/s]
Map (num_proc=8):  43%|####3     | 5091/11824 [00:05<00:06, 1102.23 examples/s]
Map (num_proc=8):  46%|####5     | 5434/11824 [00:05<00:05, 1234.90 examples/s]
Map (num_proc=8):  56%|#####5    | 6593/11824 [00:06<00:04, 1188.31 examples/s]
Map (num_proc=8):  62%|######2   | 7390/11824 [00:06<00:02, 1594.04 examples/s]
Map (num_proc=8):  68%|######8   | 8048/11824 [00:07<00:03, 1143.06 examples/s]
Map (num_proc=8):  71%|#######1  | 8428/11824 [00:07<00:02, 1304.75 examples/s]
Map (num_proc=8):  81%|########  | 9522/11824 [00:08<00:01, 1186.89 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:08<00:00, 1619.73 examples/s]
Map (num_proc=8):  93%|#########3| 11037/11824 [00:09<00:00, 1236.05 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:09<00:00, 1197.18 examples/s]INFO 2025-05-20 11:26:01,270 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1__Creating_virtual_machines_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 6876 examples [00:00, 72249.28 examples/s]INFO 2025-05-20 11:26:01,404 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:26:01,404 instructlab.sdg.datamixing:161: Dataset loaded with 6876 samples
INFO 2025-05-20 11:26:01,404 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 632/11824 [00:01<00:30, 372.08 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:01<00:10, 997.22 examples/s]
Map (num_proc=8):  18%|#8        | 2139/11824 [00:02<00:12, 791.52 examples/s]
Map (num_proc=8):  22%|##1       | 2558/11824 [00:02<00:09, 1009.36 examples/s]
Map (num_proc=8):  31%|###       | 3619/11824 [00:04<00:08, 1008.58 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:04<00:05, 1448.10 examples/s]
Map (num_proc=8):  43%|####3     | 5096/11824 [00:05<00:06, 1045.20 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:05<00:04, 1469.18 examples/s]
Map (num_proc=8):  56%|#####5    | 6586/11824 [00:06<00:04, 1066.11 examples/s]
Map (num_proc=8):  62%|######2   | 7390/11824 [00:06<00:03, 1473.27 examples/s]
Map (num_proc=8):  68%|######8   | 8058/11824 [00:07<00:03, 1069.50 examples/s]
Map (num_proc=8):  71%|#######1  | 8427/11824 [00:07<00:02, 1214.75 examples/s]
Map (num_proc=8):  81%|########  | 9545/11824 [00:08<00:01, 1142.44 examples/s]
Map (num_proc=8):  84%|########3 | 9907/11824 [00:08<00:01, 1279.40 examples/s]
Map (num_proc=8):  93%|#########3| 11050/11824 [00:09<00:00, 1245.98 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1158.48 examples/s]INFO 2025-05-20 11:26:13,668 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_4_1__Deleting_a_virtual_machine_using_the_web_console_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1544 examples [00:00, 121001.60 examples/s]INFO 2025-05-20 11:26:13,722 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:26:13,722 instructlab.sdg.datamixing:161: Dataset loaded with 1544 samples
INFO 2025-05-20 11:26:13,722 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|6         | 738/11824 [00:01<00:17, 628.33 examples/s]
Map (num_proc=8):  19%|#8        | 2205/11824 [00:01<00:07, 1243.21 examples/s]
Map (num_proc=8):  31%|###1      | 3695/11824 [00:02<00:05, 1528.02 examples/s]
Map (num_proc=8):  44%|####3     | 5175/11824 [00:03<00:03, 1675.18 examples/s]
Map (num_proc=8):  56%|#####6    | 6654/11824 [00:04<00:02, 1758.57 examples/s]
Map (num_proc=8):  69%|######8   | 8131/11824 [00:05<00:02, 1811.30 examples/s]
Map (num_proc=8):  80%|########  | 9477/11824 [00:05<00:01, 1791.63 examples/s]
Map (num_proc=8):  93%|#########3| 10998/11824 [00:06<00:00, 1906.06 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:06<00:00, 1738.00 examples/s]INFO 2025-05-20 11:26:22,294 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_12_1_1__About_node_maintenance_mode_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 478 examples [00:00, 68021.89 examples/s]INFO 2025-05-20 11:26:22,333 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:26:22,333 instructlab.sdg.datamixing:161: Dataset loaded with 478 samples
INFO 2025-05-20 11:26:22,333 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|4         | 575/11824 [00:01<00:33, 332.29 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:01<00:11, 926.30 examples/s]
Map (num_proc=8):  18%|#8        | 2143/11824 [00:02<00:12, 783.53 examples/s]
Map (num_proc=8):  21%|##        | 2478/11824 [00:03<00:10, 895.67 examples/s]
Map (num_proc=8):  31%|###       | 3616/11824 [00:04<00:08, 991.65 examples/s]
Map (num_proc=8):  33%|###3      | 3956/11824 [00:04<00:07, 1095.50 examples/s]
Map (num_proc=8):  43%|####3     | 5089/11824 [00:05<00:06, 1086.49 examples/s]
Map (num_proc=8):  46%|####5     | 5434/11824 [00:05<00:05, 1173.38 examples/s]
Map (num_proc=8):  56%|#####5    | 6569/11824 [00:06<00:04, 1137.45 examples/s]
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:06<00:04, 1216.43 examples/s]
Map (num_proc=8):  68%|######8   | 8062/11824 [00:07<00:03, 1166.00 examples/s]
Map (num_proc=8):  71%|#######   | 8390/11824 [00:08<00:02, 1232.32 examples/s]
Map (num_proc=8):  81%|########  | 9543/11824 [00:09<00:01, 1181.79 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:09<00:00, 1501.62 examples/s]
Map (num_proc=8):  93%|#########3| 11053/11824 [00:10<00:00, 1180.63 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:10<00:00, 1118.60 examples/s]INFO 2025-05-20 11:26:34,900 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_14_3__Using_the_must_gather_tool_for_OpenShift_Virtualization_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1043 examples [00:00, 90583.90 examples/s]INFO 2025-05-20 11:26:34,937 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:26:34,938 instructlab.sdg.datamixing:161: Dataset loaded with 1043 samples
INFO 2025-05-20 11:26:34,938 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 679/11824 [00:01<00:24, 459.77 examples/s]
Map (num_proc=8):  18%|#8        | 2171/11824 [00:02<00:10, 957.49 examples/s]
Map (num_proc=8):  31%|###       | 3657/11824 [00:03<00:06, 1168.71 examples/s]
Map (num_proc=8):  43%|####3     | 5142/11824 [00:04<00:05, 1280.27 examples/s]
Map (num_proc=8):  56%|#####6    | 6632/11824 [00:05<00:03, 1346.03 examples/s]
Map (num_proc=8):  69%|######8   | 8108/11824 [00:06<00:02, 1383.32 examples/s]
Map (num_proc=8):  81%|########1 | 9585/11824 [00:07<00:01, 1406.47 examples/s]
Map (num_proc=8):  94%|#########3| 11082/11824 [00:08<00:00, 1468.82 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1338.54 examples/s]INFO 2025-05-20 11:26:45,668 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_6__Uninstalling_OpenShift_Virtualization_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 3858 examples [00:00, 86274.39 examples/s]INFO 2025-05-20 11:26:45,743 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:26:45,743 instructlab.sdg.datamixing:161: Dataset loaded with 3858 samples
INFO 2025-05-20 11:26:45,743 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 708/11824 [00:01<00:21, 506.73 examples/s]
Map (num_proc=8):  19%|#8        | 2189/11824 [00:02<00:09, 1049.03 examples/s]
Map (num_proc=8):  31%|###       | 3664/11824 [00:03<00:06, 1280.83 examples/s]
Map (num_proc=8):  43%|####3     | 5140/11824 [00:04<00:04, 1404.58 examples/s]
Map (num_proc=8):  56%|#####5    | 6621/11824 [00:05<00:03, 1479.41 examples/s]
Map (num_proc=8):  69%|######8   | 8107/11824 [00:05<00:02, 1539.23 examples/s]
Map (num_proc=8):  75%|#######5  | 8868/11824 [00:06<00:01, 1827.34 examples/s]
Map (num_proc=8):  81%|########1 | 9585/11824 [00:06<00:01, 1476.30 examples/s]
Map (num_proc=8):  94%|#########3| 11093/11824 [00:07<00:00, 1598.70 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1470.23 examples/s]INFO 2025-05-20 11:26:55,735 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_14_1__Prerequisites_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2197 examples [00:00, 64390.23 examples/s]INFO 2025-05-20 11:26:55,801 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:26:55,801 instructlab.sdg.datamixing:161: Dataset loaded with 2197 samples
INFO 2025-05-20 11:26:55,801 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 673/11824 [00:01<00:25, 442.26 examples/s]
Map (num_proc=8):   9%|8         | 1025/11824 [00:01<00:15, 682.00 examples/s]
Map (num_proc=8):  18%|#7        | 2090/11824 [00:02<00:10, 969.36 examples/s]
Map (num_proc=8):  21%|##1       | 2503/11824 [00:02<00:08, 1139.90 examples/s]
Map (num_proc=8):  30%|###       | 3576/11824 [00:03<00:06, 1224.81 examples/s]
Map (num_proc=8):  34%|###3      | 3985/11824 [00:03<00:05, 1387.93 examples/s]
Map (num_proc=8):  43%|####2     | 5041/11824 [00:04<00:05, 1306.11 examples/s]
Map (num_proc=8):  46%|####5     | 5434/11824 [00:04<00:04, 1442.85 examples/s]
Map (num_proc=8):  55%|#####5    | 6519/11824 [00:05<00:03, 1367.39 examples/s]
Map (num_proc=8):  58%|#####8    | 6912/11824 [00:05<00:03, 1479.31 examples/s]
Map (num_proc=8):  68%|######7   | 8005/11824 [00:06<00:02, 1414.95 examples/s]
Map (num_proc=8):  71%|#######   | 8390/11824 [00:06<00:02, 1511.71 examples/s]
Map (num_proc=8):  80%|########  | 9481/11824 [00:07<00:01, 1408.76 examples/s]
Map (num_proc=8):  83%|########3 | 9868/11824 [00:07<00:01, 1524.38 examples/s]
Map (num_proc=8):  93%|#########2| 10979/11824 [00:08<00:00, 1503.65 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1342.50 examples/s]INFO 2025-05-20 11:27:06,689 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_3_2__Editing_a_boot_order_list_in_the_web_console_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2348 examples [00:00, 77535.32 examples/s]INFO 2025-05-20 11:27:06,752 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:27:06,752 instructlab.sdg.datamixing:161: Dataset loaded with 2348 samples
INFO 2025-05-20 11:27:06,752 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 644/11824 [00:01<00:24, 463.80 examples/s]
Map (num_proc=8):  18%|#8        | 2170/11824 [00:02<00:09, 1041.39 examples/s]
Map (num_proc=8):  31%|###       | 3623/11824 [00:03<00:06, 1259.77 examples/s]
Map (num_proc=8):  43%|####3     | 5097/11824 [00:04<00:04, 1382.89 examples/s]
Map (num_proc=8):  56%|#####5    | 6584/11824 [00:05<00:03, 1461.16 examples/s]
Map (num_proc=8):  68%|######7   | 8025/11824 [00:06<00:02, 1488.73 examples/s]
Map (num_proc=8):  81%|########  | 9546/11824 [00:06<00:01, 1538.05 examples/s]
Map (num_proc=8):  93%|#########2| 10938/11824 [00:07<00:00, 1574.68 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:08<00:00, 1452.84 examples/s]INFO 2025-05-20 11:27:16,894 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_16_5__Configuring_PXE_booting_for_virtual_machines_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 6621 examples [00:00, 63872.06 examples/s]
Generating train split: 17531 examples [00:00, 85913.89 examples/s]
Generating train split: 17531 examples [00:00, 82428.04 examples/s]INFO 2025-05-20 11:27:17,141 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:27:17,141 instructlab.sdg.datamixing:161: Dataset loaded with 17531 samples


Map (num_proc=8):   0%|          | 0/17531 [00:00<?, ? examples/s]
Map (num_proc=8):   2%|2         | 358/17531 [00:00<00:13, 1259.83 examples/s]
Map (num_proc=8):  12%|#2        | 2105/17531 [00:00<00:02, 6632.71 examples/s]
Map (num_proc=8):  24%|##4       | 4260/17531 [00:00<00:01, 11243.08 examples/s]
Map (num_proc=8):  33%|###3      | 5811/17531 [00:00<00:00, 12160.02 examples/s]
Map (num_proc=8):  42%|####1     | 7354/17531 [00:00<00:00, 11761.48 examples/s]
Map (num_proc=8):  50%|####9     | 8710/17531 [00:00<00:00, 10662.02 examples/s]
Map (num_proc=8):  57%|#####6    | 9938/17531 [00:01<00:00, 10670.00 examples/s]
Map (num_proc=8):  65%|######5   | 11475/17531 [00:01<00:00, 11863.38 examples/s]
Map (num_proc=8):  75%|#######4  | 13062/17531 [00:01<00:00, 12808.68 examples/s]
Map (num_proc=8):  83%|########2 | 14522/17531 [00:01<00:00, 13159.61 examples/s]
Map (num_proc=8):  91%|######### | 15943/17531 [00:01<00:00, 12250.27 examples/s]
Map (num_proc=8):  99%|#########8| 17340/17531 [00:01<00:00, 11252.25 examples/s]
Map (num_proc=8): 100%|##########| 17531/17531 [00:01<00:00, 9822.86 examples/s] INFO 2025-05-20 11:27:20,274 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_6_4_1__Monitoring_OpenShift_Virtualization_upgrade_status_p10.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 949 examples [00:00, 71124.20 examples/s]INFO 2025-05-20 11:27:20,304 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:27:20,304 instructlab.sdg.datamixing:161: Dataset loaded with 949 samples
INFO 2025-05-20 11:27:20,304 instructlab.sdg.datamixing:44: Rebalancing dataset to have 11824 samples ...


Map (num_proc=8):   0%|          | 0/11824 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 656/11824 [00:01<00:26, 415.15 examples/s]
Map (num_proc=8):  12%|#2        | 1478/11824 [00:01<00:10, 965.39 examples/s]
Map (num_proc=8):  18%|#8        | 2144/11824 [00:02<00:11, 876.66 examples/s]
Map (num_proc=8):  21%|##        | 2478/11824 [00:02<00:09, 986.21 examples/s]
Map (num_proc=8):  31%|###       | 3630/11824 [00:03<00:07, 1150.25 examples/s]
Map (num_proc=8):  38%|###7      | 4434/11824 [00:03<00:04, 1507.68 examples/s]
Map (num_proc=8):  43%|####3     | 5105/11824 [00:04<00:05, 1192.30 examples/s]
Map (num_proc=8):  50%|#####     | 5912/11824 [00:04<00:03, 1552.53 examples/s]
Map (num_proc=8):  56%|#####5    | 6591/11824 [00:05<00:04, 1215.58 examples/s]
Map (num_proc=8):  59%|#####8    | 6955/11824 [00:06<00:03, 1292.89 examples/s]
Map (num_proc=8):  68%|######8   | 8067/11824 [00:06<00:02, 1295.28 examples/s]
Map (num_proc=8):  71%|#######1  | 8410/11824 [00:07<00:02, 1350.91 examples/s]
Map (num_proc=8):  81%|########  | 9540/11824 [00:07<00:01, 1339.53 examples/s]
Map (num_proc=8):  88%|########7 | 10346/11824 [00:08<00:00, 1661.24 examples/s]
Map (num_proc=8):  93%|#########3| 11040/11824 [00:08<00:00, 1360.81 examples/s]
Map (num_proc=8): 100%|##########| 11824/11824 [00:09<00:00, 1271.35 examples/s]

Map (num_proc=8):   0%|          | 0/2104277 [00:00<?, ? examples/s]
Map (num_proc=8):   0%|          | 558/2104277 [00:00<15:44, 2227.76 examples/s]
Map (num_proc=8):   0%|          | 1687/2104277 [00:00<06:41, 5242.89 examples/s]
Map (num_proc=8):   0%|          | 2721/2104277 [00:00<05:34, 6287.21 examples/s]
Map (num_proc=8):   0%|          | 3729/2104277 [00:00<05:13, 6698.87 examples/s]
Map (num_proc=8):   0%|          | 4703/2104277 [00:00<04:56, 7074.46 examples/s]
Map (num_proc=8):   0%|          | 5711/2104277 [00:00<04:56, 7072.08 examples/s]
Map (num_proc=8):   0%|          | 6731/2104277 [00:01<04:59, 6999.01 examples/s]
Map (num_proc=8):   0%|          | 7721/2104277 [00:01<04:45, 7330.88 examples/s]
Map (num_proc=8):   0%|          | 8703/2104277 [00:01<04:47, 7282.48 examples/s]
Map (num_proc=8):   0%|          | 9832/2104277 [00:01<05:13, 6682.22 examples/s]
Map (num_proc=8):   1%|          | 10764/2104277 [00:01<05:53, 5924.61 examples/s]
Map (num_proc=8):   1%|          | 11756/2104277 [00:01<06:31, 5346.15 examples/s]
Map (num_proc=8):   1%|          | 12355/2104277 [00:02<07:29, 4649.92 examples/s]
Map (num_proc=8):   1%|          | 13000/2104277 [00:02<07:48, 4466.48 examples/s]
Map (num_proc=8):   1%|          | 13810/2104277 [00:02<06:44, 5165.11 examples/s]
Map (num_proc=8):   1%|          | 14401/2104277 [00:02<07:08, 4876.39 examples/s]
Map (num_proc=8):   1%|          | 15000/2104277 [00:03<16:29, 2111.52 examples/s]
Map (num_proc=8):   1%|          | 15770/2104277 [00:03<12:38, 2752.09 examples/s]
Map (num_proc=8):   1%|          | 16388/2104277 [00:03<11:53, 2925.93 examples/s]
Map (num_proc=8):   1%|          | 17000/2104277 [00:03<11:23, 3054.06 examples/s]
Map (num_proc=8):   1%|          | 17801/2104277 [00:03<08:59, 3867.83 examples/s]
Map (num_proc=8):   1%|          | 18400/2104277 [00:03<08:48, 3950.16 examples/s]
Map (num_proc=8):   1%|          | 19000/2104277 [00:04<09:03, 3837.30 examples/s]
Map (num_proc=8):   1%|          | 20000/2104277 [00:04<08:40, 4006.23 examples/s]
Map (num_proc=8):   1%|          | 20812/2104277 [00:04<07:17, 4767.21 examples/s]
Map (num_proc=8):   1%|1         | 21423/2104277 [00:04<07:30, 4623.15 examples/s]
Map (num_proc=8):   1%|1         | 22000/2104277 [00:04<07:41, 4514.09 examples/s]
Map (num_proc=8):   1%|1         | 22802/2104277 [00:04<06:33, 5289.13 examples/s]
Map (num_proc=8):   1%|1         | 23401/2104277 [00:04<07:00, 4944.23 examples/s]
Map (num_proc=8):   1%|1         | 24000/2104277 [00:05<07:30, 4614.75 examples/s]
Map (num_proc=8):   1%|1         | 24796/2104277 [00:05<06:26, 5387.14 examples/s]
Map (num_proc=8):   1%|1         | 25397/2104277 [00:05<07:18, 4742.74 examples/s]
Map (num_proc=8):   1%|1         | 26000/2104277 [00:05<08:08, 4253.87 examples/s]
Map (num_proc=8):   1%|1         | 26784/2104277 [00:05<06:52, 5038.73 examples/s]
Map (num_proc=8):   1%|1         | 27363/2104277 [00:05<07:34, 4566.39 examples/s]
Map (num_proc=8):   1%|1         | 27958/2104277 [00:05<07:33, 4583.05 examples/s]
Map (num_proc=8):   1%|1         | 28546/2104277 [00:06<09:03, 3817.32 examples/s]
Map (num_proc=8):   1%|1         | 28999/2104277 [00:06<08:45, 3946.68 examples/s]
Map (num_proc=8):   1%|1         | 29745/2104277 [00:06<09:06, 3795.77 examples/s]
Map (num_proc=8):   1%|1         | 30673/2104277 [00:06<07:43, 4474.99 examples/s]
Map (num_proc=8):   2%|1         | 32454/2104277 [00:06<05:04, 6805.13 examples/s]
Map (num_proc=8):   2%|1         | 33387/2104277 [00:06<05:13, 6600.75 examples/s]
Map (num_proc=8):   2%|1         | 34758/2104277 [00:07<04:49, 7160.30 examples/s]
Map (num_proc=8):   2%|1         | 36448/2104277 [00:07<03:55, 8768.42 examples/s]
Map (num_proc=8):   2%|1         | 37745/2104277 [00:07<04:14, 8117.29 examples/s]
Map (num_proc=8):   2%|1         | 38759/2104277 [00:07<07:01, 4901.04 examples/s]
Map (num_proc=8):   2%|1         | 40765/2104277 [00:07<04:52, 7056.42 examples/s]
Map (num_proc=8):   2%|2         | 42763/2104277 [00:08<03:46, 9096.13 examples/s]
Map (num_proc=8):   2%|2         | 44746/2104277 [00:08<03:29, 9811.43 examples/s]
Map (num_proc=8):   2%|2         | 46218/2104277 [00:08<03:17, 10414.65 examples/s]
Map (num_proc=8):   2%|2         | 47650/2104277 [00:08<04:34, 7492.38 examples/s] 
Map (num_proc=8):   2%|2         | 48640/2104277 [00:08<04:43, 7239.05 examples/s]
Map (num_proc=8):   2%|2         | 49689/2104277 [00:09<05:21, 6386.08 examples/s]
Map (num_proc=8):   2%|2         | 50788/2104277 [00:09<05:20, 6403.23 examples/s]
Map (num_proc=8):   2%|2         | 51670/2104277 [00:09<05:01, 6819.18 examples/s]
Map (num_proc=8):   3%|2         | 52633/2104277 [00:09<06:09, 5555.92 examples/s]
Map (num_proc=8):   3%|2         | 53448/2104277 [00:09<05:51, 5839.16 examples/s]
Map (num_proc=8):   3%|2         | 54267/2104277 [00:09<05:58, 5715.79 examples/s]
Map (num_proc=8):   3%|2         | 55136/2104277 [00:10<06:20, 5379.19 examples/s]
Map (num_proc=8):   3%|2         | 55835/2104277 [00:10<07:55, 4308.79 examples/s]
Map (num_proc=8):   3%|2         | 56411/2104277 [00:10<09:42, 3514.66 examples/s]
Map (num_proc=8):   3%|2         | 56833/2104277 [00:10<09:24, 3629.40 examples/s]
Map (num_proc=8):   3%|2         | 57374/2104277 [00:10<09:25, 3622.14 examples/s]
Map (num_proc=8):   3%|2         | 58000/2104277 [00:11<10:16, 3318.54 examples/s]
Map (num_proc=8):   3%|2         | 58942/2104277 [00:11<07:46, 4380.39 examples/s]
Map (num_proc=8):   3%|2         | 59807/2104277 [00:11<06:39, 5118.19 examples/s]
Map (num_proc=8):   3%|2         | 60564/2104277 [00:11<06:37, 5136.51 examples/s]
Map (num_proc=8):   3%|2         | 61599/2104277 [00:11<05:25, 6272.48 examples/s]
Map (num_proc=8):   3%|2         | 62386/2104277 [00:11<06:51, 4966.49 examples/s]
Map (num_proc=8):   3%|3         | 63355/2104277 [00:11<06:15, 5430.26 examples/s]
Map (num_proc=8):   3%|3         | 64199/2104277 [00:12<05:38, 6029.63 examples/s]
Map (num_proc=8):   3%|3         | 64890/2104277 [00:12<05:55, 5743.79 examples/s]
Map (num_proc=8):   3%|3         | 65866/2104277 [00:12<05:33, 6113.31 examples/s]
Map (num_proc=8):   3%|3         | 66763/2104277 [00:12<05:17, 6418.36 examples/s]
Map (num_proc=8):   3%|3         | 67560/2104277 [00:12<06:08, 5520.68 examples/s]
Map (num_proc=8):   3%|3         | 68383/2104277 [00:12<06:02, 5616.94 examples/s]
Map (num_proc=8):   3%|3         | 69141/2104277 [00:12<06:12, 5458.65 examples/s]
Map (num_proc=8):   3%|3         | 69744/2104277 [00:13<06:55, 4899.43 examples/s]
Map (num_proc=8):   3%|3         | 70860/2104277 [00:13<05:27, 6212.71 examples/s]
Map (num_proc=8):   3%|3         | 71683/2104277 [00:13<07:41, 4406.28 examples/s]
Map (num_proc=8):   3%|3         | 72328/2104277 [00:13<07:05, 4775.23 examples/s]
Map (num_proc=8):   3%|3         | 72960/2104277 [00:13<08:06, 4174.65 examples/s]
Map (num_proc=8):   3%|3         | 73566/2104277 [00:13<07:55, 4269.93 examples/s]
Map (num_proc=8):   4%|3         | 74681/2104277 [00:14<05:56, 5693.64 examples/s]
Map (num_proc=8):   4%|3         | 75387/2104277 [00:14<07:30, 4500.03 examples/s]
Map (num_proc=8):   4%|3         | 76732/2104277 [00:14<05:24, 6256.17 examples/s]
Map (num_proc=8):   4%|3         | 77589/2104277 [00:14<06:31, 5180.67 examples/s]
Map (num_proc=8):   4%|3         | 78495/2104277 [00:14<05:46, 5843.29 examples/s]
Map (num_proc=8):   4%|3         | 79362/2104277 [00:14<06:25, 5247.74 examples/s]
Map (num_proc=8):   4%|3         | 80129/2104277 [00:15<05:53, 5728.51 examples/s]
Map (num_proc=8):   4%|3         | 80987/2104277 [00:15<05:57, 5655.76 examples/s]
Map (num_proc=8):   4%|3         | 81724/2104277 [00:15<05:58, 5641.29 examples/s]
Map (num_proc=8):   4%|3         | 82499/2104277 [00:15<05:31, 6105.05 examples/s]
Map (num_proc=8):   4%|3         | 83307/2104277 [00:15<06:33, 5133.07 examples/s]
Map (num_proc=8):   4%|4         | 84215/2104277 [00:15<05:42, 5899.04 examples/s]
Map (num_proc=8):   4%|4         | 84999/2104277 [00:15<06:34, 5118.37 examples/s]
Map (num_proc=8):   4%|4         | 86036/2104277 [00:16<05:50, 5761.45 examples/s]
Map (num_proc=8):   4%|4         | 86872/2104277 [00:16<06:40, 5032.78 examples/s]
Map (num_proc=8):   4%|4         | 87983/2104277 [00:16<05:23, 6237.86 examples/s]
Map (num_proc=8):   4%|4         | 88887/2104277 [00:16<06:18, 5320.92 examples/s]
Map (num_proc=8):   4%|4         | 89600/2104277 [00:16<06:03, 5535.03 examples/s]
Map (num_proc=8):   4%|4         | 90510/2104277 [00:16<05:20, 6273.58 examples/s]
Map (num_proc=8):   4%|4         | 91381/2104277 [00:17<05:57, 5633.42 examples/s]
Map (num_proc=8):   4%|4         | 92532/2104277 [00:17<04:58, 6745.57 examples/s]
Map (num_proc=8):   4%|4         | 93490/2104277 [00:17<05:56, 5643.93 examples/s]
Map (num_proc=8):   4%|4         | 94680/2104277 [00:17<05:02, 6644.39 examples/s]
Map (num_proc=8):   5%|4         | 95721/2104277 [00:17<05:15, 6363.94 examples/s]
Map (num_proc=8):   5%|4         | 96799/2104277 [00:17<05:23, 6209.43 examples/s]
Map (num_proc=8):   5%|4         | 97625/2104277 [00:18<05:03, 6622.01 examples/s]
Map (num_proc=8):   5%|4         | 98638/2104277 [00:18<04:57, 6750.46 examples/s]
Map (num_proc=8):   5%|4         | 99379/2104277 [00:18<05:32, 6025.26 examples/s]
Map (num_proc=8):   5%|4         | 100479/2104277 [00:18<04:41, 7119.95 examples/s]
Map (num_proc=8):   5%|4         | 101303/2104277 [00:18<05:42, 5850.19 examples/s]
Map (num_proc=8):   5%|4         | 102193/2104277 [00:18<05:09, 6474.97 examples/s]
Map (num_proc=8):   5%|4         | 103132/2104277 [00:18<05:21, 6223.29 examples/s]
Map (num_proc=8):   5%|4         | 104004/2104277 [00:19<05:03, 6589.21 examples/s]
Map (num_proc=8):   5%|4         | 104783/2104277 [00:19<05:51, 5687.46 examples/s]
Map (num_proc=8):   5%|5         | 105529/2104277 [00:19<05:29, 6074.48 examples/s]
Map (num_proc=8):   5%|5         | 106415/2104277 [00:19<05:02, 6610.69 examples/s]
Map (num_proc=8):   5%|5         | 107416/2104277 [00:19<05:28, 6077.10 examples/s]
Map (num_proc=8):   5%|5         | 108373/2104277 [00:19<04:53, 6789.71 examples/s]
Map (num_proc=8):   5%|5         | 109378/2104277 [00:19<06:18, 5264.92 examples/s]
Map (num_proc=8):   5%|5         | 110151/2104277 [00:20<05:48, 5721.35 examples/s]
Map (num_proc=8):   5%|5         | 110962/2104277 [00:20<06:10, 5377.96 examples/s]
Map (num_proc=8):   5%|5         | 112013/2104277 [00:20<05:10, 6406.36 examples/s]
Map (num_proc=8):   5%|5         | 112945/2104277 [00:20<06:09, 5382.02 examples/s]
Map (num_proc=8):   5%|5         | 114037/2104277 [00:20<05:16, 6290.34 examples/s]
Map (num_proc=8):   5%|5         | 115000/2104277 [00:21<06:39, 4976.71 examples/s]
Map (num_proc=8):   6%|5         | 115983/2104277 [00:21<05:40, 5843.63 examples/s]
Map (num_proc=8):   6%|5         | 116995/2104277 [00:21<06:04, 5453.68 examples/s]
Map (num_proc=8):   6%|5         | 117740/2104277 [00:21<06:32, 5054.99 examples/s]
Map (num_proc=8):   6%|5         | 118512/2104277 [00:21<05:58, 5541.32 examples/s]
Map (num_proc=8):   6%|5         | 119233/2104277 [00:21<06:57, 4758.92 examples/s]
Map (num_proc=8):   6%|5         | 120533/2104277 [00:21<05:11, 6374.74 examples/s]
Map (num_proc=8):   6%|5         | 121451/2104277 [00:22<04:46, 6929.47 examples/s]
Map (num_proc=8):   6%|5         | 122397/2104277 [00:22<04:44, 6958.97 examples/s]
Map (num_proc=8):   6%|5         | 124122/2104277 [00:22<03:33, 9261.20 examples/s]
Map (num_proc=8):   6%|5         | 125357/2104277 [00:22<03:29, 9431.63 examples/s]
Map (num_proc=8):   6%|6         | 126526/2104277 [00:22<03:33, 9256.35 examples/s]
Map (num_proc=8):   6%|6         | 127773/2104277 [00:22<03:27, 9532.98 examples/s]
Map (num_proc=8):   6%|6         | 128790/2104277 [00:22<03:41, 8938.55 examples/s]
Map (num_proc=8):   6%|6         | 129918/2104277 [00:22<03:44, 8804.95 examples/s]
Map (num_proc=8):   6%|6         | 131126/2104277 [00:23<03:25, 9614.11 examples/s]
Map (num_proc=8):   6%|6         | 132147/2104277 [00:23<03:31, 9324.43 examples/s]
Map (num_proc=8):   6%|6         | 133167/2104277 [00:23<04:44, 6928.73 examples/s]
Map (num_proc=8):   6%|6         | 134406/2104277 [00:23<04:05, 8032.01 examples/s]
Map (num_proc=8):   6%|6         | 135726/2104277 [00:23<03:35, 9147.03 examples/s]
Map (num_proc=8):   7%|6         | 136914/2104277 [00:23<04:28, 7319.52 examples/s]
Map (num_proc=8):   7%|6         | 137884/2104277 [00:23<04:15, 7683.81 examples/s]
Map (num_proc=8):   7%|6         | 139171/2104277 [00:24<03:43, 8787.56 examples/s]
Map (num_proc=8):   7%|6         | 140167/2104277 [00:24<04:35, 7126.20 examples/s]
Map (num_proc=8):   7%|6         | 140997/2104277 [00:24<04:58, 6569.44 examples/s]
Map (num_proc=8):   7%|6         | 141846/2104277 [00:24<08:06, 4032.81 examples/s]
Map (num_proc=8):   7%|6         | 142577/2104277 [00:24<07:47, 4197.46 examples/s]
Map (num_proc=8):   7%|6         | 143239/2104277 [00:25<07:57, 4109.82 examples/s]
Map (num_proc=8):   7%|6         | 143932/2104277 [00:25<07:10, 4555.44 examples/s]
Map (num_proc=8):   7%|6         | 144569/2104277 [00:25<06:39, 4904.41 examples/s]
Map (num_proc=8):   7%|6         | 145294/2104277 [00:25<06:38, 4920.19 examples/s]
Map (num_proc=8):   7%|6         | 146324/2104277 [00:25<05:22, 6061.96 examples/s]
Map (num_proc=8):   7%|7         | 147558/2104277 [00:25<04:31, 7215.84 examples/s]
Map (num_proc=8):   7%|7         | 148368/2104277 [00:25<04:30, 7237.56 examples/s]
Map (num_proc=8):   7%|7         | 149277/2104277 [00:26<04:57, 6567.42 examples/s]
Map (num_proc=8):   7%|7         | 150383/2104277 [00:26<04:18, 7550.30 examples/s]
Map (num_proc=8):   7%|7         | 151432/2104277 [00:26<03:55, 8284.13 examples/s]
Map (num_proc=8):   7%|7         | 152463/2104277 [00:26<04:29, 7255.66 examples/s]
Map (num_proc=8):   7%|7         | 153352/2104277 [00:26<04:26, 7307.09 examples/s]
Map (num_proc=8):   7%|7         | 154451/2104277 [00:26<04:02, 8033.16 examples/s]
Map (num_proc=8):   7%|7         | 155312/2104277 [00:26<04:00, 8097.84 examples/s]
Map (num_proc=8):   7%|7         | 156261/2104277 [00:26<04:19, 7494.45 examples/s]
Map (num_proc=8):   7%|7         | 157377/2104277 [00:26<03:57, 8197.21 examples/s]
Map (num_proc=8):   8%|7         | 158231/2104277 [00:27<04:03, 7978.87 examples/s]
Map (num_proc=8):   8%|7         | 159250/2104277 [00:27<03:47, 8554.17 examples/s]
Map (num_proc=8):   8%|7         | 160281/2104277 [00:27<04:15, 7600.62 examples/s]
Map (num_proc=8):   8%|7         | 161129/2104277 [00:27<04:08, 7816.50 examples/s]
Map (num_proc=8):   8%|7         | 162048/2104277 [00:27<05:16, 6139.76 examples/s]
Map (num_proc=8):   8%|7         | 163048/2104277 [00:27<04:42, 6877.99 examples/s]
Map (num_proc=8):   8%|7         | 164003/2104277 [00:27<04:37, 6981.12 examples/s]
Map (num_proc=8):   8%|7         | 164757/2104277 [00:28<04:35, 7043.15 examples/s]
Map (num_proc=8):   8%|7         | 165593/2104277 [00:28<04:58, 6492.96 examples/s]
Map (num_proc=8):   8%|7         | 166383/2104277 [00:28<04:44, 6800.91 examples/s]
Map (num_proc=8):   8%|7         | 167435/2104277 [00:28<04:13, 7651.34 examples/s]
Map (num_proc=8):   8%|7         | 168337/2104277 [00:28<04:03, 7959.97 examples/s]
Map (num_proc=8):   8%|8         | 169166/2104277 [00:28<04:39, 6922.91 examples/s]
Map (num_proc=8):   8%|8         | 170483/2104277 [00:28<03:53, 8268.80 examples/s]
Map (num_proc=8):   8%|8         | 171572/2104277 [00:28<03:59, 8068.72 examples/s]
Map (num_proc=8):   8%|8         | 172481/2104277 [00:29<04:05, 7860.68 examples/s]
Map (num_proc=8):   8%|8         | 173741/2104277 [00:29<03:54, 8238.15 examples/s]
Map (num_proc=8):   8%|8         | 174869/2104277 [00:29<03:41, 8707.36 examples/s]
Map (num_proc=8):   8%|8         | 175810/2104277 [00:29<04:26, 7233.26 examples/s]
Map (num_proc=8):   8%|8         | 176833/2104277 [00:29<04:14, 7567.63 examples/s]
Map (num_proc=8):   8%|8         | 177925/2104277 [00:29<03:51, 8324.84 examples/s]
Map (num_proc=8):   9%|8         | 178986/2104277 [00:29<03:42, 8645.33 examples/s]
Map (num_proc=8):   9%|8         | 179974/2104277 [00:30<04:24, 7283.98 examples/s]
Map (num_proc=8):   9%|8         | 181053/2104277 [00:30<03:57, 8089.94 examples/s]
Map (num_proc=8):   9%|8         | 182254/2104277 [00:30<03:32, 9047.00 examples/s]
Map (num_proc=8):   9%|8         | 183405/2104277 [00:30<04:05, 7825.58 examples/s]
Map (num_proc=8):   9%|8         | 184420/2104277 [00:30<04:04, 7855.80 examples/s]
Map (num_proc=8):   9%|8         | 185476/2104277 [00:30<03:52, 8239.88 examples/s]
Map (num_proc=8):   9%|8         | 186434/2104277 [00:30<03:51, 8299.53 examples/s]
Map (num_proc=8):   9%|8         | 187453/2104277 [00:30<04:28, 7145.66 examples/s]
Map (num_proc=8):   9%|8         | 188716/2104277 [00:31<03:55, 8122.93 examples/s]
Map (num_proc=8):   9%|9         | 189745/2104277 [00:31<03:49, 8342.77 examples/s]
Map (num_proc=8):   9%|9         | 190674/2104277 [00:31<04:01, 7909.72 examples/s]
Map (num_proc=8):   9%|9         | 191584/2104277 [00:31<04:05, 7785.43 examples/s]
Map (num_proc=8):   9%|9         | 192621/2104277 [00:31<03:47, 8387.94 examples/s]
Map (num_proc=8):   9%|9         | 193577/2104277 [00:31<03:39, 8689.25 examples/s]
Map (num_proc=8):   9%|9         | 194622/2104277 [00:31<04:21, 7299.80 examples/s]
Map (num_proc=8):   9%|9         | 195714/2104277 [00:31<03:58, 7989.37 examples/s]
Map (num_proc=8):   9%|9         | 196742/2104277 [00:32<04:05, 7764.14 examples/s]
Map (num_proc=8):   9%|9         | 197703/2104277 [00:32<03:58, 7996.19 examples/s]
Map (num_proc=8):   9%|9         | 198639/2104277 [00:32<04:28, 7085.19 examples/s]
Map (num_proc=8):   9%|9         | 199858/2104277 [00:32<03:51, 8217.59 examples/s]
Map (num_proc=8):  10%|9         | 200926/2104277 [00:32<03:59, 7954.52 examples/s]
Map (num_proc=8):  10%|9         | 201907/2104277 [00:32<04:17, 7399.83 examples/s]
Map (num_proc=8):  10%|9         | 202749/2104277 [00:32<04:17, 7395.61 examples/s]
Map (num_proc=8):  10%|9         | 204028/2104277 [00:32<03:38, 8713.68 examples/s]
Map (num_proc=8):  10%|9         | 205093/2104277 [00:33<04:28, 7062.87 examples/s]
Map (num_proc=8):  10%|9         | 205938/2104277 [00:33<04:17, 7367.67 examples/s]
Map (num_proc=8):  10%|9         | 207182/2104277 [00:33<03:49, 8282.58 examples/s]
Map (num_proc=8):  10%|9         | 208085/2104277 [00:33<03:48, 8303.57 examples/s]
Map (num_proc=8):  10%|9         | 209024/2104277 [00:33<04:32, 6947.60 examples/s]
Map (num_proc=8):  10%|#         | 210481/2104277 [00:33<03:50, 8216.37 examples/s]
Map (num_proc=8):  10%|#         | 211506/2104277 [00:33<03:44, 8431.11 examples/s]
Map (num_proc=8):  10%|#         | 212442/2104277 [00:34<04:10, 7562.12 examples/s]
Map (num_proc=8):  10%|#         | 213357/2104277 [00:34<04:14, 7424.92 examples/s]
Map (num_proc=8):  10%|#         | 214511/2104277 [00:34<03:55, 8029.67 examples/s]
Map (num_proc=8):  10%|#         | 215599/2104277 [00:34<03:50, 8188.61 examples/s]
Map (num_proc=8):  10%|#         | 216628/2104277 [00:34<04:02, 7784.51 examples/s]
Map (num_proc=8):  10%|#         | 217635/2104277 [00:34<03:47, 8292.32 examples/s]
Map (num_proc=8):  10%|#         | 218507/2104277 [00:34<03:53, 8086.97 examples/s]
Map (num_proc=8):  10%|#         | 219470/2104277 [00:34<03:42, 8468.61 examples/s]
Map (num_proc=8):  10%|#         | 220592/2104277 [00:35<04:02, 7770.57 examples/s]
Map (num_proc=8):  11%|#         | 221508/2104277 [00:35<03:52, 8110.11 examples/s]
Map (num_proc=8):  11%|#         | 222559/2104277 [00:35<03:37, 8671.20 examples/s]
Map (num_proc=8):  11%|#         | 223507/2104277 [00:35<04:00, 7816.03 examples/s]
Map (num_proc=8):  11%|#         | 224375/2104277 [00:35<04:34, 6846.30 examples/s]
Map (num_proc=8):  11%|#         | 225691/2104277 [00:35<03:47, 8262.80 examples/s]
Map (num_proc=8):  11%|#         | 226738/2104277 [00:35<03:35, 8724.05 examples/s]
Map (num_proc=8):  11%|#         | 227800/2104277 [00:36<03:59, 7847.39 examples/s]
Map (num_proc=8):  11%|#         | 228787/2104277 [00:36<03:50, 8138.02 examples/s]
Map (num_proc=8):  11%|#         | 229710/2104277 [00:36<04:10, 7468.77 examples/s]
Map (num_proc=8):  11%|#         | 230652/2104277 [00:36<05:50, 5345.43 examples/s]
Map (num_proc=8):  11%|#1        | 231563/2104277 [00:36<05:17, 5889.71 examples/s]
Map (num_proc=8):  11%|#1        | 232525/2104277 [00:36<06:11, 5032.28 examples/s]
Map (num_proc=8):  11%|#1        | 233571/2104277 [00:37<05:16, 5909.01 examples/s]
Map (num_proc=8):  11%|#1        | 234669/2104277 [00:37<04:35, 6775.81 examples/s]
Map (num_proc=8):  11%|#1        | 235797/2104277 [00:37<04:08, 7508.64 examples/s]
Map (num_proc=8):  11%|#1        | 236640/2104277 [00:37<05:02, 6172.63 examples/s]
Map (num_proc=8):  11%|#1        | 237690/2104277 [00:37<04:24, 7045.58 examples/s]
Map (num_proc=8):  11%|#1        | 238937/2104277 [00:37<03:48, 8171.50 examples/s]
Map (num_proc=8):  11%|#1        | 239868/2104277 [00:37<03:41, 8400.92 examples/s]
Map (num_proc=8):  11%|#1        | 240863/2104277 [00:38<04:32, 6827.98 examples/s]
Map (num_proc=8):  12%|#1        | 242053/2104277 [00:38<04:01, 7695.31 examples/s]
Map (num_proc=8):  12%|#1        | 243240/2104277 [00:38<03:37, 8540.19 examples/s]
Map (num_proc=8):  12%|#1        | 244182/2104277 [00:38<04:40, 6634.79 examples/s]
Map (num_proc=8):  12%|#1        | 245292/2104277 [00:38<04:07, 7517.27 examples/s]
Map (num_proc=8):  12%|#1        | 246571/2104277 [00:38<03:34, 8651.03 examples/s]
Map (num_proc=8):  12%|#1        | 247642/2104277 [00:38<04:14, 7307.79 examples/s]
Map (num_proc=8):  12%|#1        | 248548/2104277 [00:38<04:02, 7640.12 examples/s]
Map (num_proc=8):  12%|#1        | 249516/2104277 [00:39<03:51, 8019.98 examples/s]
Map (num_proc=8):  12%|#1        | 250665/2104277 [00:39<03:44, 8254.85 examples/s]
Map (num_proc=8):  12%|#1        | 251575/2104277 [00:39<04:25, 6973.24 examples/s]
Map (num_proc=8):  12%|#2        | 252867/2104277 [00:39<03:44, 8244.86 examples/s]
Map (num_proc=8):  12%|#2        | 253932/2104277 [00:39<03:31, 8744.66 examples/s]
Map (num_proc=8):  12%|#2        | 254988/2104277 [00:39<04:18, 7144.92 examples/s]
Map (num_proc=8):  12%|#2        | 255976/2104277 [00:39<04:17, 7175.85 examples/s]
Map (num_proc=8):  12%|#2        | 257095/2104277 [00:40<03:51, 7962.93 examples/s]
Map (num_proc=8):  12%|#2        | 258000/2104277 [00:40<04:30, 6835.88 examples/s]
Map (num_proc=8):  12%|#2        | 259193/2104277 [00:40<03:54, 7883.19 examples/s]
Map (num_proc=8):  12%|#2        | 260509/2104277 [00:40<03:21, 9145.14 examples/s]
Map (num_proc=8):  12%|#2        | 261521/2104277 [00:40<04:27, 6882.52 examples/s]
Map (num_proc=8):  12%|#2        | 262527/2104277 [00:40<04:09, 7386.10 examples/s]
Map (num_proc=8):  13%|#2        | 263641/2104277 [00:40<03:47, 8088.25 examples/s]
Map (num_proc=8):  13%|#2        | 264613/2104277 [00:41<04:04, 7535.31 examples/s]
Map (num_proc=8):  13%|#2        | 265581/2104277 [00:41<04:05, 7489.92 examples/s]
Map (num_proc=8):  13%|#2        | 266823/2104277 [00:41<03:32, 8658.46 examples/s]
Map (num_proc=8):  13%|#2        | 267875/2104277 [00:41<03:25, 8943.09 examples/s]
Map (num_proc=8):  13%|#2        | 268993/2104277 [00:41<04:15, 7192.06 examples/s]
Map (num_proc=8):  13%|#2        | 270048/2104277 [00:41<03:51, 7913.03 examples/s]
Map (num_proc=8):  13%|#2        | 271040/2104277 [00:41<03:58, 7700.22 examples/s]
Map (num_proc=8):  13%|#2        | 271960/2104277 [00:42<04:32, 6735.63 examples/s]
Map (num_proc=8):  13%|#2        | 272971/2104277 [00:42<04:06, 7418.01 examples/s]
Map (num_proc=8):  13%|#3        | 273981/2104277 [00:42<04:34, 6656.63 examples/s]
Map (num_proc=8):  13%|#3        | 274815/2104277 [00:42<04:44, 6426.15 examples/s]
Map (num_proc=8):  13%|#3        | 275592/2104277 [00:42<04:43, 6448.32 examples/s]
Map (num_proc=8):  13%|#3        | 276417/2104277 [00:42<04:26, 6855.43 examples/s]
Map (num_proc=8):  13%|#3        | 277326/2104277 [00:42<04:09, 7337.14 examples/s]
Map (num_proc=8):  13%|#3        | 278156/2104277 [00:42<04:15, 7146.88 examples/s]
Map (num_proc=8):  13%|#3        | 279233/2104277 [00:43<03:46, 8042.67 examples/s]
Map (num_proc=8):  13%|#3        | 280191/2104277 [00:43<03:50, 7900.36 examples/s]
Map (num_proc=8):  13%|#3        | 281135/2104277 [00:43<03:56, 7697.19 examples/s]
Map (num_proc=8):  13%|#3        | 281992/2104277 [00:43<04:11, 7247.62 examples/s]
Map (num_proc=8):  13%|#3        | 283000/2104277 [00:43<03:50, 7888.45 examples/s]
Map (num_proc=8):  13%|#3        | 283966/2104277 [00:43<03:41, 8208.51 examples/s]
Map (num_proc=8):  14%|#3        | 284863/2104277 [00:43<03:37, 8359.25 examples/s]
Map (num_proc=8):  14%|#3        | 285737/2104277 [00:43<03:42, 8178.15 examples/s]
Map (num_proc=8):  14%|#3        | 286639/2104277 [00:43<03:39, 8282.23 examples/s]
Map (num_proc=8):  14%|#3        | 287619/2104277 [00:44<03:36, 8409.53 examples/s]
Map (num_proc=8):  14%|#3        | 288472/2104277 [00:44<04:01, 7512.91 examples/s]
Map (num_proc=8):  14%|#3        | 289447/2104277 [00:44<03:44, 8096.97 examples/s]
Map (num_proc=8):  14%|#3        | 290280/2104277 [00:44<03:55, 7695.33 examples/s]
Map (num_proc=8):  14%|#3        | 291170/2104277 [00:44<03:47, 7965.68 examples/s]
Map (num_proc=8):  14%|#3        | 292019/2104277 [00:44<04:22, 6892.41 examples/s]
Map (num_proc=8):  14%|#3        | 292838/2104277 [00:44<04:29, 6727.21 examples/s]
Map (num_proc=8):  14%|#3        | 293545/2104277 [00:44<04:29, 6711.34 examples/s]
Map (num_proc=8):  14%|#3        | 294427/2104277 [00:45<04:49, 6245.28 examples/s]
Map (num_proc=8):  14%|#4        | 295132/2104277 [00:45<05:00, 6016.06 examples/s]
Map (num_proc=8):  14%|#4        | 295909/2104277 [00:45<04:43, 6388.19 examples/s]
Map (num_proc=8):  14%|#4        | 296826/2104277 [00:45<05:02, 5977.89 examples/s]
Map (num_proc=8):  14%|#4        | 297633/2104277 [00:45<05:16, 5711.38 examples/s]
Map (num_proc=8):  14%|#4        | 298451/2104277 [00:45<04:56, 6084.43 examples/s]
Map (num_proc=8):  14%|#4        | 299661/2104277 [00:45<04:08, 7269.86 examples/s]
Map (num_proc=8):  14%|#4        | 300616/2104277 [00:46<03:50, 7832.09 examples/s]
Map (num_proc=8):  14%|#4        | 301469/2104277 [00:46<04:33, 6593.90 examples/s]
Map (num_proc=8):  14%|#4        | 302337/2104277 [00:46<04:39, 6451.60 examples/s]
Map (num_proc=8):  14%|#4        | 303434/2104277 [00:46<04:04, 7352.75 examples/s]
Map (num_proc=8):  14%|#4        | 304623/2104277 [00:46<03:51, 7775.07 examples/s]
Map (num_proc=8):  15%|#4        | 305432/2104277 [00:46<04:27, 6729.84 examples/s]
Map (num_proc=8):  15%|#4        | 306299/2104277 [00:46<04:34, 6552.68 examples/s]
Map (num_proc=8):  15%|#4        | 307431/2104277 [00:46<04:03, 7394.20 examples/s]
Map (num_proc=8):  15%|#4        | 308321/2104277 [00:47<03:57, 7552.05 examples/s]
Map (num_proc=8):  15%|#4        | 309145/2104277 [00:47<03:54, 7644.76 examples/s]
Map (num_proc=8):  15%|#4        | 310103/2104277 [00:47<04:30, 6624.48 examples/s]
Map (num_proc=8):  15%|#4        | 310899/2104277 [00:47<04:51, 6160.60 examples/s]
Map (num_proc=8):  15%|#4        | 312152/2104277 [00:47<03:55, 7614.72 examples/s]
Map (num_proc=8):  15%|#4        | 313182/2104277 [00:47<03:43, 8000.26 examples/s]
Map (num_proc=8):  15%|#4        | 314121/2104277 [00:47<04:04, 7330.95 examples/s]
Map (num_proc=8):  15%|#4        | 315241/2104277 [00:48<03:38, 8183.32 examples/s]
Map (num_proc=8):  15%|#5        | 316593/2104277 [00:48<03:10, 9378.01 examples/s]
Map (num_proc=8):  15%|#5        | 317656/2104277 [00:48<03:16, 9104.39 examples/s]
Map (num_proc=8):  15%|#5        | 318634/2104277 [00:48<03:13, 9221.83 examples/s]
Map (num_proc=8):  15%|#5        | 319769/2104277 [00:48<03:06, 9576.90 examples/s]
Map (num_proc=8):  15%|#5        | 320931/2104277 [00:48<03:26, 8629.79 examples/s]
Map (num_proc=8):  15%|#5        | 321861/2104277 [00:48<03:23, 8749.94 examples/s]
Map (num_proc=8):  15%|#5        | 322992/2104277 [00:48<03:23, 8731.97 examples/s]
Map (num_proc=8):  15%|#5        | 323971/2104277 [00:48<03:24, 8719.36 examples/s]
Map (num_proc=8):  15%|#5        | 325021/2104277 [00:49<03:15, 9083.95 examples/s]
Map (num_proc=8):  15%|#5        | 325997/2104277 [00:49<04:26, 6681.12 examples/s]
Map (num_proc=8):  16%|#5        | 326979/2104277 [00:49<04:01, 7364.80 examples/s]
Map (num_proc=8):  16%|#5        | 327890/2104277 [00:49<03:58, 7446.40 examples/s]
Map (num_proc=8):  16%|#5        | 328826/2104277 [00:49<04:06, 7189.02 examples/s]
Map (num_proc=8):  16%|#5        | 329593/2104277 [00:49<04:06, 7203.69 examples/s]
Map (num_proc=8):  16%|#5        | 330443/2104277 [00:49<04:24, 6706.21 examples/s]
Map (num_proc=8):  16%|#5        | 331528/2104277 [00:50<03:52, 7614.78 examples/s]
Map (num_proc=8):  16%|#5        | 332848/2104277 [00:50<03:16, 8999.21 examples/s]
Map (num_proc=8):  16%|#5        | 333937/2104277 [00:50<03:33, 8286.48 examples/s]
Map (num_proc=8):  16%|#5        | 335038/2104277 [00:50<03:22, 8734.19 examples/s]
Map (num_proc=8):  16%|#5        | 335965/2104277 [00:50<04:01, 7313.28 examples/s]
Map (num_proc=8):  16%|#6        | 336945/2104277 [00:50<03:49, 7703.91 examples/s]
Map (num_proc=8):  16%|#6        | 337983/2104277 [00:50<03:50, 7664.64 examples/s]
Map (num_proc=8):  16%|#6        | 338880/2104277 [00:51<04:43, 6237.40 examples/s]
Map (num_proc=8):  16%|#6        | 339605/2104277 [00:51<04:33, 6451.32 examples/s]
Map (num_proc=8):  16%|#6        | 340730/2104277 [00:51<03:53, 7540.81 examples/s]
Map (num_proc=8):  16%|#6        | 341554/2104277 [00:51<04:20, 6768.47 examples/s]
Map (num_proc=8):  16%|#6        | 342423/2104277 [00:51<04:14, 6924.44 examples/s]
Map (num_proc=8):  16%|#6        | 343395/2104277 [00:51<04:35, 6391.54 examples/s]
Map (num_proc=8):  16%|#6        | 344345/2104277 [00:51<04:10, 7034.94 examples/s]
Map (num_proc=8):  16%|#6        | 345549/2104277 [00:51<03:35, 8175.87 examples/s]
Map (num_proc=8):  16%|#6        | 346626/2104277 [00:52<03:23, 8643.47 examples/s]
Map (num_proc=8):  17%|#6        | 347656/2104277 [00:52<03:46, 7749.57 examples/s]
Map (num_proc=8):  17%|#6        | 348646/2104277 [00:52<03:36, 8114.89 examples/s]
Map (num_proc=8):  17%|#6        | 349906/2104277 [00:52<03:13, 9068.40 examples/s]
Map (num_proc=8):  17%|#6        | 350990/2104277 [00:52<03:11, 9152.75 examples/s]
Map (num_proc=8):  17%|#6        | 352147/2104277 [00:52<03:01, 9634.89 examples/s]
Map (num_proc=8):  17%|#6        | 353204/2104277 [00:52<03:41, 7913.32 examples/s]
Map (num_proc=8):  17%|#6        | 354655/2104277 [00:52<03:06, 9368.73 examples/s]
Map (num_proc=8):  17%|#6        | 356020/2104277 [00:53<02:49, 10339.78 examples/s]
Map (num_proc=8):  17%|#6        | 357140/2104277 [00:53<03:18, 8786.82 examples/s] 
Map (num_proc=8):  17%|#7        | 358277/2104277 [00:53<03:15, 8915.65 examples/s]
Map (num_proc=8):  17%|#7        | 359306/2104277 [00:53<03:11, 9121.35 examples/s]
Map (num_proc=8):  17%|#7        | 360781/2104277 [00:53<02:47, 10421.51 examples/s]
Map (num_proc=8):  17%|#7        | 361966/2104277 [00:53<03:03, 9491.06 examples/s] 
Map (num_proc=8):  17%|#7        | 363139/2104277 [00:53<03:25, 8453.24 examples/s]
Map (num_proc=8):  17%|#7        | 364169/2104277 [00:54<03:41, 7845.67 examples/s]
Map (num_proc=8):  17%|#7        | 365193/2104277 [00:54<03:27, 8384.83 examples/s]
Map (num_proc=8):  17%|#7        | 366621/2104277 [00:54<02:57, 9817.19 examples/s]
Map (num_proc=8):  17%|#7        | 367892/2104277 [00:54<02:56, 9854.22 examples/s]
Map (num_proc=8):  18%|#7        | 368933/2104277 [00:54<03:59, 7253.16 examples/s]
Map (num_proc=8):  18%|#7        | 370014/2104277 [00:54<03:43, 7750.61 examples/s]
Map (num_proc=8):  18%|#7        | 371527/2104277 [00:54<03:05, 9326.10 examples/s]
Map (num_proc=8):  18%|#7        | 372761/2104277 [00:54<03:02, 9500.94 examples/s]
Map (num_proc=8):  18%|#7        | 374010/2104277 [00:55<03:37, 7967.63 examples/s]
Map (num_proc=8):  18%|#7        | 375287/2104277 [00:55<03:15, 8828.23 examples/s]
Map (num_proc=8):  18%|#7        | 376301/2104277 [00:55<03:12, 8987.89 examples/s]
Map (num_proc=8):  18%|#7        | 377329/2104277 [00:55<03:26, 8357.24 examples/s]
Map (num_proc=8):  18%|#7        | 378460/2104277 [00:55<03:25, 8387.73 examples/s]
Map (num_proc=8):  18%|#8        | 379340/2104277 [00:55<03:27, 8323.87 examples/s]
Map (num_proc=8):  18%|#8        | 380553/2104277 [00:55<03:09, 9077.93 examples/s]
Map (num_proc=8):  18%|#8        | 381538/2104277 [00:56<03:19, 8649.73 examples/s]
Map (num_proc=8):  18%|#8        | 382647/2104277 [00:56<03:08, 9120.24 examples/s]
Map (num_proc=8):  18%|#8        | 383648/2104277 [00:56<03:21, 8551.79 examples/s]
Map (num_proc=8):  18%|#8        | 384656/2104277 [00:56<03:29, 8203.62 examples/s]
Map (num_proc=8):  18%|#8        | 385996/2104277 [00:56<03:01, 9448.08 examples/s]
Map (num_proc=8):  18%|#8        | 387379/2104277 [00:56<02:46, 10302.54 examples/s]
Map (num_proc=8):  18%|#8        | 388653/2104277 [00:56<03:25, 8359.35 examples/s] 
Map (num_proc=8):  19%|#8        | 389716/2104277 [00:56<03:21, 8490.03 examples/s]
Map (num_proc=8):  19%|#8        | 391094/2104277 [00:57<02:58, 9600.35 examples/s]
Map (num_proc=8):  19%|#8        | 392335/2104277 [00:57<03:01, 9440.87 examples/s]
Map (num_proc=8):  19%|#8        | 393475/2104277 [00:57<03:37, 7869.50 examples/s]
Map (num_proc=8):  19%|#8        | 394837/2104277 [00:57<03:16, 8695.92 examples/s]
Map (num_proc=8):  19%|#8        | 396045/2104277 [00:57<03:05, 9215.49 examples/s]
Map (num_proc=8):  19%|#8        | 397026/2104277 [00:57<03:36, 7897.11 examples/s]
Map (num_proc=8):  19%|#8        | 397946/2104277 [00:57<03:34, 7950.79 examples/s]
Map (num_proc=8):  19%|#8        | 399317/2104277 [00:58<03:02, 9321.65 examples/s]
Map (num_proc=8):  19%|#9        | 400398/2104277 [00:58<03:22, 8403.80 examples/s]
Map (num_proc=8):  19%|#9        | 401734/2104277 [00:58<03:08, 9024.73 examples/s]
Map (num_proc=8):  19%|#9        | 402710/2104277 [00:58<03:15, 8717.51 examples/s]
Map (num_proc=8):  19%|#9        | 403661/2104277 [00:58<03:56, 7186.60 examples/s]
Map (num_proc=8):  19%|#9        | 404794/2104277 [00:58<03:34, 7909.93 examples/s]
Map (num_proc=8):  19%|#9        | 406203/2104277 [00:58<03:07, 9053.33 examples/s]
Map (num_proc=8):  19%|#9        | 407216/2104277 [00:59<03:53, 7277.03 examples/s]
Map (num_proc=8):  19%|#9        | 408535/2104277 [00:59<03:19, 8480.20 examples/s]
Map (num_proc=8):  19%|#9        | 409738/2104277 [00:59<03:06, 9098.29 examples/s]
Map (num_proc=8):  20%|#9        | 410863/2104277 [00:59<03:13, 8744.75 examples/s]
Map (num_proc=8):  20%|#9        | 411891/2104277 [00:59<03:20, 8454.88 examples/s]
Map (num_proc=8):  20%|#9        | 412920/2104277 [00:59<03:21, 8396.84 examples/s]
Map (num_proc=8):  20%|#9        | 414393/2104277 [00:59<03:04, 9147.09 examples/s]
Map (num_proc=8):  20%|#9        | 415690/2104277 [00:59<02:51, 9818.21 examples/s]
Map (num_proc=8):  20%|#9        | 416835/2104277 [01:00<03:18, 8482.39 examples/s]
Map (num_proc=8):  20%|#9        | 417893/2104277 [01:00<03:27, 8108.05 examples/s]
Map (num_proc=8):  20%|#9        | 419375/2104277 [01:00<02:58, 9456.70 examples/s]
Map (num_proc=8):  20%|#9        | 420432/2104277 [01:00<02:56, 9549.16 examples/s]
Map (num_proc=8):  20%|##        | 421600/2104277 [01:00<03:18, 8464.96 examples/s]
Map (num_proc=8):  20%|##        | 422668/2104277 [01:00<03:14, 8624.06 examples/s]
Map (num_proc=8):  20%|##        | 424134/2104277 [01:00<03:01, 9243.90 examples/s]
Map (num_proc=8):  20%|##        | 425540/2104277 [01:01<02:47, 10035.88 examples/s]
Map (num_proc=8):  20%|##        | 426741/2104277 [01:01<02:57, 9465.73 examples/s] 
Map (num_proc=8):  20%|##        | 427804/2104277 [01:01<03:20, 8349.11 examples/s]
Map (num_proc=8):  20%|##        | 429264/2104277 [01:01<02:56, 9498.84 examples/s]
Map (num_proc=8):  20%|##        | 430269/2104277 [01:01<02:59, 9325.64 examples/s]
Map (num_proc=8):  20%|##        | 431328/2104277 [01:01<02:58, 9352.19 examples/s]
Map (num_proc=8):  21%|##        | 432294/2104277 [01:01<03:07, 8904.79 examples/s]
Map (num_proc=8):  21%|##        | 433291/2104277 [01:01<03:26, 8078.43 examples/s]
Map (num_proc=8):  21%|##        | 434508/2104277 [01:02<03:06, 8962.55 examples/s]
Map (num_proc=8):  21%|##        | 435764/2104277 [01:02<02:53, 9611.46 examples/s]
Map (num_proc=8):  21%|##        | 436798/2104277 [01:02<03:32, 7864.17 examples/s]
Map (num_proc=8):  21%|##        | 437799/2104277 [01:02<03:22, 8249.62 examples/s]
Map (num_proc=8):  21%|##        | 438990/2104277 [01:02<03:02, 9128.73 examples/s]
Map (num_proc=8):  21%|##        | 440108/2104277 [01:02<02:58, 9323.95 examples/s]
Map (num_proc=8):  21%|##        | 441095/2104277 [01:02<02:57, 9389.22 examples/s]
Map (num_proc=8):  21%|##1       | 442377/2104277 [01:02<02:44, 10077.25 examples/s]
Map (num_proc=8):  21%|##1       | 443567/2104277 [01:03<03:28, 7957.82 examples/s] 
Map (num_proc=8):  21%|##1       | 444532/2104277 [01:03<03:20, 8281.03 examples/s]
Map (num_proc=8):  21%|##1       | 445583/2104277 [01:03<03:18, 8374.71 examples/s]
Map (num_proc=8):  21%|##1       | 446534/2104277 [01:03<03:42, 7452.54 examples/s]
Map (num_proc=8):  21%|##1       | 447459/2104277 [01:03<03:37, 7623.70 examples/s]
Map (num_proc=8):  21%|##1       | 448277/2104277 [01:03<03:54, 7049.28 examples/s]
Map (num_proc=8):  21%|##1       | 449040/2104277 [01:03<04:14, 6503.32 examples/s]
Map (num_proc=8):  21%|##1       | 449879/2104277 [01:03<04:03, 6791.01 examples/s]
Map (num_proc=8):  21%|##1       | 450857/2104277 [01:04<03:47, 7263.18 examples/s]
Map (num_proc=8):  21%|##1       | 451755/2104277 [01:04<04:20, 6344.53 examples/s]
Map (num_proc=8):  22%|##1       | 452560/2104277 [01:04<04:07, 6666.44 examples/s]
Map (num_proc=8):  22%|##1       | 453340/2104277 [01:04<04:07, 6668.84 examples/s]
Map (num_proc=8):  22%|##1       | 454649/2104277 [01:04<03:34, 7692.95 examples/s]
Map (num_proc=8):  22%|##1       | 455878/2104277 [01:04<03:06, 8844.68 examples/s]
Map (num_proc=8):  22%|##1       | 457011/2104277 [01:04<02:56, 9331.26 examples/s]
Map (num_proc=8):  22%|##1       | 457986/2104277 [01:04<03:11, 8606.79 examples/s]
Map (num_proc=8):  22%|##1       | 458940/2104277 [01:05<03:07, 8794.40 examples/s]
Map (num_proc=8):  22%|##1       | 460052/2104277 [01:05<02:58, 9196.37 examples/s]
Map (num_proc=8):  22%|##1       | 461028/2104277 [01:05<02:55, 9339.15 examples/s]
Map (num_proc=8):  22%|##1       | 462086/2104277 [01:05<03:09, 8665.50 examples/s]
Map (num_proc=8):  22%|##2       | 463115/2104277 [01:05<03:05, 8858.24 examples/s]
Map (num_proc=8):  22%|##2       | 464083/2104277 [01:05<03:19, 8236.14 examples/s]
Map (num_proc=8):  22%|##2       | 465197/2104277 [01:05<03:08, 8709.80 examples/s]
Map (num_proc=8):  22%|##2       | 466235/2104277 [01:05<03:38, 7497.21 examples/s]
Map (num_proc=8):  22%|##2       | 467060/2104277 [01:06<03:35, 7596.29 examples/s]
Map (num_proc=8):  22%|##2       | 468345/2104277 [01:06<03:14, 8414.81 examples/s]
Map (num_proc=8):  22%|##2       | 469350/2104277 [01:06<03:18, 8229.74 examples/s]
Map (num_proc=8):  22%|##2       | 470232/2104277 [01:06<04:04, 6676.33 examples/s]
Map (num_proc=8):  22%|##2       | 471485/2104277 [01:06<03:24, 7978.34 examples/s]
Map (num_proc=8):  22%|##2       | 472580/2104277 [01:06<03:14, 8374.91 examples/s]
Map (num_proc=8):  23%|##2       | 473542/2104277 [01:06<04:02, 6723.88 examples/s]
Map (num_proc=8):  23%|##2       | 474434/2104277 [01:07<04:25, 6136.83 examples/s]
Map (num_proc=8):  23%|##2       | 475360/2104277 [01:07<04:04, 6666.08 examples/s]
Map (num_proc=8):  23%|##2       | 476414/2104277 [01:07<03:35, 7539.97 examples/s]
Map (num_proc=8):  23%|##2       | 477322/2104277 [01:07<03:31, 7707.84 examples/s]
Map (num_proc=8):  23%|##2       | 478185/2104277 [01:07<03:40, 7381.53 examples/s]
Map (num_proc=8):  23%|##2       | 479006/2104277 [01:07<03:35, 7549.98 examples/s]
Map (num_proc=8):  23%|##2       | 480139/2104277 [01:07<03:10, 8537.90 examples/s]
Map (num_proc=8):  23%|##2       | 481383/2104277 [01:07<03:05, 8755.81 examples/s]
Map (num_proc=8):  23%|##2       | 482360/2104277 [01:08<03:03, 8857.28 examples/s]
Map (num_proc=8):  23%|##2       | 483612/2104277 [01:08<02:59, 9006.31 examples/s]
Map (num_proc=8):  23%|##3       | 484764/2104277 [01:08<02:50, 9479.19 examples/s]
Map (num_proc=8):  23%|##3       | 485753/2104277 [01:08<02:51, 9437.46 examples/s]
Map (num_proc=8):  23%|##3       | 486863/2104277 [01:08<02:48, 9595.85 examples/s]
Map (num_proc=8):  23%|##3       | 487882/2104277 [01:08<03:08, 8595.71 examples/s]
Map (num_proc=8):  23%|##3       | 489372/2104277 [01:08<02:41, 10018.96 examples/s]
Map (num_proc=8):  23%|##3       | 490660/2104277 [01:08<02:34, 10447.52 examples/s]
Map (num_proc=8):  23%|##3       | 491850/2104277 [01:09<02:41, 9997.49 examples/s] 
Map (num_proc=8):  23%|##3       | 493011/2104277 [01:09<02:58, 9015.70 examples/s]
Map (num_proc=8):  23%|##3       | 494033/2104277 [01:09<02:58, 9028.52 examples/s]
Map (num_proc=8):  24%|##3       | 495041/2104277 [01:09<02:58, 8992.34 examples/s]
Map (num_proc=8):  24%|##3       | 496067/2104277 [01:09<02:55, 9157.69 examples/s]
Map (num_proc=8):  24%|##3       | 497154/2104277 [01:09<03:01, 8846.13 examples/s]
Map (num_proc=8):  24%|##3       | 498185/2104277 [01:09<03:00, 8914.41 examples/s]
Map (num_proc=8):  24%|##3       | 499347/2104277 [01:09<02:52, 9325.72 examples/s]
Map (num_proc=8):  24%|##3       | 500362/2104277 [01:09<02:55, 9161.59 examples/s]
Map (num_proc=8):  24%|##3       | 501458/2104277 [01:10<02:49, 9458.74 examples/s]
Map (num_proc=8):  24%|##3       | 502410/2104277 [01:10<02:54, 9190.16 examples/s]
Map (num_proc=8):  24%|##3       | 503627/2104277 [01:10<02:45, 9677.79 examples/s]
Map (num_proc=8):  24%|##3       | 504670/2104277 [01:10<03:00, 8844.48 examples/s]
Map (num_proc=8):  24%|##4       | 505617/2104277 [01:10<03:00, 8867.07 examples/s]
Map (num_proc=8):  24%|##4       | 506556/2104277 [01:10<03:02, 8759.11 examples/s]
Map (num_proc=8):  24%|##4       | 507661/2104277 [01:10<02:53, 9196.61 examples/s]
Map (num_proc=8):  24%|##4       | 508618/2104277 [01:10<03:00, 8832.59 examples/s]
Map (num_proc=8):  24%|##4       | 509602/2104277 [01:11<05:39, 4693.05 examples/s]
Map (num_proc=8):  24%|##4       | 510661/2104277 [01:11<04:45, 5586.69 examples/s]
Map (num_proc=8):  24%|##4       | 511494/2104277 [01:11<04:40, 5681.81 examples/s]
Map (num_proc=8):  25%|##4       | 516486/2104277 [01:11<01:49, 14481.51 examples/s]
Map (num_proc=8):  25%|##4       | 518516/2104277 [01:11<02:04, 12735.42 examples/s]
Map (num_proc=8):  25%|##4       | 520222/2104277 [01:12<02:07, 12434.25 examples/s]
Map (num_proc=8):  25%|##4       | 521788/2104277 [01:12<02:05, 12659.66 examples/s]
Map (num_proc=8):  25%|##4       | 523264/2104277 [01:12<02:20, 11237.40 examples/s]
Map (num_proc=8):  25%|##4       | 524548/2104277 [01:12<02:18, 11435.20 examples/s]
Map (num_proc=8):  25%|##5       | 526190/2104277 [01:12<02:07, 12368.46 examples/s]
Map (num_proc=8):  25%|##5       | 527617/2104277 [01:12<02:16, 11574.17 examples/s]
Map (num_proc=8):  25%|##5       | 528853/2104277 [01:12<02:34, 10170.76 examples/s]
Map (num_proc=8):  25%|##5       | 530377/2104277 [01:12<02:20, 11236.71 examples/s]
Map (num_proc=8):  25%|##5       | 531735/2104277 [01:13<02:16, 11554.66 examples/s]
Map (num_proc=8):  25%|##5       | 533125/2104277 [01:13<02:17, 11393.09 examples/s]
Map (num_proc=8):  25%|##5       | 534439/2104277 [01:13<02:15, 11549.25 examples/s]
Map (num_proc=8):  25%|##5       | 535762/2104277 [01:13<02:29, 10526.85 examples/s]
Map (num_proc=8):  26%|##5       | 536983/2104277 [01:13<02:27, 10651.07 examples/s]
Map (num_proc=8):  26%|##5       | 538207/2104277 [01:13<02:29, 10449.52 examples/s]
Map (num_proc=8):  26%|##5       | 539854/2104277 [01:13<02:13, 11724.75 examples/s]
Map (num_proc=8):  26%|##5       | 541088/2104277 [01:13<02:14, 11637.30 examples/s]
Map (num_proc=8):  26%|##5       | 542334/2104277 [01:14<02:13, 11691.50 examples/s]
Map (num_proc=8):  26%|##5       | 543674/2104277 [01:14<02:27, 10570.37 examples/s]
Map (num_proc=8):  26%|##5       | 544906/2104277 [01:14<02:22, 10960.48 examples/s]
Map (num_proc=8):  26%|##5       | 546163/2104277 [01:14<02:19, 11151.60 examples/s]
Map (num_proc=8):  26%|##6       | 547343/2104277 [01:14<02:18, 11261.91 examples/s]
Map (num_proc=8):  26%|##6       | 548623/2104277 [01:14<02:13, 11660.38 examples/s]
Map (num_proc=8):  26%|##6       | 549881/2104277 [01:14<02:14, 11558.74 examples/s]
Map (num_proc=8):  26%|##6       | 551190/2104277 [01:14<02:24, 10776.31 examples/s]
Map (num_proc=8):  26%|##6       | 552328/2104277 [01:14<02:35, 9984.17 examples/s] 
Map (num_proc=8):  26%|##6       | 553779/2104277 [01:15<02:21, 10980.27 examples/s]
Map (num_proc=8):  26%|##6       | 555451/2104277 [01:15<02:08, 12054.96 examples/s]
Map (num_proc=8):  26%|##6       | 556832/2104277 [01:15<02:17, 11256.28 examples/s]
Map (num_proc=8):  27%|##6       | 558140/2104277 [01:15<02:24, 10721.37 examples/s]
Map (num_proc=8):  27%|##6       | 559480/2104277 [01:15<02:18, 11155.18 examples/s]
Map (num_proc=8):  27%|##6       | 560619/2104277 [01:15<02:28, 10407.53 examples/s]
Map (num_proc=8):  27%|##6       | 561784/2104277 [01:15<02:25, 10616.01 examples/s]
Map (num_proc=8):  27%|##6       | 562936/2104277 [01:15<02:23, 10766.77 examples/s]
Map (num_proc=8):  27%|##6       | 564475/2104277 [01:16<02:08, 12006.45 examples/s]
Map (num_proc=8):  27%|##6       | 565754/2104277 [01:16<02:26, 10511.53 examples/s]
Map (num_proc=8):  27%|##6       | 567017/2104277 [01:16<02:35, 9866.34 examples/s] 
Map (num_proc=8):  27%|##6       | 568110/2104277 [01:16<02:33, 10006.62 examples/s]
Map (num_proc=8):  27%|##7       | 569272/2104277 [01:16<02:57, 8672.01 examples/s] 
Map (num_proc=8):  27%|##7       | 570290/2104277 [01:16<02:53, 8826.20 examples/s]
Map (num_proc=8):  27%|##7       | 571557/2104277 [01:16<02:47, 9159.36 examples/s]
Map (num_proc=8):  27%|##7       | 572588/2104277 [01:16<02:45, 9269.84 examples/s]
Map (num_proc=8):  27%|##7       | 573545/2104277 [01:17<03:19, 7657.59 examples/s]
Map (num_proc=8):  27%|##7       | 574686/2104277 [01:17<02:59, 8527.01 examples/s]
Map (num_proc=8):  27%|##7       | 575812/2104277 [01:17<02:46, 9177.14 examples/s]
Map (num_proc=8):  27%|##7       | 576928/2104277 [01:17<02:59, 8505.24 examples/s]
Map (num_proc=8):  27%|##7       | 578004/2104277 [01:17<03:02, 8362.26 examples/s]
Map (num_proc=8):  28%|##7       | 578996/2104277 [01:17<03:01, 8387.08 examples/s]
Map (num_proc=8):  28%|##7       | 579904/2104277 [01:17<03:07, 8133.66 examples/s]
Map (num_proc=8):  28%|##7       | 581108/2104277 [01:17<02:48, 9020.81 examples/s]
Map (num_proc=8):  28%|##7       | 582718/2104277 [01:18<02:22, 10669.92 examples/s]
Map (num_proc=8):  28%|##7       | 584267/2104277 [01:18<02:07, 11906.68 examples/s]
Map (num_proc=8):  28%|##7       | 585542/2104277 [01:18<02:41, 9397.20 examples/s] 
Map (num_proc=8):  28%|##7       | 586895/2104277 [01:18<02:29, 10179.97 examples/s]
Map (num_proc=8):  28%|##7       | 588343/2104277 [01:18<02:16, 11123.80 examples/s]
Map (num_proc=8):  28%|##8       | 589672/2104277 [01:18<02:26, 10353.29 examples/s]
Map (num_proc=8):  28%|##8       | 590872/2104277 [01:18<02:22, 10604.03 examples/s]
Map (num_proc=8):  28%|##8       | 592200/2104277 [01:18<02:15, 11175.33 examples/s]
Map (num_proc=8):  28%|##8       | 593362/2104277 [01:19<02:27, 10234.37 examples/s]
Map (num_proc=8):  28%|##8       | 594660/2104277 [01:19<02:27, 10225.62 examples/s]
Map (num_proc=8):  28%|##8       | 596084/2104277 [01:19<02:20, 10732.52 examples/s]
Map (num_proc=8):  28%|##8       | 597675/2104277 [01:19<02:09, 11596.14 examples/s]
Map (num_proc=8):  28%|##8       | 598859/2104277 [01:19<02:18, 10855.07 examples/s]
Map (num_proc=8):  29%|##8       | 600027/2104277 [01:19<02:26, 10234.61 examples/s]
Map (num_proc=8):  29%|##8       | 601682/2104277 [01:19<02:12, 11341.51 examples/s]
Map (num_proc=8):  29%|##8       | 602880/2104277 [01:19<02:17, 10885.12 examples/s]
Map (num_proc=8):  29%|##8       | 604387/2104277 [01:20<02:09, 11551.75 examples/s]
Map (num_proc=8):  29%|##8       | 605712/2104277 [01:20<02:08, 11663.22 examples/s]
Map (num_proc=8):  29%|##8       | 607005/2104277 [01:20<02:18, 10837.22 examples/s]
Map (num_proc=8):  29%|##8       | 608296/2104277 [01:20<02:18, 10790.69 examples/s]
Map (num_proc=8):  29%|##8       | 609626/2104277 [01:20<02:11, 11380.50 examples/s]
Map (num_proc=8):  29%|##9       | 610860/2104277 [01:20<02:11, 11391.47 examples/s]
Map (num_proc=8):  29%|##9       | 612182/2104277 [01:20<02:06, 11827.07 examples/s]
Map (num_proc=8):  29%|##9       | 613485/2104277 [01:20<02:22, 10493.03 examples/s]
Map (num_proc=8):  29%|##9       | 614631/2104277 [01:21<02:21, 10515.95 examples/s]
Map (num_proc=8):  29%|##9       | 615753/2104277 [01:21<02:27, 10096.46 examples/s]
Map (num_proc=8):  29%|##9       | 616817/2104277 [01:21<02:50, 8730.42 examples/s] 
Map (num_proc=8):  29%|##9       | 618132/2104277 [01:21<02:34, 9636.49 examples/s]
Map (num_proc=8):  29%|##9       | 619268/2104277 [01:21<02:30, 9871.46 examples/s]
Map (num_proc=8):  29%|##9       | 620469/2104277 [01:21<02:58, 8300.65 examples/s]
Map (num_proc=8):  30%|##9       | 621508/2104277 [01:21<02:55, 8471.42 examples/s]
Map (num_proc=8):  30%|##9       | 622789/2104277 [01:21<02:38, 9335.15 examples/s]
Map (num_proc=8):  30%|##9       | 623789/2104277 [01:22<02:57, 8340.02 examples/s]
Map (num_proc=8):  30%|##9       | 624751/2104277 [01:22<02:52, 8588.23 examples/s]
Map (num_proc=8):  30%|##9       | 625711/2104277 [01:22<02:50, 8668.87 examples/s]
Map (num_proc=8):  30%|##9       | 626777/2104277 [01:22<02:51, 8626.23 examples/s]
Map (num_proc=8):  30%|##9       | 627762/2104277 [01:22<02:52, 8564.75 examples/s]
Map (num_proc=8):  30%|##9       | 629005/2104277 [01:22<02:37, 9362.41 examples/s]
Map (num_proc=8):  30%|##9       | 630173/2104277 [01:22<02:30, 9773.44 examples/s]
Map (num_proc=8):  30%|##9       | 631252/2104277 [01:22<02:55, 8378.17 examples/s]
Map (num_proc=8):  30%|###       | 632345/2104277 [01:23<03:26, 7119.78 examples/s]
Map (num_proc=8):  30%|###       | 633156/2104277 [01:23<03:28, 7059.75 examples/s]
Map (num_proc=8):  30%|###       | 634326/2104277 [01:23<03:02, 8034.93 examples/s]
Map (num_proc=8):  30%|###       | 635561/2104277 [01:23<02:45, 8853.56 examples/s]
Map (num_proc=8):  30%|###       | 636544/2104277 [01:23<02:57, 8262.49 examples/s]
Map (num_proc=8):  30%|###       | 637416/2104277 [01:23<03:02, 8053.59 examples/s]
Map (num_proc=8):  30%|###       | 638385/2104277 [01:23<02:56, 8303.99 examples/s]
Map (num_proc=8):  30%|###       | 639434/2104277 [01:23<03:00, 8128.25 examples/s]
Map (num_proc=8):  30%|###       | 640504/2104277 [01:24<02:47, 8745.05 examples/s]
Map (num_proc=8):  31%|###       | 642032/2104277 [01:24<02:22, 10262.80 examples/s]
Map (num_proc=8):  31%|###       | 643357/2104277 [01:24<02:15, 10753.10 examples/s]
Map (num_proc=8):  31%|###       | 644504/2104277 [01:24<02:39, 9143.13 examples/s] 
Map (num_proc=8):  31%|###       | 645600/2104277 [01:24<02:34, 9412.78 examples/s]
Map (num_proc=8):  31%|###       | 647034/2104277 [01:24<02:19, 10460.22 examples/s]
Map (num_proc=8):  31%|###       | 648493/2104277 [01:24<02:10, 11167.21 examples/s]
Map (num_proc=8):  31%|###       | 649712/2104277 [01:24<02:10, 11115.24 examples/s]
Map (num_proc=8):  31%|###       | 650904/2104277 [01:25<02:22, 10163.94 examples/s]
Map (num_proc=8):  31%|###       | 652244/2104277 [01:25<02:17, 10522.09 examples/s]
Map (num_proc=8):  31%|###1      | 653565/2104277 [01:25<02:14, 10746.61 examples/s]
Map (num_proc=8):  31%|###1      | 654778/2104277 [01:25<02:18, 10502.33 examples/s]
Map (num_proc=8):  31%|###1      | 655996/2104277 [01:25<02:15, 10726.69 examples/s]
Map (num_proc=8):  31%|###1      | 657547/2104277 [01:25<02:00, 11962.69 examples/s]
Map (num_proc=8):  31%|###1      | 658832/2104277 [01:25<02:23, 10090.36 examples/s]
Map (num_proc=8):  31%|###1      | 659911/2104277 [01:25<02:25, 9921.70 examples/s] 
Map (num_proc=8):  31%|###1      | 661349/2104277 [01:26<02:10, 11024.23 examples/s]
Map (num_proc=8):  31%|###1      | 662625/2104277 [01:26<02:11, 10989.79 examples/s]
Map (num_proc=8):  32%|###1      | 663768/2104277 [01:26<02:12, 10843.95 examples/s]
Map (num_proc=8):  32%|###1      | 665060/2104277 [01:26<02:12, 10822.13 examples/s]
Map (num_proc=8):  32%|###1      | 666455/2104277 [01:26<02:05, 11416.29 examples/s]
Map (num_proc=8):  32%|###1      | 667660/2104277 [01:26<02:15, 10591.07 examples/s]
Map (num_proc=8):  32%|###1      | 668821/2104277 [01:26<02:22, 10067.90 examples/s]
Map (num_proc=8):  32%|###1      | 670484/2104277 [01:26<02:08, 11142.73 examples/s]
Map (num_proc=8):  32%|###1      | 672118/2104277 [01:26<01:57, 12203.82 examples/s]
Map (num_proc=8):  32%|###2      | 673468/2104277 [01:27<02:22, 10039.17 examples/s]
Map (num_proc=8):  32%|###2      | 674572/2104277 [01:27<02:22, 10044.52 examples/s]
Map (num_proc=8):  32%|###2      | 675951/2104277 [01:27<02:12, 10776.27 examples/s]
Map (num_proc=8):  32%|###2      | 677446/2104277 [01:27<02:01, 11782.86 examples/s]
Map (num_proc=8):  32%|###2      | 678774/2104277 [01:27<02:15, 10499.53 examples/s]
Map (num_proc=8):  32%|###2      | 680264/2104277 [01:27<02:07, 11174.19 examples/s]
Map (num_proc=8):  32%|###2      | 681489/2104277 [01:27<02:06, 11215.08 examples/s]
Map (num_proc=8):  32%|###2      | 682673/2104277 [01:28<02:22, 9971.16 examples/s] 
Map (num_proc=8):  33%|###2      | 684103/2104277 [01:28<02:10, 10883.37 examples/s]
Map (num_proc=8):  33%|###2      | 685746/2104277 [01:28<01:59, 11883.09 examples/s]
Map (num_proc=8):  33%|###2      | 687037/2104277 [01:28<02:10, 10890.35 examples/s]
Map (num_proc=8):  33%|###2      | 688290/2104277 [01:28<02:26, 9676.29 examples/s] 
Map (num_proc=8):  33%|###2      | 689776/2104277 [01:28<02:12, 10696.17 examples/s]
Map (num_proc=8):  33%|###2      | 691366/2104277 [01:28<02:00, 11723.25 examples/s]
Map (num_proc=8):  33%|###2      | 692659/2104277 [01:28<02:12, 10633.10 examples/s]
Map (num_proc=8):  33%|###2      | 694136/2104277 [01:29<02:11, 10732.37 examples/s]
Map (num_proc=8):  33%|###3      | 695470/2104277 [01:29<02:09, 10886.26 examples/s]
Map (num_proc=8):  33%|###3      | 696791/2104277 [01:29<02:13, 10571.29 examples/s]
Map (num_proc=8):  33%|###3      | 698002/2104277 [01:29<02:09, 10820.12 examples/s]
Map (num_proc=8):  33%|###3      | 699479/2104277 [01:29<02:02, 11485.40 examples/s]
Map (num_proc=8):  33%|###3      | 700832/2104277 [01:29<01:59, 11776.18 examples/s]
Map (num_proc=8):  33%|###3      | 702191/2104277 [01:29<02:23, 9741.75 examples/s] 
Map (num_proc=8):  33%|###3      | 703414/2104277 [01:29<02:18, 10137.97 examples/s]
Map (num_proc=8):  33%|###3      | 704769/2104277 [01:30<02:13, 10459.10 examples/s]
Map (num_proc=8):  34%|###3      | 705907/2104277 [01:30<02:31, 9244.56 examples/s] 
Map (num_proc=8):  34%|###3      | 707164/2104277 [01:30<02:19, 10031.69 examples/s]
Map (num_proc=8):  34%|###3      | 708470/2104277 [01:30<02:14, 10363.37 examples/s]
Map (num_proc=8):  34%|###3      | 709803/2104277 [01:30<02:17, 10109.04 examples/s]
Map (num_proc=8):  34%|###3      | 710956/2104277 [01:30<02:32, 9159.89 examples/s] 
Map (num_proc=8):  34%|###3      | 712231/2104277 [01:30<02:20, 9918.10 examples/s]
Map (num_proc=8):  34%|###3      | 713363/2104277 [01:30<02:18, 10054.27 examples/s]
Map (num_proc=8):  34%|###3      | 714518/2104277 [01:31<02:20, 9918.26 examples/s] 
Map (num_proc=8):  34%|###4      | 715729/2104277 [01:31<02:12, 10478.75 examples/s]
Map (num_proc=8):  34%|###4      | 717129/2104277 [01:31<02:05, 11024.20 examples/s]
Map (num_proc=8):  34%|###4      | 718434/2104277 [01:31<02:01, 11428.43 examples/s]
Map (num_proc=8):  34%|###4      | 719722/2104277 [01:31<02:23, 9679.07 examples/s] 
Map (num_proc=8):  34%|###4      | 720965/2104277 [01:31<02:15, 10226.27 examples/s]
Map (num_proc=8):  34%|###4      | 722322/2104277 [01:31<02:04, 11073.22 examples/s]
Map (num_proc=8):  34%|###4      | 723828/2104277 [01:31<01:55, 11964.49 examples/s]
Map (num_proc=8):  34%|###4      | 725231/2104277 [01:32<02:04, 11075.05 examples/s]
Map (num_proc=8):  35%|###4      | 726562/2104277 [01:32<02:08, 10739.45 examples/s]
Map (num_proc=8):  35%|###4      | 727833/2104277 [01:32<02:05, 10933.41 examples/s]
Map (num_proc=8):  35%|###4      | 728998/2104277 [01:32<02:12, 10390.00 examples/s]
Map (num_proc=8):  35%|###4      | 730325/2104277 [01:32<02:06, 10851.57 examples/s]
Map (num_proc=8):  35%|###4      | 731702/2104277 [01:32<01:59, 11502.52 examples/s]
Map (num_proc=8):  35%|###4      | 733089/2104277 [01:32<01:58, 11605.17 examples/s]
Map (num_proc=8):  35%|###4      | 734352/2104277 [01:32<02:21, 9671.95 examples/s] 
Map (num_proc=8):  35%|###4      | 735503/2104277 [01:33<02:17, 9958.19 examples/s]
Map (num_proc=8):  35%|###5      | 737003/2104277 [01:33<02:03, 11069.59 examples/s]
Map (num_proc=8):  35%|###5      | 738178/2104277 [01:33<02:07, 10728.11 examples/s]
Map (num_proc=8):  35%|###5      | 739559/2104277 [01:33<02:03, 11059.62 examples/s]
Map (num_proc=8):  35%|###5      | 740756/2104277 [01:33<02:02, 11143.45 examples/s]
Map (num_proc=8):  35%|###5      | 741899/2104277 [01:33<02:09, 10480.83 examples/s]
Map (num_proc=8):  35%|###5      | 742986/2104277 [01:33<02:19, 9729.93 examples/s] 
Map (num_proc=8):  35%|###5      | 744551/2104277 [01:33<02:00, 11259.53 examples/s]
Map (num_proc=8):  35%|###5      | 745732/2104277 [01:33<02:06, 10772.57 examples/s]
Map (num_proc=8):  35%|###5      | 746872/2104277 [01:34<02:09, 10493.79 examples/s]
Map (num_proc=8):  36%|###5      | 748044/2104277 [01:34<02:26, 9257.59 examples/s] 
Map (num_proc=8):  36%|###5      | 749055/2104277 [01:34<02:27, 9172.65 examples/s]
Map (num_proc=8):  36%|###5      | 750129/2104277 [01:34<02:32, 8852.73 examples/s]
Map (num_proc=8):  36%|###5      | 751100/2104277 [01:34<02:30, 8993.22 examples/s]
Map (num_proc=8):  36%|###5      | 752084/2104277 [01:34<02:29, 9032.82 examples/s]
Map (num_proc=8):  36%|###5      | 753026/2104277 [01:34<02:31, 8943.21 examples/s]
Map (num_proc=8):  36%|###5      | 754051/2104277 [01:34<03:00, 7463.35 examples/s]
Map (num_proc=8):  36%|###5      | 755237/2104277 [01:35<02:41, 8334.13 examples/s]
Map (num_proc=8):  36%|###5      | 756633/2104277 [01:35<02:20, 9601.92 examples/s]
Map (num_proc=8):  36%|###6      | 757971/2104277 [01:35<02:10, 10286.35 examples/s]
Map (num_proc=8):  36%|###6      | 759187/2104277 [01:35<02:19, 9613.08 examples/s] 
Map (num_proc=8):  36%|###6      | 760227/2104277 [01:35<02:19, 9617.59 examples/s]
Map (num_proc=8):  36%|###6      | 761558/2104277 [01:35<02:07, 10498.80 examples/s]
Map (num_proc=8):  36%|###6      | 762881/2104277 [01:35<02:07, 10533.22 examples/s]
Map (num_proc=8):  36%|###6      | 764105/2104277 [01:35<02:08, 10403.38 examples/s]
Map (num_proc=8):  36%|###6      | 765253/2104277 [01:36<02:07, 10496.53 examples/s]
Map (num_proc=8):  36%|###6      | 766453/2104277 [01:36<02:03, 10808.52 examples/s]
Map (num_proc=8):  36%|###6      | 767658/2104277 [01:36<02:18, 9639.10 examples/s] 
Map (num_proc=8):  37%|###6      | 768732/2104277 [01:36<02:17, 9715.51 examples/s]
Map (num_proc=8):  37%|###6      | 770212/2104277 [01:36<02:02, 10869.72 examples/s]
Map (num_proc=8):  37%|###6      | 771622/2104277 [01:36<01:53, 11715.99 examples/s]
Map (num_proc=8):  37%|###6      | 772914/2104277 [01:36<02:22, 9337.09 examples/s] 
Map (num_proc=8):  37%|###6      | 774045/2104277 [01:36<02:42, 8189.77 examples/s]
Map (num_proc=8):  37%|###6      | 774974/2104277 [01:37<02:41, 8216.15 examples/s]
Map (num_proc=8):  37%|###6      | 775959/2104277 [01:37<02:44, 8068.55 examples/s]
Map (num_proc=8):  37%|###6      | 776916/2104277 [01:37<02:43, 8097.43 examples/s]
Map (num_proc=8):  37%|###6      | 777812/2104277 [01:37<02:49, 7804.43 examples/s]
Map (num_proc=8):  37%|###7      | 778798/2104277 [01:37<02:45, 8007.68 examples/s]
Map (num_proc=8):  37%|###7      | 779719/2104277 [01:37<02:54, 7594.61 examples/s]
Map (num_proc=8):  37%|###7      | 780569/2104277 [01:37<02:56, 7501.34 examples/s]
Map (num_proc=8):  37%|###7      | 781708/2104277 [01:37<02:42, 8148.51 examples/s]
Map (num_proc=8):  37%|###7      | 782604/2104277 [01:38<02:51, 7716.45 examples/s]
Map (num_proc=8):  37%|###7      | 783517/2104277 [01:38<02:48, 7831.38 examples/s]
Map (num_proc=8):  37%|###7      | 784329/2104277 [01:38<02:49, 7770.20 examples/s]
Map (num_proc=8):  37%|###7      | 785235/2104277 [01:38<02:48, 7844.86 examples/s]
Map (num_proc=8):  37%|###7      | 786057/2104277 [01:38<02:49, 7789.81 examples/s]
Map (num_proc=8):  37%|###7      | 787042/2104277 [01:38<02:41, 8162.00 examples/s]
Map (num_proc=8):  37%|###7      | 787995/2104277 [01:38<02:48, 7813.67 examples/s]
Map (num_proc=8):  37%|###7      | 788911/2104277 [01:38<02:42, 8072.14 examples/s]
Map (num_proc=8):  38%|###7      | 789891/2104277 [01:38<02:46, 7905.42 examples/s]
Map (num_proc=8):  38%|###7      | 790765/2104277 [01:39<02:45, 7957.52 examples/s]
Map (num_proc=8):  38%|###7      | 791601/2104277 [01:39<02:47, 7848.48 examples/s]
Map (num_proc=8):  38%|###7      | 792389/2104277 [01:39<02:54, 7517.02 examples/s]
Map (num_proc=8):  38%|###7      | 793285/2104277 [01:39<02:50, 7699.12 examples/s]
Map (num_proc=8):  38%|###7      | 798356/2104277 [01:39<01:10, 18614.07 examples/s]
Map (num_proc=8):  38%|###8      | 800242/2104277 [01:39<01:36, 13580.44 examples/s]
Map (num_proc=8):  38%|###8      | 801819/2104277 [01:40<01:58, 10986.13 examples/s]
Map (num_proc=8):  38%|###8      | 803139/2104277 [01:40<01:59, 10868.23 examples/s]
Map (num_proc=8):  38%|###8      | 804394/2104277 [01:40<02:02, 10633.73 examples/s]
Map (num_proc=8):  38%|###8      | 805619/2104277 [01:40<01:59, 10853.96 examples/s]
Map (num_proc=8):  38%|###8      | 806880/2104277 [01:40<02:02, 10612.90 examples/s]
Map (num_proc=8):  38%|###8      | 808091/2104277 [01:40<02:03, 10496.31 examples/s]
Map (num_proc=8):  38%|###8      | 809176/2104277 [01:40<02:14, 9601.47 examples/s] 
Map (num_proc=8):  39%|###8      | 810276/2104277 [01:40<02:12, 9754.93 examples/s]
Map (num_proc=8):  39%|###8      | 811283/2104277 [01:40<02:14, 9612.14 examples/s]
Map (num_proc=8):  39%|###8      | 812581/2104277 [01:41<02:05, 10309.47 examples/s]
Map (num_proc=8):  39%|###8      | 813765/2104277 [01:41<02:05, 10302.82 examples/s]
Map (num_proc=8):  39%|###8      | 815120/2104277 [01:41<02:01, 10579.33 examples/s]
Map (num_proc=8):  39%|###8      | 816189/2104277 [01:41<02:08, 10019.41 examples/s]
Map (num_proc=8):  39%|###8      | 817351/2104277 [01:41<02:08, 10043.24 examples/s]
Map (num_proc=8):  39%|###8      | 818581/2104277 [01:41<02:01, 10564.72 examples/s]
Map (num_proc=8):  39%|###8      | 819962/2104277 [01:41<01:53, 11332.35 examples/s]
Map (num_proc=8):  39%|###9      | 821252/2104277 [01:41<01:49, 11710.09 examples/s]
Map (num_proc=8):  39%|###9      | 822545/2104277 [01:41<01:50, 11599.37 examples/s]
Map (num_proc=8):  39%|###9      | 823833/2104277 [01:42<01:48, 11784.57 examples/s]
Map (num_proc=8):  39%|###9      | 825060/2104277 [01:42<01:48, 11785.03 examples/s]
Map (num_proc=8):  39%|###9      | 826346/2104277 [01:42<02:02, 10441.78 examples/s]
Map (num_proc=8):  39%|###9      | 827555/2104277 [01:42<02:17, 9258.78 examples/s] 
Map (num_proc=8):  39%|###9      | 828593/2104277 [01:42<02:14, 9462.39 examples/s]
Map (num_proc=8):  39%|###9      | 830137/2104277 [01:42<01:56, 10902.61 examples/s]
Map (num_proc=8):  40%|###9      | 831361/2104277 [01:42<01:56, 10883.07 examples/s]
Map (num_proc=8):  40%|###9      | 832486/2104277 [01:43<02:33, 8307.71 examples/s] 
Map (num_proc=8):  40%|###9      | 833877/2104277 [01:43<02:15, 9397.12 examples/s]
Map (num_proc=8):  40%|###9      | 835276/2104277 [01:43<02:01, 10417.77 examples/s]
Map (num_proc=8):  40%|###9      | 836443/2104277 [01:43<02:16, 9316.65 examples/s] 
Map (num_proc=8):  40%|###9      | 837464/2104277 [01:43<02:13, 9471.97 examples/s]
Map (num_proc=8):  40%|###9      | 838825/2104277 [01:43<02:01, 10412.40 examples/s]
Map (num_proc=8):  40%|###9      | 840175/2104277 [01:43<01:53, 11112.69 examples/s]
Map (num_proc=8):  40%|###9      | 841443/2104277 [01:43<02:00, 10477.99 examples/s]
Map (num_proc=8):  40%|####      | 842663/2104277 [01:43<02:05, 10088.60 examples/s]
Map (num_proc=8):  40%|####      | 844133/2104277 [01:44<01:53, 11127.37 examples/s]
Map (num_proc=8):  40%|####      | 845725/2104277 [01:44<01:43, 12203.49 examples/s]
Map (num_proc=8):  40%|####      | 847207/2104277 [01:44<01:39, 12639.46 examples/s]
Map (num_proc=8):  40%|####      | 848535/2104277 [01:44<02:03, 10136.93 examples/s]
Map (num_proc=8):  40%|####      | 849698/2104277 [01:44<01:59, 10458.50 examples/s]
Map (num_proc=8):  40%|####      | 851407/2104277 [01:44<01:44, 11962.43 examples/s]
Map (num_proc=8):  41%|####      | 852815/2104277 [01:44<01:42, 12196.85 examples/s]
Map (num_proc=8):  41%|####      | 854093/2104277 [01:44<01:57, 10668.29 examples/s]
Map (num_proc=8):  41%|####      | 855733/2104277 [01:45<01:44, 11943.59 examples/s]
Map (num_proc=8):  41%|####      | 857067/2104277 [01:45<01:48, 11499.20 examples/s]
Map (num_proc=8):  41%|####      | 858278/2104277 [01:45<02:15, 9188.71 examples/s] 
Map (num_proc=8):  41%|####      | 859387/2104277 [01:45<02:24, 8635.00 examples/s]
Map (num_proc=8):  41%|####      | 860757/2104277 [01:45<02:09, 9596.25 examples/s]
Map (num_proc=8):  41%|####      | 862194/2104277 [01:45<02:00, 10341.05 examples/s]
Map (num_proc=8):  41%|####1     | 863394/2104277 [01:45<02:08, 9651.09 examples/s] 
Map (num_proc=8):  41%|####1     | 864478/2104277 [01:46<02:13, 9311.68 examples/s]
Map (num_proc=8):  41%|####1     | 865523/2104277 [01:46<02:09, 9589.09 examples/s]
Map (num_proc=8):  41%|####1     | 866803/2104277 [01:46<02:04, 9906.88 examples/s]
Map (num_proc=8):  41%|####1     | 867940/2104277 [01:46<02:08, 9586.95 examples/s]
Map (num_proc=8):  41%|####1     | 868935/2104277 [01:46<02:09, 9513.68 examples/s]
Map (num_proc=8):  41%|####1     | 870465/2104277 [01:46<01:53, 10840.46 examples/s]
Map (num_proc=8):  41%|####1     | 871608/2104277 [01:46<01:53, 10886.25 examples/s]
Map (num_proc=8):  41%|####1     | 872774/2104277 [01:46<02:23, 8600.31 examples/s] 
Map (num_proc=8):  42%|####1     | 874201/2104277 [01:47<02:11, 9338.41 examples/s]
Map (num_proc=8):  42%|####1     | 875972/2104277 [01:47<01:50, 11120.80 examples/s]
Map (num_proc=8):  42%|####1     | 877249/2104277 [01:47<01:51, 11004.34 examples/s]
Map (num_proc=8):  42%|####1     | 878458/2104277 [01:47<01:57, 10445.97 examples/s]
Map (num_proc=8):  42%|####1     | 879774/2104277 [01:47<01:50, 11045.94 examples/s]
Map (num_proc=8):  42%|####1     | 880993/2104277 [01:47<01:48, 11286.78 examples/s]
Map (num_proc=8):  42%|####1     | 882337/2104277 [01:47<01:49, 11173.60 examples/s]
Map (num_proc=8):  42%|####1     | 883576/2104277 [01:47<02:00, 10117.79 examples/s]
Map (num_proc=8):  42%|####2     | 884884/2104277 [01:48<01:55, 10598.83 examples/s]
Map (num_proc=8):  42%|####2     | 886486/2104277 [01:48<01:43, 11731.50 examples/s]
Map (num_proc=8):  42%|####2     | 887720/2104277 [01:48<01:45, 11554.63 examples/s]
Map (num_proc=8):  42%|####2     | 888959/2104277 [01:48<01:45, 11501.56 examples/s]
Map (num_proc=8):  42%|####2     | 890184/2104277 [01:48<01:49, 11047.19 examples/s]
Map (num_proc=8):  42%|####2     | 891422/2104277 [01:48<01:46, 11400.16 examples/s]
Map (num_proc=8):  42%|####2     | 892607/2104277 [01:48<01:46, 11364.41 examples/s]
Map (num_proc=8):  42%|####2     | 893780/2104277 [01:48<01:47, 11280.34 examples/s]
Map (num_proc=8):  43%|####2     | 894975/2104277 [01:48<01:49, 11001.07 examples/s]
Map (num_proc=8):  43%|####2     | 896318/2104277 [01:49<01:44, 11518.59 examples/s]
Map (num_proc=8):  43%|####2     | 898130/2104277 [01:49<01:33, 12879.55 examples/s]
Map (num_proc=8):  43%|####2     | 899548/2104277 [01:49<01:43, 11614.47 examples/s]
Map (num_proc=8):  43%|####2     | 900867/2104277 [01:49<01:57, 10220.47 examples/s]
Map (num_proc=8):  43%|####2     | 902401/2104277 [01:49<01:45, 11445.74 examples/s]
Map (num_proc=8):  43%|####2     | 903895/2104277 [01:49<01:39, 12006.31 examples/s]
Map (num_proc=8):  43%|####3     | 905287/2104277 [01:49<01:40, 11944.23 examples/s]
Map (num_proc=8):  43%|####3     | 906616/2104277 [01:49<01:46, 11201.14 examples/s]
Map (num_proc=8):  43%|####3     | 908097/2104277 [01:50<01:38, 12095.71 examples/s]
Map (num_proc=8):  43%|####3     | 909429/2104277 [01:50<01:37, 12223.61 examples/s]
Map (num_proc=8):  43%|####3     | 910702/2104277 [01:50<01:49, 10905.18 examples/s]
Map (num_proc=8):  43%|####3     | 911864/2104277 [01:50<01:53, 10475.54 examples/s]
Map (num_proc=8):  43%|####3     | 913174/2104277 [01:50<01:46, 11144.41 examples/s]
Map (num_proc=8):  43%|####3     | 914907/2104277 [01:50<01:36, 12343.35 examples/s]
Map (num_proc=8):  44%|####3     | 916211/2104277 [01:50<01:38, 12107.46 examples/s]
Map (num_proc=8):  44%|####3     | 917454/2104277 [01:50<01:46, 11135.58 examples/s]
Map (num_proc=8):  44%|####3     | 918679/2104277 [01:50<01:47, 11026.38 examples/s]
Map (num_proc=8):  44%|####3     | 920102/2104277 [01:51<01:43, 11476.56 examples/s]
Map (num_proc=8):  44%|####3     | 921388/2104277 [01:51<01:43, 11416.62 examples/s]
Map (num_proc=8):  44%|####3     | 922741/2104277 [01:51<01:39, 11901.81 examples/s]
Map (num_proc=8):  44%|####3     | 923990/2104277 [01:51<01:41, 11636.08 examples/s]
Map (num_proc=8):  44%|####3     | 925316/2104277 [01:51<01:39, 11887.60 examples/s]
Map (num_proc=8):  44%|####4     | 926513/2104277 [01:51<01:47, 10905.98 examples/s]
Map (num_proc=8):  44%|####4     | 927717/2104277 [01:51<01:47, 10907.71 examples/s]
Map (num_proc=8):  44%|####4     | 928946/2104277 [01:51<01:49, 10769.75 examples/s]
Map (num_proc=8):  44%|####4     | 930173/2104277 [01:51<01:47, 10962.92 examples/s]
Map (num_proc=8):  44%|####4     | 931578/2104277 [01:52<01:40, 11685.03 examples/s]
Map (num_proc=8):  44%|####4     | 932875/2104277 [01:52<01:48, 10766.48 examples/s]
Map (num_proc=8):  44%|####4     | 934102/2104277 [01:52<01:46, 10972.85 examples/s]
Map (num_proc=8):  44%|####4     | 935640/2104277 [01:52<01:41, 11536.47 examples/s]
Map (num_proc=8):  45%|####4     | 937043/2104277 [01:52<01:38, 11856.50 examples/s]
Map (num_proc=8):  45%|####4     | 938255/2104277 [01:52<01:53, 10249.99 examples/s]
Map (num_proc=8):  45%|####4     | 939636/2104277 [01:52<01:48, 10775.26 examples/s]
Map (num_proc=8):  45%|####4     | 941052/2104277 [01:52<01:40, 11523.74 examples/s]
Map (num_proc=8):  45%|####4     | 942608/2104277 [01:53<01:35, 12115.08 examples/s]
Map (num_proc=8):  45%|####4     | 944078/2104277 [01:53<01:34, 12247.37 examples/s]
Map (num_proc=8):  45%|####4     | 945460/2104277 [01:53<01:45, 10946.75 examples/s]
Map (num_proc=8):  45%|####4     | 946686/2104277 [01:53<01:45, 10941.23 examples/s]
Map (num_proc=8):  45%|####5     | 948059/2104277 [01:53<01:39, 11568.01 examples/s]
Map (num_proc=8):  45%|####5     | 949319/2104277 [01:53<01:39, 11654.25 examples/s]
Map (num_proc=8):  45%|####5     | 950554/2104277 [01:53<01:49, 10512.48 examples/s]
Map (num_proc=8):  45%|####5     | 952005/2104277 [01:53<01:41, 11405.46 examples/s]
Map (num_proc=8):  45%|####5     | 953393/2104277 [01:53<01:35, 12034.48 examples/s]
Map (num_proc=8):  45%|####5     | 954781/2104277 [01:54<01:34, 12141.60 examples/s]
Map (num_proc=8):  45%|####5     | 956070/2104277 [01:54<01:53, 10095.13 examples/s]
Map (num_proc=8):  46%|####5     | 957505/2104277 [01:54<01:44, 10993.58 examples/s]
Map (num_proc=8):  46%|####5     | 958816/2104277 [01:54<01:39, 11508.89 examples/s]
Map (num_proc=8):  46%|####5     | 960401/2104277 [01:54<01:30, 12635.34 examples/s]
Map (num_proc=8):  46%|####5     | 961789/2104277 [01:54<01:43, 11007.52 examples/s]
Map (num_proc=8):  46%|####5     | 963090/2104277 [01:54<01:44, 10879.76 examples/s]
Map (num_proc=8):  46%|####5     | 964399/2104277 [01:55<01:43, 10968.60 examples/s]
Map (num_proc=8):  46%|####5     | 965829/2104277 [01:55<01:40, 11303.69 examples/s]
Map (num_proc=8):  46%|####5     | 967115/2104277 [01:55<01:50, 10266.07 examples/s]
Map (num_proc=8):  46%|####6     | 968253/2104277 [01:55<02:01, 9343.32 examples/s] 
Map (num_proc=8):  46%|####6     | 969548/2104277 [01:55<01:52, 10060.02 examples/s]
Map (num_proc=8):  46%|####6     | 970857/2104277 [01:55<01:51, 10179.00 examples/s]
Map (num_proc=8):  46%|####6     | 972029/2104277 [01:55<02:04, 9096.10 examples/s] 
Map (num_proc=8):  46%|####6     | 973113/2104277 [01:55<02:05, 8980.30 examples/s]
Map (num_proc=8):  46%|####6     | 974234/2104277 [01:56<01:59, 9429.72 examples/s]
Map (num_proc=8):  46%|####6     | 975367/2104277 [01:56<01:55, 9767.10 examples/s]
Map (num_proc=8):  46%|####6     | 976409/2104277 [01:56<01:59, 9414.58 examples/s]
Map (num_proc=8):  46%|####6     | 977489/2104277 [01:56<02:00, 9337.92 examples/s]
Map (num_proc=8):  47%|####6     | 978906/2104277 [01:56<01:47, 10422.90 examples/s]
Map (num_proc=8):  47%|####6     | 980172/2104277 [01:56<01:42, 10998.49 examples/s]
Map (num_proc=8):  47%|####6     | 981421/2104277 [01:56<01:48, 10324.25 examples/s]
Map (num_proc=8):  47%|####6     | 982582/2104277 [01:56<01:47, 10398.27 examples/s]
Map (num_proc=8):  47%|####6     | 983819/2104277 [01:56<01:42, 10925.87 examples/s]
Map (num_proc=8):  47%|####6     | 985237/2104277 [01:57<01:35, 11659.66 examples/s]
Map (num_proc=8):  47%|####6     | 986493/2104277 [01:57<01:38, 11339.02 examples/s]
Map (num_proc=8):  47%|####6     | 987671/2104277 [01:57<01:40, 11094.94 examples/s]
Map (num_proc=8):  47%|####6     | 988909/2104277 [01:57<01:42, 10901.43 examples/s]
Map (num_proc=8):  47%|####7     | 990096/2104277 [01:57<01:46, 10456.13 examples/s]
Map (num_proc=8):  47%|####7     | 991512/2104277 [01:57<01:41, 10918.75 examples/s]
Map (num_proc=8):  47%|####7     | 992895/2104277 [01:57<01:38, 11335.47 examples/s]
Map (num_proc=8):  47%|####7     | 994223/2104277 [01:57<01:37, 11343.72 examples/s]
Map (num_proc=8):  47%|####7     | 995662/2104277 [01:57<01:32, 12050.16 examples/s]
Map (num_proc=8):  47%|####7     | 996948/2104277 [01:58<01:31, 12119.76 examples/s]
Map (num_proc=8):  47%|####7     | 998260/2104277 [01:58<01:33, 11803.60 examples/s]
Map (num_proc=8):  47%|####7     | 999509/2104277 [01:58<01:40, 11047.14 examples/s]
Map (num_proc=8):  48%|####7     | 1000737/2104277 [01:58<01:42, 10724.20 examples/s]
Map (num_proc=8):  48%|####7     | 1001889/2104277 [01:58<01:42, 10731.36 examples/s]
Map (num_proc=8):  48%|####7     | 1003203/2104277 [01:58<01:38, 11201.25 examples/s]
Map (num_proc=8):  48%|####7     | 1004355/2104277 [01:58<01:37, 11285.60 examples/s]
Map (num_proc=8):  48%|####7     | 1005589/2104277 [01:58<01:40, 10901.29 examples/s]
Map (num_proc=8):  48%|####7     | 1006835/2104277 [01:59<01:41, 10833.01 examples/s]
Map (num_proc=8):  48%|####7     | 1008107/2104277 [01:59<01:37, 11196.42 examples/s]
Map (num_proc=8):  48%|####7     | 1009359/2104277 [01:59<01:42, 10678.34 examples/s]
Map (num_proc=8):  48%|####8     | 1010438/2104277 [01:59<02:01, 9032.38 examples/s] 
Map (num_proc=8):  48%|####8     | 1011443/2104277 [01:59<01:57, 9269.29 examples/s]
Map (num_proc=8):  48%|####8     | 1012605/2104277 [01:59<01:52, 9715.54 examples/s]
Map (num_proc=8):  48%|####8     | 1013643/2104277 [01:59<01:50, 9867.11 examples/s]
Map (num_proc=8):  48%|####8     | 1014731/2104277 [01:59<02:01, 8960.06 examples/s]
Map (num_proc=8):  48%|####8     | 1015915/2104277 [01:59<01:53, 9627.08 examples/s]
Map (num_proc=8):  48%|####8     | 1017274/2104277 [02:00<01:47, 10092.13 examples/s]
Map (num_proc=8):  48%|####8     | 1018648/2104277 [02:00<01:40, 10803.82 examples/s]
Map (num_proc=8):  48%|####8     | 1019828/2104277 [02:00<02:06, 8592.58 examples/s] 
Map (num_proc=8):  49%|####8     | 1020820/2104277 [02:00<02:08, 8464.12 examples/s]
Map (num_proc=8):  49%|####8     | 1021904/2104277 [02:00<02:05, 8641.15 examples/s]
Map (num_proc=8):  49%|####8     | 1023220/2104277 [02:00<01:56, 9274.00 examples/s]
Map (num_proc=8):  49%|####8     | 1024315/2104277 [02:00<01:53, 9549.21 examples/s]
Map (num_proc=8):  49%|####8     | 1025467/2104277 [02:00<01:52, 9620.90 examples/s]
Map (num_proc=8):  49%|####8     | 1026601/2104277 [02:01<01:49, 9802.65 examples/s]
Map (num_proc=8):  49%|####8     | 1027667/2104277 [02:01<01:57, 9151.49 examples/s]
Map (num_proc=8):  49%|####8     | 1028707/2104277 [02:01<01:58, 9055.44 examples/s]
Map (num_proc=8):  49%|####8     | 1029661/2104277 [02:01<01:58, 9055.25 examples/s]
Map (num_proc=8):  49%|####8     | 1031027/2104277 [02:01<01:44, 10291.67 examples/s]
Map (num_proc=8):  49%|####9     | 1032126/2104277 [02:01<01:42, 10464.84 examples/s]
Map (num_proc=8):  49%|####9     | 1033335/2104277 [02:01<01:51, 9636.23 examples/s] 
Map (num_proc=8):  49%|####9     | 1034418/2104277 [02:01<01:50, 9654.06 examples/s]
Map (num_proc=8):  49%|####9     | 1035945/2104277 [02:02<01:39, 10738.54 examples/s]
Map (num_proc=8):  49%|####9     | 1037258/2104277 [02:02<01:37, 10898.28 examples/s]
Map (num_proc=8):  49%|####9     | 1038507/2104277 [02:02<01:35, 11199.47 examples/s]
Map (num_proc=8):  49%|####9     | 1039717/2104277 [02:02<01:38, 10830.53 examples/s]
Map (num_proc=8):  49%|####9     | 1040861/2104277 [02:02<01:40, 10561.80 examples/s]
Map (num_proc=8):  50%|####9     | 1041988/2104277 [02:02<01:47, 9925.63 examples/s] 
Map (num_proc=8):  50%|####9     | 1043079/2104277 [02:02<02:01, 8728.73 examples/s]
Map (num_proc=8):  50%|####9     | 1044012/2104277 [02:02<02:03, 8584.34 examples/s]
Map (num_proc=8):  50%|####9     | 1045390/2104277 [02:03<01:54, 9284.11 examples/s]
Map (num_proc=8):  50%|####9     | 1046724/2104277 [02:03<01:43, 10266.59 examples/s]
Map (num_proc=8):  50%|####9     | 1047935/2104277 [02:03<01:41, 10360.97 examples/s]
Map (num_proc=8):  50%|####9     | 1049094/2104277 [02:03<01:53, 9322.17 examples/s] 
Map (num_proc=8):  50%|####9     | 1050152/2104277 [02:03<01:53, 9310.13 examples/s]
Map (num_proc=8):  50%|####9     | 1051268/2104277 [02:03<01:56, 9021.87 examples/s]
Map (num_proc=8):  50%|#####     | 1052783/2104277 [02:03<01:40, 10514.15 examples/s]
Map (num_proc=8):  50%|#####     | 1053959/2104277 [02:03<01:38, 10648.27 examples/s]
Map (num_proc=8):  50%|#####     | 1055427/2104277 [02:03<01:30, 11612.61 examples/s]
Map (num_proc=8):  50%|#####     | 1056650/2104277 [02:04<01:35, 10968.73 examples/s]
Map (num_proc=8):  50%|#####     | 1057822/2104277 [02:04<01:35, 10902.94 examples/s]
Map (num_proc=8):  50%|#####     | 1058937/2104277 [02:04<01:37, 10756.37 examples/s]
Map (num_proc=8):  50%|#####     | 1060260/2104277 [02:04<01:31, 11419.88 examples/s]
Map (num_proc=8):  50%|#####     | 1061493/2104277 [02:04<01:32, 11292.32 examples/s]
Map (num_proc=8):  51%|#####     | 1062673/2104277 [02:04<01:35, 10944.09 examples/s]
Map (num_proc=8):  51%|#####     | 1064122/2104277 [02:04<01:30, 11541.43 examples/s]
Map (num_proc=8):  51%|#####     | 1065291/2104277 [02:04<01:31, 11302.12 examples/s]
Map (num_proc=8):  51%|#####     | 1066726/2104277 [02:04<01:26, 12016.79 examples/s]
Map (num_proc=8):  51%|#####     | 1067948/2104277 [02:05<01:35, 10875.33 examples/s]
Map (num_proc=8):  51%|#####     | 1069237/2104277 [02:05<01:33, 11060.03 examples/s]
Map (num_proc=8):  51%|#####     | 1070437/2104277 [02:05<01:31, 11275.20 examples/s]
Map (num_proc=8):  51%|#####     | 1071818/2104277 [02:05<01:27, 11854.15 examples/s]
Map (num_proc=8):  51%|#####     | 1073135/2104277 [02:05<01:26, 11857.45 examples/s]
Map (num_proc=8):  51%|#####1    | 1074503/2104277 [02:05<01:26, 11966.45 examples/s]
Map (num_proc=8):  51%|#####1    | 1075762/2104277 [02:05<01:32, 11115.97 examples/s]
Map (num_proc=8):  51%|#####1    | 1077067/2104277 [02:05<01:29, 11447.41 examples/s]
Map (num_proc=8):  51%|#####1    | 1078604/2104277 [02:05<01:25, 11931.11 examples/s]
Map (num_proc=8):  51%|#####1    | 1079879/2104277 [02:06<01:36, 10627.76 examples/s]
Map (num_proc=8):  51%|#####1    | 1081171/2104277 [02:06<01:32, 11041.81 examples/s]
Map (num_proc=8):  51%|#####1    | 1082522/2104277 [02:06<01:30, 11271.44 examples/s]
Map (num_proc=8):  52%|#####1    | 1084184/2104277 [02:06<01:25, 11900.88 examples/s]
Map (num_proc=8):  52%|#####1    | 1085460/2104277 [02:06<01:25, 11943.14 examples/s]
Map (num_proc=8):  52%|#####1    | 1086668/2104277 [02:06<01:29, 11421.34 examples/s]
Map (num_proc=8):  52%|#####1    | 1087909/2104277 [02:06<01:29, 11382.87 examples/s]
Map (num_proc=8):  52%|#####1    | 1089208/2104277 [02:06<01:29, 11302.80 examples/s]
Map (num_proc=8):  52%|#####1    | 1090446/2104277 [02:07<01:32, 10970.27 examples/s]
Map (num_proc=8):  52%|#####1    | 1091594/2104277 [02:07<01:44, 9699.93 examples/s] 
Map (num_proc=8):  52%|#####1    | 1092646/2104277 [02:07<01:43, 9767.30 examples/s]
Map (num_proc=8):  52%|#####1    | 1093727/2104277 [02:07<01:43, 9800.78 examples/s]
Map (num_proc=8):  52%|#####2    | 1094969/2104277 [02:07<01:40, 10071.53 examples/s]
Map (num_proc=8):  52%|#####2    | 1096057/2104277 [02:07<01:50, 9089.43 examples/s] 
Map (num_proc=8):  52%|#####2    | 1097050/2104277 [02:07<01:53, 8879.27 examples/s]
Map (num_proc=8):  52%|#####2    | 1098070/2104277 [02:07<01:55, 8684.97 examples/s]
Map (num_proc=8):  52%|#####2    | 1099054/2104277 [02:08<01:52, 8956.76 examples/s]
Map (num_proc=8):  52%|#####2    | 1100191/2104277 [02:08<01:45, 9500.55 examples/s]
Map (num_proc=8):  52%|#####2    | 1101284/2104277 [02:08<02:08, 7795.29 examples/s]
Map (num_proc=8):  52%|#####2    | 1102643/2104277 [02:08<01:52, 8912.02 examples/s]
Map (num_proc=8):  52%|#####2    | 1104382/2104277 [02:08<01:31, 10899.58 examples/s]
Map (num_proc=8):  53%|#####2    | 1105685/2104277 [02:08<01:40, 9897.63 examples/s] 
Map (num_proc=8):  53%|#####2    | 1106853/2104277 [02:08<01:39, 9979.66 examples/s]
Map (num_proc=8):  53%|#####2    | 1108234/2104277 [02:08<01:32, 10726.06 examples/s]
Map (num_proc=8):  53%|#####2    | 1109536/2104277 [02:09<01:37, 10251.67 examples/s]
Map (num_proc=8):  53%|#####2    | 1110686/2104277 [02:09<01:48, 9140.79 examples/s] 
Map (num_proc=8):  53%|#####2    | 1112158/2104277 [02:09<01:36, 10248.15 examples/s]
Map (num_proc=8):  53%|#####2    | 1113632/2104277 [02:09<01:28, 11146.81 examples/s]
Map (num_proc=8):  53%|#####2    | 1114881/2104277 [02:09<01:30, 10981.45 examples/s]
Map (num_proc=8):  53%|#####3    | 1116015/2104277 [02:09<01:36, 10245.72 examples/s]
Map (num_proc=8):  53%|#####3    | 1117229/2104277 [02:09<01:37, 10108.73 examples/s]
Map (num_proc=8):  53%|#####3    | 1118356/2104277 [02:09<01:36, 10199.48 examples/s]
Map (num_proc=8):  53%|#####3    | 1119830/2104277 [02:10<01:35, 10281.14 examples/s]
Map (num_proc=8):  53%|#####3    | 1120885/2104277 [02:10<01:36, 10173.78 examples/s]
Map (num_proc=8):  53%|#####3    | 1122531/2104277 [02:10<01:24, 11568.81 examples/s]
Map (num_proc=8):  53%|#####3    | 1123757/2104277 [02:10<01:29, 10970.37 examples/s]
Map (num_proc=8):  53%|#####3    | 1124904/2104277 [02:10<01:39, 9799.58 examples/s] 
Map (num_proc=8):  54%|#####3    | 1126234/2104277 [02:10<01:32, 10613.35 examples/s]
Map (num_proc=8):  54%|#####3    | 1127416/2104277 [02:10<01:35, 10220.73 examples/s]
Map (num_proc=8):  54%|#####3    | 1128854/2104277 [02:10<01:26, 11243.62 examples/s]
Map (num_proc=8):  54%|#####3    | 1130152/2104277 [02:11<01:37, 10036.52 examples/s]
Map (num_proc=8):  54%|#####3    | 1131415/2104277 [02:11<01:34, 10319.63 examples/s]
Map (num_proc=8):  54%|#####3    | 1132583/2104277 [02:11<01:40, 9634.98 examples/s] 
Map (num_proc=8):  54%|#####3    | 1133805/2104277 [02:11<01:35, 10118.36 examples/s]
Map (num_proc=8):  54%|#####3    | 1135007/2104277 [02:11<01:31, 10607.64 examples/s]
Map (num_proc=8):  54%|#####4    | 1136382/2104277 [02:11<01:24, 11433.68 examples/s]
Map (num_proc=8):  54%|#####4    | 1137580/2104277 [02:11<01:34, 10245.30 examples/s]
Map (num_proc=8):  54%|#####4    | 1138803/2104277 [02:11<01:40, 9642.16 examples/s] 
Map (num_proc=8):  54%|#####4    | 1139984/2104277 [02:12<01:35, 10079.13 examples/s]
Map (num_proc=8):  54%|#####4    | 1141111/2104277 [02:12<01:34, 10174.88 examples/s]
Map (num_proc=8):  54%|#####4    | 1142246/2104277 [02:12<01:34, 10162.95 examples/s]
Map (num_proc=8):  54%|#####4    | 1143449/2104277 [02:12<01:32, 10400.53 examples/s]
Map (num_proc=8):  54%|#####4    | 1144607/2104277 [02:12<01:32, 10343.37 examples/s]
Map (num_proc=8):  54%|#####4    | 1145673/2104277 [02:12<01:35, 10000.10 examples/s]
Map (num_proc=8):  54%|#####4    | 1146742/2104277 [02:12<01:36, 9911.68 examples/s] 
Map (num_proc=8):  55%|#####4    | 1148084/2104277 [02:12<01:27, 10871.73 examples/s]
Map (num_proc=8):  55%|#####4    | 1149416/2104277 [02:12<01:35, 9989.40 examples/s] 
Map (num_proc=8):  55%|#####4    | 1150543/2104277 [02:13<01:39, 9600.08 examples/s]
Map (num_proc=8):  55%|#####4    | 1151593/2104277 [02:13<01:49, 8675.85 examples/s]
Map (num_proc=8):  55%|#####4    | 1152573/2104277 [02:13<01:54, 8326.93 examples/s]
Map (num_proc=8):  55%|#####4    | 1153938/2104277 [02:13<01:39, 9530.23 examples/s]
Map (num_proc=8):  55%|#####4    | 1155160/2104277 [02:13<01:33, 10168.97 examples/s]
Map (num_proc=8):  55%|#####4    | 1156460/2104277 [02:13<01:33, 10141.83 examples/s]
Map (num_proc=8):  55%|#####5    | 1157577/2104277 [02:13<01:36, 9840.83 examples/s] 
Map (num_proc=8):  55%|#####5    | 1158665/2104277 [02:13<01:33, 10109.91 examples/s]
Map (num_proc=8):  55%|#####5    | 1160010/2104277 [02:14<01:27, 10845.25 examples/s]
Map (num_proc=8):  55%|#####5    | 1161187/2104277 [02:14<01:26, 10941.06 examples/s]
Map (num_proc=8):  55%|#####5    | 1162473/2104277 [02:14<01:38, 9576.44 examples/s] 
Map (num_proc=8):  55%|#####5    | 1163649/2104277 [02:14<01:52, 8339.17 examples/s]
Map (num_proc=8):  55%|#####5    | 1164678/2104277 [02:14<01:57, 7997.34 examples/s]
Map (num_proc=8):  55%|#####5    | 1165946/2104277 [02:14<01:47, 8755.16 examples/s]
Map (num_proc=8):  55%|#####5    | 1166882/2104277 [02:14<02:19, 6704.88 examples/s]
Map (num_proc=8):  56%|#####5    | 1167975/2104277 [02:15<02:07, 7371.65 examples/s]
Map (num_proc=8):  56%|#####5    | 1169023/2104277 [02:15<01:58, 7865.69 examples/s]
Map (num_proc=8):  56%|#####5    | 1169975/2104277 [02:15<02:11, 7129.09 examples/s]
Map (num_proc=8):  56%|#####5    | 1170853/2104277 [02:15<02:08, 7251.90 examples/s]
Map (num_proc=8):  56%|#####5    | 1171828/2104277 [02:15<02:03, 7535.17 examples/s]
Map (num_proc=8):  56%|#####5    | 1172783/2104277 [02:15<01:59, 7826.91 examples/s]
Map (num_proc=8):  56%|#####5    | 1173821/2104277 [02:15<01:53, 8170.90 examples/s]
Map (num_proc=8):  56%|#####5    | 1174818/2104277 [02:15<01:51, 8356.65 examples/s]
Map (num_proc=8):  56%|#####5    | 1176008/2104277 [02:16<01:40, 9199.46 examples/s]
Map (num_proc=8):  56%|#####5    | 1177163/2104277 [02:16<01:41, 9098.61 examples/s]
Map (num_proc=8):  56%|#####5    | 1178224/2104277 [02:16<01:48, 8566.99 examples/s]
Map (num_proc=8):  56%|#####6    | 1179220/2104277 [02:16<01:46, 8724.48 examples/s]
Map (num_proc=8):  56%|#####6    | 1180273/2104277 [02:16<01:42, 9005.60 examples/s]
Map (num_proc=8):  56%|#####6    | 1181301/2104277 [02:16<01:40, 9190.06 examples/s]
Map (num_proc=8):  56%|#####6    | 1182234/2104277 [02:16<01:47, 8549.91 examples/s]
Map (num_proc=8):  56%|#####6    | 1183922/2104277 [02:16<01:28, 10359.75 examples/s]
Map (num_proc=8):  56%|#####6    | 1185059/2104277 [02:17<01:41, 9031.01 examples/s] 
Map (num_proc=8):  56%|#####6    | 1186313/2104277 [02:17<01:35, 9640.24 examples/s]
Map (num_proc=8):  56%|#####6    | 1187424/2104277 [02:17<01:33, 9812.54 examples/s]
Map (num_proc=8):  56%|#####6    | 1188438/2104277 [02:17<02:01, 7559.02 examples/s]
Map (num_proc=8):  57%|#####6    | 1189556/2104277 [02:17<01:50, 8308.57 examples/s]
Map (num_proc=8):  57%|#####6    | 1190795/2104277 [02:17<01:42, 8923.97 examples/s]
Map (num_proc=8):  57%|#####6    | 1191818/2104277 [02:17<01:46, 8585.85 examples/s]
Map (num_proc=8):  57%|#####6    | 1192802/2104277 [02:17<01:43, 8810.14 examples/s]
Map (num_proc=8):  57%|#####6    | 1193943/2104277 [02:18<01:36, 9463.39 examples/s]
Map (num_proc=8):  57%|#####6    | 1195209/2104277 [02:18<01:30, 10079.08 examples/s]
Map (num_proc=8):  57%|#####6    | 1196403/2104277 [02:18<01:27, 10324.26 examples/s]
Map (num_proc=8):  57%|#####6    | 1197460/2104277 [02:18<01:29, 10096.19 examples/s]
Map (num_proc=8):  57%|#####6    | 1198710/2104277 [02:18<01:24, 10761.62 examples/s]
Map (num_proc=8):  57%|#####7    | 1199886/2104277 [02:18<01:22, 10981.65 examples/s]
Map (num_proc=8):  57%|#####7    | 1201089/2104277 [02:18<01:27, 10361.57 examples/s]
Map (num_proc=8):  57%|#####7    | 1202288/2104277 [02:18<01:35, 9487.47 examples/s] 
Map (num_proc=8):  57%|#####7    | 1203772/2104277 [02:18<01:23, 10744.38 examples/s]
Map (num_proc=8):  57%|#####7    | 1205425/2104277 [02:19<01:14, 12022.89 examples/s]
Map (num_proc=8):  57%|#####7    | 1206698/2104277 [02:19<01:22, 10912.14 examples/s]
Map (num_proc=8):  57%|#####7    | 1207862/2104277 [02:19<01:26, 10349.77 examples/s]
Map (num_proc=8):  57%|#####7    | 1209140/2104277 [02:19<01:22, 10866.61 examples/s]
Map (num_proc=8):  58%|#####7    | 1210585/2104277 [02:19<01:19, 11245.61 examples/s]
Map (num_proc=8):  58%|#####7    | 1211812/2104277 [02:19<01:26, 10309.14 examples/s]
Map (num_proc=8):  58%|#####7    | 1213331/2104277 [02:19<01:19, 11183.30 examples/s]
Map (num_proc=8):  58%|#####7    | 1214816/2104277 [02:19<01:15, 11856.97 examples/s]
Map (num_proc=8):  58%|#####7    | 1216046/2104277 [02:20<01:33, 9517.37 examples/s] 
Map (num_proc=8):  58%|#####7    | 1217441/2104277 [02:20<01:26, 10253.50 examples/s]
Map (num_proc=8):  58%|#####7    | 1219001/2104277 [02:20<01:17, 11422.29 examples/s]
Map (num_proc=8):  58%|#####7    | 1220333/2104277 [02:20<01:15, 11673.65 examples/s]
Map (num_proc=8):  58%|#####8    | 1221731/2104277 [02:20<01:22, 10744.58 examples/s]
Map (num_proc=8):  58%|#####8    | 1222993/2104277 [02:20<01:23, 10540.39 examples/s]
Map (num_proc=8):  58%|#####8    | 1224132/2104277 [02:20<01:24, 10436.87 examples/s]
Map (num_proc=8):  58%|#####8    | 1225229/2104277 [02:20<01:24, 10441.17 examples/s]
Map (num_proc=8):  58%|#####8    | 1226411/2104277 [02:21<01:26, 10182.60 examples/s]
Map (num_proc=8):  58%|#####8    | 1228058/2104277 [02:21<01:14, 11838.61 examples/s]
Map (num_proc=8):  58%|#####8    | 1229284/2104277 [02:21<01:24, 10317.94 examples/s]
Map (num_proc=8):  58%|#####8    | 1230398/2104277 [02:21<01:25, 10193.14 examples/s]
Map (num_proc=8):  59%|#####8    | 1231469/2104277 [02:21<01:37, 8958.69 examples/s] 
Map (num_proc=8):  59%|#####8    | 1232423/2104277 [02:21<01:50, 7875.57 examples/s]
Map (num_proc=8):  59%|#####8    | 1233286/2104277 [02:21<02:00, 7246.35 examples/s]
Map (num_proc=8):  59%|#####8    | 1234056/2104277 [02:22<02:32, 5712.70 examples/s]
Map (num_proc=8):  59%|#####8    | 1235087/2104277 [02:22<02:13, 6529.79 examples/s]
Map (num_proc=8):  59%|#####8    | 1235839/2104277 [02:22<02:31, 5725.68 examples/s]
Map (num_proc=8):  59%|#####8    | 1236477/2104277 [02:22<02:28, 5860.48 examples/s]
Map (num_proc=8):  59%|#####8    | 1237932/2104277 [02:22<01:52, 7713.65 examples/s]
Map (num_proc=8):  59%|#####8    | 1238852/2104277 [02:22<02:02, 7066.03 examples/s]
Map (num_proc=8):  59%|#####8    | 1239728/2104277 [02:22<01:55, 7454.93 examples/s]
Map (num_proc=8):  59%|#####8    | 1240970/2104277 [02:23<01:40, 8549.44 examples/s]
Map (num_proc=8):  59%|#####9    | 1242028/2104277 [02:23<01:51, 7760.16 examples/s]
Map (num_proc=8):  59%|#####9    | 1243509/2104277 [02:23<01:32, 9308.54 examples/s]
Map (num_proc=8):  59%|#####9    | 1244907/2104277 [02:23<01:22, 10361.28 examples/s]
Map (num_proc=8):  59%|#####9    | 1246105/2104277 [02:23<01:37, 8794.85 examples/s] 
Map (num_proc=8):  59%|#####9    | 1247580/2104277 [02:23<01:24, 10189.00 examples/s]
Map (num_proc=8):  59%|#####9    | 1249246/2104277 [02:23<01:15, 11389.18 examples/s]
Map (num_proc=8):  59%|#####9    | 1250501/2104277 [02:23<01:25, 10039.10 examples/s]
Map (num_proc=8):  60%|#####9    | 1252181/2104277 [02:24<01:15, 11304.78 examples/s]
Map (num_proc=8):  60%|#####9    | 1253592/2104277 [02:24<01:13, 11582.16 examples/s]
Map (num_proc=8):  60%|#####9    | 1254973/2104277 [02:24<01:23, 10144.97 examples/s]
Map (num_proc=8):  60%|#####9    | 1256662/2104277 [02:24<01:13, 11610.85 examples/s]
Map (num_proc=8):  60%|#####9    | 1258196/2104277 [02:24<01:14, 11431.75 examples/s]
Map (num_proc=8):  60%|#####9    | 1259445/2104277 [02:24<01:16, 11017.13 examples/s]
Map (num_proc=8):  60%|#####9    | 1260807/2104277 [02:24<01:12, 11650.02 examples/s]
Map (num_proc=8):  60%|#####9    | 1262044/2104277 [02:24<01:11, 11835.61 examples/s]
Map (num_proc=8):  60%|######    | 1263419/2104277 [02:25<01:15, 11158.49 examples/s]
Map (num_proc=8):  60%|######    | 1264645/2104277 [02:25<01:15, 11070.15 examples/s]
Map (num_proc=8):  60%|######    | 1265983/2104277 [02:25<01:13, 11333.17 examples/s]
Map (num_proc=8):  60%|######    | 1267191/2104277 [02:25<01:14, 11249.40 examples/s]
Map (num_proc=8):  60%|######    | 1268380/2104277 [02:25<01:23, 10023.61 examples/s]
Map (num_proc=8):  60%|######    | 1269552/2104277 [02:25<01:20, 10331.59 examples/s]
Map (num_proc=8):  60%|######    | 1270690/2104277 [02:25<01:34, 8782.50 examples/s] 
Map (num_proc=8):  60%|######    | 1271796/2104277 [02:25<01:33, 8919.82 examples/s]
Map (num_proc=8):  60%|######    | 1273036/2104277 [02:26<01:26, 9606.66 examples/s]
Map (num_proc=8):  61%|######    | 1274141/2104277 [02:26<01:45, 7843.58 examples/s]
Map (num_proc=8):  61%|######    | 1275254/2104277 [02:26<01:39, 8319.71 examples/s]
Map (num_proc=8):  61%|######    | 1276744/2104277 [02:26<01:28, 9385.08 examples/s]
Map (num_proc=8):  61%|######    | 1277778/2104277 [02:26<01:36, 8555.93 examples/s]
Map (num_proc=8):  61%|######    | 1279397/2104277 [02:26<01:20, 10189.83 examples/s]
Map (num_proc=8):  61%|######    | 1280512/2104277 [02:26<01:20, 10230.08 examples/s]
Map (num_proc=8):  61%|######    | 1281645/2104277 [02:26<01:18, 10457.46 examples/s]
Map (num_proc=8):  61%|######    | 1283046/2104277 [02:27<01:12, 11320.71 examples/s]
Map (num_proc=8):  61%|######1   | 1284215/2104277 [02:27<01:15, 10798.17 examples/s]
Map (num_proc=8):  61%|######1   | 1285588/2104277 [02:27<01:13, 11112.91 examples/s]
Map (num_proc=8):  61%|######1   | 1286948/2104277 [02:27<01:10, 11671.99 examples/s]
Map (num_proc=8):  61%|######1   | 1288343/2104277 [02:27<01:08, 11894.28 examples/s]
Map (num_proc=8):  61%|######1   | 1289692/2104277 [02:27<01:11, 11356.17 examples/s]
Map (num_proc=8):  61%|######1   | 1291065/2104277 [02:27<01:10, 11520.91 examples/s]
Map (num_proc=8):  61%|######1   | 1292459/2104277 [02:27<01:15, 10822.13 examples/s]
Map (num_proc=8):  61%|######1   | 1293606/2104277 [02:28<01:13, 10986.72 examples/s]
Map (num_proc=8):  62%|######1   | 1295001/2104277 [02:28<01:11, 11346.74 examples/s]
Map (num_proc=8):  62%|######1   | 1296263/2104277 [02:28<01:11, 11325.94 examples/s]
Map (num_proc=8):  62%|######1   | 1297662/2104277 [02:28<01:10, 11489.11 examples/s]
Map (num_proc=8):  62%|######1   | 1298960/2104277 [02:28<01:16, 10471.40 examples/s]
Map (num_proc=8):  62%|######1   | 1300137/2104277 [02:28<01:14, 10798.46 examples/s]
Map (num_proc=8):  62%|######1   | 1301757/2104277 [02:28<01:06, 11990.07 examples/s]
Map (num_proc=8):  62%|######1   | 1303013/2104277 [02:28<01:13, 10927.95 examples/s]
Map (num_proc=8):  62%|######1   | 1304225/2104277 [02:28<01:13, 10951.13 examples/s]
Map (num_proc=8):  62%|######2   | 1305593/2104277 [02:29<01:10, 11258.35 examples/s]
Map (num_proc=8):  62%|######2   | 1306963/2104277 [02:29<01:08, 11569.12 examples/s]
Map (num_proc=8):  62%|######2   | 1308149/2104277 [02:29<01:16, 10401.73 examples/s]
Map (num_proc=8):  62%|######2   | 1309506/2104277 [02:29<01:12, 11030.71 examples/s]
Map (num_proc=8):  62%|######2   | 1310952/2104277 [02:29<01:07, 11807.99 examples/s]
Map (num_proc=8):  62%|######2   | 1312320/2104277 [02:29<01:13, 10813.88 examples/s]
Map (num_proc=8):  62%|######2   | 1313544/2104277 [02:29<01:11, 11132.87 examples/s]
Map (num_proc=8):  62%|######2   | 1314963/2104277 [02:29<01:08, 11566.02 examples/s]
Map (num_proc=8):  63%|######2   | 1316165/2104277 [02:30<01:07, 11685.44 examples/s]
Map (num_proc=8):  63%|######2   | 1317378/2104277 [02:30<01:12, 10816.40 examples/s]
Map (num_proc=8):  63%|######2   | 1318551/2104277 [02:30<01:13, 10650.07 examples/s]
Map (num_proc=8):  63%|######2   | 1319926/2104277 [02:30<01:09, 11275.17 examples/s]
Map (num_proc=8):  63%|######2   | 1321073/2104277 [02:30<01:20, 9716.64 examples/s] 
Map (num_proc=8):  63%|######2   | 1322132/2104277 [02:30<01:18, 9935.36 examples/s]
Map (num_proc=8):  63%|######2   | 1323299/2104277 [02:30<01:21, 9624.96 examples/s]
Map (num_proc=8):  63%|######2   | 1324410/2104277 [02:30<01:22, 9503.22 examples/s]
Map (num_proc=8):  63%|######2   | 1325575/2104277 [02:30<01:20, 9730.51 examples/s]
Map (num_proc=8):  63%|######3   | 1326650/2104277 [02:31<01:23, 9259.18 examples/s]
Map (num_proc=8):  63%|######3   | 1327637/2104277 [02:31<01:26, 8981.14 examples/s]
Map (num_proc=8):  63%|######3   | 1328919/2104277 [02:31<01:19, 9704.33 examples/s]
Map (num_proc=8):  63%|######3   | 1330089/2104277 [02:31<01:18, 9921.57 examples/s]
Map (num_proc=8):  63%|######3   | 1331472/2104277 [02:31<01:13, 10462.97 examples/s]
Map (num_proc=8):  63%|######3   | 1332599/2104277 [02:31<01:15, 10240.85 examples/s]
Map (num_proc=8):  63%|######3   | 1334106/2104277 [02:31<01:07, 11415.12 examples/s]
Map (num_proc=8):  63%|######3   | 1335460/2104277 [02:31<01:04, 11871.16 examples/s]
Map (num_proc=8):  64%|######3   | 1336801/2104277 [02:32<01:13, 10453.65 examples/s]
Map (num_proc=8):  64%|######3   | 1338158/2104277 [02:32<01:09, 11022.51 examples/s]
Map (num_proc=8):  64%|######3   | 1339643/2104277 [02:32<01:04, 11903.33 examples/s]
Map (num_proc=8):  64%|######3   | 1340936/2104277 [02:32<01:08, 11073.15 examples/s]
Map (num_proc=8):  64%|######3   | 1342210/2104277 [02:32<01:07, 11358.24 examples/s]
Map (num_proc=8):  64%|######3   | 1343629/2104277 [02:32<01:05, 11561.54 examples/s]
Map (num_proc=8):  64%|######3   | 1344951/2104277 [02:32<01:04, 11747.30 examples/s]
Map (num_proc=8):  64%|######3   | 1346204/2104277 [02:32<01:08, 11004.43 examples/s]
Map (num_proc=8):  64%|######4   | 1347682/2104277 [02:33<01:06, 11431.29 examples/s]
Map (num_proc=8):  64%|######4   | 1349049/2104277 [02:33<01:04, 11650.14 examples/s]
Map (num_proc=8):  64%|######4   | 1350534/2104277 [02:33<01:00, 12405.91 examples/s]
Map (num_proc=8):  64%|######4   | 1351792/2104277 [02:33<01:06, 11333.45 examples/s]
Map (num_proc=8):  64%|######4   | 1353043/2104277 [02:33<01:07, 11174.69 examples/s]
Map (num_proc=8):  64%|######4   | 1354266/2104277 [02:33<01:07, 11153.18 examples/s]
Map (num_proc=8):  64%|######4   | 1355625/2104277 [02:33<01:05, 11504.74 examples/s]
Map (num_proc=8):  64%|######4   | 1356950/2104277 [02:33<01:04, 11501.22 examples/s]
Map (num_proc=8):  65%|######4   | 1358233/2104277 [02:33<01:06, 11170.81 examples/s]
Map (num_proc=8):  65%|######4   | 1359521/2104277 [02:34<01:04, 11627.84 examples/s]
Map (num_proc=8):  65%|######4   | 1360840/2104277 [02:34<01:03, 11671.66 examples/s]
Map (num_proc=8):  65%|######4   | 1362254/2104277 [02:34<01:06, 11096.19 examples/s]
Map (num_proc=8):  65%|######4   | 1363567/2104277 [02:34<01:04, 11539.17 examples/s]
Map (num_proc=8):  65%|######4   | 1364759/2104277 [02:34<01:06, 11172.61 examples/s]
Map (num_proc=8):  65%|######4   | 1366028/2104277 [02:34<01:04, 11374.20 examples/s]
Map (num_proc=8):  65%|######4   | 1367438/2104277 [02:34<01:02, 11780.00 examples/s]
Map (num_proc=8):  65%|######5   | 1368827/2104277 [02:34<01:03, 11623.93 examples/s]
Map (num_proc=8):  65%|######5   | 1370082/2104277 [02:34<01:04, 11321.49 examples/s]
Map (num_proc=8):  65%|######5   | 1371392/2104277 [02:35<01:03, 11581.90 examples/s]
Map (num_proc=8):  65%|######5   | 1372826/2104277 [02:35<01:01, 11819.08 examples/s]
Map (num_proc=8):  65%|######5   | 1374167/2104277 [02:35<01:01, 11803.37 examples/s]
Map (num_proc=8):  65%|######5   | 1375546/2104277 [02:35<01:01, 11922.59 examples/s]
Map (num_proc=8):  65%|######5   | 1376906/2104277 [02:35<00:59, 12249.31 examples/s]
Map (num_proc=8):  65%|######5   | 1378213/2104277 [02:35<01:03, 11468.50 examples/s]
Map (num_proc=8):  66%|######5   | 1379541/2104277 [02:35<01:00, 11951.23 examples/s]
Map (num_proc=8):  66%|######5   | 1380889/2104277 [02:35<01:00, 11900.76 examples/s]
Map (num_proc=8):  66%|######5   | 1382108/2104277 [02:35<01:04, 11125.24 examples/s]
Map (num_proc=8):  66%|######5   | 1383430/2104277 [02:36<01:09, 10406.63 examples/s]
Map (num_proc=8):  66%|######5   | 1384526/2104277 [02:36<01:10, 10254.82 examples/s]
Map (num_proc=8):  66%|######5   | 1385620/2104277 [02:36<01:12, 9900.04 examples/s] 
Map (num_proc=8):  66%|######5   | 1386753/2104277 [02:36<01:10, 10132.50 examples/s]
Map (num_proc=8):  66%|######5   | 1387964/2104277 [02:36<01:14, 9598.84 examples/s] 
Map (num_proc=8):  66%|######6   | 1389080/2104277 [02:36<01:13, 9733.09 examples/s]
Map (num_proc=8):  66%|######6   | 1390105/2104277 [02:36<01:13, 9764.24 examples/s]
Map (num_proc=8):  66%|######6   | 1391188/2104277 [02:36<01:12, 9787.15 examples/s]
Map (num_proc=8):  66%|######6   | 1392817/2104277 [02:37<01:02, 11383.83 examples/s]
Map (num_proc=8):  66%|######6   | 1394173/2104277 [02:37<01:12, 9832.73 examples/s] 
Map (num_proc=8):  66%|######6   | 1395560/2104277 [02:37<01:09, 10196.60 examples/s]
Map (num_proc=8):  66%|######6   | 1396777/2104277 [02:37<01:14, 9485.16 examples/s] 
Map (num_proc=8):  66%|######6   | 1397970/2104277 [02:37<01:15, 9353.62 examples/s]
Map (num_proc=8):  66%|######6   | 1399105/2104277 [02:37<01:13, 9604.90 examples/s]
Map (num_proc=8):  67%|######6   | 1400352/2104277 [02:37<01:14, 9510.17 examples/s]
Map (num_proc=8):  67%|######6   | 1401448/2104277 [02:37<01:13, 9597.62 examples/s]
Map (num_proc=8):  67%|######6   | 1402733/2104277 [02:38<01:12, 9742.76 examples/s]
Map (num_proc=8):  67%|######6   | 1404360/2104277 [02:38<01:04, 10803.03 examples/s]
Map (num_proc=8):  67%|######6   | 1405649/2104277 [02:38<01:01, 11337.54 examples/s]
Map (num_proc=8):  67%|######6   | 1407169/2104277 [02:38<00:57, 12214.14 examples/s]
Map (num_proc=8):  67%|######6   | 1408447/2104277 [02:38<01:01, 11399.59 examples/s]
Map (num_proc=8):  67%|######6   | 1409770/2104277 [02:38<00:59, 11606.96 examples/s]
Map (num_proc=8):  67%|######7   | 1411045/2104277 [02:38<01:02, 11092.13 examples/s]
Map (num_proc=8):  67%|######7   | 1412488/2104277 [02:38<00:58, 11740.02 examples/s]
Map (num_proc=8):  67%|######7   | 1413781/2104277 [02:39<00:59, 11562.38 examples/s]
Map (num_proc=8):  67%|######7   | 1415326/2104277 [02:39<00:56, 12213.77 examples/s]
Map (num_proc=8):  67%|######7   | 1416707/2104277 [02:39<00:59, 11518.79 examples/s]
Map (num_proc=8):  67%|######7   | 1417952/2104277 [02:39<01:00, 11320.48 examples/s]
Map (num_proc=8):  67%|######7   | 1419385/2104277 [02:39<00:57, 11858.38 examples/s]
Map (num_proc=8):  68%|######7   | 1420681/2104277 [02:39<00:57, 11961.02 examples/s]
Map (num_proc=8):  68%|######7   | 1422038/2104277 [02:39<00:56, 12047.22 examples/s]
Map (num_proc=8):  68%|######7   | 1423435/2104277 [02:39<01:02, 10915.84 examples/s]
Map (num_proc=8):  68%|######7   | 1424669/2104277 [02:39<01:01, 10970.58 examples/s]
Map (num_proc=8):  68%|######7   | 1426271/2104277 [02:40<00:57, 11857.72 examples/s]
Map (num_proc=8):  68%|######7   | 1427482/2104277 [02:40<01:01, 11077.10 examples/s]
Map (num_proc=8):  68%|######7   | 1428677/2104277 [02:40<00:59, 11272.91 examples/s]
Map (num_proc=8):  68%|######7   | 1429977/2104277 [02:40<00:59, 11408.68 examples/s]
Map (num_proc=8):  68%|######8   | 1431236/2104277 [02:40<01:06, 10099.80 examples/s]
Map (num_proc=8):  68%|######8   | 1432369/2104277 [02:40<01:05, 10202.25 examples/s]
Map (num_proc=8):  68%|######8   | 1433617/2104277 [02:40<01:12, 9242.80 examples/s] 
Map (num_proc=8):  68%|######8   | 1434692/2104277 [02:41<01:16, 8789.22 examples/s]
Map (num_proc=8):  68%|######8   | 1435726/2104277 [02:41<01:17, 8622.06 examples/s]
Map (num_proc=8):  68%|######8   | 1436630/2104277 [02:41<01:17, 8560.37 examples/s]
Map (num_proc=8):  68%|######8   | 1437584/2104277 [02:41<01:18, 8484.05 examples/s]
Map (num_proc=8):  68%|######8   | 1438764/2104277 [02:41<01:15, 8819.86 examples/s]
Map (num_proc=8):  68%|######8   | 1440000/2104277 [02:41<01:11, 9305.53 examples/s]
Map (num_proc=8):  68%|######8   | 1441267/2104277 [02:41<01:10, 9382.06 examples/s]
Map (num_proc=8):  69%|######8   | 1442754/2104277 [02:41<01:03, 10402.84 examples/s]
Map (num_proc=8):  69%|######8   | 1443896/2104277 [02:41<01:03, 10400.62 examples/s]
Map (num_proc=8):  69%|######8   | 1445085/2104277 [02:42<01:01, 10682.74 examples/s]
Map (num_proc=8):  69%|######8   | 1446203/2104277 [02:42<01:01, 10616.25 examples/s]
Map (num_proc=8):  69%|######8   | 1447378/2104277 [02:42<01:05, 10057.82 examples/s]
Map (num_proc=8):  69%|######8   | 1448817/2104277 [02:42<00:59, 10958.19 examples/s]
Map (num_proc=8):  69%|######8   | 1449935/2104277 [02:42<01:00, 10758.26 examples/s]
Map (num_proc=8):  69%|######8   | 1451055/2104277 [02:42<01:06, 9861.76 examples/s] 
Map (num_proc=8):  69%|######9   | 1452508/2104277 [02:42<01:01, 10518.95 examples/s]
Map (num_proc=8):  69%|######9   | 1453678/2104277 [02:42<01:02, 10391.70 examples/s]
Map (num_proc=8):  69%|######9   | 1454798/2104277 [02:43<01:04, 10086.86 examples/s]
Map (num_proc=8):  69%|######9   | 1456017/2104277 [02:43<01:02, 10410.97 examples/s]
Map (num_proc=8):  69%|######9   | 1457201/2104277 [02:43<01:01, 10533.61 examples/s]
Map (num_proc=8):  69%|######9   | 1458671/2104277 [02:43<00:56, 11371.38 examples/s]
Map (num_proc=8):  69%|######9   | 1460051/2104277 [02:43<00:54, 11845.42 examples/s]
Map (num_proc=8):  69%|######9   | 1461565/2104277 [02:43<00:51, 12383.72 examples/s]
Map (num_proc=8):  70%|######9   | 1462884/2104277 [02:43<00:54, 11666.32 examples/s]
Map (num_proc=8):  70%|######9   | 1464244/2104277 [02:43<00:58, 11004.13 examples/s]
Map (num_proc=8):  70%|######9   | 1466373/2104277 [02:43<00:48, 13209.40 examples/s]
Map (num_proc=8):  70%|######9   | 1468160/2104277 [02:44<00:44, 14432.65 examples/s]
Map (num_proc=8):  70%|######9   | 1469678/2104277 [02:44<00:45, 13854.94 examples/s]
Map (num_proc=8):  70%|######9   | 1471313/2104277 [02:44<00:44, 14249.92 examples/s]
Map (num_proc=8):  70%|######9   | 1472806/2104277 [02:44<00:45, 13958.98 examples/s]
Map (num_proc=8):  70%|#######   | 1474408/2104277 [02:44<00:45, 13780.00 examples/s]
Map (num_proc=8):  70%|#######   | 1476345/2104277 [02:44<00:45, 13668.70 examples/s]
Map (num_proc=8):  70%|#######   | 1477869/2104277 [02:44<00:45, 13758.77 examples/s]
Map (num_proc=8):  70%|#######   | 1479911/2104277 [02:44<00:41, 15119.15 examples/s]
Map (num_proc=8):  70%|#######   | 1481663/2104277 [02:44<00:43, 14290.70 examples/s]
Map (num_proc=8):  71%|#######   | 1483585/2104277 [02:45<00:41, 15068.23 examples/s]
Map (num_proc=8):  71%|#######   | 1485113/2104277 [02:45<00:45, 13499.06 examples/s]
Map (num_proc=8):  71%|#######   | 1486624/2104277 [02:45<00:44, 13741.17 examples/s]
Map (num_proc=8):  71%|#######   | 1488245/2104277 [02:45<00:49, 12475.69 examples/s]
Map (num_proc=8):  71%|#######   | 1489730/2104277 [02:45<00:48, 12686.61 examples/s]
Map (num_proc=8):  71%|#######   | 1491151/2104277 [02:45<00:50, 12031.93 examples/s]
Map (num_proc=8):  71%|#######   | 1492395/2104277 [02:45<00:50, 12040.68 examples/s]
Map (num_proc=8):  71%|#######   | 1493635/2104277 [02:45<00:51, 11940.52 examples/s]
Map (num_proc=8):  71%|#######1  | 1495185/2104277 [02:46<00:47, 12875.55 examples/s]
Map (num_proc=8):  71%|#######1  | 1496893/2104277 [02:46<00:43, 13962.13 examples/s]
Map (num_proc=8):  71%|#######1  | 1498605/2104277 [02:46<00:43, 13925.67 examples/s]
Map (num_proc=8):  71%|#######1  | 1500196/2104277 [02:46<00:44, 13598.67 examples/s]
Map (num_proc=8):  71%|#######1  | 1501686/2104277 [02:46<00:47, 12748.38 examples/s]
Map (num_proc=8):  71%|#######1  | 1503059/2104277 [02:46<00:50, 11915.56 examples/s]
Map (num_proc=8):  71%|#######1  | 1504332/2104277 [02:46<00:49, 12119.62 examples/s]
Map (num_proc=8):  72%|#######1  | 1505658/2104277 [02:46<00:51, 11717.99 examples/s]
Map (num_proc=8):  72%|#######1  | 1507116/2104277 [02:47<00:49, 12109.48 examples/s]
Map (num_proc=8):  72%|#######1  | 1508445/2104277 [02:47<00:54, 10856.20 examples/s]
Map (num_proc=8):  72%|#######1  | 1510082/2104277 [02:47<00:49, 12120.73 examples/s]
Map (num_proc=8):  72%|#######1  | 1511601/2104277 [02:47<00:52, 11218.49 examples/s]
Map (num_proc=8):  72%|#######1  | 1513289/2104277 [02:47<00:47, 12482.58 examples/s]
Map (num_proc=8):  72%|#######1  | 1514670/2104277 [02:47<00:47, 12321.92 examples/s]
Map (num_proc=8):  72%|#######2  | 1516413/2104277 [02:47<00:43, 13653.95 examples/s]
Map (num_proc=8):  72%|#######2  | 1517935/2104277 [02:47<00:42, 13929.84 examples/s]
Map (num_proc=8):  72%|#######2  | 1519509/2104277 [02:47<00:41, 14041.31 examples/s]
Map (num_proc=8):  72%|#######2  | 1521227/2104277 [02:48<00:39, 14749.29 examples/s]
Map (num_proc=8):  72%|#######2  | 1522913/2104277 [02:48<00:42, 13789.78 examples/s]
Map (num_proc=8):  72%|#######2  | 1524646/2104277 [02:48<00:41, 14094.18 examples/s]
Map (num_proc=8):  73%|#######2  | 1526281/2104277 [02:48<00:39, 14694.72 examples/s]
Map (num_proc=8):  73%|#######2  | 1527862/2104277 [02:48<00:39, 14482.42 examples/s]
Map (num_proc=8):  73%|#######2  | 1529530/2104277 [02:48<00:39, 14678.64 examples/s]
Map (num_proc=8):  73%|#######2  | 1531190/2104277 [02:48<00:39, 14530.13 examples/s]
Map (num_proc=8):  73%|#######2  | 1532823/2104277 [02:48<00:38, 14891.48 examples/s]
Map (num_proc=8):  73%|#######2  | 1534474/2104277 [02:49<00:40, 14040.69 examples/s]
Map (num_proc=8):  73%|#######3  | 1536191/2104277 [02:49<00:38, 14662.09 examples/s]
Map (num_proc=8):  73%|#######3  | 1538094/2104277 [02:49<00:36, 15463.50 examples/s]
Map (num_proc=8):  73%|#######3  | 1539788/2104277 [02:49<00:39, 14361.64 examples/s]
Map (num_proc=8):  73%|#######3  | 1541435/2104277 [02:49<00:45, 12469.87 examples/s]
Map (num_proc=8):  73%|#######3  | 1543083/2104277 [02:49<00:44, 12500.89 examples/s]
Map (num_proc=8):  73%|#######3  | 1544409/2104277 [02:49<00:47, 11800.22 examples/s]
Map (num_proc=8):  73%|#######3  | 1545983/2104277 [02:49<00:43, 12755.49 examples/s]
Map (num_proc=8):  74%|#######3  | 1547425/2104277 [02:50<00:47, 11680.59 examples/s]
Map (num_proc=8):  74%|#######3  | 1549361/2104277 [02:50<00:41, 13531.63 examples/s]
Map (num_proc=8):  74%|#######3  | 1550886/2104277 [02:50<00:41, 13259.71 examples/s]
Map (num_proc=8):  74%|#######3  | 1552585/2104277 [02:50<00:39, 13938.22 examples/s]
Map (num_proc=8):  74%|#######3  | 1554423/2104277 [02:50<00:37, 14543.61 examples/s]
Map (num_proc=8):  74%|#######3  | 1556122/2104277 [02:50<00:39, 14035.97 examples/s]
Map (num_proc=8):  74%|#######4  | 1558140/2104277 [02:50<00:35, 15439.25 examples/s]
Map (num_proc=8):  74%|#######4  | 1559894/2104277 [02:50<00:38, 13992.68 examples/s]
Map (num_proc=8):  74%|#######4  | 1561974/2104277 [02:51<00:35, 15147.56 examples/s]
Map (num_proc=8):  74%|#######4  | 1563624/2104277 [02:51<00:37, 14543.00 examples/s]
Map (num_proc=8):  74%|#######4  | 1565112/2104277 [02:51<00:37, 14553.20 examples/s]
Map (num_proc=8):  74%|#######4  | 1566635/2104277 [02:51<00:36, 14723.73 examples/s]
Map (num_proc=8):  75%|#######4  | 1568352/2104277 [02:51<00:37, 14434.89 examples/s]
Map (num_proc=8):  75%|#######4  | 1570374/2104277 [02:51<00:34, 15574.08 examples/s]
Map (num_proc=8):  75%|#######4  | 1572102/2104277 [02:51<00:38, 13784.13 examples/s]
Map (num_proc=8):  75%|#######4  | 1574188/2104277 [02:51<00:34, 15324.50 examples/s]
Map (num_proc=8):  75%|#######4  | 1575852/2104277 [02:51<00:34, 15481.22 examples/s]
Map (num_proc=8):  75%|#######4  | 1577529/2104277 [02:52<00:37, 13869.44 examples/s]
Map (num_proc=8):  75%|#######5  | 1579285/2104277 [02:52<00:36, 14372.41 examples/s]
Map (num_proc=8):  75%|#######5  | 1581039/2104277 [02:52<00:36, 14511.99 examples/s]
Map (num_proc=8):  75%|#######5  | 1582644/2104277 [02:52<00:36, 14418.90 examples/s]
Map (num_proc=8):  75%|#######5  | 1584159/2104277 [02:52<00:35, 14543.67 examples/s]
Map (num_proc=8):  75%|#######5  | 1585854/2104277 [02:52<00:35, 14810.05 examples/s]
Map (num_proc=8):  75%|#######5  | 1587363/2104277 [02:52<00:36, 14169.66 examples/s]
Map (num_proc=8):  76%|#######5  | 1589165/2104277 [02:52<00:35, 14638.94 examples/s]
Map (num_proc=8):  76%|#######5  | 1590652/2104277 [02:52<00:36, 14025.13 examples/s]
Map (num_proc=8):  76%|#######5  | 1592462/2104277 [02:53<00:34, 14740.64 examples/s]
Map (num_proc=8):  76%|#######5  | 1594166/2104277 [02:53<00:35, 14482.14 examples/s]
Map (num_proc=8):  76%|#######5  | 1595905/2104277 [02:53<00:34, 14728.16 examples/s]
Map (num_proc=8):  76%|#######5  | 1597428/2104277 [02:53<00:34, 14704.20 examples/s]
Map (num_proc=8):  76%|#######5  | 1599135/2104277 [02:53<00:36, 14028.92 examples/s]
Map (num_proc=8):  76%|#######6  | 1600620/2104277 [02:53<00:35, 14105.81 examples/s]
Map (num_proc=8):  76%|#######6  | 1602243/2104277 [02:53<00:40, 12388.62 examples/s]
Map (num_proc=8):  76%|#######6  | 1603751/2104277 [02:53<00:38, 12860.65 examples/s]
Map (num_proc=8):  76%|#######6  | 1605236/2104277 [02:54<00:39, 12573.37 examples/s]
Map (num_proc=8):  76%|#######6  | 1606533/2104277 [02:54<00:40, 12442.38 examples/s]
Map (num_proc=8):  76%|#######6  | 1607916/2104277 [02:54<00:42, 11609.91 examples/s]
Map (num_proc=8):  76%|#######6  | 1609627/2104277 [02:54<00:38, 12757.76 examples/s]
Map (num_proc=8):  77%|#######6  | 1611257/2104277 [02:54<00:37, 13306.16 examples/s]
Map (num_proc=8):  77%|#######6  | 1612668/2104277 [02:54<00:36, 13376.74 examples/s]
Map (num_proc=8):  77%|#######6  | 1614180/2104277 [02:54<00:37, 13234.51 examples/s]
Map (num_proc=8):  77%|#######6  | 1615667/2104277 [02:54<00:37, 13135.65 examples/s]
Map (num_proc=8):  77%|#######6  | 1617537/2104277 [02:54<00:33, 14421.38 examples/s]
Map (num_proc=8):  77%|#######6  | 1619160/2104277 [02:55<00:33, 14369.23 examples/s]
Map (num_proc=8):  77%|#######7  | 1620631/2104277 [02:55<00:33, 14227.11 examples/s]
Map (num_proc=8):  77%|#######7  | 1622244/2104277 [02:55<00:33, 14554.45 examples/s]
Map (num_proc=8):  77%|#######7  | 1623881/2104277 [02:55<00:33, 14530.85 examples/s]
Map (num_proc=8):  77%|#######7  | 1625363/2104277 [02:55<00:34, 14076.78 examples/s]
Map (num_proc=8):  77%|#######7  | 1627190/2104277 [02:55<00:31, 15104.73 examples/s]
Map (num_proc=8):  77%|#######7  | 1628768/2104277 [02:55<00:33, 14097.54 examples/s]
Map (num_proc=8):  77%|#######7  | 1630623/2104277 [02:55<00:31, 14938.69 examples/s]
Map (num_proc=8):  78%|#######7  | 1632138/2104277 [02:56<00:33, 14111.57 examples/s]
Map (num_proc=8):  78%|#######7  | 1633800/2104277 [02:56<00:32, 14283.42 examples/s]
Map (num_proc=8):  78%|#######7  | 1635401/2104277 [02:56<00:32, 14628.03 examples/s]
Map (num_proc=8):  78%|#######7  | 1637035/2104277 [02:56<00:37, 12543.63 examples/s]
Map (num_proc=8):  78%|#######7  | 1638707/2104277 [02:56<00:34, 13499.90 examples/s]
Map (num_proc=8):  78%|#######7  | 1640128/2104277 [02:56<00:38, 12053.17 examples/s]
Map (num_proc=8):  78%|#######8  | 1641658/2104277 [02:56<00:37, 12319.62 examples/s]
Map (num_proc=8):  78%|#######8  | 1643123/2104277 [02:56<00:39, 11793.19 examples/s]
Map (num_proc=8):  78%|#######8  | 1644404/2104277 [02:57<00:38, 11944.53 examples/s]
Map (num_proc=8):  78%|#######8  | 1646328/2104277 [02:57<00:33, 13470.25 examples/s]
Map (num_proc=8):  78%|#######8  | 1647850/2104277 [02:57<00:33, 13754.80 examples/s]
Map (num_proc=8):  78%|#######8  | 1649405/2104277 [02:57<00:34, 13064.19 examples/s]
Map (num_proc=8):  78%|#######8  | 1651225/2104277 [02:57<00:32, 13978.68 examples/s]
Map (num_proc=8):  79%|#######8  | 1652677/2104277 [02:57<00:32, 13973.98 examples/s]
Map (num_proc=8):  79%|#######8  | 1654202/2104277 [02:57<00:31, 14245.36 examples/s]
Map (num_proc=8):  79%|#######8  | 1656054/2104277 [02:57<00:31, 14030.14 examples/s]
Map (num_proc=8):  79%|#######8  | 1657795/2104277 [02:57<00:29, 14927.35 examples/s]
Map (num_proc=8):  79%|#######8  | 1659422/2104277 [02:58<00:30, 14361.53 examples/s]
Map (num_proc=8):  79%|#######8  | 1660936/2104277 [02:58<00:30, 14442.64 examples/s]
Map (num_proc=8):  79%|#######9  | 1662858/2104277 [02:58<00:28, 15754.69 examples/s]
Map (num_proc=8):  79%|#######9  | 1664454/2104277 [02:58<00:28, 15245.46 examples/s]
Map (num_proc=8):  79%|#######9  | 1666159/2104277 [02:58<00:27, 15734.57 examples/s]
Map (num_proc=8):  79%|#######9  | 1667796/2104277 [02:58<00:31, 14019.91 examples/s]
Map (num_proc=8):  79%|#######9  | 1669576/2104277 [02:58<00:29, 14632.03 examples/s]
Map (num_proc=8):  79%|#######9  | 1671106/2104277 [02:58<00:30, 14330.70 examples/s]
Map (num_proc=8):  79%|#######9  | 1672575/2104277 [02:58<00:33, 13069.02 examples/s]
Map (num_proc=8):  80%|#######9  | 1674004/2104277 [02:59<00:33, 12688.08 examples/s]
Map (num_proc=8):  80%|#######9  | 1675310/2104277 [02:59<00:33, 12782.24 examples/s]
Map (num_proc=8):  80%|#######9  | 1676641/2104277 [02:59<00:35, 12127.44 examples/s]
Map (num_proc=8):  80%|#######9  | 1677975/2104277 [02:59<00:35, 11908.44 examples/s]
Map (num_proc=8):  80%|#######9  | 1679324/2104277 [02:59<00:35, 12067.67 examples/s]
Map (num_proc=8):  80%|#######9  | 1680635/2104277 [02:59<00:35, 11982.47 examples/s]
Map (num_proc=8):  80%|#######9  | 1682692/2104277 [02:59<00:30, 13955.74 examples/s]
Map (num_proc=8):  80%|########  | 1684218/2104277 [02:59<00:30, 13634.63 examples/s]
Map (num_proc=8):  80%|########  | 1686196/2104277 [02:59<00:27, 15146.95 examples/s]
Map (num_proc=8):  80%|########  | 1687905/2104277 [03:00<00:30, 13858.01 examples/s]
Map (num_proc=8):  80%|########  | 1689674/2104277 [03:00<00:27, 14844.91 examples/s]
Map (num_proc=8):  80%|########  | 1691425/2104277 [03:00<00:27, 15166.14 examples/s]
Map (num_proc=8):  80%|########  | 1693137/2104277 [03:00<00:29, 14062.17 examples/s]
Map (num_proc=8):  81%|########  | 1695019/2104277 [03:00<00:27, 14869.92 examples/s]
Map (num_proc=8):  81%|########  | 1696688/2104277 [03:00<00:28, 14083.32 examples/s]
Map (num_proc=8):  81%|########  | 1698690/2104277 [03:00<00:26, 15443.43 examples/s]
Map (num_proc=8):  81%|########  | 1700477/2104277 [03:00<00:28, 14377.36 examples/s]
Map (num_proc=8):  81%|########  | 1702181/2104277 [03:01<00:28, 14321.79 examples/s]
Map (num_proc=8):  81%|########  | 1703758/2104277 [03:01<00:28, 13978.64 examples/s]
Map (num_proc=8):  81%|########1 | 1705185/2104277 [03:01<00:29, 13479.10 examples/s]
Map (num_proc=8):  81%|########1 | 1707043/2104277 [03:01<00:29, 13395.54 examples/s]
Map (num_proc=8):  81%|########1 | 1708513/2104277 [03:01<00:34, 11393.01 examples/s]
Map (num_proc=8):  81%|########1 | 1709840/2104277 [03:01<00:34, 11324.55 examples/s]
Map (num_proc=8):  81%|########1 | 1711296/2104277 [03:01<00:33, 11884.04 examples/s]
Map (num_proc=8):  81%|########1 | 1712742/2104277 [03:01<00:31, 12531.86 examples/s]
Map (num_proc=8):  81%|########1 | 1714563/2104277 [03:02<00:27, 14032.52 examples/s]
Map (num_proc=8):  82%|########1 | 1716246/2104277 [03:02<00:28, 13520.25 examples/s]
Map (num_proc=8):  82%|########1 | 1718101/2104277 [03:02<00:26, 14554.82 examples/s]
Map (num_proc=8):  82%|########1 | 1719697/2104277 [03:02<00:27, 14203.34 examples/s]
Map (num_proc=8):  82%|########1 | 1721508/2104277 [03:02<00:25, 14891.81 examples/s]
Map (num_proc=8):  82%|########1 | 1723046/2104277 [03:02<00:27, 13929.19 examples/s]
Map (num_proc=8):  82%|########1 | 1724735/2104277 [03:02<00:26, 14341.32 examples/s]
Map (num_proc=8):  82%|########2 | 1726749/2104277 [03:02<00:23, 15763.44 examples/s]
Map (num_proc=8):  82%|########2 | 1728412/2104277 [03:03<00:26, 14289.89 examples/s]
Map (num_proc=8):  82%|########2 | 1730261/2104277 [03:03<00:25, 14712.46 examples/s]
Map (num_proc=8):  82%|########2 | 1731800/2104277 [03:03<00:25, 14807.05 examples/s]
Map (num_proc=8):  82%|########2 | 1733348/2104277 [03:03<00:24, 14910.58 examples/s]
Map (num_proc=8):  82%|########2 | 1734882/2104277 [03:03<00:25, 14679.13 examples/s]
Map (num_proc=8):  83%|########2 | 1736735/2104277 [03:03<00:23, 15564.66 examples/s]
Map (num_proc=8):  83%|########2 | 1738389/2104277 [03:03<00:24, 14882.83 examples/s]
Map (num_proc=8):  83%|########2 | 1740068/2104277 [03:03<00:24, 14841.78 examples/s]
Map (num_proc=8):  83%|########2 | 1741679/2104277 [03:03<00:25, 14228.56 examples/s]
Map (num_proc=8):  83%|########2 | 1743352/2104277 [03:04<00:24, 14714.29 examples/s]
Map (num_proc=8):  83%|########2 | 1745080/2104277 [03:04<00:24, 14421.53 examples/s]
Map (num_proc=8):  83%|########3 | 1746992/2104277 [03:04<00:25, 14153.40 examples/s]
Map (num_proc=8):  83%|########3 | 1748724/2104277 [03:04<00:23, 14967.74 examples/s]
Map (num_proc=8):  83%|########3 | 1750465/2104277 [03:04<00:27, 12730.50 examples/s]
Map (num_proc=8):  83%|########3 | 1752219/2104277 [03:04<00:27, 12748.23 examples/s]
Map (num_proc=8):  83%|########3 | 1753955/2104277 [03:04<00:27, 12953.05 examples/s]
Map (num_proc=8):  83%|########3 | 1755399/2104277 [03:04<00:28, 12069.03 examples/s]
Map (num_proc=8):  84%|########3 | 1757236/2104277 [03:05<00:26, 13139.03 examples/s]
Map (num_proc=8):  84%|########3 | 1758667/2104277 [03:05<00:26, 13169.48 examples/s]
Map (num_proc=8):  84%|########3 | 1760164/2104277 [03:05<00:26, 13208.48 examples/s]
Map (num_proc=8):  84%|########3 | 1761788/2104277 [03:05<00:24, 13897.07 examples/s]
Map (num_proc=8):  84%|########3 | 1763385/2104277 [03:05<00:23, 14457.85 examples/s]
Map (num_proc=8):  84%|########3 | 1765168/2104277 [03:05<00:23, 14433.74 examples/s]
Map (num_proc=8):  84%|########3 | 1766969/2104277 [03:05<00:23, 14096.57 examples/s]
Map (num_proc=8):  84%|########4 | 1768447/2104277 [03:05<00:24, 13950.88 examples/s]
Map (num_proc=8):  84%|########4 | 1770307/2104277 [03:05<00:22, 14999.62 examples/s]
Map (num_proc=8):  84%|########4 | 1772190/2104277 [03:06<00:23, 14130.79 examples/s]
Map (num_proc=8):  84%|########4 | 1773928/2104277 [03:06<00:22, 14365.28 examples/s]
Map (num_proc=8):  84%|########4 | 1775700/2104277 [03:06<00:21, 15232.30 examples/s]
Map (num_proc=8):  84%|########4 | 1777333/2104277 [03:06<00:21, 14862.89 examples/s]
Map (num_proc=8):  85%|########4 | 1778966/2104277 [03:06<00:21, 15164.66 examples/s]
Map (num_proc=8):  85%|########4 | 1780558/2104277 [03:06<00:21, 15360.29 examples/s]
Map (num_proc=8):  85%|########4 | 1782198/2104277 [03:06<00:22, 14579.33 examples/s]
Map (num_proc=8):  85%|########4 | 1784139/2104277 [03:06<00:20, 15481.04 examples/s]
Map (num_proc=8):  85%|########4 | 1785828/2104277 [03:07<00:22, 14012.43 examples/s]
Map (num_proc=8):  85%|########4 | 1787489/2104277 [03:07<00:22, 13929.03 examples/s]
Map (num_proc=8):  85%|########5 | 1789007/2104277 [03:07<00:22, 14252.06 examples/s]
Map (num_proc=8):  85%|########5 | 1790588/2104277 [03:07<00:21, 14664.50 examples/s]
Map (num_proc=8):  85%|########5 | 1792212/2104277 [03:07<00:20, 14898.40 examples/s]
Map (num_proc=8):  85%|########5 | 1794175/2104277 [03:07<00:21, 14685.28 examples/s]
Map (num_proc=8):  85%|########5 | 1795821/2104277 [03:07<00:20, 15156.36 examples/s]
Map (num_proc=8):  85%|########5 | 1797754/2104277 [03:07<00:20, 14818.04 examples/s]
Map (num_proc=8):  86%|########5 | 1799477/2104277 [03:07<00:20, 14944.26 examples/s]
Map (num_proc=8):  86%|########5 | 1801332/2104277 [03:08<00:22, 13513.67 examples/s]
Map (num_proc=8):  86%|########5 | 1802985/2104277 [03:08<00:22, 13521.82 examples/s]
Map (num_proc=8):  86%|########5 | 1804391/2104277 [03:08<00:22, 13371.52 examples/s]
Map (num_proc=8):  86%|########5 | 1806021/2104277 [03:08<00:26, 11368.15 examples/s]
Map (num_proc=8):  86%|########5 | 1807310/2104277 [03:08<00:25, 11581.25 examples/s]
Map (num_proc=8):  86%|########5 | 1808995/2104277 [03:08<00:23, 12642.40 examples/s]
Map (num_proc=8):  86%|########6 | 1810539/2104277 [03:08<00:22, 13129.53 examples/s]
Map (num_proc=8):  86%|########6 | 1811962/2104277 [03:08<00:21, 13386.56 examples/s]
Map (num_proc=8):  86%|########6 | 1813652/2104277 [03:09<00:20, 13893.90 examples/s]
Map (num_proc=8):  86%|########6 | 1815165/2104277 [03:09<00:21, 13741.12 examples/s]
Map (num_proc=8):  86%|########6 | 1816903/2104277 [03:09<00:20, 13929.52 examples/s]
Map (num_proc=8):  86%|########6 | 1818437/2104277 [03:09<00:19, 14309.50 examples/s]
Map (num_proc=8):  86%|########6 | 1820086/2104277 [03:09<00:19, 14402.92 examples/s]
Map (num_proc=8):  87%|########6 | 1821679/2104277 [03:09<00:20, 13869.46 examples/s]
Map (num_proc=8):  87%|########6 | 1823231/2104277 [03:09<00:23, 12083.09 examples/s]
Map (num_proc=8):  87%|########6 | 1824792/2104277 [03:09<00:22, 12407.48 examples/s]
Map (num_proc=8):  87%|########6 | 1826533/2104277 [03:10<00:23, 11697.76 examples/s]
Map (num_proc=8):  87%|########6 | 1828177/2104277 [03:10<00:22, 12053.86 examples/s]
Map (num_proc=8):  87%|########6 | 1829928/2104277 [03:10<00:21, 13043.72 examples/s]
Map (num_proc=8):  87%|########7 | 1831470/2104277 [03:10<00:20, 13372.51 examples/s]
Map (num_proc=8):  87%|########7 | 1833281/2104277 [03:10<00:19, 13829.21 examples/s]
Map (num_proc=8):  87%|########7 | 1834711/2104277 [03:10<00:20, 13334.78 examples/s]
Map (num_proc=8):  87%|########7 | 1836566/2104277 [03:10<00:18, 14442.29 examples/s]
Map (num_proc=8):  87%|########7 | 1838111/2104277 [03:10<00:18, 14711.45 examples/s]
Map (num_proc=8):  87%|########7 | 1839631/2104277 [03:11<00:17, 14846.33 examples/s]
Map (num_proc=8):  87%|########7 | 1841230/2104277 [03:11<00:18, 13865.51 examples/s]
Map (num_proc=8):  88%|########7 | 1843365/2104277 [03:11<00:17, 14539.72 examples/s]
Map (num_proc=8):  88%|########7 | 1844972/2104277 [03:11<00:17, 14853.10 examples/s]
Map (num_proc=8):  88%|########7 | 1846558/2104277 [03:11<00:17, 14720.51 examples/s]
Map (num_proc=8):  88%|########7 | 1848323/2104277 [03:11<00:17, 14413.47 examples/s]
Map (num_proc=8):  88%|########7 | 1849822/2104277 [03:11<00:17, 14223.66 examples/s]
Map (num_proc=8):  88%|########8 | 1851815/2104277 [03:11<00:16, 15733.14 examples/s]
Map (num_proc=8):  88%|########8 | 1853428/2104277 [03:12<00:18, 13261.84 examples/s]
Map (num_proc=8):  88%|########8 | 1855271/2104277 [03:12<00:17, 14388.71 examples/s]
Map (num_proc=8):  88%|########8 | 1856828/2104277 [03:12<00:17, 14023.46 examples/s]
Map (num_proc=8):  88%|########8 | 1858803/2104277 [03:12<00:16, 15313.46 examples/s]
Map (num_proc=8):  88%|########8 | 1860393/2104277 [03:12<00:16, 14912.38 examples/s]
Map (num_proc=8):  88%|########8 | 1861927/2104277 [03:12<00:17, 14231.28 examples/s]
Map (num_proc=8):  89%|########8 | 1863630/2104277 [03:12<00:16, 14817.65 examples/s]
Map (num_proc=8):  89%|########8 | 1865634/2104277 [03:12<00:16, 14738.13 examples/s]
Map (num_proc=8):  89%|########8 | 1867441/2104277 [03:12<00:16, 14564.11 examples/s]
Map (num_proc=8):  89%|########8 | 1868949/2104277 [03:13<00:17, 13658.88 examples/s]
Map (num_proc=8):  89%|########8 | 1870471/2104277 [03:13<00:16, 14049.44 examples/s]
Map (num_proc=8):  89%|########8 | 1871958/2104277 [03:13<00:16, 14268.07 examples/s]
Map (num_proc=8):  89%|########9 | 1873558/2104277 [03:13<00:16, 14265.36 examples/s]
Map (num_proc=8):  89%|########9 | 1875351/2104277 [03:13<00:16, 14240.78 examples/s]
Map (num_proc=8):  89%|########9 | 1876959/2104277 [03:13<00:16, 13846.34 examples/s]
Map (num_proc=8):  89%|########9 | 1878510/2104277 [03:13<00:17, 12746.48 examples/s]
Map (num_proc=8):  89%|########9 | 1879817/2104277 [03:13<00:18, 12399.29 examples/s]
Map (num_proc=8):  89%|########9 | 1881454/2104277 [03:14<00:18, 12376.12 examples/s]
Map (num_proc=8):  89%|########9 | 1883206/2104277 [03:14<00:19, 11167.42 examples/s]
Map (num_proc=8):  90%|########9 | 1884801/2104277 [03:14<00:18, 11713.86 examples/s]
Map (num_proc=8):  90%|########9 | 1886283/2104277 [03:14<00:17, 12241.50 examples/s]
Map (num_proc=8):  90%|########9 | 1887941/2104277 [03:14<00:17, 12203.74 examples/s]
Map (num_proc=8):  90%|########9 | 1889385/2104277 [03:14<00:18, 11716.75 examples/s]
Map (num_proc=8):  90%|########9 | 1890974/2104277 [03:14<00:16, 12734.60 examples/s]
Map (num_proc=8):  90%|########9 | 1892737/2104277 [03:14<00:16, 13092.09 examples/s]
Map (num_proc=8):  90%|######### | 1894256/2104277 [03:15<00:15, 13248.02 examples/s]
Map (num_proc=8):  90%|######### | 1895801/2104277 [03:15<00:16, 12951.05 examples/s]
Map (num_proc=8):  90%|######### | 1897209/2104277 [03:15<00:15, 13033.19 examples/s]
Map (num_proc=8):  90%|######### | 1898695/2104277 [03:15<00:15, 13345.25 examples/s]
Map (num_proc=8):  90%|######### | 1900691/2104277 [03:15<00:14, 14105.73 examples/s]
Map (num_proc=8):  90%|######### | 1902695/2104277 [03:15<00:13, 14698.17 examples/s]
Map (num_proc=8):  91%|######### | 1904422/2104277 [03:15<00:13, 15046.00 examples/s]
Map (num_proc=8):  91%|######### | 1906230/2104277 [03:15<00:13, 15046.00 examples/s]
Map (num_proc=8):  91%|######### | 1908235/2104277 [03:15<00:12, 15133.44 examples/s]
Map (num_proc=8):  91%|######### | 1910241/2104277 [03:16<00:12, 15155.24 examples/s]
Map (num_proc=8):  91%|######### | 1912252/2104277 [03:16<00:12, 15174.29 examples/s]
Map (num_proc=8):  91%|######### | 1914223/2104277 [03:16<00:12, 15178.01 examples/s]
Map (num_proc=8):  91%|#########1| 1916320/2104277 [03:16<00:15, 12452.76 examples/s]
Map (num_proc=8):  91%|#########1| 1917788/2104277 [03:16<00:15, 12059.74 examples/s]
Map (num_proc=8):  91%|#########1| 1919294/2104277 [03:16<00:15, 12172.69 examples/s]
Map (num_proc=8):  91%|#########1| 1921209/2104277 [03:17<00:14, 12949.09 examples/s]
Map (num_proc=8):  91%|#########1| 1923209/2104277 [03:17<00:13, 13452.26 examples/s]
Map (num_proc=8):  91%|#########1| 1925168/2104277 [03:17<00:12, 14902.54 examples/s]
Map (num_proc=8):  92%|#########1| 1926786/2104277 [03:17<00:11, 15224.79 examples/s]
Map (num_proc=8):  92%|#########1| 1928824/2104277 [03:17<00:14, 12260.50 examples/s]
Map (num_proc=8):  92%|#########1| 1930262/2104277 [03:17<00:16, 10553.92 examples/s]
Map (num_proc=8):  92%|#########1| 1932344/2104277 [03:17<00:14, 11772.49 examples/s]
Map (num_proc=8):  92%|#########1| 1934226/2104277 [03:18<00:13, 12351.02 examples/s]
Map (num_proc=8):  92%|#########1| 1935835/2104277 [03:18<00:12, 13177.01 examples/s]
Map (num_proc=8):  92%|#########2| 1937703/2104277 [03:18<00:12, 13596.96 examples/s]
Map (num_proc=8):  92%|#########2| 1939782/2104277 [03:18<00:12, 13680.33 examples/s]
Map (num_proc=8):  92%|#########2| 1941699/2104277 [03:18<00:11, 14079.64 examples/s]
Map (num_proc=8):  92%|#########2| 1943703/2104277 [03:18<00:11, 14565.66 examples/s]
Map (num_proc=8):  92%|#########2| 1945706/2104277 [03:18<00:10, 14931.52 examples/s]
Map (num_proc=8):  93%|#########2| 1947705/2104277 [03:18<00:10, 15125.35 examples/s]
Map (num_proc=8):  93%|#########2| 1949705/2104277 [03:19<00:10, 14911.30 examples/s]
Map (num_proc=8):  93%|#########2| 1951706/2104277 [03:19<00:10, 15107.96 examples/s]
Map (num_proc=8):  93%|#########2| 1953351/2104277 [03:19<00:09, 15374.81 examples/s]
Map (num_proc=8):  93%|#########2| 1955235/2104277 [03:19<00:09, 15487.99 examples/s]
Map (num_proc=8):  93%|#########3| 1957233/2104277 [03:19<00:09, 15453.19 examples/s]
Map (num_proc=8):  93%|#########3| 1959230/2104277 [03:19<00:09, 15445.63 examples/s]
Map (num_proc=8):  93%|#########3| 1960837/2104277 [03:19<00:09, 15599.41 examples/s]
Map (num_proc=8):  93%|#########3| 1962888/2104277 [03:19<00:09, 15627.74 examples/s]
Map (num_proc=8):  93%|#########3| 1964743/2104277 [03:20<00:08, 15551.42 examples/s]
Map (num_proc=8):  93%|#########3| 1966743/2104277 [03:20<00:08, 15461.17 examples/s]
Map (num_proc=8):  94%|#########3| 1968741/2104277 [03:20<00:08, 15677.43 examples/s]
Map (num_proc=8):  94%|#########3| 1970767/2104277 [03:20<00:08, 15424.90 examples/s]
Map (num_proc=8):  94%|#########3| 1972726/2104277 [03:20<00:08, 15482.93 examples/s]
Map (num_proc=8):  94%|#########3| 1974730/2104277 [03:20<00:08, 15618.78 examples/s]
Map (num_proc=8):  94%|#########3| 1976734/2104277 [03:20<00:08, 15751.12 examples/s]
Map (num_proc=8):  94%|#########4| 1978740/2104277 [03:20<00:07, 15820.28 examples/s]
Map (num_proc=8):  94%|#########4| 1980654/2104277 [03:21<00:07, 16040.77 examples/s]
Map (num_proc=8):  94%|#########4| 1982368/2104277 [03:21<00:07, 15377.47 examples/s]
Map (num_proc=8):  94%|#########4| 1984264/2104277 [03:21<00:07, 15141.98 examples/s]
Map (num_proc=8):  94%|#########4| 1986263/2104277 [03:21<00:07, 15335.19 examples/s]
Map (num_proc=8):  94%|#########4| 1988260/2104277 [03:21<00:07, 15499.99 examples/s]
Map (num_proc=8):  95%|#########4| 1990263/2104277 [03:21<00:07, 15460.72 examples/s]
Map (num_proc=8):  95%|#########4| 1992263/2104277 [03:21<00:07, 15475.41 examples/s]
Map (num_proc=8):  95%|#########4| 1994258/2104277 [03:21<00:07, 15542.41 examples/s]
Map (num_proc=8):  95%|#########4| 1996256/2104277 [03:22<00:06, 15467.79 examples/s]
Map (num_proc=8):  95%|#########4| 1998241/2104277 [03:22<00:06, 15342.89 examples/s]
Map (num_proc=8):  95%|#########5| 2000348/2104277 [03:22<00:08, 12531.69 examples/s]
Map (num_proc=8):  95%|#########5| 2001753/2104277 [03:22<00:08, 12076.37 examples/s]
Map (num_proc=8):  95%|#########5| 2003398/2104277 [03:22<00:08, 11226.87 examples/s]
Map (num_proc=8):  95%|#########5| 2004752/2104277 [03:22<00:09, 10070.27 examples/s]
Map (num_proc=8):  95%|#########5| 2006266/2104277 [03:23<00:09, 10827.59 examples/s]
Map (num_proc=8):  95%|#########5| 2008256/2104277 [03:23<00:08, 11999.74 examples/s]
Map (num_proc=8):  96%|#########5| 2010262/2104277 [03:23<00:07, 12951.90 examples/s]
Map (num_proc=8):  96%|#########5| 2012248/2104277 [03:23<00:06, 13659.84 examples/s]
Map (num_proc=8):  96%|#########5| 2014258/2104277 [03:23<00:06, 14158.68 examples/s]
Map (num_proc=8):  96%|#########5| 2016264/2104277 [03:23<00:06, 14424.40 examples/s]
Map (num_proc=8):  96%|#########5| 2018253/2104277 [03:23<00:05, 14619.78 examples/s]
Map (num_proc=8):  96%|#########6| 2020209/2104277 [03:23<00:05, 14698.94 examples/s]
Map (num_proc=8):  96%|#########6| 2022209/2104277 [03:24<00:05, 14959.65 examples/s]
Map (num_proc=8):  96%|#########6| 2024209/2104277 [03:24<00:05, 15121.69 examples/s]
Map (num_proc=8):  96%|#########6| 2026209/2104277 [03:24<00:05, 15248.80 examples/s]
Map (num_proc=8):  96%|#########6| 2028209/2104277 [03:24<00:04, 15436.21 examples/s]
Map (num_proc=8):  96%|#########6| 2030209/2104277 [03:24<00:04, 15538.62 examples/s]
Map (num_proc=8):  97%|#########6| 2031857/2104277 [03:24<00:04, 14667.83 examples/s]
Map (num_proc=8):  97%|#########6| 2033713/2104277 [03:24<00:04, 14896.99 examples/s]
Map (num_proc=8):  97%|#########6| 2036208/2104277 [03:24<00:04, 15681.46 examples/s]
Map (num_proc=8):  97%|#########6| 2037833/2104277 [03:25<00:04, 14669.83 examples/s]
Map (num_proc=8):  97%|#########6| 2040125/2104277 [03:25<00:04, 15460.66 examples/s]
Map (num_proc=8):  97%|#########7| 2041807/2104277 [03:25<00:04, 14600.03 examples/s]
Map (num_proc=8):  97%|#########7| 2044124/2104277 [03:25<00:04, 14557.16 examples/s]
Map (num_proc=8):  97%|#########7| 2046243/2104277 [03:25<00:04, 14043.53 examples/s]
Map (num_proc=8):  97%|#########7| 2047850/2104277 [03:25<00:03, 14497.71 examples/s]
Map (num_proc=8):  97%|#########7| 2050231/2104277 [03:25<00:03, 14616.94 examples/s]
Map (num_proc=8):  98%|#########7| 2052243/2104277 [03:26<00:03, 13815.49 examples/s]
Map (num_proc=8):  98%|#########7| 2054396/2104277 [03:26<00:03, 13976.89 examples/s]
Map (num_proc=8):  98%|#########7| 2056223/2104277 [03:26<00:03, 14493.79 examples/s]
Map (num_proc=8):  98%|#########7| 2058243/2104277 [03:26<00:03, 13701.40 examples/s]
Map (num_proc=8):  98%|#########7| 2059835/2104277 [03:26<00:03, 14202.20 examples/s]
Map (num_proc=8):  98%|#########7| 2062123/2104277 [03:27<00:04, 9481.38 examples/s] 
Map (num_proc=8):  98%|#########8| 2064232/2104277 [03:27<00:03, 10496.23 examples/s]
Map (num_proc=8):  98%|#########8| 2065487/2104277 [03:27<00:03, 10866.09 examples/s]
Map (num_proc=8):  98%|#########8| 2067232/2104277 [03:27<00:03, 12024.69 examples/s]
Map (num_proc=8):  98%|#########8| 2069243/2104277 [03:27<00:02, 12290.29 examples/s]
Map (num_proc=8):  98%|#########8| 2070868/2104277 [03:27<00:02, 13165.24 examples/s]
Map (num_proc=8):  99%|#########8| 2073237/2104277 [03:27<00:02, 13810.50 examples/s]
Map (num_proc=8):  99%|#########8| 2075243/2104277 [03:27<00:02, 13432.32 examples/s]
Map (num_proc=8):  99%|#########8| 2076729/2104277 [03:28<00:02, 13751.44 examples/s]
Map (num_proc=8):  99%|#########8| 2078243/2104277 [03:28<00:01, 13291.18 examples/s]
Map (num_proc=8):  99%|#########8| 2079677/2104277 [03:28<00:01, 13540.00 examples/s]
Map (num_proc=8):  99%|#########8| 2081243/2104277 [03:28<00:01, 13361.41 examples/s]
Map (num_proc=8):  99%|#########8| 2082712/2104277 [03:28<00:01, 13705.54 examples/s]
Map (num_proc=8):  99%|#########9| 2084236/2104277 [03:28<00:01, 13568.11 examples/s]
Map (num_proc=8):  99%|#########9| 2085628/2104277 [03:28<00:01, 13662.89 examples/s]
Map (num_proc=8):  99%|#########9| 2087243/2104277 [03:28<00:01, 13969.89 examples/s]
Map (num_proc=8):  99%|#########9| 2089243/2104277 [03:28<00:01, 14621.00 examples/s]
Map (num_proc=8):  99%|#########9| 2090950/2104277 [03:29<00:00, 15285.66 examples/s]
Map (num_proc=8):  99%|#########9| 2093219/2104277 [03:29<00:00, 15075.27 examples/s]
Map (num_proc=8): 100%|#########9| 2095243/2104277 [03:29<00:00, 14156.05 examples/s]
Map (num_proc=8): 100%|#########9| 2096845/2104277 [03:29<00:00, 14602.90 examples/s]
Map (num_proc=8): 100%|#########9| 2099230/2104277 [03:29<00:00, 14672.02 examples/s]
Map (num_proc=8): 100%|#########9| 2101243/2104277 [03:29<00:00, 13982.60 examples/s]
Map (num_proc=8): 100%|#########9| 2103243/2104277 [03:29<00:00, 13713.40 examples/s]
Map (num_proc=8): 100%|##########| 2104277/2104277 [03:30<00:00, 9997.44 examples/s] 

Creating json from Arrow format:   0%|          | 0/2105 [00:00<?, ?ba/s]
Creating json from Arrow format:   0%|          | 1/2105 [00:00<08:59,  3.90ba/s]
Creating json from Arrow format:   0%|          | 2/2105 [00:00<08:20,  4.20ba/s]
Creating json from Arrow format:   0%|          | 3/2105 [00:00<08:43,  4.01ba/s]
Creating json from Arrow format:   0%|          | 4/2105 [00:00<07:47,  4.50ba/s]
Creating json from Arrow format:   0%|          | 5/2105 [00:01<08:30,  4.11ba/s]
Creating json from Arrow format:   0%|          | 6/2105 [00:01<09:07,  3.84ba/s]
Creating json from Arrow format:   0%|          | 7/2105 [00:01<08:28,  4.13ba/s]
Creating json from Arrow format:   0%|          | 8/2105 [00:01<08:14,  4.24ba/s]
Creating json from Arrow format:   0%|          | 9/2105 [00:02<08:45,  3.99ba/s]
Creating json from Arrow format:   0%|          | 10/2105 [00:02<08:10,  4.27ba/s]
Creating json from Arrow format:   1%|          | 11/2105 [00:02<08:13,  4.24ba/s]
Creating json from Arrow format:   1%|          | 12/2105 [00:02<08:49,  3.96ba/s]
Creating json from Arrow format:   1%|          | 13/2105 [00:03<07:56,  4.39ba/s]
Creating json from Arrow format:   1%|          | 14/2105 [00:03<07:21,  4.74ba/s]
Creating json from Arrow format:   1%|          | 15/2105 [00:03<07:10,  4.85ba/s]
Creating json from Arrow format:   1%|          | 16/2105 [00:03<07:54,  4.40ba/s]
Creating json from Arrow format:   1%|          | 17/2105 [00:03<07:47,  4.46ba/s]
Creating json from Arrow format:   1%|          | 18/2105 [00:04<07:19,  4.75ba/s]
Creating json from Arrow format:   1%|          | 19/2105 [00:04<07:21,  4.73ba/s]
Creating json from Arrow format:   1%|          | 20/2105 [00:04<08:09,  4.26ba/s]
Creating json from Arrow format:   1%|          | 21/2105 [00:04<07:46,  4.47ba/s]
Creating json from Arrow format:   1%|1         | 22/2105 [00:05<07:23,  4.70ba/s]
Creating json from Arrow format:   1%|1         | 23/2105 [00:05<07:19,  4.74ba/s]
Creating json from Arrow format:   1%|1         | 24/2105 [00:05<07:11,  4.82ba/s]
Creating json from Arrow format:   1%|1         | 25/2105 [00:05<07:23,  4.69ba/s]
Creating json from Arrow format:   1%|1         | 26/2105 [00:05<08:03,  4.30ba/s]
Creating json from Arrow format:   1%|1         | 27/2105 [00:06<07:45,  4.47ba/s]
Creating json from Arrow format:   1%|1         | 28/2105 [00:06<08:09,  4.25ba/s]
Creating json from Arrow format:   1%|1         | 29/2105 [00:06<08:44,  3.96ba/s]
Creating json from Arrow format:   1%|1         | 30/2105 [00:06<08:20,  4.15ba/s]
Creating json from Arrow format:   1%|1         | 31/2105 [00:07<08:10,  4.23ba/s]
Creating json from Arrow format:   2%|1         | 32/2105 [00:07<07:30,  4.61ba/s]
Creating json from Arrow format:   2%|1         | 33/2105 [00:07<07:33,  4.57ba/s]
Creating json from Arrow format:   2%|1         | 34/2105 [00:07<07:13,  4.77ba/s]
Creating json from Arrow format:   2%|1         | 35/2105 [00:07<07:50,  4.40ba/s]
Creating json from Arrow format:   2%|1         | 36/2105 [00:08<08:15,  4.17ba/s]
Creating json from Arrow format:   2%|1         | 37/2105 [00:08<08:03,  4.28ba/s]
Creating json from Arrow format:   2%|1         | 38/2105 [00:08<08:20,  4.13ba/s]
Creating json from Arrow format:   2%|1         | 39/2105 [00:09<08:33,  4.02ba/s]
Creating json from Arrow format:   2%|1         | 40/2105 [00:09<07:39,  4.49ba/s]
Creating json from Arrow format:   2%|1         | 41/2105 [00:09<08:05,  4.25ba/s]
Creating json from Arrow format:   2%|1         | 42/2105 [00:09<07:59,  4.30ba/s]
Creating json from Arrow format:   2%|2         | 43/2105 [00:09<07:51,  4.37ba/s]
Creating json from Arrow format:   2%|2         | 44/2105 [00:10<08:07,  4.23ba/s]
Creating json from Arrow format:   2%|2         | 45/2105 [00:10<07:45,  4.43ba/s]
Creating json from Arrow format:   2%|2         | 46/2105 [00:10<07:10,  4.78ba/s]
Creating json from Arrow format:   2%|2         | 47/2105 [00:10<07:30,  4.57ba/s]
Creating json from Arrow format:   2%|2         | 48/2105 [00:10<07:28,  4.58ba/s]
Creating json from Arrow format:   2%|2         | 49/2105 [00:11<07:46,  4.41ba/s]
Creating json from Arrow format:   2%|2         | 50/2105 [00:11<07:57,  4.30ba/s]
Creating json from Arrow format:   2%|2         | 51/2105 [00:11<08:09,  4.20ba/s]
Creating json from Arrow format:   2%|2         | 52/2105 [00:11<07:35,  4.50ba/s]
Creating json from Arrow format:   3%|2         | 53/2105 [00:12<07:35,  4.51ba/s]
Creating json from Arrow format:   3%|2         | 54/2105 [00:12<07:47,  4.39ba/s]
Creating json from Arrow format:   3%|2         | 55/2105 [00:12<07:32,  4.53ba/s]
Creating json from Arrow format:   3%|2         | 56/2105 [00:12<07:23,  4.62ba/s]
Creating json from Arrow format:   3%|2         | 57/2105 [00:12<07:23,  4.62ba/s]
Creating json from Arrow format:   3%|2         | 58/2105 [00:13<07:34,  4.51ba/s]
Creating json from Arrow format:   3%|2         | 59/2105 [00:13<07:21,  4.64ba/s]
Creating json from Arrow format:   3%|2         | 60/2105 [00:13<07:12,  4.73ba/s]
Creating json from Arrow format:   3%|2         | 61/2105 [00:13<07:45,  4.39ba/s]
Creating json from Arrow format:   3%|2         | 62/2105 [00:14<06:57,  4.89ba/s]
Creating json from Arrow format:   3%|2         | 63/2105 [00:14<06:52,  4.95ba/s]
Creating json from Arrow format:   3%|3         | 64/2105 [00:14<07:29,  4.54ba/s]
Creating json from Arrow format:   3%|3         | 65/2105 [00:14<07:27,  4.55ba/s]
Creating json from Arrow format:   3%|3         | 66/2105 [00:14<07:54,  4.29ba/s]
Creating json from Arrow format:   3%|3         | 67/2105 [00:15<07:36,  4.47ba/s]
Creating json from Arrow format:   3%|3         | 68/2105 [00:15<07:28,  4.54ba/s]
Creating json from Arrow format:   3%|3         | 69/2105 [00:15<07:08,  4.75ba/s]
Creating json from Arrow format:   3%|3         | 70/2105 [00:15<07:48,  4.34ba/s]
Creating json from Arrow format:   3%|3         | 71/2105 [00:16<07:16,  4.66ba/s]
Creating json from Arrow format:   3%|3         | 72/2105 [00:16<07:35,  4.46ba/s]
Creating json from Arrow format:   3%|3         | 73/2105 [00:16<07:31,  4.50ba/s]
Creating json from Arrow format:   4%|3         | 74/2105 [00:16<07:26,  4.55ba/s]
Creating json from Arrow format:   4%|3         | 75/2105 [00:16<07:39,  4.42ba/s]
Creating json from Arrow format:   4%|3         | 76/2105 [00:17<07:30,  4.51ba/s]
Creating json from Arrow format:   4%|3         | 77/2105 [00:17<07:50,  4.31ba/s]
Creating json from Arrow format:   4%|3         | 78/2105 [00:17<08:14,  4.10ba/s]
Creating json from Arrow format:   4%|3         | 79/2105 [00:17<08:08,  4.15ba/s]
Creating json from Arrow format:   4%|3         | 80/2105 [00:18<08:22,  4.03ba/s]
Creating json from Arrow format:   4%|3         | 81/2105 [00:18<08:10,  4.13ba/s]
Creating json from Arrow format:   4%|3         | 82/2105 [00:18<08:12,  4.11ba/s]
Creating json from Arrow format:   4%|3         | 83/2105 [00:18<07:37,  4.42ba/s]
Creating json from Arrow format:   4%|3         | 84/2105 [00:19<07:18,  4.61ba/s]
Creating json from Arrow format:   4%|4         | 85/2105 [00:19<07:15,  4.64ba/s]
Creating json from Arrow format:   4%|4         | 86/2105 [00:19<07:45,  4.33ba/s]
Creating json from Arrow format:   4%|4         | 87/2105 [00:19<07:53,  4.26ba/s]
Creating json from Arrow format:   4%|4         | 88/2105 [00:20<08:09,  4.12ba/s]
Creating json from Arrow format:   4%|4         | 89/2105 [00:20<07:53,  4.26ba/s]
Creating json from Arrow format:   4%|4         | 90/2105 [00:20<07:12,  4.66ba/s]
Creating json from Arrow format:   4%|4         | 91/2105 [00:20<06:44,  4.98ba/s]
Creating json from Arrow format:   4%|4         | 92/2105 [00:20<07:03,  4.76ba/s]
Creating json from Arrow format:   4%|4         | 93/2105 [00:21<07:41,  4.36ba/s]
Creating json from Arrow format:   4%|4         | 94/2105 [00:21<08:02,  4.17ba/s]
Creating json from Arrow format:   5%|4         | 95/2105 [00:21<07:38,  4.38ba/s]
Creating json from Arrow format:   5%|4         | 96/2105 [00:21<07:33,  4.43ba/s]
Creating json from Arrow format:   5%|4         | 97/2105 [00:22<08:04,  4.15ba/s]
Creating json from Arrow format:   5%|4         | 98/2105 [00:22<08:33,  3.91ba/s]
Creating json from Arrow format:   5%|4         | 99/2105 [00:22<08:17,  4.03ba/s]
Creating json from Arrow format:   5%|4         | 100/2105 [00:22<08:06,  4.12ba/s]
Creating json from Arrow format:   5%|4         | 101/2105 [00:23<08:18,  4.02ba/s]
Creating json from Arrow format:   5%|4         | 102/2105 [00:23<08:03,  4.14ba/s]
Creating json from Arrow format:   5%|4         | 103/2105 [00:23<08:12,  4.07ba/s]
Creating json from Arrow format:   5%|4         | 104/2105 [00:23<08:47,  3.79ba/s]
Creating json from Arrow format:   5%|4         | 105/2105 [00:24<08:32,  3.90ba/s]
Creating json from Arrow format:   5%|5         | 106/2105 [00:24<08:33,  3.89ba/s]
Creating json from Arrow format:   5%|5         | 107/2105 [00:24<08:36,  3.87ba/s]
Creating json from Arrow format:   5%|5         | 108/2105 [00:24<08:56,  3.72ba/s]
Creating json from Arrow format:   5%|5         | 109/2105 [00:25<09:12,  3.62ba/s]
Creating json from Arrow format:   5%|5         | 110/2105 [00:25<08:27,  3.93ba/s]
Creating json from Arrow format:   5%|5         | 111/2105 [00:25<08:07,  4.09ba/s]
Creating json from Arrow format:   5%|5         | 112/2105 [00:25<07:58,  4.16ba/s]
Creating json from Arrow format:   5%|5         | 113/2105 [00:26<07:48,  4.25ba/s]
Creating json from Arrow format:   5%|5         | 114/2105 [00:26<07:57,  4.17ba/s]
Creating json from Arrow format:   5%|5         | 115/2105 [00:26<08:16,  4.01ba/s]
Creating json from Arrow format:   6%|5         | 116/2105 [00:26<08:22,  3.96ba/s]
Creating json from Arrow format:   6%|5         | 117/2105 [00:27<07:40,  4.31ba/s]
Creating json from Arrow format:   6%|5         | 118/2105 [00:27<07:15,  4.56ba/s]
Creating json from Arrow format:   6%|5         | 119/2105 [00:27<07:14,  4.57ba/s]
Creating json from Arrow format:   6%|5         | 120/2105 [00:27<07:27,  4.43ba/s]
Creating json from Arrow format:   6%|5         | 121/2105 [00:27<07:48,  4.24ba/s]
Creating json from Arrow format:   6%|5         | 122/2105 [00:28<08:19,  3.97ba/s]
Creating json from Arrow format:   6%|5         | 123/2105 [00:28<08:28,  3.89ba/s]
Creating json from Arrow format:   6%|5         | 124/2105 [00:28<08:01,  4.12ba/s]
Creating json from Arrow format:   6%|5         | 125/2105 [00:28<08:04,  4.09ba/s]
Creating json from Arrow format:   6%|5         | 126/2105 [00:29<08:11,  4.03ba/s]
Creating json from Arrow format:   6%|6         | 127/2105 [00:29<07:55,  4.16ba/s]
Creating json from Arrow format:   6%|6         | 128/2105 [00:29<07:35,  4.34ba/s]
Creating json from Arrow format:   6%|6         | 129/2105 [00:29<07:12,  4.57ba/s]
Creating json from Arrow format:   6%|6         | 130/2105 [00:30<06:42,  4.91ba/s]
Creating json from Arrow format:   6%|6         | 131/2105 [00:30<06:46,  4.85ba/s]
Creating json from Arrow format:   6%|6         | 132/2105 [00:30<07:28,  4.40ba/s]
Creating json from Arrow format:   6%|6         | 133/2105 [00:30<06:35,  4.98ba/s]
Creating json from Arrow format:   6%|6         | 134/2105 [00:30<06:38,  4.95ba/s]
Creating json from Arrow format:   6%|6         | 135/2105 [00:31<07:11,  4.57ba/s]
Creating json from Arrow format:   6%|6         | 136/2105 [00:31<07:13,  4.54ba/s]
Creating json from Arrow format:   7%|6         | 137/2105 [00:31<06:48,  4.82ba/s]
Creating json from Arrow format:   7%|6         | 138/2105 [00:31<07:15,  4.52ba/s]
Creating json from Arrow format:   7%|6         | 139/2105 [00:32<07:26,  4.41ba/s]
Creating json from Arrow format:   7%|6         | 140/2105 [00:32<07:06,  4.61ba/s]
Creating json from Arrow format:   7%|6         | 141/2105 [00:32<07:21,  4.45ba/s]
Creating json from Arrow format:   7%|6         | 142/2105 [00:32<07:12,  4.54ba/s]
Creating json from Arrow format:   7%|6         | 143/2105 [00:32<06:49,  4.79ba/s]
Creating json from Arrow format:   7%|6         | 144/2105 [00:33<06:33,  4.99ba/s]
Creating json from Arrow format:   7%|6         | 145/2105 [00:33<06:33,  4.98ba/s]
Creating json from Arrow format:   7%|6         | 146/2105 [00:33<07:11,  4.54ba/s]
Creating json from Arrow format:   7%|6         | 147/2105 [00:33<07:52,  4.15ba/s]
Creating json from Arrow format:   7%|7         | 148/2105 [00:34<07:56,  4.11ba/s]
Creating json from Arrow format:   7%|7         | 149/2105 [00:34<07:28,  4.36ba/s]
Creating json from Arrow format:   7%|7         | 150/2105 [00:34<07:48,  4.18ba/s]
Creating json from Arrow format:   7%|7         | 151/2105 [00:34<07:50,  4.15ba/s]
Creating json from Arrow format:   7%|7         | 152/2105 [00:34<08:08,  4.00ba/s]
Creating json from Arrow format:   7%|7         | 153/2105 [00:35<08:16,  3.93ba/s]
Creating json from Arrow format:   7%|7         | 154/2105 [00:35<07:54,  4.11ba/s]
Creating json from Arrow format:   7%|7         | 155/2105 [00:35<07:26,  4.36ba/s]
Creating json from Arrow format:   7%|7         | 156/2105 [00:35<06:55,  4.69ba/s]
Creating json from Arrow format:   7%|7         | 157/2105 [00:36<06:48,  4.77ba/s]
Creating json from Arrow format:   8%|7         | 158/2105 [00:36<06:45,  4.80ba/s]
Creating json from Arrow format:   8%|7         | 159/2105 [00:36<06:42,  4.83ba/s]
Creating json from Arrow format:   8%|7         | 160/2105 [00:36<06:41,  4.85ba/s]
Creating json from Arrow format:   8%|7         | 161/2105 [00:36<06:21,  5.10ba/s]
Creating json from Arrow format:   8%|7         | 162/2105 [00:37<06:53,  4.69ba/s]
Creating json from Arrow format:   8%|7         | 163/2105 [00:37<06:46,  4.78ba/s]
Creating json from Arrow format:   8%|7         | 164/2105 [00:37<07:26,  4.35ba/s]
Creating json from Arrow format:   8%|7         | 165/2105 [00:37<07:16,  4.44ba/s]
Creating json from Arrow format:   8%|7         | 166/2105 [00:38<07:33,  4.27ba/s]
Creating json from Arrow format:   8%|7         | 167/2105 [00:38<06:50,  4.72ba/s]
Creating json from Arrow format:   8%|7         | 168/2105 [00:38<07:22,  4.38ba/s]
Creating json from Arrow format:   8%|8         | 169/2105 [00:38<07:13,  4.47ba/s]
Creating json from Arrow format:   8%|8         | 170/2105 [00:38<07:06,  4.54ba/s]
Creating json from Arrow format:   8%|8         | 171/2105 [00:39<07:00,  4.60ba/s]
Creating json from Arrow format:   8%|8         | 172/2105 [00:39<07:43,  4.17ba/s]
Creating json from Arrow format:   8%|8         | 173/2105 [00:39<06:51,  4.69ba/s]
Creating json from Arrow format:   8%|8         | 174/2105 [00:39<07:09,  4.50ba/s]
Creating json from Arrow format:   8%|8         | 175/2105 [00:40<07:13,  4.45ba/s]
Creating json from Arrow format:   8%|8         | 176/2105 [00:40<07:27,  4.31ba/s]
Creating json from Arrow format:   8%|8         | 177/2105 [00:40<07:15,  4.42ba/s]
Creating json from Arrow format:   8%|8         | 178/2105 [00:40<07:08,  4.49ba/s]
Creating json from Arrow format:   9%|8         | 179/2105 [00:40<07:22,  4.36ba/s]
Creating json from Arrow format:   9%|8         | 180/2105 [00:41<06:49,  4.70ba/s]
Creating json from Arrow format:   9%|8         | 181/2105 [00:41<06:38,  4.83ba/s]
Creating json from Arrow format:   9%|8         | 182/2105 [00:41<06:22,  5.02ba/s]
Creating json from Arrow format:   9%|8         | 183/2105 [00:41<06:43,  4.77ba/s]
Creating json from Arrow format:   9%|8         | 184/2105 [00:41<06:55,  4.62ba/s]
Creating json from Arrow format:   9%|8         | 185/2105 [00:42<07:03,  4.53ba/s]
Creating json from Arrow format:   9%|8         | 186/2105 [00:42<07:35,  4.21ba/s]
Creating json from Arrow format:   9%|8         | 187/2105 [00:42<07:51,  4.07ba/s]
Creating json from Arrow format:   9%|8         | 188/2105 [00:42<07:59,  4.00ba/s]
Creating json from Arrow format:   9%|8         | 189/2105 [00:43<07:47,  4.10ba/s]
Creating json from Arrow format:   9%|9         | 190/2105 [00:43<07:48,  4.08ba/s]
Creating json from Arrow format:   9%|9         | 191/2105 [00:43<07:15,  4.39ba/s]
Creating json from Arrow format:   9%|9         | 192/2105 [00:43<07:40,  4.16ba/s]
Creating json from Arrow format:   9%|9         | 193/2105 [00:44<06:59,  4.56ba/s]
Creating json from Arrow format:   9%|9         | 194/2105 [00:44<07:10,  4.44ba/s]
Creating json from Arrow format:   9%|9         | 195/2105 [00:44<07:02,  4.52ba/s]
Creating json from Arrow format:   9%|9         | 196/2105 [00:44<06:31,  4.88ba/s]
Creating json from Arrow format:   9%|9         | 197/2105 [00:44<06:23,  4.97ba/s]
Creating json from Arrow format:   9%|9         | 198/2105 [00:45<06:38,  4.78ba/s]
Creating json from Arrow format:   9%|9         | 199/2105 [00:45<07:02,  4.51ba/s]
Creating json from Arrow format:  10%|9         | 200/2105 [00:45<07:22,  4.31ba/s]
Creating json from Arrow format:  10%|9         | 201/2105 [00:45<07:04,  4.48ba/s]
Creating json from Arrow format:  10%|9         | 202/2105 [00:46<06:49,  4.65ba/s]
Creating json from Arrow format:  10%|9         | 203/2105 [00:46<06:44,  4.70ba/s]
Creating json from Arrow format:  10%|9         | 204/2105 [00:46<07:25,  4.27ba/s]
Creating json from Arrow format:  10%|9         | 205/2105 [00:46<07:10,  4.42ba/s]
Creating json from Arrow format:  10%|9         | 206/2105 [00:46<06:45,  4.69ba/s]
Creating json from Arrow format:  10%|9         | 207/2105 [00:47<06:55,  4.56ba/s]
Creating json from Arrow format:  10%|9         | 208/2105 [00:47<06:56,  4.56ba/s]
Creating json from Arrow format:  10%|9         | 209/2105 [00:47<06:47,  4.65ba/s]
Creating json from Arrow format:  10%|9         | 210/2105 [00:47<06:57,  4.54ba/s]
Creating json from Arrow format:  10%|#         | 211/2105 [00:47<06:26,  4.90ba/s]
Creating json from Arrow format:  10%|#         | 212/2105 [00:48<06:29,  4.86ba/s]
Creating json from Arrow format:  10%|#         | 213/2105 [00:48<06:40,  4.72ba/s]
Creating json from Arrow format:  10%|#         | 214/2105 [00:48<06:22,  4.95ba/s]
Creating json from Arrow format:  10%|#         | 215/2105 [00:48<06:13,  5.05ba/s]
Creating json from Arrow format:  10%|#         | 216/2105 [00:49<06:46,  4.65ba/s]
Creating json from Arrow format:  10%|#         | 217/2105 [00:49<06:33,  4.80ba/s]
Creating json from Arrow format:  10%|#         | 218/2105 [00:49<07:03,  4.45ba/s]
Creating json from Arrow format:  10%|#         | 219/2105 [00:49<06:59,  4.50ba/s]
Creating json from Arrow format:  10%|#         | 220/2105 [00:49<06:52,  4.57ba/s]
Creating json from Arrow format:  10%|#         | 221/2105 [00:50<07:32,  4.16ba/s]
Creating json from Arrow format:  11%|#         | 222/2105 [00:50<07:19,  4.28ba/s]
Creating json from Arrow format:  11%|#         | 223/2105 [00:50<06:54,  4.54ba/s]
Creating json from Arrow format:  11%|#         | 224/2105 [00:50<07:15,  4.32ba/s]
Creating json from Arrow format:  11%|#         | 225/2105 [00:51<07:57,  3.94ba/s]
Creating json from Arrow format:  11%|#         | 226/2105 [00:51<07:32,  4.15ba/s]
Creating json from Arrow format:  11%|#         | 227/2105 [00:51<07:18,  4.29ba/s]
Creating json from Arrow format:  11%|#         | 228/2105 [00:51<07:45,  4.03ba/s]
Creating json from Arrow format:  11%|#         | 229/2105 [00:52<08:08,  3.84ba/s]
Creating json from Arrow format:  11%|#         | 230/2105 [00:52<07:41,  4.07ba/s]
Creating json from Arrow format:  11%|#         | 231/2105 [00:52<07:53,  3.96ba/s]
Creating json from Arrow format:  11%|#1        | 232/2105 [00:52<07:33,  4.13ba/s]
Creating json from Arrow format:  11%|#1        | 233/2105 [00:53<07:55,  3.94ba/s]
Creating json from Arrow format:  11%|#1        | 234/2105 [00:53<07:38,  4.08ba/s]
Creating json from Arrow format:  11%|#1        | 235/2105 [00:53<06:42,  4.65ba/s]
Creating json from Arrow format:  11%|#1        | 236/2105 [00:53<06:54,  4.51ba/s]
Creating json from Arrow format:  11%|#1        | 237/2105 [00:53<06:48,  4.57ba/s]
Creating json from Arrow format:  11%|#1        | 238/2105 [00:54<06:27,  4.81ba/s]
Creating json from Arrow format:  11%|#1        | 239/2105 [00:54<06:57,  4.47ba/s]
Creating json from Arrow format:  11%|#1        | 240/2105 [00:54<07:25,  4.19ba/s]
Creating json from Arrow format:  11%|#1        | 241/2105 [00:54<07:09,  4.34ba/s]
Creating json from Arrow format:  11%|#1        | 242/2105 [00:55<06:46,  4.58ba/s]
Creating json from Arrow format:  12%|#1        | 243/2105 [00:55<06:49,  4.54ba/s]
Creating json from Arrow format:  12%|#1        | 244/2105 [00:55<06:53,  4.50ba/s]
Creating json from Arrow format:  12%|#1        | 245/2105 [00:55<07:36,  4.08ba/s]
Creating json from Arrow format:  12%|#1        | 246/2105 [00:56<07:20,  4.22ba/s]
Creating json from Arrow format:  12%|#1        | 247/2105 [00:56<06:58,  4.44ba/s]
Creating json from Arrow format:  12%|#1        | 248/2105 [00:56<06:56,  4.45ba/s]
Creating json from Arrow format:  12%|#1        | 249/2105 [00:56<06:52,  4.50ba/s]
Creating json from Arrow format:  12%|#1        | 250/2105 [00:56<07:08,  4.33ba/s]
Creating json from Arrow format:  12%|#1        | 251/2105 [00:57<07:17,  4.24ba/s]
Creating json from Arrow format:  12%|#1        | 252/2105 [00:57<07:04,  4.37ba/s]
Creating json from Arrow format:  12%|#2        | 253/2105 [00:57<06:34,  4.70ba/s]
Creating json from Arrow format:  12%|#2        | 254/2105 [00:57<06:39,  4.64ba/s]
Creating json from Arrow format:  12%|#2        | 255/2105 [00:58<07:39,  4.03ba/s]
Creating json from Arrow format:  12%|#2        | 256/2105 [00:58<07:11,  4.28ba/s]
Creating json from Arrow format:  12%|#2        | 257/2105 [00:58<07:00,  4.40ba/s]
Creating json from Arrow format:  12%|#2        | 258/2105 [00:58<06:54,  4.45ba/s]
Creating json from Arrow format:  12%|#2        | 259/2105 [00:58<06:36,  4.66ba/s]
Creating json from Arrow format:  12%|#2        | 260/2105 [00:59<06:27,  4.77ba/s]
Creating json from Arrow format:  12%|#2        | 261/2105 [00:59<06:20,  4.85ba/s]
Creating json from Arrow format:  12%|#2        | 262/2105 [00:59<06:03,  5.07ba/s]
Creating json from Arrow format:  12%|#2        | 263/2105 [00:59<06:05,  5.04ba/s]
Creating json from Arrow format:  13%|#2        | 264/2105 [00:59<06:00,  5.11ba/s]
Creating json from Arrow format:  13%|#2        | 265/2105 [01:00<06:14,  4.92ba/s]
Creating json from Arrow format:  13%|#2        | 266/2105 [01:00<06:18,  4.86ba/s]
Creating json from Arrow format:  13%|#2        | 267/2105 [01:00<06:40,  4.58ba/s]
Creating json from Arrow format:  13%|#2        | 268/2105 [01:00<06:31,  4.69ba/s]
Creating json from Arrow format:  13%|#2        | 269/2105 [01:01<06:58,  4.39ba/s]
Creating json from Arrow format:  13%|#2        | 270/2105 [01:01<06:26,  4.75ba/s]
Creating json from Arrow format:  13%|#2        | 271/2105 [01:01<06:09,  4.96ba/s]
Creating json from Arrow format:  13%|#2        | 272/2105 [01:01<06:39,  4.59ba/s]
Creating json from Arrow format:  13%|#2        | 273/2105 [01:01<07:00,  4.36ba/s]
Creating json from Arrow format:  13%|#3        | 274/2105 [01:02<07:02,  4.33ba/s]
Creating json from Arrow format:  13%|#3        | 275/2105 [01:02<07:13,  4.22ba/s]
Creating json from Arrow format:  13%|#3        | 276/2105 [01:02<08:03,  3.78ba/s]
Creating json from Arrow format:  13%|#3        | 277/2105 [01:02<07:54,  3.85ba/s]
Creating json from Arrow format:  13%|#3        | 278/2105 [01:03<07:23,  4.12ba/s]
Creating json from Arrow format:  13%|#3        | 279/2105 [01:03<06:48,  4.47ba/s]
Creating json from Arrow format:  13%|#3        | 280/2105 [01:03<07:06,  4.28ba/s]
Creating json from Arrow format:  13%|#3        | 281/2105 [01:03<06:55,  4.39ba/s]
Creating json from Arrow format:  13%|#3        | 282/2105 [01:04<06:38,  4.57ba/s]
Creating json from Arrow format:  13%|#3        | 283/2105 [01:04<07:03,  4.30ba/s]
Creating json from Arrow format:  13%|#3        | 284/2105 [01:04<06:55,  4.38ba/s]
Creating json from Arrow format:  14%|#3        | 285/2105 [01:04<06:43,  4.51ba/s]
Creating json from Arrow format:  14%|#3        | 286/2105 [01:04<07:05,  4.28ba/s]
Creating json from Arrow format:  14%|#3        | 287/2105 [01:05<06:34,  4.61ba/s]
Creating json from Arrow format:  14%|#3        | 288/2105 [01:05<06:50,  4.43ba/s]
Creating json from Arrow format:  14%|#3        | 289/2105 [01:05<06:50,  4.42ba/s]
Creating json from Arrow format:  14%|#3        | 290/2105 [01:05<06:28,  4.67ba/s]
Creating json from Arrow format:  14%|#3        | 291/2105 [01:06<06:39,  4.54ba/s]
Creating json from Arrow format:  14%|#3        | 292/2105 [01:06<06:36,  4.57ba/s]
Creating json from Arrow format:  14%|#3        | 293/2105 [01:06<06:20,  4.77ba/s]
Creating json from Arrow format:  14%|#3        | 294/2105 [01:06<06:41,  4.51ba/s]
Creating json from Arrow format:  14%|#4        | 295/2105 [01:06<06:17,  4.80ba/s]
Creating json from Arrow format:  14%|#4        | 296/2105 [01:07<05:56,  5.07ba/s]
Creating json from Arrow format:  14%|#4        | 297/2105 [01:07<05:56,  5.07ba/s]
Creating json from Arrow format:  14%|#4        | 298/2105 [01:07<06:12,  4.85ba/s]
Creating json from Arrow format:  14%|#4        | 299/2105 [01:07<05:57,  5.05ba/s]
Creating json from Arrow format:  14%|#4        | 300/2105 [01:07<06:00,  5.00ba/s]
Creating json from Arrow format:  14%|#4        | 301/2105 [01:08<05:53,  5.10ba/s]
Creating json from Arrow format:  14%|#4        | 302/2105 [01:08<06:37,  4.53ba/s]
Creating json from Arrow format:  14%|#4        | 303/2105 [01:08<06:32,  4.59ba/s]
Creating json from Arrow format:  14%|#4        | 304/2105 [01:08<06:43,  4.47ba/s]
Creating json from Arrow format:  14%|#4        | 305/2105 [01:08<06:27,  4.64ba/s]
Creating json from Arrow format:  15%|#4        | 306/2105 [01:09<06:09,  4.87ba/s]
Creating json from Arrow format:  15%|#4        | 307/2105 [01:09<06:31,  4.59ba/s]
Creating json from Arrow format:  15%|#4        | 308/2105 [01:09<06:54,  4.33ba/s]
Creating json from Arrow format:  15%|#4        | 309/2105 [01:09<07:01,  4.26ba/s]
Creating json from Arrow format:  15%|#4        | 310/2105 [01:10<06:39,  4.50ba/s]
Creating json from Arrow format:  15%|#4        | 311/2105 [01:10<06:51,  4.36ba/s]
Creating json from Arrow format:  15%|#4        | 312/2105 [01:10<06:41,  4.46ba/s]
Creating json from Arrow format:  15%|#4        | 313/2105 [01:10<06:00,  4.97ba/s]
Creating json from Arrow format:  15%|#4        | 314/2105 [01:10<06:38,  4.50ba/s]
Creating json from Arrow format:  15%|#4        | 315/2105 [01:11<06:45,  4.41ba/s]
Creating json from Arrow format:  15%|#5        | 316/2105 [01:11<06:29,  4.59ba/s]
Creating json from Arrow format:  15%|#5        | 317/2105 [01:11<06:14,  4.77ba/s]
Creating json from Arrow format:  15%|#5        | 318/2105 [01:11<06:19,  4.71ba/s]
Creating json from Arrow format:  15%|#5        | 319/2105 [01:12<06:13,  4.78ba/s]
Creating json from Arrow format:  15%|#5        | 320/2105 [01:12<06:02,  4.93ba/s]
Creating json from Arrow format:  15%|#5        | 321/2105 [01:12<06:17,  4.72ba/s]
Creating json from Arrow format:  15%|#5        | 322/2105 [01:12<06:22,  4.67ba/s]
Creating json from Arrow format:  15%|#5        | 323/2105 [01:12<07:19,  4.05ba/s]
Creating json from Arrow format:  15%|#5        | 324/2105 [01:13<07:29,  3.96ba/s]
Creating json from Arrow format:  15%|#5        | 325/2105 [01:13<07:02,  4.21ba/s]
Creating json from Arrow format:  15%|#5        | 326/2105 [01:13<06:57,  4.27ba/s]
Creating json from Arrow format:  16%|#5        | 327/2105 [01:13<07:20,  4.04ba/s]
Creating json from Arrow format:  16%|#5        | 328/2105 [01:14<06:56,  4.26ba/s]
Creating json from Arrow format:  16%|#5        | 329/2105 [01:14<07:02,  4.20ba/s]
Creating json from Arrow format:  16%|#5        | 330/2105 [01:14<06:51,  4.32ba/s]
Creating json from Arrow format:  16%|#5        | 331/2105 [01:14<06:58,  4.24ba/s]
Creating json from Arrow format:  16%|#5        | 332/2105 [01:15<06:42,  4.40ba/s]
Creating json from Arrow format:  16%|#5        | 333/2105 [01:15<06:40,  4.43ba/s]
Creating json from Arrow format:  16%|#5        | 334/2105 [01:15<06:36,  4.47ba/s]
Creating json from Arrow format:  16%|#5        | 335/2105 [01:15<06:20,  4.65ba/s]
Creating json from Arrow format:  16%|#5        | 336/2105 [01:15<05:55,  4.98ba/s]
Creating json from Arrow format:  16%|#6        | 337/2105 [01:16<05:35,  5.26ba/s]
Creating json from Arrow format:  16%|#6        | 338/2105 [01:16<06:05,  4.83ba/s]
Creating json from Arrow format:  16%|#6        | 339/2105 [01:16<06:30,  4.53ba/s]
Creating json from Arrow format:  16%|#6        | 340/2105 [01:16<06:29,  4.53ba/s]
Creating json from Arrow format:  16%|#6        | 341/2105 [01:17<07:02,  4.17ba/s]
Creating json from Arrow format:  16%|#6        | 342/2105 [01:17<06:53,  4.26ba/s]
Creating json from Arrow format:  16%|#6        | 343/2105 [01:17<06:07,  4.79ba/s]
Creating json from Arrow format:  16%|#6        | 344/2105 [01:17<06:07,  4.79ba/s]
Creating json from Arrow format:  16%|#6        | 345/2105 [01:17<06:02,  4.85ba/s]
Creating json from Arrow format:  16%|#6        | 346/2105 [01:18<05:53,  4.98ba/s]
Creating json from Arrow format:  16%|#6        | 347/2105 [01:18<06:10,  4.75ba/s]
Creating json from Arrow format:  17%|#6        | 348/2105 [01:18<06:11,  4.73ba/s]
Creating json from Arrow format:  17%|#6        | 349/2105 [01:18<06:59,  4.19ba/s]
Creating json from Arrow format:  17%|#6        | 350/2105 [01:18<06:37,  4.42ba/s]
Creating json from Arrow format:  17%|#6        | 351/2105 [01:19<07:20,  3.98ba/s]
Creating json from Arrow format:  17%|#6        | 352/2105 [01:19<06:52,  4.24ba/s]
Creating json from Arrow format:  17%|#6        | 353/2105 [01:19<06:54,  4.22ba/s]
Creating json from Arrow format:  17%|#6        | 354/2105 [01:19<06:35,  4.42ba/s]
Creating json from Arrow format:  17%|#6        | 355/2105 [01:20<06:31,  4.47ba/s]
Creating json from Arrow format:  17%|#6        | 356/2105 [01:20<08:04,  3.61ba/s]
Creating json from Arrow format:  17%|#6        | 357/2105 [01:20<07:53,  3.69ba/s]
Creating json from Arrow format:  17%|#7        | 358/2105 [01:20<07:19,  3.98ba/s]
Creating json from Arrow format:  17%|#7        | 359/2105 [01:21<07:33,  3.85ba/s]
Creating json from Arrow format:  17%|#7        | 360/2105 [01:21<07:33,  3.84ba/s]
Creating json from Arrow format:  17%|#7        | 361/2105 [01:21<07:48,  3.73ba/s]
Creating json from Arrow format:  17%|#7        | 362/2105 [01:22<06:58,  4.16ba/s]
Creating json from Arrow format:  17%|#7        | 363/2105 [01:22<06:48,  4.26ba/s]
Creating json from Arrow format:  17%|#7        | 364/2105 [01:22<07:05,  4.09ba/s]
Creating json from Arrow format:  17%|#7        | 365/2105 [01:22<06:48,  4.26ba/s]
Creating json from Arrow format:  17%|#7        | 366/2105 [01:22<06:18,  4.59ba/s]
Creating json from Arrow format:  17%|#7        | 367/2105 [01:23<05:46,  5.02ba/s]
Creating json from Arrow format:  17%|#7        | 368/2105 [01:23<05:36,  5.16ba/s]
Creating json from Arrow format:  18%|#7        | 369/2105 [01:23<05:42,  5.07ba/s]
Creating json from Arrow format:  18%|#7        | 370/2105 [01:23<06:21,  4.54ba/s]
Creating json from Arrow format:  18%|#7        | 371/2105 [01:23<06:35,  4.38ba/s]
Creating json from Arrow format:  18%|#7        | 372/2105 [01:24<06:14,  4.63ba/s]
Creating json from Arrow format:  18%|#7        | 373/2105 [01:24<05:49,  4.95ba/s]
Creating json from Arrow format:  18%|#7        | 374/2105 [01:24<05:30,  5.24ba/s]
Creating json from Arrow format:  18%|#7        | 375/2105 [01:24<05:25,  5.32ba/s]
Creating json from Arrow format:  18%|#7        | 376/2105 [01:24<05:27,  5.28ba/s]
Creating json from Arrow format:  18%|#7        | 377/2105 [01:25<05:24,  5.32ba/s]
Creating json from Arrow format:  18%|#7        | 378/2105 [01:25<06:02,  4.77ba/s]
Creating json from Arrow format:  18%|#8        | 379/2105 [01:25<05:43,  5.02ba/s]
Creating json from Arrow format:  18%|#8        | 380/2105 [01:25<05:40,  5.07ba/s]
Creating json from Arrow format:  18%|#8        | 381/2105 [01:25<06:11,  4.64ba/s]
Creating json from Arrow format:  18%|#8        | 382/2105 [01:26<05:54,  4.86ba/s]
Creating json from Arrow format:  18%|#8        | 383/2105 [01:26<05:19,  5.39ba/s]
Creating json from Arrow format:  18%|#8        | 384/2105 [01:26<05:28,  5.24ba/s]
Creating json from Arrow format:  18%|#8        | 385/2105 [01:26<06:16,  4.56ba/s]
Creating json from Arrow format:  18%|#8        | 386/2105 [01:26<06:12,  4.62ba/s]
Creating json from Arrow format:  18%|#8        | 387/2105 [01:27<05:47,  4.95ba/s]
Creating json from Arrow format:  18%|#8        | 388/2105 [01:27<05:29,  5.21ba/s]
Creating json from Arrow format:  18%|#8        | 389/2105 [01:27<06:07,  4.67ba/s]
Creating json from Arrow format:  19%|#8        | 390/2105 [01:27<06:12,  4.60ba/s]
Creating json from Arrow format:  19%|#8        | 391/2105 [01:28<06:25,  4.44ba/s]
Creating json from Arrow format:  19%|#8        | 392/2105 [01:28<06:56,  4.12ba/s]
Creating json from Arrow format:  19%|#8        | 393/2105 [01:28<06:43,  4.24ba/s]
Creating json from Arrow format:  19%|#8        | 394/2105 [01:28<06:24,  4.45ba/s]
Creating json from Arrow format:  19%|#8        | 396/2105 [01:28<04:15,  6.68ba/s]
Creating json from Arrow format:  19%|#8        | 398/2105 [01:28<03:11,  8.93ba/s]
Creating json from Arrow format:  19%|#9        | 400/2105 [01:29<02:35, 10.96ba/s]
Creating json from Arrow format:  19%|#9        | 402/2105 [01:29<02:14, 12.65ba/s]
Creating json from Arrow format:  19%|#9        | 404/2105 [01:29<02:00, 14.09ba/s]
Creating json from Arrow format:  19%|#9        | 406/2105 [01:29<01:52, 15.15ba/s]
Creating json from Arrow format:  19%|#9        | 408/2105 [01:29<01:45, 16.16ba/s]
Creating json from Arrow format:  19%|#9        | 410/2105 [01:29<01:40, 16.93ba/s]
Creating json from Arrow format:  20%|#9        | 412/2105 [01:29<01:37, 17.43ba/s]
Creating json from Arrow format:  20%|#9        | 414/2105 [01:29<01:34, 17.86ba/s]
Creating json from Arrow format:  20%|#9        | 416/2105 [01:29<01:32, 18.20ba/s]
Creating json from Arrow format:  20%|#9        | 418/2105 [01:30<01:30, 18.67ba/s]
Creating json from Arrow format:  20%|##        | 423/2105 [01:30<01:05, 25.84ba/s]
Creating json from Arrow format:  20%|##        | 428/2105 [01:30<00:54, 30.66ba/s]
Creating json from Arrow format:  20%|##        | 431/2105 [01:30<00:58, 28.84ba/s]
Creating json from Arrow format:  21%|##        | 434/2105 [01:30<01:05, 25.54ba/s]
Creating json from Arrow format:  21%|##        | 437/2105 [01:30<01:11, 23.45ba/s]
Creating json from Arrow format:  21%|##        | 440/2105 [01:30<01:15, 22.08ba/s]
Creating json from Arrow format:  21%|##1       | 443/2105 [01:30<01:15, 21.95ba/s]
Creating json from Arrow format:  21%|##1       | 446/2105 [01:31<01:15, 21.88ba/s]
Creating json from Arrow format:  21%|##1       | 449/2105 [01:31<01:15, 22.05ba/s]
Creating json from Arrow format:  21%|##1       | 452/2105 [01:31<01:14, 22.20ba/s]
Creating json from Arrow format:  22%|##1       | 455/2105 [01:31<01:17, 21.18ba/s]
Creating json from Arrow format:  22%|##1       | 458/2105 [01:31<01:20, 20.40ba/s]
Creating json from Arrow format:  22%|##1       | 461/2105 [01:31<01:23, 19.69ba/s]
Creating json from Arrow format:  22%|##1       | 463/2105 [01:31<01:23, 19.64ba/s]
Creating json from Arrow format:  22%|##2       | 467/2105 [01:32<01:08, 23.96ba/s]
Creating json from Arrow format:  22%|##2       | 471/2105 [01:32<00:59, 27.52ba/s]
Creating json from Arrow format:  23%|##2       | 476/2105 [01:32<00:51, 31.73ba/s]
Creating json from Arrow format:  23%|##2       | 481/2105 [01:32<00:44, 36.26ba/s]
Creating json from Arrow format:  23%|##3       | 486/2105 [01:32<00:43, 37.06ba/s]
Creating json from Arrow format:  23%|##3       | 490/2105 [01:32<00:52, 31.01ba/s]
Creating json from Arrow format:  23%|##3       | 494/2105 [01:32<00:58, 27.47ba/s]
Creating json from Arrow format:  24%|##3       | 497/2105 [01:33<01:02, 25.85ba/s]
Creating json from Arrow format:  24%|##3       | 500/2105 [01:33<01:01, 26.12ba/s]
Creating json from Arrow format:  24%|##3       | 503/2105 [01:33<01:01, 26.20ba/s]
Creating json from Arrow format:  24%|##4       | 506/2105 [01:33<01:00, 26.42ba/s]
Creating json from Arrow format:  24%|##4       | 509/2105 [01:33<01:00, 26.29ba/s]
Creating json from Arrow format:  24%|##4       | 512/2105 [01:33<01:03, 24.93ba/s]
Creating json from Arrow format:  24%|##4       | 515/2105 [01:33<01:06, 24.04ba/s]
Creating json from Arrow format:  25%|##4       | 518/2105 [01:33<01:07, 23.41ba/s]
Creating json from Arrow format:  25%|##4       | 521/2105 [01:34<01:14, 21.12ba/s]
Creating json from Arrow format:  25%|##4       | 524/2105 [01:34<01:37, 16.23ba/s]
Creating json from Arrow format:  25%|##4       | 526/2105 [01:34<01:49, 14.44ba/s]
Creating json from Arrow format:  25%|##5       | 528/2105 [01:34<01:59, 13.17ba/s]
Creating json from Arrow format:  25%|##5       | 530/2105 [01:34<02:08, 12.25ba/s]
Creating json from Arrow format:  25%|##5       | 532/2105 [01:35<02:14, 11.69ba/s]
Creating json from Arrow format:  25%|##5       | 535/2105 [01:35<01:52, 14.01ba/s]
Creating json from Arrow format:  26%|##5       | 538/2105 [01:35<01:36, 16.21ba/s]
Creating json from Arrow format:  26%|##5       | 541/2105 [01:35<01:27, 17.96ba/s]
Creating json from Arrow format:  26%|##5       | 544/2105 [01:35<01:20, 19.36ba/s]
Creating json from Arrow format:  26%|##6       | 548/2105 [01:35<01:06, 23.33ba/s]
Creating json from Arrow format:  26%|##6       | 552/2105 [01:35<00:59, 26.06ba/s]
Creating json from Arrow format:  26%|##6       | 556/2105 [01:36<00:55, 27.80ba/s]
Creating json from Arrow format:  27%|##6       | 559/2105 [01:36<00:58, 26.38ba/s]
Creating json from Arrow format:  27%|##6       | 562/2105 [01:36<01:00, 25.36ba/s]
Creating json from Arrow format:  27%|##6       | 565/2105 [01:36<01:01, 25.19ba/s]
Creating json from Arrow format:  27%|##6       | 568/2105 [01:36<01:01, 24.99ba/s]
Creating json from Arrow format:  27%|##7       | 573/2105 [01:36<00:49, 31.04ba/s]
Creating json from Arrow format:  27%|##7       | 578/2105 [01:36<00:42, 35.74ba/s]
Creating json from Arrow format:  28%|##7       | 582/2105 [01:36<00:51, 29.76ba/s]
Creating json from Arrow format:  28%|##7       | 586/2105 [01:37<01:01, 24.85ba/s]
Creating json from Arrow format:  28%|##7       | 589/2105 [01:37<01:05, 23.06ba/s]
Creating json from Arrow format:  28%|##8       | 592/2105 [01:37<01:09, 21.81ba/s]
Creating json from Arrow format:  28%|##8       | 595/2105 [01:37<01:09, 21.61ba/s]
Creating json from Arrow format:  28%|##8       | 599/2105 [01:37<00:59, 25.12ba/s]
Creating json from Arrow format:  29%|##8       | 603/2105 [01:37<00:53, 28.27ba/s]
Creating json from Arrow format:  29%|##8       | 607/2105 [01:37<00:48, 30.99ba/s]
Creating json from Arrow format:  29%|##9       | 612/2105 [01:38<00:45, 33.11ba/s]
Creating json from Arrow format:  29%|##9       | 616/2105 [01:38<00:47, 31.31ba/s]
Creating json from Arrow format:  29%|##9       | 620/2105 [01:38<00:49, 30.22ba/s]
Creating json from Arrow format:  30%|##9       | 624/2105 [01:38<00:54, 26.96ba/s]
Creating json from Arrow format:  30%|##9       | 627/2105 [01:38<01:01, 24.18ba/s]
Creating json from Arrow format:  30%|##9       | 630/2105 [01:38<01:05, 22.68ba/s]
Creating json from Arrow format:  30%|###       | 634/2105 [01:38<00:58, 25.26ba/s]
Creating json from Arrow format:  30%|###       | 639/2105 [01:39<00:50, 29.12ba/s]
Creating json from Arrow format:  31%|###       | 643/2105 [01:39<00:54, 26.61ba/s]
Creating json from Arrow format:  31%|###       | 646/2105 [01:39<00:57, 25.31ba/s]
Creating json from Arrow format:  31%|###       | 649/2105 [01:39<00:59, 24.31ba/s]
Creating json from Arrow format:  31%|###       | 652/2105 [01:39<01:00, 24.13ba/s]
Creating json from Arrow format:  31%|###1      | 655/2105 [01:39<00:58, 24.77ba/s]
Creating json from Arrow format:  31%|###1      | 658/2105 [01:39<00:57, 25.22ba/s]
Creating json from Arrow format:  31%|###1      | 661/2105 [01:40<00:56, 25.57ba/s]
Creating json from Arrow format:  32%|###1      | 665/2105 [01:40<00:52, 27.48ba/s]
Creating json from Arrow format:  32%|###1      | 669/2105 [01:40<00:48, 29.70ba/s]
Creating json from Arrow format:  32%|###1      | 673/2105 [01:40<00:45, 31.33ba/s]
Creating json from Arrow format:  32%|###2      | 677/2105 [01:40<00:43, 32.67ba/s]
Creating json from Arrow format:  32%|###2      | 681/2105 [01:40<00:42, 33.60ba/s]
Creating json from Arrow format:  33%|###2      | 685/2105 [01:40<00:41, 34.35ba/s]
Creating json from Arrow format:  33%|###2      | 689/2105 [01:40<00:47, 29.52ba/s]
Creating json from Arrow format:  33%|###2      | 693/2105 [01:41<00:55, 25.39ba/s]
Creating json from Arrow format:  33%|###3      | 696/2105 [01:41<00:59, 23.64ba/s]
Creating json from Arrow format:  33%|###3      | 699/2105 [01:41<00:59, 23.48ba/s]
Creating json from Arrow format:  33%|###3      | 704/2105 [01:41<00:50, 28.01ba/s]
Creating json from Arrow format:  34%|###3      | 709/2105 [01:41<00:44, 31.48ba/s]
Creating json from Arrow format:  34%|###3      | 713/2105 [01:41<00:44, 31.09ba/s]
Creating json from Arrow format:  34%|###4      | 717/2105 [01:41<00:44, 31.07ba/s]
Creating json from Arrow format:  34%|###4      | 721/2105 [01:42<00:44, 31.05ba/s]
Creating json from Arrow format:  34%|###4      | 725/2105 [01:42<01:02, 21.91ba/s]
Creating json from Arrow format:  35%|###4      | 728/2105 [01:42<01:15, 18.24ba/s]
Creating json from Arrow format:  35%|###4      | 731/2105 [01:42<01:25, 16.05ba/s]
Creating json from Arrow format:  35%|###4      | 733/2105 [01:43<01:32, 14.87ba/s]
Creating json from Arrow format:  35%|###4      | 736/2105 [01:43<01:22, 16.51ba/s]
Creating json from Arrow format:  35%|###5      | 739/2105 [01:43<01:14, 18.38ba/s]
Creating json from Arrow format:  35%|###5      | 742/2105 [01:43<01:08, 19.94ba/s]
Creating json from Arrow format:  35%|###5      | 745/2105 [01:43<01:04, 21.14ba/s]
Creating json from Arrow format:  36%|###5      | 750/2105 [01:43<00:51, 26.57ba/s]
Creating json from Arrow format:  36%|###5      | 755/2105 [01:43<00:43, 30.70ba/s]
Creating json from Arrow format:  36%|###6      | 759/2105 [01:43<00:45, 29.76ba/s]
Creating json from Arrow format:  36%|###6      | 763/2105 [01:44<00:51, 26.07ba/s]
Creating json from Arrow format:  36%|###6      | 766/2105 [01:44<00:55, 24.30ba/s]
Creating json from Arrow format:  37%|###6      | 769/2105 [01:44<00:57, 23.05ba/s]
Creating json from Arrow format:  37%|###6      | 772/2105 [01:44<00:57, 23.00ba/s]
Creating json from Arrow format:  37%|###6      | 775/2105 [01:44<00:57, 23.11ba/s]
Creating json from Arrow format:  37%|###6      | 778/2105 [01:44<00:57, 23.08ba/s]
Creating json from Arrow format:  37%|###7      | 781/2105 [01:44<00:57, 23.14ba/s]
Creating json from Arrow format:  37%|###7      | 784/2105 [01:45<00:53, 24.64ba/s]
Creating json from Arrow format:  37%|###7      | 787/2105 [01:45<00:51, 25.71ba/s]
Creating json from Arrow format:  38%|###7      | 790/2105 [01:45<00:49, 26.67ba/s]
Creating json from Arrow format:  38%|###7      | 793/2105 [01:45<00:47, 27.52ba/s]
Creating json from Arrow format:  38%|###7      | 798/2105 [01:45<00:39, 32.71ba/s]
Creating json from Arrow format:  38%|###8      | 803/2105 [01:45<00:35, 36.41ba/s]
Creating json from Arrow format:  38%|###8      | 807/2105 [01:45<01:01, 21.15ba/s]
Creating json from Arrow format:  38%|###8      | 810/2105 [01:46<01:25, 15.21ba/s]
Creating json from Arrow format:  39%|###8      | 813/2105 [01:46<01:44, 12.40ba/s]
Creating json from Arrow format:  39%|###8      | 815/2105 [01:46<01:55, 11.17ba/s]
Creating json from Arrow format:  39%|###8      | 817/2105 [01:47<01:54, 11.28ba/s]
Creating json from Arrow format:  39%|###8      | 820/2105 [01:47<01:35, 13.52ba/s]
Creating json from Arrow format:  39%|###9      | 823/2105 [01:47<01:22, 15.60ba/s]
Creating json from Arrow format:  39%|###9      | 826/2105 [01:47<01:13, 17.40ba/s]
Creating json from Arrow format:  39%|###9      | 829/2105 [01:47<01:07, 19.04ba/s]
Creating json from Arrow format:  40%|###9      | 832/2105 [01:47<01:00, 20.95ba/s]
Creating json from Arrow format:  40%|###9      | 835/2105 [01:47<00:56, 22.47ba/s]
Creating json from Arrow format:  40%|###9      | 838/2105 [01:47<00:53, 23.59ba/s]
Creating json from Arrow format:  40%|###9      | 841/2105 [01:48<00:54, 23.33ba/s]
Creating json from Arrow format:  40%|####      | 844/2105 [01:48<00:58, 21.60ba/s]
Creating json from Arrow format:  40%|####      | 847/2105 [01:48<00:59, 21.04ba/s]
Creating json from Arrow format:  40%|####      | 850/2105 [01:48<01:01, 20.35ba/s]
Creating json from Arrow format:  41%|####      | 853/2105 [01:48<00:59, 20.88ba/s]
Creating json from Arrow format:  41%|####      | 856/2105 [01:48<00:55, 22.35ba/s]
Creating json from Arrow format:  41%|####      | 859/2105 [01:48<00:53, 23.43ba/s]
Creating json from Arrow format:  41%|####      | 862/2105 [01:48<00:51, 24.28ba/s]
Creating json from Arrow format:  41%|####1     | 865/2105 [01:49<00:48, 25.44ba/s]
Creating json from Arrow format:  41%|####1     | 869/2105 [01:49<00:43, 28.40ba/s]
Creating json from Arrow format:  41%|####1     | 873/2105 [01:49<00:40, 30.37ba/s]
Creating json from Arrow format:  42%|####1     | 877/2105 [01:49<00:40, 30.51ba/s]
Creating json from Arrow format:  42%|####1     | 881/2105 [01:49<00:41, 29.69ba/s]
Creating json from Arrow format:  42%|####1     | 884/2105 [01:49<00:41, 29.18ba/s]
Creating json from Arrow format:  42%|####2     | 887/2105 [01:49<00:42, 28.92ba/s]
Creating json from Arrow format:  42%|####2     | 890/2105 [01:49<00:43, 27.74ba/s]
Creating json from Arrow format:  42%|####2     | 893/2105 [01:50<00:45, 26.82ba/s]
Creating json from Arrow format:  43%|####2     | 896/2105 [01:50<00:45, 26.29ba/s]
Creating json from Arrow format:  43%|####2     | 899/2105 [01:50<00:47, 25.47ba/s]
Creating json from Arrow format:  43%|####2     | 903/2105 [01:50<00:43, 27.79ba/s]
Creating json from Arrow format:  43%|####3     | 906/2105 [01:50<00:42, 28.24ba/s]
Creating json from Arrow format:  43%|####3     | 909/2105 [01:50<00:42, 28.41ba/s]
Creating json from Arrow format:  43%|####3     | 913/2105 [01:50<00:39, 30.54ba/s]
Creating json from Arrow format:  44%|####3     | 918/2105 [01:50<00:35, 33.73ba/s]
Creating json from Arrow format:  44%|####3     | 923/2105 [01:51<00:34, 34.17ba/s]
Creating json from Arrow format:  44%|####4     | 927/2105 [01:51<00:44, 26.59ba/s]
Creating json from Arrow format:  44%|####4     | 930/2105 [01:51<00:48, 24.04ba/s]
Creating json from Arrow format:  44%|####4     | 933/2105 [01:51<00:53, 22.10ba/s]
Creating json from Arrow format:  44%|####4     | 936/2105 [01:51<00:53, 21.73ba/s]
Creating json from Arrow format:  45%|####4     | 939/2105 [01:51<00:53, 21.72ba/s]
Creating json from Arrow format:  45%|####4     | 942/2105 [01:51<00:53, 21.88ba/s]
Creating json from Arrow format:  45%|####4     | 945/2105 [01:52<00:52, 22.10ba/s]
Creating json from Arrow format:  45%|####5     | 948/2105 [01:52<00:49, 23.18ba/s]
Creating json from Arrow format:  45%|####5     | 952/2105 [01:52<00:44, 25.97ba/s]
Creating json from Arrow format:  45%|####5     | 956/2105 [01:52<00:41, 27.70ba/s]
Creating json from Arrow format:  46%|####5     | 960/2105 [01:52<00:38, 29.48ba/s]
Creating json from Arrow format:  46%|####5     | 964/2105 [01:52<00:36, 31.26ba/s]
Creating json from Arrow format:  46%|####5     | 968/2105 [01:52<00:34, 32.59ba/s]
Creating json from Arrow format:  46%|####6     | 972/2105 [01:53<00:40, 28.10ba/s]
Creating json from Arrow format:  46%|####6     | 975/2105 [01:53<00:46, 24.35ba/s]
Creating json from Arrow format:  46%|####6     | 978/2105 [01:53<00:50, 22.21ba/s]
Creating json from Arrow format:  47%|####6     | 981/2105 [01:53<00:53, 20.98ba/s]
Creating json from Arrow format:  47%|####6     | 985/2105 [01:53<00:45, 24.59ba/s]
Creating json from Arrow format:  47%|####7     | 990/2105 [01:53<00:37, 29.83ba/s]
Creating json from Arrow format:  47%|####7     | 995/2105 [01:53<00:35, 31.35ba/s]
Creating json from Arrow format:  47%|####7     | 999/2105 [01:54<00:39, 27.72ba/s]
Creating json from Arrow format:  48%|####7     | 1002/2105 [01:54<00:42, 26.08ba/s]
Creating json from Arrow format:  48%|####7     | 1005/2105 [01:54<00:44, 24.89ba/s]
Creating json from Arrow format:  48%|####7     | 1008/2105 [01:54<00:51, 21.20ba/s]
Creating json from Arrow format:  48%|####8     | 1011/2105 [01:54<00:59, 18.46ba/s]
Creating json from Arrow format:  48%|####8     | 1013/2105 [01:54<01:02, 17.38ba/s]
Creating json from Arrow format:  48%|####8     | 1015/2105 [01:55<01:05, 16.52ba/s]
Creating json from Arrow format:  48%|####8     | 1017/2105 [01:55<01:08, 15.86ba/s]
Creating json from Arrow format:  48%|####8     | 1019/2105 [01:55<01:06, 16.24ba/s]
Creating json from Arrow format:  49%|####8     | 1021/2105 [01:55<01:05, 16.59ba/s]
Creating json from Arrow format:  49%|####8     | 1023/2105 [01:55<01:04, 16.85ba/s]
Creating json from Arrow format:  49%|####8     | 1025/2105 [01:55<01:03, 16.90ba/s]
Creating json from Arrow format:  49%|####8     | 1027/2105 [01:55<01:02, 17.16ba/s]
Creating json from Arrow format:  49%|####8     | 1029/2105 [01:55<01:02, 17.35ba/s]
Creating json from Arrow format:  49%|####9     | 1032/2105 [01:55<00:53, 19.92ba/s]
Creating json from Arrow format:  49%|####9     | 1035/2105 [01:56<00:49, 21.62ba/s]
Creating json from Arrow format:  49%|####9     | 1038/2105 [01:56<00:46, 22.77ba/s]
Creating json from Arrow format:  49%|####9     | 1041/2105 [01:56<00:45, 23.57ba/s]
Creating json from Arrow format:  50%|####9     | 1044/2105 [01:56<00:47, 22.45ba/s]
Creating json from Arrow format:  50%|####9     | 1047/2105 [01:56<00:47, 22.06ba/s]
Creating json from Arrow format:  50%|####9     | 1050/2105 [01:56<00:48, 21.56ba/s]
Creating json from Arrow format:  50%|#####     | 1053/2105 [01:56<00:49, 21.42ba/s]
Creating json from Arrow format:  50%|#####     | 1057/2105 [01:57<00:41, 25.18ba/s]
Creating json from Arrow format:  50%|#####     | 1061/2105 [01:57<00:36, 28.26ba/s]
Creating json from Arrow format:  51%|#####     | 1065/2105 [01:57<00:34, 30.43ba/s]
Creating json from Arrow format:  51%|#####     | 1069/2105 [01:57<00:35, 29.20ba/s]
Creating json from Arrow format:  51%|#####     | 1072/2105 [01:57<00:36, 28.56ba/s]
Creating json from Arrow format:  51%|#####1    | 1075/2105 [01:57<00:36, 28.18ba/s]
Creating json from Arrow format:  51%|#####1    | 1078/2105 [01:57<00:37, 27.10ba/s]
Creating json from Arrow format:  51%|#####1    | 1081/2105 [01:57<00:40, 25.46ba/s]
Creating json from Arrow format:  51%|#####1    | 1084/2105 [01:57<00:41, 24.78ba/s]
Creating json from Arrow format:  52%|#####1    | 1087/2105 [01:58<00:42, 23.70ba/s]
Creating json from Arrow format:  52%|#####1    | 1090/2105 [01:58<00:44, 22.73ba/s]
Creating json from Arrow format:  52%|#####1    | 1093/2105 [01:58<00:45, 22.41ba/s]
Creating json from Arrow format:  52%|#####2    | 1096/2105 [01:58<00:46, 21.75ba/s]
Creating json from Arrow format:  52%|#####2    | 1099/2105 [01:58<00:47, 21.31ba/s]
Creating json from Arrow format:  52%|#####2    | 1102/2105 [01:58<00:49, 20.37ba/s]
Creating json from Arrow format:  52%|#####2    | 1105/2105 [01:59<00:52, 19.11ba/s]
Creating json from Arrow format:  53%|#####2    | 1107/2105 [01:59<00:54, 18.26ba/s]
Creating json from Arrow format:  53%|#####2    | 1111/2105 [01:59<00:43, 22.79ba/s]
Creating json from Arrow format:  53%|#####2    | 1115/2105 [01:59<00:37, 26.15ba/s]
Creating json from Arrow format:  53%|#####3    | 1118/2105 [01:59<00:39, 24.68ba/s]
Creating json from Arrow format:  53%|#####3    | 1121/2105 [01:59<00:41, 23.85ba/s]
Creating json from Arrow format:  53%|#####3    | 1124/2105 [01:59<00:41, 23.40ba/s]
Creating json from Arrow format:  54%|#####3    | 1127/2105 [01:59<00:42, 22.78ba/s]
Creating json from Arrow format:  54%|#####3    | 1131/2105 [02:00<00:36, 26.60ba/s]
Creating json from Arrow format:  54%|#####3    | 1135/2105 [02:00<00:32, 29.58ba/s]
Creating json from Arrow format:  54%|#####4    | 1139/2105 [02:00<00:29, 32.27ba/s]
Creating json from Arrow format:  54%|#####4    | 1145/2105 [02:00<00:25, 38.02ba/s]
Creating json from Arrow format:  55%|#####4    | 1151/2105 [02:00<00:24, 39.52ba/s]
Creating json from Arrow format:  55%|#####4    | 1155/2105 [02:00<00:32, 29.04ba/s]
Creating json from Arrow format:  55%|#####5    | 1159/2105 [02:00<00:36, 26.19ba/s]
Creating json from Arrow format:  55%|#####5    | 1163/2105 [02:01<00:32, 28.92ba/s]
Creating json from Arrow format:  55%|#####5    | 1167/2105 [02:01<00:35, 26.77ba/s]
Creating json from Arrow format:  56%|#####5    | 1170/2105 [02:01<00:44, 20.97ba/s]
Creating json from Arrow format:  56%|#####5    | 1173/2105 [02:01<00:52, 17.81ba/s]
Creating json from Arrow format:  56%|#####5    | 1176/2105 [02:01<00:58, 15.91ba/s]
Creating json from Arrow format:  56%|#####5    | 1178/2105 [02:02<00:59, 15.68ba/s]
Creating json from Arrow format:  56%|#####6    | 1181/2105 [02:02<00:52, 17.57ba/s]
Creating json from Arrow format:  56%|#####6    | 1184/2105 [02:02<00:47, 19.35ba/s]
Creating json from Arrow format:  56%|#####6    | 1187/2105 [02:02<00:44, 20.74ba/s]
Creating json from Arrow format:  57%|#####6    | 1190/2105 [02:02<00:42, 21.63ba/s]
Creating json from Arrow format:  57%|#####6    | 1193/2105 [02:02<00:41, 22.04ba/s]
Creating json from Arrow format:  57%|#####6    | 1196/2105 [02:02<00:40, 22.52ba/s]
Creating json from Arrow format:  57%|#####6    | 1199/2105 [02:02<00:39, 22.77ba/s]
Creating json from Arrow format:  57%|#####7    | 1202/2105 [02:03<00:39, 22.79ba/s]
Creating json from Arrow format:  57%|#####7    | 1205/2105 [02:03<00:40, 22.47ba/s]
Creating json from Arrow format:  57%|#####7    | 1208/2105 [02:03<00:40, 22.26ba/s]
Creating json from Arrow format:  58%|#####7    | 1211/2105 [02:03<00:40, 22.15ba/s]
Creating json from Arrow format:  58%|#####7    | 1214/2105 [02:03<00:39, 22.63ba/s]
Creating json from Arrow format:  58%|#####7    | 1217/2105 [02:03<00:37, 23.50ba/s]
Creating json from Arrow format:  58%|#####7    | 1220/2105 [02:03<00:36, 24.00ba/s]
Creating json from Arrow format:  58%|#####8    | 1223/2105 [02:03<00:36, 24.44ba/s]
Creating json from Arrow format:  58%|#####8    | 1226/2105 [02:04<00:34, 25.33ba/s]
Creating json from Arrow format:  58%|#####8    | 1230/2105 [02:04<00:31, 27.44ba/s]
Creating json from Arrow format:  59%|#####8    | 1234/2105 [02:04<00:30, 28.58ba/s]
Creating json from Arrow format:  59%|#####8    | 1238/2105 [02:04<00:30, 28.45ba/s]
Creating json from Arrow format:  59%|#####8    | 1241/2105 [02:04<00:33, 25.73ba/s]
Creating json from Arrow format:  59%|#####9    | 1244/2105 [02:04<00:35, 24.30ba/s]
Creating json from Arrow format:  59%|#####9    | 1247/2105 [02:04<00:37, 23.02ba/s]
Creating json from Arrow format:  59%|#####9    | 1250/2105 [02:05<00:37, 22.54ba/s]
Creating json from Arrow format:  60%|#####9    | 1253/2105 [02:05<00:38, 22.00ba/s]
Creating json from Arrow format:  60%|#####9    | 1256/2105 [02:05<00:38, 21.85ba/s]
Creating json from Arrow format:  60%|#####9    | 1259/2105 [02:05<00:39, 21.52ba/s]
Creating json from Arrow format:  60%|#####9    | 1262/2105 [02:05<00:39, 21.47ba/s]
Creating json from Arrow format:  60%|######    | 1265/2105 [02:05<00:39, 21.04ba/s]
Creating json from Arrow format:  60%|######    | 1268/2105 [02:05<00:40, 20.46ba/s]
Creating json from Arrow format:  60%|######    | 1271/2105 [02:06<00:41, 20.31ba/s]
Creating json from Arrow format:  61%|######    | 1275/2105 [02:06<00:35, 23.67ba/s]
Creating json from Arrow format:  61%|######    | 1280/2105 [02:06<00:28, 29.24ba/s]
Creating json from Arrow format:  61%|######1   | 1285/2105 [02:06<00:25, 32.35ba/s]
Creating json from Arrow format:  61%|######1   | 1289/2105 [02:06<00:26, 30.91ba/s]
Creating json from Arrow format:  61%|######1   | 1293/2105 [02:06<00:27, 29.87ba/s]
Creating json from Arrow format:  62%|######1   | 1297/2105 [02:06<00:28, 28.06ba/s]
Creating json from Arrow format:  62%|######1   | 1300/2105 [02:07<00:31, 25.96ba/s]
Creating json from Arrow format:  62%|######1   | 1303/2105 [02:07<00:33, 24.25ba/s]
Creating json from Arrow format:  62%|######2   | 1306/2105 [02:07<00:34, 23.31ba/s]
Creating json from Arrow format:  62%|######2   | 1309/2105 [02:07<00:32, 24.79ba/s]
Creating json from Arrow format:  62%|######2   | 1315/2105 [02:07<00:24, 31.97ba/s]
Creating json from Arrow format:  63%|######2   | 1320/2105 [02:07<00:22, 34.74ba/s]
Creating json from Arrow format:  63%|######2   | 1324/2105 [02:07<00:26, 29.62ba/s]
Creating json from Arrow format:  63%|######3   | 1328/2105 [02:08<00:28, 27.16ba/s]
Creating json from Arrow format:  63%|######3   | 1331/2105 [02:08<00:30, 25.77ba/s]
Creating json from Arrow format:  63%|######3   | 1334/2105 [02:08<00:30, 25.54ba/s]
Creating json from Arrow format:  64%|######3   | 1337/2105 [02:08<00:30, 25.58ba/s]
Creating json from Arrow format:  64%|######3   | 1340/2105 [02:08<00:29, 25.53ba/s]
Creating json from Arrow format:  64%|######3   | 1343/2105 [02:08<00:29, 25.49ba/s]
Creating json from Arrow format:  64%|######4   | 1348/2105 [02:08<00:24, 30.48ba/s]
Creating json from Arrow format:  64%|######4   | 1353/2105 [02:08<00:22, 34.15ba/s]
Creating json from Arrow format:  64%|######4   | 1357/2105 [02:09<00:22, 33.91ba/s]
Creating json from Arrow format:  65%|######4   | 1361/2105 [02:09<00:23, 31.86ba/s]
Creating json from Arrow format:  65%|######4   | 1365/2105 [02:09<00:24, 30.59ba/s]
Creating json from Arrow format:  65%|######5   | 1369/2105 [02:09<00:26, 27.76ba/s]
Creating json from Arrow format:  65%|######5   | 1372/2105 [02:09<00:28, 25.59ba/s]
Creating json from Arrow format:  65%|######5   | 1375/2105 [02:09<00:30, 24.04ba/s]
Creating json from Arrow format:  65%|######5   | 1378/2105 [02:09<00:31, 23.00ba/s]
Creating json from Arrow format:  66%|######5   | 1381/2105 [02:10<00:30, 23.50ba/s]
Creating json from Arrow format:  66%|######5   | 1384/2105 [02:10<00:29, 24.26ba/s]
Creating json from Arrow format:  66%|######5   | 1387/2105 [02:10<00:28, 24.84ba/s]
Creating json from Arrow format:  66%|######6   | 1390/2105 [02:10<00:28, 25.36ba/s]
Creating json from Arrow format:  66%|######6   | 1393/2105 [02:10<00:32, 21.67ba/s]
Creating json from Arrow format:  66%|######6   | 1396/2105 [02:10<00:34, 20.30ba/s]
Creating json from Arrow format:  66%|######6   | 1399/2105 [02:10<00:36, 19.15ba/s]
Creating json from Arrow format:  67%|######6   | 1401/2105 [02:11<00:37, 18.78ba/s]
Creating json from Arrow format:  67%|######6   | 1404/2105 [02:11<00:36, 19.17ba/s]
Creating json from Arrow format:  67%|######6   | 1408/2105 [02:11<00:30, 23.16ba/s]
Creating json from Arrow format:  67%|######7   | 1412/2105 [02:11<00:25, 27.02ba/s]
Creating json from Arrow format:  67%|######7   | 1416/2105 [02:11<00:23, 29.81ba/s]
Creating json from Arrow format:  68%|######7   | 1421/2105 [02:11<00:20, 32.75ba/s]
Creating json from Arrow format:  68%|######7   | 1425/2105 [02:11<00:20, 33.44ba/s]
Creating json from Arrow format:  68%|######7   | 1429/2105 [02:11<00:20, 33.78ba/s]
Creating json from Arrow format:  68%|######8   | 1433/2105 [02:11<00:19, 34.16ba/s]
Creating json from Arrow format:  68%|######8   | 1437/2105 [02:12<00:19, 34.61ba/s]
Creating json from Arrow format:  68%|######8   | 1441/2105 [02:12<00:19, 34.93ba/s]
Creating json from Arrow format:  69%|######8   | 1445/2105 [02:12<00:20, 31.73ba/s]
Creating json from Arrow format:  69%|######8   | 1449/2105 [02:12<00:23, 27.69ba/s]
Creating json from Arrow format:  69%|######8   | 1452/2105 [02:12<00:25, 25.76ba/s]
Creating json from Arrow format:  69%|######9   | 1455/2105 [02:12<00:26, 24.47ba/s]
Creating json from Arrow format:  69%|######9   | 1458/2105 [02:12<00:27, 23.23ba/s]
Creating json from Arrow format:  69%|######9   | 1461/2105 [02:13<00:28, 22.63ba/s]
Creating json from Arrow format:  70%|######9   | 1464/2105 [02:13<00:29, 22.00ba/s]
Creating json from Arrow format:  70%|######9   | 1467/2105 [02:13<00:28, 22.06ba/s]
Creating json from Arrow format:  70%|######9   | 1470/2105 [02:13<00:26, 23.87ba/s]
Creating json from Arrow format:  70%|######9   | 1473/2105 [02:13<00:25, 25.25ba/s]
Creating json from Arrow format:  70%|#######   | 1476/2105 [02:13<00:23, 26.37ba/s]
Creating json from Arrow format:  70%|#######   | 1479/2105 [02:13<00:24, 25.97ba/s]
Creating json from Arrow format:  70%|#######   | 1482/2105 [02:13<00:26, 23.56ba/s]
Creating json from Arrow format:  71%|#######   | 1485/2105 [02:14<00:27, 22.24ba/s]
Creating json from Arrow format:  71%|#######   | 1488/2105 [02:14<00:29, 21.22ba/s]
Creating json from Arrow format:  71%|#######   | 1491/2105 [02:14<00:30, 20.10ba/s]
Creating json from Arrow format:  71%|#######   | 1494/2105 [02:14<00:31, 19.21ba/s]
Creating json from Arrow format:  71%|#######1  | 1496/2105 [02:14<00:32, 18.94ba/s]
Creating json from Arrow format:  71%|#######1  | 1498/2105 [02:14<00:32, 18.54ba/s]
Creating json from Arrow format:  71%|#######1  | 1501/2105 [02:14<00:28, 21.26ba/s]
Creating json from Arrow format:  71%|#######1  | 1505/2105 [02:15<00:23, 25.99ba/s]
Creating json from Arrow format:  72%|#######1  | 1509/2105 [02:15<00:20, 28.44ba/s]
Creating json from Arrow format:  72%|#######1  | 1512/2105 [02:15<00:20, 28.84ba/s]
Creating json from Arrow format:  72%|#######1  | 1515/2105 [02:15<00:20, 29.07ba/s]
Creating json from Arrow format:  72%|#######2  | 1518/2105 [02:15<00:20, 29.11ba/s]
Creating json from Arrow format:  72%|#######2  | 1521/2105 [02:15<00:20, 29.18ba/s]
Creating json from Arrow format:  72%|#######2  | 1524/2105 [02:15<00:19, 29.22ba/s]
Creating json from Arrow format:  73%|#######2  | 1527/2105 [02:15<00:19, 29.26ba/s]
Creating json from Arrow format:  73%|#######2  | 1530/2105 [02:15<00:19, 29.31ba/s]
Creating json from Arrow format:  73%|#######2  | 1533/2105 [02:15<00:21, 26.71ba/s]
Creating json from Arrow format:  73%|#######2  | 1536/2105 [02:16<00:22, 24.81ba/s]
Creating json from Arrow format:  73%|#######3  | 1539/2105 [02:16<00:24, 23.53ba/s]
Creating json from Arrow format:  73%|#######3  | 1542/2105 [02:16<00:24, 22.79ba/s]
Creating json from Arrow format:  73%|#######3  | 1545/2105 [02:16<00:25, 21.74ba/s]
Creating json from Arrow format:  74%|#######3  | 1548/2105 [02:16<00:27, 20.46ba/s]
Creating json from Arrow format:  74%|#######3  | 1551/2105 [02:16<00:28, 19.77ba/s]
Creating json from Arrow format:  74%|#######3  | 1554/2105 [02:17<00:28, 19.40ba/s]
Creating json from Arrow format:  74%|#######3  | 1556/2105 [02:17<00:28, 19.46ba/s]
Creating json from Arrow format:  74%|#######4  | 1559/2105 [02:17<00:27, 19.86ba/s]
Creating json from Arrow format:  74%|#######4  | 1562/2105 [02:17<00:26, 20.17ba/s]
Creating json from Arrow format:  74%|#######4  | 1565/2105 [02:17<00:26, 20.37ba/s]
Creating json from Arrow format:  74%|#######4  | 1568/2105 [02:17<00:28, 18.80ba/s]
Creating json from Arrow format:  75%|#######4  | 1570/2105 [02:17<00:31, 16.85ba/s]
Creating json from Arrow format:  75%|#######4  | 1572/2105 [02:18<00:34, 15.44ba/s]
Creating json from Arrow format:  75%|#######4  | 1574/2105 [02:18<00:36, 14.48ba/s]
Creating json from Arrow format:  75%|#######4  | 1576/2105 [02:18<00:38, 13.81ba/s]
Creating json from Arrow format:  75%|#######4  | 1578/2105 [02:18<00:39, 13.35ba/s]
Creating json from Arrow format:  75%|#######5  | 1580/2105 [02:18<00:35, 14.66ba/s]
Creating json from Arrow format:  75%|#######5  | 1583/2105 [02:18<00:31, 16.55ba/s]
Creating json from Arrow format:  75%|#######5  | 1586/2105 [02:18<00:28, 17.97ba/s]
Creating json from Arrow format:  75%|#######5  | 1589/2105 [02:19<00:27, 19.03ba/s]
Creating json from Arrow format:  76%|#######5  | 1591/2105 [02:19<00:27, 18.91ba/s]
Creating json from Arrow format:  76%|#######5  | 1593/2105 [02:19<00:27, 18.95ba/s]
Creating json from Arrow format:  76%|#######5  | 1595/2105 [02:19<00:26, 18.96ba/s]
Creating json from Arrow format:  76%|#######5  | 1597/2105 [02:19<00:26, 18.96ba/s]
Creating json from Arrow format:  76%|#######5  | 1599/2105 [02:19<00:26, 18.88ba/s]
Creating json from Arrow format:  76%|#######6  | 1601/2105 [02:19<00:26, 18.86ba/s]
Creating json from Arrow format:  76%|#######6  | 1604/2105 [02:19<00:25, 19.67ba/s]
Creating json from Arrow format:  76%|#######6  | 1607/2105 [02:20<00:24, 20.62ba/s]
Creating json from Arrow format:  76%|#######6  | 1610/2105 [02:20<00:23, 21.29ba/s]
Creating json from Arrow format:  77%|#######6  | 1613/2105 [02:20<00:22, 21.66ba/s]
Creating json from Arrow format:  77%|#######6  | 1616/2105 [02:20<00:22, 22.09ba/s]
Creating json from Arrow format:  77%|#######6  | 1619/2105 [02:20<00:21, 22.33ba/s]
Creating json from Arrow format:  77%|#######7  | 1622/2105 [02:20<00:21, 22.45ba/s]
Creating json from Arrow format:  77%|#######7  | 1625/2105 [02:20<00:21, 22.56ba/s]
Creating json from Arrow format:  77%|#######7  | 1629/2105 [02:20<00:17, 26.57ba/s]
Creating json from Arrow format:  78%|#######7  | 1634/2105 [02:21<00:14, 31.66ba/s]
Creating json from Arrow format:  78%|#######7  | 1639/2105 [02:21<00:13, 33.31ba/s]
Creating json from Arrow format:  78%|#######8  | 1643/2105 [02:21<00:15, 30.59ba/s]
Creating json from Arrow format:  78%|#######8  | 1647/2105 [02:21<00:15, 28.99ba/s]
Creating json from Arrow format:  78%|#######8  | 1650/2105 [02:21<00:15, 28.84ba/s]
Creating json from Arrow format:  79%|#######8  | 1654/2105 [02:21<00:14, 31.46ba/s]
Creating json from Arrow format:  79%|#######8  | 1658/2105 [02:21<00:13, 33.52ba/s]
Creating json from Arrow format:  79%|#######8  | 1662/2105 [02:21<00:12, 34.37ba/s]
Creating json from Arrow format:  79%|#######9  | 1666/2105 [02:22<00:13, 32.50ba/s]
Creating json from Arrow format:  79%|#######9  | 1670/2105 [02:22<00:13, 31.60ba/s]
Creating json from Arrow format:  80%|#######9  | 1674/2105 [02:22<00:13, 31.75ba/s]
Creating json from Arrow format:  80%|#######9  | 1678/2105 [02:22<00:13, 32.78ba/s]
Creating json from Arrow format:  80%|#######9  | 1682/2105 [02:22<00:12, 33.43ba/s]
Creating json from Arrow format:  80%|########  | 1686/2105 [02:22<00:13, 31.75ba/s]
Creating json from Arrow format:  80%|########  | 1690/2105 [02:22<00:15, 27.55ba/s]
Creating json from Arrow format:  80%|########  | 1693/2105 [02:23<00:16, 25.43ba/s]
Creating json from Arrow format:  81%|########  | 1696/2105 [02:23<00:16, 24.17ba/s]
Creating json from Arrow format:  81%|########  | 1700/2105 [02:23<00:15, 26.23ba/s]
Creating json from Arrow format:  81%|########  | 1704/2105 [02:23<00:13, 28.70ba/s]
Creating json from Arrow format:  81%|########1 | 1708/2105 [02:23<00:12, 30.56ba/s]
Creating json from Arrow format:  81%|########1 | 1712/2105 [02:23<00:13, 29.99ba/s]
Creating json from Arrow format:  82%|########1 | 1716/2105 [02:23<00:13, 29.20ba/s]
Creating json from Arrow format:  82%|########1 | 1719/2105 [02:23<00:13, 28.75ba/s]
Creating json from Arrow format:  82%|########1 | 1722/2105 [02:24<00:14, 26.48ba/s]
Creating json from Arrow format:  82%|########1 | 1725/2105 [02:24<00:15, 24.63ba/s]
Creating json from Arrow format:  82%|########2 | 1728/2105 [02:24<00:16, 23.19ba/s]
Creating json from Arrow format:  82%|########2 | 1731/2105 [02:24<00:16, 22.47ba/s]
Creating json from Arrow format:  82%|########2 | 1734/2105 [02:24<00:16, 21.84ba/s]
Creating json from Arrow format:  83%|########2 | 1737/2105 [02:24<00:17, 21.56ba/s]
Creating json from Arrow format:  83%|########2 | 1740/2105 [02:24<00:17, 21.03ba/s]
Creating json from Arrow format:  83%|########2 | 1743/2105 [02:25<00:17, 20.82ba/s]
Creating json from Arrow format:  83%|########2 | 1746/2105 [02:25<00:19, 18.05ba/s]
Creating json from Arrow format:  83%|########3 | 1748/2105 [02:25<00:21, 16.41ba/s]
Creating json from Arrow format:  83%|########3 | 1750/2105 [02:25<00:23, 15.17ba/s]
Creating json from Arrow format:  83%|########3 | 1752/2105 [02:25<00:24, 14.30ba/s]
Creating json from Arrow format:  83%|########3 | 1754/2105 [02:25<00:25, 13.72ba/s]
Creating json from Arrow format:  83%|########3 | 1756/2105 [02:26<00:25, 13.66ba/s]
Creating json from Arrow format:  84%|########3 | 1758/2105 [02:26<00:23, 14.86ba/s]
Creating json from Arrow format:  84%|########3 | 1760/2105 [02:26<00:21, 15.90ba/s]
Creating json from Arrow format:  84%|########3 | 1762/2105 [02:26<00:20, 16.74ba/s]
Creating json from Arrow format:  84%|########3 | 1764/2105 [02:26<00:19, 17.42ba/s]
Creating json from Arrow format:  84%|########3 | 1766/2105 [02:26<00:18, 17.96ba/s]
Creating json from Arrow format:  84%|########3 | 1768/2105 [02:26<00:18, 18.49ba/s]
Creating json from Arrow format:  84%|########4 | 1771/2105 [02:26<00:16, 20.53ba/s]
Creating json from Arrow format:  84%|########4 | 1774/2105 [02:26<00:14, 22.11ba/s]
Creating json from Arrow format:  84%|########4 | 1777/2105 [02:27<00:14, 23.20ba/s]
Creating json from Arrow format:  85%|########4 | 1780/2105 [02:27<00:13, 23.26ba/s]
Creating json from Arrow format:  85%|########4 | 1783/2105 [02:27<00:15, 21.37ba/s]
Creating json from Arrow format:  85%|########4 | 1786/2105 [02:27<00:15, 20.73ba/s]
Creating json from Arrow format:  85%|########4 | 1789/2105 [02:27<00:15, 20.06ba/s]
Creating json from Arrow format:  85%|########5 | 1792/2105 [02:27<00:15, 20.54ba/s]
Creating json from Arrow format:  85%|########5 | 1796/2105 [02:27<00:12, 24.09ba/s]
Creating json from Arrow format:  86%|########5 | 1800/2105 [02:28<00:11, 27.00ba/s]
Creating json from Arrow format:  86%|########5 | 1804/2105 [02:28<00:10, 28.51ba/s]
Creating json from Arrow format:  86%|########5 | 1807/2105 [02:28<00:10, 27.47ba/s]
Creating json from Arrow format:  86%|########5 | 1810/2105 [02:28<00:11, 26.71ba/s]
Creating json from Arrow format:  86%|########6 | 1813/2105 [02:28<00:11, 26.15ba/s]
Creating json from Arrow format:  86%|########6 | 1816/2105 [02:28<00:11, 25.84ba/s]
Creating json from Arrow format:  86%|########6 | 1819/2105 [02:28<00:11, 25.51ba/s]
Creating json from Arrow format:  87%|########6 | 1822/2105 [02:28<00:11, 25.37ba/s]
Creating json from Arrow format:  87%|########6 | 1825/2105 [02:28<00:11, 25.21ba/s]
Creating json from Arrow format:  87%|########6 | 1828/2105 [02:29<00:11, 23.81ba/s]
Creating json from Arrow format:  87%|########6 | 1831/2105 [02:29<00:12, 22.37ba/s]
Creating json from Arrow format:  87%|########7 | 1834/2105 [02:29<00:12, 21.48ba/s]
Creating json from Arrow format:  87%|########7 | 1837/2105 [02:29<00:12, 21.03ba/s]
Creating json from Arrow format:  87%|########7 | 1840/2105 [02:29<00:12, 21.44ba/s]
Creating json from Arrow format:  88%|########7 | 1843/2105 [02:29<00:11, 22.62ba/s]
Creating json from Arrow format:  88%|########7 | 1846/2105 [02:29<00:11, 23.49ba/s]
Creating json from Arrow format:  88%|########7 | 1849/2105 [02:30<00:10, 24.11ba/s]
Creating json from Arrow format:  88%|########7 | 1852/2105 [02:30<00:10, 23.85ba/s]
Creating json from Arrow format:  88%|########8 | 1855/2105 [02:30<00:11, 22.66ba/s]
Creating json from Arrow format:  88%|########8 | 1858/2105 [02:30<00:11, 21.85ba/s]
Creating json from Arrow format:  88%|########8 | 1861/2105 [02:30<00:11, 21.37ba/s]
Creating json from Arrow format:  89%|########8 | 1864/2105 [02:30<00:11, 21.82ba/s]
Creating json from Arrow format:  89%|########8 | 1867/2105 [02:30<00:10, 22.78ba/s]
Creating json from Arrow format:  89%|########8 | 1870/2105 [02:31<00:10, 23.46ba/s]
Creating json from Arrow format:  89%|########8 | 1873/2105 [02:31<00:09, 23.98ba/s]
Creating json from Arrow format:  89%|########9 | 1877/2105 [02:31<00:08, 27.36ba/s]
Creating json from Arrow format:  89%|########9 | 1882/2105 [02:31<00:06, 31.97ba/s]
Creating json from Arrow format:  90%|########9 | 1887/2105 [02:31<00:06, 32.37ba/s]
Creating json from Arrow format:  90%|########9 | 1891/2105 [02:31<00:07, 27.12ba/s]
Creating json from Arrow format:  90%|########9 | 1894/2105 [02:31<00:08, 24.60ba/s]
Creating json from Arrow format:  90%|######### | 1897/2105 [02:32<00:09, 22.91ba/s]
Creating json from Arrow format:  90%|######### | 1900/2105 [02:32<00:08, 24.28ba/s]
Creating json from Arrow format:  90%|######### | 1904/2105 [02:32<00:07, 26.71ba/s]
Creating json from Arrow format:  91%|######### | 1908/2105 [02:32<00:06, 28.19ba/s]
Creating json from Arrow format:  91%|######### | 1911/2105 [02:32<00:06, 28.05ba/s]
Creating json from Arrow format:  91%|######### | 1914/2105 [02:32<00:07, 26.71ba/s]
Creating json from Arrow format:  91%|#########1| 1917/2105 [02:32<00:07, 25.69ba/s]
Creating json from Arrow format:  91%|#########1| 1920/2105 [02:32<00:07, 24.78ba/s]
Creating json from Arrow format:  91%|#########1| 1923/2105 [02:32<00:07, 24.88ba/s]
Creating json from Arrow format:  91%|#########1| 1926/2105 [02:33<00:07, 25.17ba/s]
Creating json from Arrow format:  92%|#########1| 1929/2105 [02:33<00:06, 25.32ba/s]
Creating json from Arrow format:  92%|#########1| 1932/2105 [02:33<00:06, 25.43ba/s]
Creating json from Arrow format:  92%|#########1| 1935/2105 [02:33<00:06, 26.04ba/s]
Creating json from Arrow format:  92%|#########2| 1938/2105 [02:33<00:06, 26.86ba/s]
Creating json from Arrow format:  92%|#########2| 1941/2105 [02:33<00:06, 27.29ba/s]
Creating json from Arrow format:  92%|#########2| 1944/2105 [02:33<00:05, 27.53ba/s]
Creating json from Arrow format:  93%|#########2| 1948/2105 [02:33<00:05, 29.31ba/s]
Creating json from Arrow format:  93%|#########2| 1952/2105 [02:34<00:04, 31.05ba/s]
Creating json from Arrow format:  93%|#########2| 1956/2105 [02:34<00:04, 32.47ba/s]
Creating json from Arrow format:  93%|#########3| 1960/2105 [02:34<00:05, 25.03ba/s]
Creating json from Arrow format:  93%|#########3| 1963/2105 [02:34<00:06, 21.86ba/s]
Creating json from Arrow format:  93%|#########3| 1966/2105 [02:34<00:06, 20.07ba/s]
Creating json from Arrow format:  94%|#########3| 1969/2105 [02:34<00:07, 19.25ba/s]
Creating json from Arrow format:  94%|#########3| 1974/2105 [02:35<00:05, 24.32ba/s]
Creating json from Arrow format:  94%|#########4| 1979/2105 [02:35<00:04, 28.53ba/s]
Creating json from Arrow format:  94%|#########4| 1983/2105 [02:35<00:04, 27.18ba/s]
Creating json from Arrow format:  94%|#########4| 1986/2105 [02:35<00:04, 25.33ba/s]
Creating json from Arrow format:  94%|#########4| 1989/2105 [02:35<00:04, 24.04ba/s]
Creating json from Arrow format:  95%|#########4| 1992/2105 [02:35<00:04, 23.09ba/s]
Creating json from Arrow format:  95%|#########4| 1995/2105 [02:35<00:04, 22.52ba/s]
Creating json from Arrow format:  95%|#########4| 1998/2105 [02:36<00:04, 22.30ba/s]
Creating json from Arrow format:  95%|#########5| 2001/2105 [02:36<00:04, 22.06ba/s]
Creating json from Arrow format:  95%|#########5| 2004/2105 [02:36<00:04, 21.93ba/s]
Creating json from Arrow format:  95%|#########5| 2008/2105 [02:36<00:03, 25.87ba/s]
Creating json from Arrow format:  96%|#########5| 2012/2105 [02:36<00:03, 29.12ba/s]
Creating json from Arrow format:  96%|#########5| 2016/2105 [02:36<00:02, 31.16ba/s]
Creating json from Arrow format:  96%|#########5| 2020/2105 [02:36<00:03, 27.45ba/s]
Creating json from Arrow format:  96%|#########6| 2023/2105 [02:36<00:03, 25.78ba/s]
Creating json from Arrow format:  96%|#########6| 2026/2105 [02:37<00:03, 24.65ba/s]
Creating json from Arrow format:  96%|#########6| 2029/2105 [02:37<00:03, 24.36ba/s]
Creating json from Arrow format:  97%|#########6| 2032/2105 [02:37<00:02, 24.54ba/s]
Creating json from Arrow format:  97%|#########6| 2035/2105 [02:37<00:02, 24.68ba/s]
Creating json from Arrow format:  97%|#########6| 2038/2105 [02:37<00:02, 24.86ba/s]
Creating json from Arrow format:  97%|#########6| 2041/2105 [02:37<00:02, 24.91ba/s]
Creating json from Arrow format:  97%|#########7| 2044/2105 [02:37<00:02, 24.88ba/s]
Creating json from Arrow format:  97%|#########7| 2047/2105 [02:37<00:02, 25.02ba/s]
Creating json from Arrow format:  97%|#########7| 2050/2105 [02:38<00:02, 24.92ba/s]
Creating json from Arrow format:  98%|#########7| 2053/2105 [02:38<00:02, 24.27ba/s]
Creating json from Arrow format:  98%|#########7| 2056/2105 [02:38<00:02, 23.49ba/s]
Creating json from Arrow format:  98%|#########7| 2059/2105 [02:38<00:02, 22.87ba/s]
Creating json from Arrow format:  98%|#########7| 2062/2105 [02:38<00:01, 22.51ba/s]
Creating json from Arrow format:  98%|#########8| 2065/2105 [02:38<00:01, 22.91ba/s]
Creating json from Arrow format:  98%|#########8| 2068/2105 [02:38<00:01, 23.48ba/s]
Creating json from Arrow format:  98%|#########8| 2071/2105 [02:38<00:01, 23.90ba/s]
Creating json from Arrow format:  99%|#########8| 2074/2105 [02:39<00:01, 24.30ba/s]
Creating json from Arrow format:  99%|#########8| 2077/2105 [02:39<00:01, 22.17ba/s]
Creating json from Arrow format:  99%|#########8| 2080/2105 [02:39<00:01, 20.65ba/s]
Creating json from Arrow format:  99%|#########8| 2083/2105 [02:39<00:01, 19.38ba/s]
Creating json from Arrow format:  99%|#########9| 2087/2105 [02:39<00:00, 22.89ba/s]
Creating json from Arrow format:  99%|#########9| 2091/2105 [02:39<00:00, 26.18ba/s]
Creating json from Arrow format:  99%|#########9| 2094/2105 [02:39<00:00, 26.59ba/s]
Creating json from Arrow format: 100%|#########9| 2097/2105 [02:40<00:00, 24.53ba/s]
Creating json from Arrow format: 100%|#########9| 2100/2105 [02:40<00:00, 23.17ba/s]
Creating json from Arrow format: 100%|#########9| 2103/2105 [02:40<00:00, 22.30ba/s]
Creating json from Arrow format: 100%|##########| 2105/2105 [02:40<00:00, 13.12ba/s]INFO 2025-05-20 11:33:44,311 instructlab.sdg.datamixing:235: Mixed Dataset saved to /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/skills_train_msgs_2025-05-19T13_31_09.jsonl
INFO 2025-05-20 11:33:44,604 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_6_2_1__About_workload_updates_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 649 examples [00:00, 88108.21 examples/s]INFO 2025-05-20 11:33:44,650 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:44,650 instructlab.sdg.datamixing:161: Dataset loaded with 649 samples


Map (num_proc=8):   0%|          | 0/649 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 81/649 [00:00<00:01, 504.96 examples/s]
Map (num_proc=8): 100%|##########| 649/649 [00:00<00:00, 2329.32 examples/s]INFO 2025-05-20 11:33:45,351 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1_6__About_RunStrategies_for_virtual_machines_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2341 examples [00:00, 104960.72 examples/s]INFO 2025-05-20 11:33:45,391 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:45,391 instructlab.sdg.datamixing:161: Dataset loaded with 2341 samples


Map (num_proc=8):   0%|          | 0/2341 [00:00<?, ? examples/s]
Map (num_proc=8):  10%|#         | 241/2341 [00:00<00:01, 1234.52 examples/s]
Map (num_proc=8):  75%|#######5  | 1767/2341 [00:00<00:00, 7128.54 examples/s]
Map (num_proc=8): 100%|##########| 2341/2341 [00:00<00:00, 5498.89 examples/s]INFO 2025-05-20 11:33:46,246 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_3_1__About_virtual_machine_events_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 121 examples [00:00, 43856.79 examples/s]INFO 2025-05-20 11:33:46,265 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:46,265 instructlab.sdg.datamixing:161: Dataset loaded with 121 samples


Map (num_proc=8):   0%|          | 0/121 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 15/121 [00:00<00:00, 131.55 examples/s]
Map (num_proc=8): 100%|##########| 121/121 [00:00<00:00, 545.23 examples/s]INFO 2025-05-20 11:33:46,904 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_8_5__Libguestfs_tools_and_virtctl_guestfs_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 274 examples [00:00, 51179.66 examples/s]INFO 2025-05-20 11:33:46,927 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:46,927 instructlab.sdg.datamixing:161: Dataset loaded with 274 samples


Map (num_proc=8):   0%|          | 0/274 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 35/274 [00:00<00:00, 281.74 examples/s]
Map (num_proc=8): 100%|##########| 274/274 [00:00<00:00, 1136.86 examples/s]INFO 2025-05-20 11:33:47,585 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_12_2__Removing_a_secret__config_map__or_service_account_from_a_virtual_machine_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 739 examples [00:00, 89604.26 examples/s]INFO 2025-05-20 11:33:47,610 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:47,610 instructlab.sdg.datamixing:161: Dataset loaded with 739 samples


Map (num_proc=8):   0%|          | 0/739 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 93/739 [00:00<00:01, 534.17 examples/s]
Map (num_proc=8): 100%|##########| 739/739 [00:00<00:00, 2571.51 examples/s]INFO 2025-05-20 11:33:48,321 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_16_10__Configuring_vGPU_passthrough_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 10306 examples [00:00, 114801.96 examples/s]INFO 2025-05-20 11:33:48,429 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:48,429 instructlab.sdg.datamixing:161: Dataset loaded with 10306 samples


Map (num_proc=8):   0%|          | 0/10306 [00:00<?, ? examples/s]
Map (num_proc=8):   2%|1         | 206/10306 [00:00<00:09, 1081.65 examples/s]
Map (num_proc=8):  18%|#8        | 1871/10306 [00:00<00:01, 7578.98 examples/s]
Map (num_proc=8):  38%|###7      | 3867/10306 [00:00<00:00, 11771.08 examples/s]
Map (num_proc=8):  54%|#####3    | 5553/10306 [00:00<00:00, 13278.96 examples/s]
Map (num_proc=8):  68%|######8   | 7032/10306 [00:00<00:00, 12610.90 examples/s]
Map (num_proc=8):  81%|########1 | 8383/10306 [00:00<00:00, 10458.46 examples/s]
Map (num_proc=8):  94%|#########3| 9647/10306 [00:00<00:00, 10638.77 examples/s]
Map (num_proc=8): 100%|##########| 10306/10306 [00:01<00:00, 9748.23 examples/s]INFO 2025-05-20 11:33:49,939 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_3_2__Storage_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 310 examples [00:00, 83492.86 examples/s]INFO 2025-05-20 11:33:49,958 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:49,958 instructlab.sdg.datamixing:161: Dataset loaded with 310 samples


Map (num_proc=8):   0%|          | 0/310 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 39/310 [00:00<00:00, 314.36 examples/s]
Map (num_proc=8): 100%|##########| 310/310 [00:00<00:00, 1168.26 examples/s]INFO 2025-05-20 11:33:50,642 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_4_1__Prerequisites_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1360 examples [00:00, 101184.10 examples/s]INFO 2025-05-20 11:33:50,673 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:50,673 instructlab.sdg.datamixing:161: Dataset loaded with 1360 samples


Map (num_proc=8):   0%|          | 0/1360 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 170/1360 [00:00<00:01, 942.17 examples/s]
Map (num_proc=8): 100%|##########| 1360/1360 [00:00<00:00, 4103.76 examples/s]INFO 2025-05-20 11:33:51,428 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_6_2__Restarting_a_virtual_machine_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 255 examples [00:00, 54250.44 examples/s]INFO 2025-05-20 11:33:51,449 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:51,450 instructlab.sdg.datamixing:161: Dataset loaded with 255 samples


Map (num_proc=8):   0%|          | 0/255 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 32/255 [00:00<00:00, 253.08 examples/s]
Map (num_proc=8): 100%|##########| 255/255 [00:00<00:00, 1060.60 examples/s]INFO 2025-05-20 11:33:52,109 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_2_2__Example_manifests_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2652 examples [00:00, 106542.92 examples/s]INFO 2025-05-20 11:33:52,152 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:52,152 instructlab.sdg.datamixing:161: Dataset loaded with 2652 samples


Map (num_proc=8):   0%|          | 0/2652 [00:00<?, ? examples/s]
Map (num_proc=8):   7%|6         | 178/2652 [00:00<00:02, 1004.70 examples/s]
Map (num_proc=8):  61%|######1   | 1628/2652 [00:00<00:00, 6755.91 examples/s]
Map (num_proc=8): 100%|##########| 2652/2652 [00:00<00:00, 5894.21 examples/s]INFO 2025-05-20 11:33:53,032 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_2_2__Viewing_virtual_machine_logs_in_the_web_console_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 272 examples [00:00, 39076.92 examples/s]INFO 2025-05-20 11:33:53,057 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:53,057 instructlab.sdg.datamixing:161: Dataset loaded with 272 samples


Map (num_proc=8):   0%|          | 0/272 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 34/272 [00:00<00:00, 261.86 examples/s]
Map (num_proc=8): 100%|##########| 272/272 [00:00<00:00, 1113.15 examples/s]INFO 2025-05-20 11:33:53,723 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_12__Managing_config_maps__secrets__and_service_accounts_in_virtual_machines_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1034 examples [00:00, 98235.71 examples/s]INFO 2025-05-20 11:33:53,750 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:53,750 instructlab.sdg.datamixing:161: Dataset loaded with 1034 samples


Map (num_proc=8):   0%|          | 0/1034 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 130/1034 [00:00<00:01, 804.46 examples/s]
Map (num_proc=8): 100%|##########| 1034/1034 [00:00<00:00, 3369.68 examples/s]INFO 2025-05-20 11:33:54,479 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_5_6__Deleting_a_standalone_virtual_machine_instance_using_the_web_console_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1480 examples [00:00, 130570.23 examples/s]INFO 2025-05-20 11:33:54,514 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:54,514 instructlab.sdg.datamixing:161: Dataset loaded with 1480 samples


Map (num_proc=8):   0%|          | 0/1480 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 185/1480 [00:00<00:01, 1043.85 examples/s]
Map (num_proc=8): 100%|##########| 1480/1480 [00:00<00:00, 4394.55 examples/s]INFO 2025-05-20 11:33:55,280 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_OpenShift_Virtualization_installation__usage__and_release_notes_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 150 examples [00:00, 41679.07 examples/s]INFO 2025-05-20 11:33:55,300 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:55,300 instructlab.sdg.datamixing:161: Dataset loaded with 150 samples


Map (num_proc=8):   0%|          | 0/150 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 19/150 [00:00<00:00, 167.92 examples/s]
Map (num_proc=8): 100%|##########| 150/150 [00:00<00:00, 663.66 examples/s]INFO 2025-05-20 11:33:55,939 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_1__Making_open_source_more_inclusive_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 251 examples [00:00, 67732.76 examples/s]INFO 2025-05-20 11:33:55,960 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:55,960 instructlab.sdg.datamixing:161: Dataset loaded with 251 samples


Map (num_proc=8):   0%|          | 0/251 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 32/251 [00:00<00:00, 254.42 examples/s]
Map (num_proc=8): 100%|##########| 251/251 [00:00<00:00, 1062.76 examples/s]INFO 2025-05-20 11:33:56,610 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_20_20__Expanding_a_virtual_machine_disk_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 11757 examples [00:00, 107473.60 examples/s]
Generating train split: 16300 examples [00:00, 115898.73 examples/s]INFO 2025-05-20 11:33:56,769 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:56,769 instructlab.sdg.datamixing:161: Dataset loaded with 16300 samples


Map (num_proc=8):   0%|          | 0/16300 [00:00<?, ? examples/s]
Map (num_proc=8):   2%|2         | 358/16300 [00:00<00:08, 1783.79 examples/s]
Map (num_proc=8):  14%|#3        | 2202/16300 [00:00<00:01, 8558.94 examples/s]
Map (num_proc=8):  25%|##5       | 4097/16300 [00:00<00:01, 11866.08 examples/s]
Map (num_proc=8):  36%|###5      | 5793/16300 [00:00<00:00, 13056.96 examples/s]
Map (num_proc=8):  45%|####4     | 7314/16300 [00:00<00:00, 12585.07 examples/s]
Map (num_proc=8):  54%|#####3    | 8770/16300 [00:00<00:00, 11396.33 examples/s]
Map (num_proc=8):  61%|######1   | 10023/16300 [00:00<00:00, 11513.59 examples/s]
Map (num_proc=8):  72%|#######1  | 11732/16300 [00:01<00:00, 12854.75 examples/s]
Map (num_proc=8):  83%|########2 | 13455/16300 [00:01<00:00, 13576.15 examples/s]
Map (num_proc=8):  92%|#########2| 15060/16300 [00:01<00:00, 13326.79 examples/s]
Map (num_proc=8): 100%|##########| 16300/16300 [00:01<00:00, 11040.47 examples/s]INFO 2025-05-20 11:33:58,711 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_8_3__Generalizing_a_Windows_VM_using_sysprep_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1349 examples [00:00, 124751.76 examples/s]INFO 2025-05-20 11:33:58,740 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:58,740 instructlab.sdg.datamixing:161: Dataset loaded with 1349 samples


Map (num_proc=8):   0%|          | 0/1349 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 169/1349 [00:00<00:01, 841.27 examples/s]
Map (num_proc=8): 100%|##########| 1349/1349 [00:00<00:00, 4094.65 examples/s]INFO 2025-05-20 11:33:59,492 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_19_7__Viewing_the_IP_address_of_NICs_on_a_virtual_machine_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 8292 examples [00:00, 117099.21 examples/s]INFO 2025-05-20 11:33:59,582 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:33:59,583 instructlab.sdg.datamixing:161: Dataset loaded with 8292 samples


Map (num_proc=8):   0%|          | 0/8292 [00:00<?, ? examples/s]
Map (num_proc=8):   4%|4         | 364/8292 [00:00<00:03, 2133.80 examples/s]
Map (num_proc=8):  23%|##2       | 1870/8292 [00:00<00:00, 7832.55 examples/s]
Map (num_proc=8):  44%|####3     | 3619/8292 [00:00<00:00, 11359.41 examples/s]
Map (num_proc=8):  62%|######1   | 5123/8292 [00:00<00:00, 12573.83 examples/s]
Map (num_proc=8):  80%|########  | 6645/8292 [00:00<00:00, 12935.45 examples/s]
Map (num_proc=8):  97%|#########6| 8023/8292 [00:00<00:00, 10278.84 examples/s]
Map (num_proc=8): 100%|##########| 8292/8292 [00:00<00:00, 9203.28 examples/s] INFO 2025-05-20 11:34:00,925 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_1_4__Restricted_network_environments_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2260 examples [00:00, 113893.49 examples/s]INFO 2025-05-20 11:34:00,964 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:00,964 instructlab.sdg.datamixing:161: Dataset loaded with 2260 samples


Map (num_proc=8):   0%|          | 0/2260 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#1        | 263/2260 [00:00<00:01, 1410.65 examples/s]
Map (num_proc=8):  72%|#######2  | 1633/2260 [00:00<00:00, 6701.32 examples/s]
Map (num_proc=8): 100%|##########| 2260/2260 [00:00<00:00, 5417.17 examples/s]INFO 2025-05-20 11:34:01,804 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_2__Editing_virtual_machine_templates_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1366 examples [00:00, 120436.80 examples/s]INFO 2025-05-20 11:34:01,832 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:01,832 instructlab.sdg.datamixing:161: Dataset loaded with 1366 samples


Map (num_proc=8):   0%|          | 0/1366 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 171/1366 [00:00<00:01, 941.44 examples/s]
Map (num_proc=8): 100%|##########| 1366/1366 [00:00<00:00, 4213.85 examples/s]INFO 2025-05-20 11:34:02,576 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_11_2__About_the_QEMU_guest_agent_information_in_the_web_console_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 750 examples [00:00, 112733.94 examples/s]INFO 2025-05-20 11:34:02,603 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:02,603 instructlab.sdg.datamixing:161: Dataset loaded with 750 samples


Map (num_proc=8):   0%|          | 0/750 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 94/750 [00:00<00:01, 594.74 examples/s]
Map (num_proc=8): 100%|##########| 750/750 [00:00<00:00, 2703.52 examples/s]INFO 2025-05-20 11:34:03,293 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_8_1__About_reviewing_top_consumers_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 722 examples [00:00, 112287.72 examples/s]INFO 2025-05-20 11:34:03,317 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:03,317 instructlab.sdg.datamixing:161: Dataset loaded with 722 samples


Map (num_proc=8):   0%|          | 0/722 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 91/722 [00:00<00:01, 547.89 examples/s]
Map (num_proc=8): 100%|##########| 722/722 [00:00<00:00, 2631.76 examples/s]INFO 2025-05-20 11:34:04,002 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_3__Installing_OpenShift_Virtualization_using_the_web_console_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 725 examples [00:00, 82473.23 examples/s]INFO 2025-05-20 11:34:04,028 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:04,028 instructlab.sdg.datamixing:161: Dataset loaded with 725 samples


Map (num_proc=8):   0%|          | 0/725 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 91/725 [00:00<00:01, 543.86 examples/s]
Map (num_proc=8): 100%|##########| 725/725 [00:00<00:00, 2584.80 examples/s]INFO 2025-05-20 11:34:04,721 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_3_4__Viewing_resource_events_in_the_CLI_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 198 examples [00:00, 59087.31 examples/s]INFO 2025-05-20 11:34:04,742 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:04,742 instructlab.sdg.datamixing:161: Dataset loaded with 198 samples


Map (num_proc=8):   0%|          | 0/198 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 25/198 [00:00<00:00, 236.73 examples/s]
Map (num_proc=8): 100%|##########| 198/198 [00:00<00:00, 854.88 examples/s]INFO 2025-05-20 11:34:05,385 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_1_2__Resources_monitored_actively_in_the_Virtualization_Overview_page_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 881 examples [00:00, 111792.27 examples/s]INFO 2025-05-20 11:34:05,410 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:05,411 instructlab.sdg.datamixing:161: Dataset loaded with 881 samples


Map (num_proc=8):   0%|          | 0/881 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 110/881 [00:00<00:01, 673.55 examples/s]
Map (num_proc=8): 100%|##########| 881/881 [00:00<00:00, 3067.67 examples/s]INFO 2025-05-20 11:34:06,112 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_1_1__What_you_can_do_with_OpenShift_Virtualization_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 164 examples [00:00, 33121.43 examples/s]INFO 2025-05-20 11:34:06,134 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:06,134 instructlab.sdg.datamixing:161: Dataset loaded with 164 samples


Map (num_proc=8):   0%|          | 0/164 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 21/164 [00:00<00:00, 173.07 examples/s]
Map (num_proc=8): 100%|##########| 164/164 [00:00<00:00, 700.37 examples/s]INFO 2025-05-20 11:34:06,775 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_2_5__Adding_a_network_interface_to_a_virtual_machine_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2243 examples [00:00, 112660.45 examples/s]INFO 2025-05-20 11:34:06,812 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:06,812 instructlab.sdg.datamixing:161: Dataset loaded with 2243 samples


Map (num_proc=8):   0%|          | 0/2243 [00:00<?, ? examples/s]
Map (num_proc=8):   8%|8         | 190/2243 [00:00<00:02, 928.06 examples/s]
Map (num_proc=8):  95%|#########5| 2137/2243 [00:00<00:00, 8348.81 examples/s]
Map (num_proc=8): 100%|##########| 2243/2243 [00:00<00:00, 5626.81 examples/s]INFO 2025-05-20 11:34:07,633 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_2_2__Supported_guest_operating_systems_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 269 examples [00:00, 67423.67 examples/s]INFO 2025-05-20 11:34:07,655 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:07,655 instructlab.sdg.datamixing:161: Dataset loaded with 269 samples


Map (num_proc=8):   0%|          | 0/269 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 34/269 [00:00<00:00, 273.96 examples/s]
Map (num_proc=8): 100%|##########| 269/269 [00:00<00:00, 1156.05 examples/s]INFO 2025-05-20 11:34:08,300 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_6_6__Additional_resources_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1504 examples [00:00, 103215.69 examples/s]INFO 2025-05-20 11:34:08,333 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:08,333 instructlab.sdg.datamixing:161: Dataset loaded with 1504 samples


Map (num_proc=8):   0%|          | 0/1504 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 188/1504 [00:00<00:01, 1051.94 examples/s]
Map (num_proc=8): 100%|##########| 1504/1504 [00:00<00:00, 4333.70 examples/s]INFO 2025-05-20 11:34:09,097 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_14_3_3__Additional_resources_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1810 examples [00:00, 110424.59 examples/s]INFO 2025-05-20 11:34:09,132 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:09,132 instructlab.sdg.datamixing:161: Dataset loaded with 1810 samples


Map (num_proc=8):   0%|          | 0/1810 [00:00<?, ? examples/s]
Map (num_proc=8):   7%|7         | 134/1810 [00:00<00:02, 665.21 examples/s]
Map (num_proc=8): 100%|##########| 1810/1810 [00:00<00:00, 4882.61 examples/s]INFO 2025-05-20 11:34:09,922 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_11_4__Additional_resources_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1646 examples [00:00, 122590.82 examples/s]INFO 2025-05-20 11:34:09,953 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:09,954 instructlab.sdg.datamixing:161: Dataset loaded with 1646 samples


Map (num_proc=8):   0%|          | 0/1646 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 206/1646 [00:00<00:01, 1178.83 examples/s]
Map (num_proc=8): 100%|##########| 1646/1646 [00:00<00:00, 4731.76 examples/s]INFO 2025-05-20 11:34:10,716 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_5__Deleting_virtual_machine_templates_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 667 examples [00:00, 129165.74 examples/s]INFO 2025-05-20 11:34:10,739 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:10,739 instructlab.sdg.datamixing:161: Dataset loaded with 667 samples


Map (num_proc=8):   0%|          | 0/667 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 84/667 [00:00<00:01, 553.01 examples/s]
Map (num_proc=8): 100%|##########| 667/667 [00:00<00:00, 2524.91 examples/s]INFO 2025-05-20 11:34:11,417 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_7__About_the_tekton_tasks_operator_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 120 examples [00:00, 26170.78 examples/s]INFO 2025-05-20 11:34:11,441 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:11,441 instructlab.sdg.datamixing:161: Dataset loaded with 120 samples


Map (num_proc=8):   0%|          | 0/120 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 15/120 [00:00<00:00, 142.02 examples/s]
Map (num_proc=8): 100%|##########| 120/120 [00:00<00:00, 520.37 examples/s]INFO 2025-05-20 11:34:12,084 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_2_2__Editing_a_virtual_machine_YAML_configuration_using_the_web_console_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2544 examples [00:00, 122402.43 examples/s]INFO 2025-05-20 11:34:12,122 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:12,122 instructlab.sdg.datamixing:161: Dataset loaded with 2544 samples


Map (num_proc=8):   0%|          | 0/2544 [00:00<?, ? examples/s]
Map (num_proc=8):  10%|#         | 258/2544 [00:00<00:01, 1390.52 examples/s]
Map (num_proc=8):  74%|#######3  | 1873/2544 [00:00<00:00, 7809.50 examples/s]
Map (num_proc=8): 100%|##########| 2544/2544 [00:00<00:00, 6053.48 examples/s]INFO 2025-05-20 11:34:12,963 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_5__Viewing_information_about_virtual_machine_workloads_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 396 examples [00:00, 77923.73 examples/s]INFO 2025-05-20 11:34:12,986 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:12,986 instructlab.sdg.datamixing:161: Dataset loaded with 396 samples


Map (num_proc=8):   0%|          | 0/396 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 50/396 [00:00<00:00, 395.75 examples/s]
Map (num_proc=8): 100%|##########| 396/396 [00:00<00:00, 1586.53 examples/s]INFO 2025-05-20 11:34:13,644 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_10_1__About_the_OpenShift_Container_Platform_cluster_checkup_framework_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1558 examples [00:00, 88813.58 examples/s]INFO 2025-05-20 11:34:13,679 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:13,679 instructlab.sdg.datamixing:161: Dataset loaded with 1558 samples


Map (num_proc=8):   0%|          | 0/1558 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 195/1558 [00:00<00:01, 1099.22 examples/s]
Map (num_proc=8): 100%|##########| 1558/1558 [00:00<00:00, 4359.46 examples/s]INFO 2025-05-20 11:34:14,451 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_9_3__About_Telemetry_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1653 examples [00:00, 117449.89 examples/s]INFO 2025-05-20 11:34:14,488 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:14,488 instructlab.sdg.datamixing:161: Dataset loaded with 1653 samples


Map (num_proc=8):   0%|          | 0/1653 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 207/1653 [00:00<00:01, 1063.26 examples/s]
Map (num_proc=8): 100%|##########| 1653/1653 [00:00<00:00, 4714.43 examples/s]INFO 2025-05-20 11:34:15,255 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_4_1__Deprecated_features_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 343 examples [00:00, 76022.31 examples/s]INFO 2025-05-20 11:34:15,277 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:15,277 instructlab.sdg.datamixing:161: Dataset loaded with 343 samples


Map (num_proc=8):   0%|          | 0/343 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 43/343 [00:00<00:00, 322.25 examples/s]
Map (num_proc=8): 100%|##########| 343/343 [00:00<00:00, 1401.27 examples/s]INFO 2025-05-20 11:34:15,928 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_3_1__Adding_items_to_a_boot_order_list_in_the_web_console_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1163 examples [00:00, 112191.53 examples/s]INFO 2025-05-20 11:34:15,957 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:15,957 instructlab.sdg.datamixing:161: Dataset loaded with 1163 samples


Map (num_proc=8):   0%|          | 0/1163 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 146/1163 [00:00<00:01, 810.07 examples/s]
Map (num_proc=8): 100%|##########| 1163/1163 [00:00<00:00, 3716.48 examples/s]INFO 2025-05-20 11:34:16,682 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_17_1__TLS_certificates_for_data_volume_imports_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1035 examples [00:00, 92580.61 examples/s]INFO 2025-05-20 11:34:16,713 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:16,713 instructlab.sdg.datamixing:161: Dataset loaded with 1035 samples


Map (num_proc=8):   0%|          | 0/1035 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 130/1035 [00:00<00:01, 764.39 examples/s]
Map (num_proc=8): 100%|##########| 1035/1035 [00:00<00:00, 3403.34 examples/s]INFO 2025-05-20 11:34:17,434 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_11_3__Migrating_a_virtual_machine_instance_to_another_node_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 147 examples [00:00, 41696.27 examples/s]INFO 2025-05-20 11:34:17,456 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:17,456 instructlab.sdg.datamixing:161: Dataset loaded with 147 samples


Map (num_proc=8):   0%|          | 0/147 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 19/147 [00:00<00:00, 163.73 examples/s]
Map (num_proc=8): 100%|##########| 147/147 [00:00<00:00, 640.98 examples/s]INFO 2025-05-20 11:34:18,100 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_2__About_Red_Hat_OpenShift_Virtualization_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 357 examples [00:00, 81325.58 examples/s]INFO 2025-05-20 11:34:18,123 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:18,123 instructlab.sdg.datamixing:161: Dataset loaded with 357 samples


Map (num_proc=8):   0%|          | 0/357 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 45/357 [00:00<00:00, 344.94 examples/s]
Map (num_proc=8): 100%|##########| 357/357 [00:00<00:00, 1465.29 examples/s]INFO 2025-05-20 11:34:18,775 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_8_2__OpenShift_Container_Platform_client_commands_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 408 examples [00:00, 58899.84 examples/s]INFO 2025-05-20 11:34:18,802 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:18,802 instructlab.sdg.datamixing:161: Dataset loaded with 408 samples


Map (num_proc=8):   0%|          | 0/408 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 51/408 [00:00<00:00, 361.03 examples/s]
Map (num_proc=8): 100%|##########| 408/408 [00:00<00:00, 1614.38 examples/s]INFO 2025-05-20 11:34:19,462 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_12_1__Adding_a_secret__config_map__or_service_account_to_a_virtual_machine_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 798 examples [00:00, 88586.26 examples/s]INFO 2025-05-20 11:34:19,490 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:19,490 instructlab.sdg.datamixing:161: Dataset loaded with 798 samples


Map (num_proc=8):   0%|          | 0/798 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 100/798 [00:00<00:01, 574.50 examples/s]
Map (num_proc=8): 100%|##########| 798/798 [00:00<00:00, 2775.67 examples/s]INFO 2025-05-20 11:34:20,192 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_11_3__Virtualization_metrics_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1506 examples [00:00, 126903.50 examples/s]INFO 2025-05-20 11:34:20,221 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:20,221 instructlab.sdg.datamixing:161: Dataset loaded with 1506 samples


Map (num_proc=8):   0%|          | 0/1506 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 189/1506 [00:00<00:01, 1005.08 examples/s]
Map (num_proc=8): 100%|##########| 1506/1506 [00:00<00:00, 4505.17 examples/s]INFO 2025-05-20 11:34:20,970 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_12_3__Managing_node_labeling_for_obsolete_CPU_models_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 159 examples [00:00, 47533.45 examples/s]INFO 2025-05-20 11:34:20,990 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:20,990 instructlab.sdg.datamixing:161: Dataset loaded with 159 samples


Map (num_proc=8):   0%|          | 0/159 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 20/159 [00:00<00:00, 169.58 examples/s]
Map (num_proc=8): 100%|##########| 159/159 [00:00<00:00, 685.11 examples/s]INFO 2025-05-20 11:34:21,628 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_12_5__Additional_resources_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 848 examples [00:00, 82008.02 examples/s]INFO 2025-05-20 11:34:21,655 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:21,655 instructlab.sdg.datamixing:161: Dataset loaded with 848 samples


Map (num_proc=8):   0%|          | 0/848 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 106/848 [00:00<00:01, 679.16 examples/s]
Map (num_proc=8): 100%|##########| 848/848 [00:00<00:00, 2862.36 examples/s]INFO 2025-05-20 11:34:22,366 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_12_4_1__Using_skip_node_annotation_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 156 examples [00:00, 46299.99 examples/s]INFO 2025-05-20 11:34:22,392 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:22,392 instructlab.sdg.datamixing:161: Dataset loaded with 156 samples


Map (num_proc=8):   0%|          | 0/156 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 20/156 [00:00<00:00, 183.45 examples/s]
Map (num_proc=8): 100%|##########| 156/156 [00:00<00:00, 685.69 examples/s]INFO 2025-05-20 11:34:23,025 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_6__Monitoring_virtual_machine_health_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1627 examples [00:00, 106445.78 examples/s]INFO 2025-05-20 11:34:23,059 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:23,059 instructlab.sdg.datamixing:161: Dataset loaded with 1627 samples


Map (num_proc=8):   0%|          | 0/1627 [00:00<?, ? examples/s]
Map (num_proc=8):   9%|9         | 149/1627 [00:00<00:01, 847.03 examples/s]
Map (num_proc=8): 100%|##########| 1627/1627 [00:00<00:00, 4596.31 examples/s]INFO 2025-05-20 11:34:23,830 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_1_2__About_virtual_machines_and_boot_sources_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 412 examples [00:00, 56810.22 examples/s]INFO 2025-05-20 11:34:23,855 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:23,855 instructlab.sdg.datamixing:161: Dataset loaded with 412 samples


Map (num_proc=8):   0%|          | 0/412 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 52/412 [00:00<00:00, 366.43 examples/s]
Map (num_proc=8): 100%|##########| 412/412 [00:00<00:00, 1584.77 examples/s]INFO 2025-05-20 11:34:24,526 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_2__Viewing_OpenShift_Virtualization_logs_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 307 examples [00:00, 47555.17 examples/s]INFO 2025-05-20 11:34:24,550 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:24,550 instructlab.sdg.datamixing:161: Dataset loaded with 307 samples


Map (num_proc=8):   0%|          | 0/307 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 39/307 [00:00<00:00, 287.57 examples/s]
Map (num_proc=8): 100%|##########| 307/307 [00:00<00:00, 1222.57 examples/s]INFO 2025-05-20 11:34:25,211 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_8__About_the_virt_operator_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 397 examples [00:00, 70838.88 examples/s]INFO 2025-05-20 11:34:25,234 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:25,234 instructlab.sdg.datamixing:161: Dataset loaded with 397 samples


Map (num_proc=8):   0%|          | 0/397 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 50/397 [00:00<00:00, 374.89 examples/s]
Map (num_proc=8): 100%|##########| 397/397 [00:00<00:00, 1581.72 examples/s]INFO 2025-05-20 11:34:25,892 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_14_4__Adding_VirtIO_drivers_container_disk_to_a_virtual_machine_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1431 examples [00:00, 100049.16 examples/s]INFO 2025-05-20 11:34:25,924 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:25,924 instructlab.sdg.datamixing:161: Dataset loaded with 1431 samples


Map (num_proc=8):   0%|          | 0/1431 [00:00<?, ? examples/s]
Map (num_proc=8):   8%|7         | 113/1431 [00:00<00:01, 662.73 examples/s]
Map (num_proc=8): 100%|##########| 1431/1431 [00:00<00:00, 4197.25 examples/s]INFO 2025-05-20 11:34:26,682 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_5__Managing_virtual_machine_instances_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1547 examples [00:00, 133967.63 examples/s]INFO 2025-05-20 11:34:26,712 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:26,712 instructlab.sdg.datamixing:161: Dataset loaded with 1547 samples


Map (num_proc=8):   0%|          | 0/1547 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 194/1547 [00:00<00:01, 959.68 examples/s]
Map (num_proc=8): 100%|##########| 1547/1547 [00:00<00:00, 4591.01 examples/s]INFO 2025-05-20 11:34:27,464 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_11_1__Prerequisites_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2442 examples [00:00, 129645.21 examples/s]INFO 2025-05-20 11:34:27,502 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:27,502 instructlab.sdg.datamixing:161: Dataset loaded with 2442 samples


Map (num_proc=8):   0%|          | 0/2442 [00:00<?, ? examples/s]
Map (num_proc=8):   7%|6         | 166/2442 [00:00<00:02, 838.84 examples/s]
Map (num_proc=8):  82%|########1 | 1995/2442 [00:00<00:00, 8002.43 examples/s]
Map (num_proc=8): 100%|##########| 2442/2442 [00:00<00:00, 5909.11 examples/s]INFO 2025-05-20 11:34:28,333 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_1_3__Object_maximums_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2334 examples [00:00, 117850.72 examples/s]INFO 2025-05-20 11:34:28,372 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:28,372 instructlab.sdg.datamixing:161: Dataset loaded with 2334 samples


Map (num_proc=8):   0%|          | 0/2334 [00:00<?, ? examples/s]
Map (num_proc=8):  11%|#1        | 259/2334 [00:00<00:01, 1325.39 examples/s]
Map (num_proc=8):  90%|########9 | 2090/2334 [00:00<00:00, 8205.27 examples/s]
Map (num_proc=8): 100%|##########| 2334/2334 [00:00<00:00, 5658.32 examples/s]INFO 2025-05-20 11:34:29,205 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_14_5__Installing_VirtIO_drivers_during_Windows_installation_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1524 examples [00:00, 106165.51 examples/s]INFO 2025-05-20 11:34:29,237 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:29,237 instructlab.sdg.datamixing:161: Dataset loaded with 1524 samples


Map (num_proc=8):   0%|          | 0/1524 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 191/1524 [00:00<00:01, 953.62 examples/s]
Map (num_proc=8): 100%|##########| 1524/1524 [00:00<00:00, 4373.96 examples/s]INFO 2025-05-20 11:34:29,999 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_16_12__Configuring_a_watchdog_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 7968 examples [00:00, 113759.71 examples/s]INFO 2025-05-20 11:34:30,092 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:30,092 instructlab.sdg.datamixing:161: Dataset loaded with 7968 samples


Map (num_proc=8):   0%|          | 0/7968 [00:00<?, ? examples/s]
Map (num_proc=8):   2%|1         | 149/7968 [00:00<00:10, 755.76 examples/s]
Map (num_proc=8):  22%|##1       | 1750/7968 [00:00<00:00, 7064.63 examples/s]
Map (num_proc=8):  50%|####9     | 3977/7968 [00:00<00:00, 12611.93 examples/s]
Map (num_proc=8):  69%|######9   | 5527/7968 [00:00<00:00, 13201.91 examples/s]
Map (num_proc=8):  89%|########9 | 7112/7968 [00:00<00:00, 13041.50 examples/s]
Map (num_proc=8): 100%|##########| 7968/7968 [00:00<00:00, 9139.85 examples/s] INFO 2025-05-20 11:34:31,403 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_4__About_the_cluster_network_addons_operator_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 268 examples [00:00, 59581.97 examples/s]INFO 2025-05-20 11:34:31,425 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:31,425 instructlab.sdg.datamixing:161: Dataset loaded with 268 samples


Map (num_proc=8):   0%|          | 0/268 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 34/268 [00:00<00:00, 271.55 examples/s]
Map (num_proc=8): 100%|##########| 268/268 [00:00<00:00, 1112.72 examples/s]INFO 2025-05-20 11:34:32,077 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_15_1__About_vTPM_devices_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 538 examples [00:00, 95494.52 examples/s]INFO 2025-05-20 11:34:32,099 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:32,099 instructlab.sdg.datamixing:161: Dataset loaded with 538 samples


Map (num_proc=8):   0%|          | 0/538 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 68/538 [00:00<00:00, 512.85 examples/s]
Map (num_proc=8): 100%|##########| 538/538 [00:00<00:00, 2063.51 examples/s]INFO 2025-05-20 11:34:32,771 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_12_4__Preventing_node_reconciliation_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 88 examples [00:00, 39429.41 examples/s]INFO 2025-05-20 11:34:32,797 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:32,797 instructlab.sdg.datamixing:161: Dataset loaded with 88 samples


Map (num_proc=8):   0%|          | 0/88 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 11/88 [00:00<00:00, 109.94 examples/s]
Map (num_proc=8): 100%|##########| 88/88 [00:00<00:00, 393.57 examples/s]INFO 2025-05-20 11:34:33,433 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_19__Virtual_machine_networking_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 7804 examples [00:00, 118407.13 examples/s]INFO 2025-05-20 11:34:33,518 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:33,518 instructlab.sdg.datamixing:161: Dataset loaded with 7804 samples


Map (num_proc=8):   0%|          | 0/7804 [00:00<?, ? examples/s]
Map (num_proc=8):   2%|2         | 185/7804 [00:00<00:07, 1020.51 examples/s]
Map (num_proc=8):  22%|##1       | 1713/7804 [00:00<00:00, 7288.34 examples/s]
Map (num_proc=8):  46%|####5     | 3589/7804 [00:00<00:00, 11329.82 examples/s]
Map (num_proc=8):  68%|######7   | 5295/7804 [00:00<00:00, 12790.42 examples/s]
Map (num_proc=8):  86%|########6 | 6731/7804 [00:00<00:00, 12720.04 examples/s]
Map (num_proc=8): 100%|##########| 7804/7804 [00:00<00:00, 9158.12 examples/s] INFO 2025-05-20 11:34:34,813 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_1_1_1__OpenShift_Virtualization_supported_cluster_version_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 222 examples [00:00, 43166.08 examples/s]INFO 2025-05-20 11:34:34,835 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:34,835 instructlab.sdg.datamixing:161: Dataset loaded with 222 samples


Map (num_proc=8):   0%|          | 0/222 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 28/222 [00:00<00:00, 216.77 examples/s]
Map (num_proc=8): 100%|##########| 222/222 [00:00<00:00, 921.06 examples/s]INFO 2025-05-20 11:34:35,490 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_4_1__Creating_a_custom_namespace_for_templates_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 251 examples [00:00, 56210.71 examples/s]INFO 2025-05-20 11:34:35,514 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:35,514 instructlab.sdg.datamixing:161: Dataset loaded with 251 samples


Map (num_proc=8):   0%|          | 0/251 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 32/251 [00:00<00:00, 257.82 examples/s]
Map (num_proc=8): 100%|##########| 251/251 [00:00<00:00, 1053.10 examples/s]INFO 2025-05-20 11:34:36,161 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_5__Technology_Preview_features_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 115 examples [00:00, 33547.43 examples/s]INFO 2025-05-20 11:34:36,181 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:36,181 instructlab.sdg.datamixing:161: Dataset loaded with 115 samples


Map (num_proc=8):   0%|          | 0/115 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#3        | 15/115 [00:00<00:00, 149.25 examples/s]
Map (num_proc=8): 100%|##########| 115/115 [00:00<00:00, 499.74 examples/s]INFO 2025-05-20 11:34:36,822 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_10_1__Installing_QEMU_guest_agent_on_a_Linux_virtual_machine_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1103 examples [00:00, 94176.32 examples/s]INFO 2025-05-20 11:34:36,852 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:36,852 instructlab.sdg.datamixing:161: Dataset loaded with 1103 samples


Map (num_proc=8):   0%|          | 0/1103 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 138/1103 [00:00<00:01, 800.48 examples/s]
Map (num_proc=8): 100%|##########| 1103/1103 [00:00<00:00, 3466.61 examples/s]INFO 2025-05-20 11:34:37,585 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_9_2__Deleting_nodes_from_a_bare_metal_cluster_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1057 examples [00:00, 102615.02 examples/s]INFO 2025-05-20 11:34:37,619 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:37,619 instructlab.sdg.datamixing:161: Dataset loaded with 1057 samples


Map (num_proc=8):   0%|          | 0/1057 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 132/1057 [00:00<00:01, 720.78 examples/s]
Map (num_proc=8): 100%|##########| 1057/1057 [00:00<00:00, 3421.96 examples/s]INFO 2025-05-20 11:34:38,343 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_11_2__Live_migration_limits_and_timeouts_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 563 examples [00:00, 98605.03 examples/s]INFO 2025-05-20 11:34:38,369 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:38,369 instructlab.sdg.datamixing:161: Dataset loaded with 563 samples


Map (num_proc=8):   0%|          | 0/563 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 71/563 [00:00<00:00, 494.01 examples/s]
Map (num_proc=8): 100%|##########| 563/563 [00:00<00:00, 2140.46 examples/s]INFO 2025-05-20 11:34:39,044 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1_7__Additional_resources_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 3692 examples [00:00, 98334.17 examples/s]INFO 2025-05-20 11:34:39,099 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:39,099 instructlab.sdg.datamixing:161: Dataset loaded with 3692 samples


Map (num_proc=8):   0%|          | 0/3692 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 206/3692 [00:00<00:03, 931.84 examples/s]
Map (num_proc=8):  60%|#####9    | 2198/3692 [00:00<00:00, 8261.28 examples/s]
Map (num_proc=8): 100%|##########| 3692/3692 [00:00<00:00, 10340.39 examples/s]
Map (num_proc=8): 100%|##########| 3692/3692 [00:00<00:00, 7106.22 examples/s] INFO 2025-05-20 11:34:40,046 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_14_1_1__Installing_the_OADP_Operator_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 975 examples [00:00, 95222.99 examples/s]INFO 2025-05-20 11:34:40,073 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:40,074 instructlab.sdg.datamixing:161: Dataset loaded with 975 samples


Map (num_proc=8):   0%|          | 0/975 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 122/975 [00:00<00:01, 719.95 examples/s]
Map (num_proc=8): 100%|##########| 975/975 [00:00<00:00, 3259.63 examples/s]INFO 2025-05-20 11:34:40,787 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_4_2__Subscribing_to_the_OpenShift_Virtualization_catalog_by_using_the_CLI_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1280 examples [00:00, 100177.44 examples/s]INFO 2025-05-20 11:34:40,818 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:40,818 instructlab.sdg.datamixing:161: Dataset loaded with 1280 samples


Map (num_proc=8):   0%|          | 0/1280 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 160/1280 [00:00<00:01, 833.01 examples/s]
Map (num_proc=8): 100%|##########| 1280/1280 [00:00<00:00, 3928.35 examples/s]INFO 2025-05-20 11:34:41,558 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_3__New_and_changed_features_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 427 examples [00:00, 99981.46 examples/s]INFO 2025-05-20 11:34:41,579 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:41,579 instructlab.sdg.datamixing:161: Dataset loaded with 427 samples


Map (num_proc=8):   0%|          | 0/427 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 54/427 [00:00<00:00, 418.15 examples/s]
Map (num_proc=8): 100%|##########| 427/427 [00:00<00:00, 1720.97 examples/s]INFO 2025-05-20 11:34:42,238 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_6_4__Unpausing_a_virtual_machine_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 273 examples [00:00, 63046.19 examples/s]INFO 2025-05-20 11:34:42,264 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:42,264 instructlab.sdg.datamixing:161: Dataset loaded with 273 samples


Map (num_proc=8):   0%|          | 0/273 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 35/273 [00:00<00:00, 278.65 examples/s]
Map (num_proc=8): 100%|##########| 273/273 [00:00<00:00, 1136.78 examples/s]INFO 2025-05-20 11:34:42,913 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1_1__Using_a_Quick_Start_to_create_a_virtual_machine_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 3880 examples [00:00, 94621.75 examples/s]INFO 2025-05-20 11:34:42,973 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:42,973 instructlab.sdg.datamixing:161: Dataset loaded with 3880 samples


Map (num_proc=8):   0%|          | 0/3880 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|4         | 185/3880 [00:00<00:04, 890.16 examples/s]
Map (num_proc=8):  51%|#####     | 1971/3880 [00:00<00:00, 7798.79 examples/s]
Map (num_proc=8):  91%|######### | 3517/3880 [00:00<00:00, 10567.84 examples/s]
Map (num_proc=8): 100%|##########| 3880/3880 [00:00<00:00, 7219.44 examples/s] INFO 2025-05-20 11:34:43,934 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_6__About_the_ssp_operator_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 237 examples [00:00, 69160.93 examples/s]INFO 2025-05-20 11:34:43,955 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:43,955 instructlab.sdg.datamixing:161: Dataset loaded with 237 samples


Map (num_proc=8):   0%|          | 0/237 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 30/237 [00:00<00:00, 266.06 examples/s]
Map (num_proc=8): 100%|##########| 237/237 [00:00<00:00, 1014.93 examples/s]INFO 2025-05-20 11:34:44,595 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_2_3__Editing_a_virtual_machine_YAML_configuration_using_the_CLI_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2019 examples [00:00, 112507.14 examples/s]INFO 2025-05-20 11:34:44,631 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:44,631 instructlab.sdg.datamixing:161: Dataset loaded with 2019 samples


Map (num_proc=8):   0%|          | 0/2019 [00:00<?, ? examples/s]
Map (num_proc=8):   9%|8         | 179/2019 [00:00<00:01, 927.66 examples/s]
Map (num_proc=8): 100%|##########| 2019/2019 [00:00<00:00, 7967.92 examples/s]
Map (num_proc=8): 100%|##########| 2019/2019 [00:00<00:00, 5194.99 examples/s]INFO 2025-05-20 11:34:45,440 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_7_3_1__Viewing_the_SCC_and_RBAC_definitions_for_the_kubevirt_controller_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 445 examples [00:00, 80419.89 examples/s]INFO 2025-05-20 11:34:45,464 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:45,464 instructlab.sdg.datamixing:161: Dataset loaded with 445 samples


Map (num_proc=8):   0%|          | 0/445 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 56/445 [00:00<00:00, 404.53 examples/s]
Map (num_proc=8): 100%|##########| 445/445 [00:00<00:00, 1732.76 examples/s]INFO 2025-05-20 11:34:46,130 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_2_1__OpenShift_Virtualization_supported_cluster_version_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 224 examples [00:00, 55188.21 examples/s]INFO 2025-05-20 11:34:46,152 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:46,152 instructlab.sdg.datamixing:161: Dataset loaded with 224 samples


Map (num_proc=8):   0%|          | 0/224 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 28/224 [00:00<00:00, 255.65 examples/s]
Map (num_proc=8): 100%|##########| 224/224 [00:00<00:00, 953.54 examples/s]INFO 2025-05-20 11:34:46,794 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_12_3_3__Configuring_obsolete_CPU_models_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 213 examples [00:00, 52438.03 examples/s]INFO 2025-05-20 11:34:46,827 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:46,827 instructlab.sdg.datamixing:161: Dataset loaded with 213 samples


Map (num_proc=8):   0%|          | 0/213 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 27/213 [00:00<00:00, 222.28 examples/s]
Map (num_proc=8): 100%|##########| 213/213 [00:00<00:00, 914.92 examples/s]INFO 2025-05-20 11:34:47,464 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_7__Using_the_OpenShift_Container_Platform_dashboard_to_get_cluster_information_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 183 examples [00:00, 43865.45 examples/s]INFO 2025-05-20 11:34:47,485 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:47,485 instructlab.sdg.datamixing:161: Dataset loaded with 183 samples


Map (num_proc=8):   0%|          | 0/183 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 23/183 [00:00<00:00, 189.45 examples/s]
Map (num_proc=8): 100%|##########| 183/183 [00:00<00:00, 771.67 examples/s]INFO 2025-05-20 11:34:48,127 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_9_1__Prerequisites_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1203 examples [00:00, 92981.75 examples/s]INFO 2025-05-20 11:34:48,313 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:48,313 instructlab.sdg.datamixing:161: Dataset loaded with 1203 samples


Map (num_proc=8):   0%|          | 0/1203 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 151/1203 [00:00<00:01, 765.41 examples/s]
Map (num_proc=8): 100%|##########| 1203/1203 [00:00<00:00, 3858.23 examples/s]INFO 2025-05-20 11:34:49,041 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_20_19__Re_using_persistent_volumes_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 14979 examples [00:00, 125011.65 examples/s]
Generating train split: 14979 examples [00:00, 123802.12 examples/s]INFO 2025-05-20 11:34:49,179 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:49,179 instructlab.sdg.datamixing:161: Dataset loaded with 14979 samples


Map (num_proc=8):   0%|          | 0/14979 [00:00<?, ? examples/s]
Map (num_proc=8):   1%|1         | 220/14979 [00:00<00:13, 1080.82 examples/s]
Map (num_proc=8):  13%|#2        | 1880/14979 [00:00<00:01, 7464.79 examples/s]
Map (num_proc=8):  27%|##7       | 4106/14979 [00:00<00:00, 12764.47 examples/s]
Map (num_proc=8):  38%|###8      | 5739/14979 [00:00<00:00, 13433.21 examples/s]
Map (num_proc=8):  49%|####9     | 7351/14979 [00:00<00:00, 13188.60 examples/s]
Map (num_proc=8):  59%|#####8    | 8834/14979 [00:00<00:00, 10670.73 examples/s]
Map (num_proc=8):  69%|######9   | 10396/14979 [00:00<00:00, 11874.06 examples/s]
Map (num_proc=8):  82%|########1 | 12245/14979 [00:01<00:00, 13223.08 examples/s]
Map (num_proc=8):  93%|#########2| 13876/14979 [00:01<00:00, 13609.52 examples/s]
Map (num_proc=8): 100%|##########| 14979/14979 [00:01<00:00, 10901.37 examples/s]INFO 2025-05-20 11:34:51,014 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_5__Enabling_the_virtctl_client_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1382 examples [00:00, 133128.04 examples/s]INFO 2025-05-20 11:34:51,043 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:51,043 instructlab.sdg.datamixing:161: Dataset loaded with 1382 samples


Map (num_proc=8):   0%|          | 0/1382 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 173/1382 [00:00<00:01, 912.66 examples/s]
Map (num_proc=8): 100%|##########| 1382/1382 [00:00<00:00, 4235.51 examples/s]INFO 2025-05-20 11:34:51,789 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_1_1__Prerequisites_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1126 examples [00:00, 124532.92 examples/s]INFO 2025-05-20 11:34:51,815 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:51,815 instructlab.sdg.datamixing:161: Dataset loaded with 1126 samples


Map (num_proc=8):   0%|          | 0/1126 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 141/1126 [00:00<00:01, 750.27 examples/s]
Map (num_proc=8): 100%|##########| 1126/1126 [00:00<00:00, 3702.01 examples/s]INFO 2025-05-20 11:34:52,539 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_1__Preparing_your_cluster_for_OpenShift_Virtualization_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2725 examples [00:00, 115876.50 examples/s]INFO 2025-05-20 11:34:52,580 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:52,580 instructlab.sdg.datamixing:161: Dataset loaded with 2725 samples


Map (num_proc=8):   0%|          | 0/2725 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 161/2725 [00:00<00:03, 847.64 examples/s]
Map (num_proc=8):  76%|#######5  | 2068/2725 [00:00<00:00, 8621.11 examples/s]
Map (num_proc=8): 100%|##########| 2725/2725 [00:00<00:00, 6034.92 examples/s]INFO 2025-05-20 11:34:53,457 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_14_1__Installing_and_configuring_OADP_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 923 examples [00:00, 92994.06 examples/s]INFO 2025-05-20 11:34:53,484 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:53,484 instructlab.sdg.datamixing:161: Dataset loaded with 923 samples


Map (num_proc=8):   0%|          | 0/923 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 116/923 [00:00<00:01, 677.63 examples/s]
Map (num_proc=8): 100%|##########| 923/923 [00:00<00:00, 3030.05 examples/s]INFO 2025-05-20 11:34:54,210 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_9__OpenShift_Container_Platform_cluster_monitoring__logging__and_Telemetry_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1264 examples [00:00, 116408.67 examples/s]INFO 2025-05-20 11:34:54,239 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:54,239 instructlab.sdg.datamixing:161: Dataset loaded with 1264 samples


Map (num_proc=8):   0%|          | 0/1264 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 158/1264 [00:00<00:01, 804.43 examples/s]
Map (num_proc=8): 100%|##########| 1264/1264 [00:00<00:00, 3904.07 examples/s]INFO 2025-05-20 11:34:54,981 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_3_1__Installing_the_OpenShift_Virtualization_Operator_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 613 examples [00:00, 73725.65 examples/s]INFO 2025-05-20 11:34:55,007 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:55,007 instructlab.sdg.datamixing:161: Dataset loaded with 613 samples


Map (num_proc=8):   0%|          | 0/613 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 77/613 [00:00<00:01, 502.06 examples/s]
Map (num_proc=8): 100%|##########| 613/613 [00:00<00:00, 2248.30 examples/s]INFO 2025-05-20 11:34:55,695 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_19_1__Configuring_the_virtual_machine_for_the_default_pod_network_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 8945 examples [00:00, 107879.00 examples/s]INFO 2025-05-20 11:34:55,799 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:55,799 instructlab.sdg.datamixing:161: Dataset loaded with 8945 samples


Map (num_proc=8):   0%|          | 0/8945 [00:00<?, ? examples/s]
Map (num_proc=8):   2%|1         | 148/8945 [00:00<00:11, 782.99 examples/s]
Map (num_proc=8):  24%|##3       | 2140/8945 [00:00<00:00, 8963.52 examples/s]
Map (num_proc=8):  41%|####1     | 3692/8945 [00:00<00:00, 11418.44 examples/s]
Map (num_proc=8):  59%|#####9    | 5300/8945 [00:00<00:00, 12785.32 examples/s]
Map (num_proc=8):  77%|#######6  | 6862/8945 [00:00<00:00, 13200.05 examples/s]
Map (num_proc=8):  93%|#########2| 8285/8945 [00:00<00:00, 10092.49 examples/s]
Map (num_proc=8): 100%|##########| 8945/8945 [00:00<00:00, 9286.90 examples/s] INFO 2025-05-20 11:34:57,205 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_11_7__Configuring_virtual_machine_eviction_strategy_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 426 examples [00:00, 81864.45 examples/s]INFO 2025-05-20 11:34:57,227 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:57,227 instructlab.sdg.datamixing:161: Dataset loaded with 426 samples


Map (num_proc=8):   0%|          | 0/426 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 54/426 [00:00<00:00, 383.20 examples/s]
Map (num_proc=8): 100%|##########| 426/426 [00:00<00:00, 1665.01 examples/s]INFO 2025-05-20 11:34:57,899 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_14_2_1__Additional_resources_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 125 examples [00:00, 37762.03 examples/s]INFO 2025-05-20 11:34:57,920 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:57,920 instructlab.sdg.datamixing:161: Dataset loaded with 125 samples


Map (num_proc=8):   0%|          | 0/125 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 16/125 [00:00<00:00, 136.21 examples/s]
Map (num_proc=8): 100%|##########| 125/125 [00:00<00:00, 541.17 examples/s]INFO 2025-05-20 11:34:58,568 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_10__Running_cluster_checkups_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1397 examples [00:00, 94704.19 examples/s]INFO 2025-05-20 11:34:58,600 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:58,600 instructlab.sdg.datamixing:161: Dataset loaded with 1397 samples


Map (num_proc=8):   0%|          | 0/1397 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 175/1397 [00:00<00:01, 1005.20 examples/s]
Map (num_proc=8): 100%|##########| 1397/1397 [00:00<00:00, 4055.19 examples/s]INFO 2025-05-20 11:34:59,364 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_12_2__Configuring_a_virtual_machine_with_the_node_exporter_service_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 900 examples [00:00, 84320.80 examples/s]INFO 2025-05-20 11:34:59,393 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:34:59,393 instructlab.sdg.datamixing:161: Dataset loaded with 900 samples


Map (num_proc=8):   0%|          | 0/900 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 113/900 [00:00<00:01, 649.56 examples/s]
Map (num_proc=8): 100%|##########| 900/900 [00:00<00:00, 2978.07 examples/s]INFO 2025-05-20 11:35:00,108 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_7_3__Additional_OpenShift_Container_Platform_security_context_constraints_and_Linux_capabilities_for_the_kubevirt_controller_service_account_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 383 examples [00:00, 52712.66 examples/s]INFO 2025-05-20 11:35:00,133 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:00,133 instructlab.sdg.datamixing:161: Dataset loaded with 383 samples


Map (num_proc=8):   0%|          | 0/383 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 48/383 [00:00<00:00, 335.40 examples/s]
Map (num_proc=8): 100%|##########| 383/383 [00:00<00:00, 1500.23 examples/s]INFO 2025-05-20 11:35:00,799 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_4_7__Known_issues_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 306 examples [00:00, 49458.84 examples/s]INFO 2025-05-20 11:35:00,845 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:00,845 instructlab.sdg.datamixing:161: Dataset loaded with 306 samples


Map (num_proc=8):   0%|          | 0/306 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 39/306 [00:00<00:00, 283.42 examples/s]
Map (num_proc=8): 100%|##########| 306/306 [00:00<00:00, 1232.23 examples/s]INFO 2025-05-20 11:35:01,505 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_11_4__Migrating_a_virtual_machine_over_a_dedicated_additional_network_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 207 examples [00:00, 41951.15 examples/s]INFO 2025-05-20 11:35:01,535 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:01,535 instructlab.sdg.datamixing:161: Dataset loaded with 207 samples


Map (num_proc=8):   0%|          | 0/207 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 26/207 [00:00<00:00, 225.31 examples/s]
Map (num_proc=8): 100%|##########| 207/207 [00:00<00:00, 873.03 examples/s]INFO 2025-05-20 11:35:02,181 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_1_3__Creating_a_virtual_machine_template_in_the_web_console_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 551 examples [00:00, 63649.83 examples/s]INFO 2025-05-20 11:35:02,207 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:02,207 instructlab.sdg.datamixing:161: Dataset loaded with 551 samples


Map (num_proc=8):   0%|          | 0/551 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 69/551 [00:00<00:01, 443.38 examples/s]
Map (num_proc=8): 100%|##########| 551/551 [00:00<00:00, 2066.26 examples/s]INFO 2025-05-20 11:35:02,886 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_1_1__Hardware_and_operating_system_requirements_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2126 examples [00:00, 103179.60 examples/s]INFO 2025-05-20 11:35:02,925 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:02,925 instructlab.sdg.datamixing:161: Dataset loaded with 2126 samples


Map (num_proc=8):   0%|          | 0/2126 [00:00<?, ? examples/s]
Map (num_proc=8):   8%|7         | 169/2126 [00:00<00:02, 815.74 examples/s]
Map (num_proc=8):  86%|########5 | 1828/2126 [00:00<00:00, 7228.80 examples/s]
Map (num_proc=8): 100%|##########| 2126/2126 [00:00<00:00, 5248.15 examples/s]INFO 2025-05-20 11:35:03,752 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_13_1__About_VirtIO_drivers_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1189 examples [00:00, 100042.68 examples/s]INFO 2025-05-20 11:35:03,781 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:03,781 instructlab.sdg.datamixing:161: Dataset loaded with 1189 samples


Map (num_proc=8):   0%|          | 0/1189 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 149/1189 [00:00<00:01, 868.81 examples/s]
Map (num_proc=8): 100%|##########| 1189/1189 [00:00<00:00, 3715.50 examples/s]INFO 2025-05-20 11:35:04,518 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_5_1__Deleting_a_virtual_machine_template_in_the_web_console_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 584 examples [00:00, 113982.02 examples/s]INFO 2025-05-20 11:35:04,541 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:04,541 instructlab.sdg.datamixing:161: Dataset loaded with 584 samples


Map (num_proc=8):   0%|          | 0/584 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 73/584 [00:00<00:01, 479.36 examples/s]
Map (num_proc=8): 100%|##########| 584/584 [00:00<00:00, 2191.93 examples/s]INFO 2025-05-20 11:35:05,219 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_7_1__Accessing_virtual_machine_consoles_in_the_OpenShift_Container_Platform_web_console_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 3151 examples [00:00, 126909.73 examples/s]INFO 2025-05-20 11:35:05,282 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:05,282 instructlab.sdg.datamixing:161: Dataset loaded with 3151 samples


Map (num_proc=8):   0%|          | 0/3151 [00:00<?, ? examples/s]
Map (num_proc=8):   7%|6         | 206/3151 [00:00<00:02, 1040.82 examples/s]
Map (num_proc=8):  60%|######    | 1900/3151 [00:00<00:00, 7654.78 examples/s]
Map (num_proc=8): 100%|##########| 3151/3151 [00:00<00:00, 6622.34 examples/s]INFO 2025-05-20 11:35:06,183 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_15__Using_virtual_Trusted_Platform_Module_devices_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 414 examples [00:00, 85215.78 examples/s]INFO 2025-05-20 11:35:06,206 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:06,206 instructlab.sdg.datamixing:161: Dataset loaded with 414 samples


Map (num_proc=8):   0%|          | 0/414 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 52/414 [00:00<00:00, 387.78 examples/s]
Map (num_proc=8): 100%|##########| 414/414 [00:00<00:00, 1624.34 examples/s]INFO 2025-05-20 11:35:06,874 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_11_2_2__Cluster_wide_live_migration_limits_and_timeouts_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 452 examples [00:00, 88354.64 examples/s]INFO 2025-05-20 11:35:06,897 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:06,897 instructlab.sdg.datamixing:161: Dataset loaded with 452 samples


Map (num_proc=8):   0%|          | 0/452 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 57/452 [00:00<00:00, 398.97 examples/s]
Map (num_proc=8): 100%|##########| 452/452 [00:00<00:00, 1775.17 examples/s]INFO 2025-05-20 11:35:07,562 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_8_3__Additional_resources_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 676 examples [00:00, 107493.25 examples/s]INFO 2025-05-20 11:35:07,586 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:07,586 instructlab.sdg.datamixing:161: Dataset loaded with 676 samples


Map (num_proc=8):   0%|          | 0/676 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 85/676 [00:00<00:01, 536.74 examples/s]
Map (num_proc=8): 100%|##########| 676/676 [00:00<00:00, 2513.28 examples/s]INFO 2025-05-20 11:35:08,265 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_14_2__About_VirtIO_drivers_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1211 examples [00:00, 100759.81 examples/s]INFO 2025-05-20 11:35:08,295 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:08,295 instructlab.sdg.datamixing:161: Dataset loaded with 1211 samples


Map (num_proc=8):   0%|          | 0/1211 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 152/1211 [00:00<00:01, 797.30 examples/s]
Map (num_proc=8): 100%|##########| 1211/1211 [00:00<00:00, 3772.07 examples/s]INFO 2025-05-20 11:35:09,032 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_5_4__Editing_a_standalone_virtual_machine_instance_using_the_web_console_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1469 examples [00:00, 127682.21 examples/s]INFO 2025-05-20 11:35:09,065 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:09,065 instructlab.sdg.datamixing:161: Dataset loaded with 1469 samples


Map (num_proc=8):   0%|          | 0/1469 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 184/1469 [00:00<00:01, 984.41 examples/s]
Map (num_proc=8): 100%|##########| 1469/1469 [00:00<00:00, 4407.17 examples/s]INFO 2025-05-20 11:35:09,814 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_6_1__Starting_a_virtual_machine_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 237 examples [00:00, 46864.84 examples/s]INFO 2025-05-20 11:35:09,837 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:09,837 instructlab.sdg.datamixing:161: Dataset loaded with 237 samples


Map (num_proc=8):   0%|          | 0/237 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 30/237 [00:00<00:00, 236.64 examples/s]
Map (num_proc=8): 100%|##########| 237/237 [00:00<00:00, 1000.89 examples/s]INFO 2025-05-20 11:35:10,481 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_13_5__Removing_the_VirtIO_container_disk_from_a_virtual_machine_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1156 examples [00:00, 94695.82 examples/s]INFO 2025-05-20 11:35:10,515 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:10,515 instructlab.sdg.datamixing:161: Dataset loaded with 1156 samples


Map (num_proc=8):   0%|          | 0/1156 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 145/1156 [00:00<00:01, 707.34 examples/s]
Map (num_proc=8): 100%|##########| 1156/1156 [00:00<00:00, 3630.81 examples/s]INFO 2025-05-20 11:35:11,248 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_13_4__Installing_VirtIO_drivers_on_an_existing_Windows_virtual_machine_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1304 examples [00:00, 91860.47 examples/s]INFO 2025-05-20 11:35:11,281 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:11,281 instructlab.sdg.datamixing:161: Dataset loaded with 1304 samples


Map (num_proc=8):   0%|          | 0/1304 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 163/1304 [00:00<00:01, 841.39 examples/s]
Map (num_proc=8): 100%|##########| 1304/1304 [00:00<00:00, 3988.85 examples/s]INFO 2025-05-20 11:35:12,024 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_1__How_OpenShift_Virtualization_architecture_works_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 300 examples [00:00, 48981.71 examples/s]INFO 2025-05-20 11:35:12,065 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:12,065 instructlab.sdg.datamixing:161: Dataset loaded with 300 samples


Map (num_proc=8):   0%|          | 0/300 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 38/300 [00:00<00:00, 277.56 examples/s]
Map (num_proc=8): 100%|##########| 300/300 [00:00<00:00, 1195.58 examples/s]INFO 2025-05-20 11:35:12,725 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_7_1__About_the_OpenShift_Container_Platform_dashboards_page_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 176 examples [00:00, 40369.55 examples/s]INFO 2025-05-20 11:35:12,751 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:12,751 instructlab.sdg.datamixing:161: Dataset loaded with 176 samples


Map (num_proc=8):   0%|          | 0/176 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 22/176 [00:00<00:00, 178.64 examples/s]
Map (num_proc=8): 100%|##########| 176/176 [00:00<00:00, 743.06 examples/s]INFO 2025-05-20 11:35:13,397 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_4__Deploying_a_virtual_machine_template_to_a_custom_namespace_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 203 examples [00:00, 50179.38 examples/s]INFO 2025-05-20 11:35:13,423 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:13,423 instructlab.sdg.datamixing:161: Dataset loaded with 203 samples


Map (num_proc=8):   0%|          | 0/203 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 26/203 [00:00<00:00, 209.17 examples/s]
Map (num_proc=8): 100%|##########| 203/203 [00:00<00:00, 859.72 examples/s]INFO 2025-05-20 11:35:14,070 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_13_4__Additional_resources_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 3320 examples [00:00, 85553.86 examples/s]INFO 2025-05-20 11:35:14,127 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:14,127 instructlab.sdg.datamixing:161: Dataset loaded with 3320 samples


Map (num_proc=8):   0%|          | 0/3320 [00:00<?, ? examples/s]
Map (num_proc=8):  10%|#         | 336/3320 [00:00<00:01, 1816.69 examples/s]
Map (num_proc=8):  50%|####9     | 1654/3320 [00:00<00:00, 6557.54 examples/s]
Map (num_proc=8):  95%|#########5| 3156/3320 [00:00<00:00, 9505.22 examples/s]
Map (num_proc=8): 100%|##########| 3320/3320 [00:00<00:00, 6606.49 examples/s]INFO 2025-05-20 11:35:15,057 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_1_4__Reviewing_top_consumers_for_projects__virtual_machines__and_nodes_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1101 examples [00:00, 119049.46 examples/s]INFO 2025-05-20 11:35:15,085 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:15,085 instructlab.sdg.datamixing:161: Dataset loaded with 1101 samples


Map (num_proc=8):   0%|          | 0/1101 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 138/1101 [00:00<00:01, 769.36 examples/s]
Map (num_proc=8): 100%|##########| 1101/1101 [00:00<00:00, 3586.49 examples/s]INFO 2025-05-20 11:35:15,807 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_7__Accessing_virtual_machine_consoles_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 3032 examples [00:00, 123980.05 examples/s]INFO 2025-05-20 11:35:15,848 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:15,848 instructlab.sdg.datamixing:161: Dataset loaded with 3032 samples


Map (num_proc=8):   0%|          | 0/3032 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|4         | 149/3032 [00:00<00:03, 777.59 examples/s]
Map (num_proc=8):  67%|######6   | 2031/3032 [00:00<00:00, 8478.93 examples/s]
Map (num_proc=8): 100%|##########| 3032/3032 [00:00<00:00, 6550.07 examples/s]INFO 2025-05-20 11:35:16,734 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_5__About_the_hostpath_provisioner_operator_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 205 examples [00:00, 53223.91 examples/s]INFO 2025-05-20 11:35:16,754 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:16,754 instructlab.sdg.datamixing:161: Dataset loaded with 205 samples


Map (num_proc=8):   0%|          | 0/205 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 26/205 [00:00<00:00, 204.80 examples/s]
Map (num_proc=8): 100%|##########| 205/205 [00:00<00:00, 867.56 examples/s]INFO 2025-05-20 11:35:17,407 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1_4__Using_the_CLI_to_create_a_virtual_machine_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 3383 examples [00:00, 90459.72 examples/s]INFO 2025-05-20 11:35:17,462 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:17,462 instructlab.sdg.datamixing:161: Dataset loaded with 3383 samples


Map (num_proc=8):   0%|          | 0/3383 [00:00<?, ? examples/s]
Map (num_proc=8):   5%|5         | 177/3383 [00:00<00:03, 915.67 examples/s]
Map (num_proc=8):  63%|######2   | 2124/3383 [00:00<00:00, 8557.42 examples/s]
Map (num_proc=8):  98%|#########7| 3311/3383 [00:00<00:00, 9305.74 examples/s]
Map (num_proc=8): 100%|##########| 3383/3383 [00:00<00:00, 6704.93 examples/s]INFO 2025-05-20 11:35:18,393 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_14_2__Collecting_data_about_virtual_machines_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 529 examples [00:00, 86303.89 examples/s]INFO 2025-05-20 11:35:18,418 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:18,418 instructlab.sdg.datamixing:161: Dataset loaded with 529 samples


Map (num_proc=8):   0%|          | 0/529 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 66/529 [00:00<00:01, 438.70 examples/s]
Map (num_proc=8): 100%|##########| 529/529 [00:00<00:00, 2012.27 examples/s]INFO 2025-05-20 11:35:19,101 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_14_6__Removing_the_VirtIO_container_disk_from_a_virtual_machine_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1286 examples [00:00, 102410.81 examples/s]INFO 2025-05-20 11:35:19,131 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:19,131 instructlab.sdg.datamixing:161: Dataset loaded with 1286 samples


Map (num_proc=8):   0%|          | 0/1286 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 159/1286 [00:00<00:01, 931.29 examples/s]
Map (num_proc=8): 100%|##########| 1286/1286 [00:00<00:00, 3941.45 examples/s]INFO 2025-05-20 11:35:19,876 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_14_1__Collecting_data_about_your_environment_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 514 examples [00:00, 82959.64 examples/s]INFO 2025-05-20 11:35:19,901 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:19,901 instructlab.sdg.datamixing:161: Dataset loaded with 514 samples


Map (num_proc=8):   0%|          | 0/514 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 65/514 [00:00<00:00, 468.13 examples/s]
Map (num_proc=8): 100%|##########| 514/514 [00:00<00:00, 1900.72 examples/s]INFO 2025-05-20 11:35:20,585 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_3_1__Before_you_begin_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 84 examples [00:00, 36707.81 examples/s]INFO 2025-05-20 11:35:20,604 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:20,604 instructlab.sdg.datamixing:161: Dataset loaded with 84 samples


Map (num_proc=8):   0%|          | 0/84 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#3        | 11/84 [00:00<00:00, 96.93 examples/s]
Map (num_proc=8): 100%|##########| 84/84 [00:00<00:00, 371.29 examples/s]INFO 2025-05-20 11:35:21,239 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_4_3__Deploying_the_OpenShift_Virtualization_Operator_by_using_the_CLI_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1473 examples [00:00, 96749.19 examples/s]INFO 2025-05-20 11:35:21,272 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:21,273 instructlab.sdg.datamixing:161: Dataset loaded with 1473 samples


Map (num_proc=8):   0%|          | 0/1473 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 183/1473 [00:00<00:01, 1042.51 examples/s]
Map (num_proc=8): 100%|##########| 1473/1473 [00:00<00:00, 4206.85 examples/s]INFO 2025-05-20 11:35:22,047 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_8_4__Creating_a_container_using_virtctl_guestfs_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 224 examples [00:00, 55914.07 examples/s]INFO 2025-05-20 11:35:22,070 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:22,070 instructlab.sdg.datamixing:161: Dataset loaded with 224 samples


Map (num_proc=8):   0%|          | 0/224 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 28/224 [00:00<00:00, 227.37 examples/s]
Map (num_proc=8): 100%|##########| 224/224 [00:00<00:00, 947.78 examples/s]INFO 2025-05-20 11:35:22,717 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_1_6__Snapshots_and_cloning_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1549 examples [00:00, 106026.35 examples/s]INFO 2025-05-20 11:35:22,751 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:22,751 instructlab.sdg.datamixing:161: Dataset loaded with 1549 samples


Map (num_proc=8):   0%|          | 0/1549 [00:00<?, ? examples/s]
Map (num_proc=8):   9%|9         | 140/1549 [00:00<00:01, 754.21 examples/s]
Map (num_proc=8): 100%|##########| 1549/1549 [00:00<00:00, 4406.35 examples/s]INFO 2025-05-20 11:35:23,519 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_2_1__Editing_a_virtual_machine_in_the_web_console_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2263 examples [00:00, 124297.24 examples/s]INFO 2025-05-20 11:35:23,560 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:23,560 instructlab.sdg.datamixing:161: Dataset loaded with 2263 samples


Map (num_proc=8):   0%|          | 0/2263 [00:00<?, ? examples/s]
Map (num_proc=8):   8%|8         | 190/2263 [00:00<00:02, 989.59 examples/s]
Map (num_proc=8):  81%|########1 | 1843/2263 [00:00<00:00, 7241.63 examples/s]
Map (num_proc=8): 100%|##########| 2263/2263 [00:00<00:00, 5662.96 examples/s]INFO 2025-05-20 11:35:24,380 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_9_1__About_OpenShift_Container_Platform_monitoring_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1656 examples [00:00, 119036.29 examples/s]INFO 2025-05-20 11:35:24,416 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:24,416 instructlab.sdg.datamixing:161: Dataset loaded with 1656 samples


Map (num_proc=8):   0%|          | 0/1656 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 207/1656 [00:00<00:01, 1058.29 examples/s]
Map (num_proc=8): 100%|##########| 1656/1656 [00:00<00:00, 4690.55 examples/s]INFO 2025-05-20 11:35:25,188 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_7_1__About_workload_security_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 298 examples [00:00, 67173.78 examples/s]INFO 2025-05-20 11:35:25,210 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:25,210 instructlab.sdg.datamixing:161: Dataset loaded with 298 samples


Map (num_proc=8):   0%|          | 0/298 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 38/298 [00:00<00:00, 283.78 examples/s]
Map (num_proc=8): 100%|##########| 298/298 [00:00<00:00, 1228.93 examples/s]INFO 2025-05-20 11:35:25,862 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_10_1_5__Additional_resources_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 471 examples [00:00, 63083.32 examples/s]INFO 2025-05-20 11:35:25,887 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:25,887 instructlab.sdg.datamixing:161: Dataset loaded with 471 samples


Map (num_proc=8):   0%|          | 0/471 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 59/471 [00:00<00:00, 449.53 examples/s]
Map (num_proc=8): 100%|##########| 471/471 [00:00<00:00, 1779.95 examples/s]INFO 2025-05-20 11:35:26,566 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_2_3__About_the_cdi_operator_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 287 examples [00:00, 68967.87 examples/s]INFO 2025-05-20 11:35:26,588 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:26,588 instructlab.sdg.datamixing:161: Dataset loaded with 287 samples


Map (num_proc=8):   0%|          | 0/287 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 36/287 [00:00<00:00, 274.65 examples/s]
Map (num_proc=8): 100%|##########| 287/287 [00:00<00:00, 1198.14 examples/s]INFO 2025-05-20 11:35:27,240 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1_5__Virtual_machine_storage_volume_types_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2655 examples [00:00, 101977.83 examples/s]INFO 2025-05-20 11:35:27,284 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:27,284 instructlab.sdg.datamixing:161: Dataset loaded with 2655 samples


Map (num_proc=8):   0%|          | 0/2655 [00:00<?, ? examples/s]
Map (num_proc=8):   6%|5         | 152/2655 [00:00<00:03, 757.76 examples/s]
Map (num_proc=8):  66%|######6   | 1755/2655 [00:00<00:00, 6881.53 examples/s]
Map (num_proc=8): 100%|##########| 2655/2655 [00:00<00:00, 5975.86 examples/s]INFO 2025-05-20 11:35:28,154 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_1__Creating_virtual_machines_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 3528 examples [00:00, 98903.22 examples/s]INFO 2025-05-20 11:35:28,208 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:28,208 instructlab.sdg.datamixing:161: Dataset loaded with 3528 samples


Map (num_proc=8):   0%|          | 0/3528 [00:00<?, ? examples/s]
Map (num_proc=8):  10%|#         | 362/3528 [00:00<00:01, 2091.43 examples/s]
Map (num_proc=8):  46%|####6     | 1640/3528 [00:00<00:00, 6909.61 examples/s]
Map (num_proc=8):  95%|#########5| 3356/3528 [00:00<00:00, 10860.41 examples/s]
Map (num_proc=8): 100%|##########| 3528/3528 [00:00<00:00, 6933.45 examples/s] INFO 2025-05-20 11:35:29,145 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_4_1__Deleting_a_virtual_machine_using_the_web_console_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 787 examples [00:00, 117503.82 examples/s]INFO 2025-05-20 11:35:29,170 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:29,170 instructlab.sdg.datamixing:161: Dataset loaded with 787 samples


Map (num_proc=8):   0%|          | 0/787 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 99/787 [00:00<00:01, 621.09 examples/s]
Map (num_proc=8): 100%|##########| 787/787 [00:00<00:00, 2803.48 examples/s]INFO 2025-05-20 11:35:29,863 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_12_1_1__About_node_maintenance_mode_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 258 examples [00:00, 49281.83 examples/s]INFO 2025-05-20 11:35:29,885 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:29,885 instructlab.sdg.datamixing:161: Dataset loaded with 258 samples


Map (num_proc=8):   0%|          | 0/258 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 33/258 [00:00<00:00, 261.49 examples/s]
Map (num_proc=8): 100%|##########| 258/258 [00:00<00:00, 1073.49 examples/s]INFO 2025-05-20 11:35:30,535 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_13_14_3__Using_the_must_gather_tool_for_OpenShift_Virtualization_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 568 examples [00:00, 93177.59 examples/s]INFO 2025-05-20 11:35:30,559 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:30,559 instructlab.sdg.datamixing:161: Dataset loaded with 568 samples


Map (num_proc=8):   0%|          | 0/568 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 71/568 [00:00<00:01, 461.75 examples/s]
Map (num_proc=8): 100%|##########| 568/568 [00:00<00:00, 2113.76 examples/s]INFO 2025-05-20 11:35:31,240 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_5_6__Uninstalling_OpenShift_Virtualization_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1972 examples [00:00, 130043.67 examples/s]INFO 2025-05-20 11:35:31,273 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:31,274 instructlab.sdg.datamixing:161: Dataset loaded with 1972 samples


Map (num_proc=8):   0%|          | 0/1972 [00:00<?, ? examples/s]
Map (num_proc=8):  10%|9         | 193/1972 [00:00<00:01, 1051.43 examples/s]
Map (num_proc=8):  93%|#########3| 1836/1972 [00:00<00:00, 7748.10 examples/s]
Map (num_proc=8): 100%|##########| 1972/1972 [00:00<00:00, 5253.15 examples/s]INFO 2025-05-20 11:35:32,068 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_14_1__Prerequisites_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1133 examples [00:00, 96154.47 examples/s]INFO 2025-05-20 11:35:32,102 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:32,102 instructlab.sdg.datamixing:161: Dataset loaded with 1133 samples


Map (num_proc=8):   0%|          | 0/1133 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 142/1133 [00:00<00:01, 788.93 examples/s]
Map (num_proc=8): 100%|##########| 1133/1133 [00:00<00:00, 3593.80 examples/s]INFO 2025-05-20 11:35:32,837 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_3_2__Editing_a_boot_order_list_in_the_web_console_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1200 examples [00:00, 113854.48 examples/s]INFO 2025-05-20 11:35:32,866 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:32,866 instructlab.sdg.datamixing:161: Dataset loaded with 1200 samples


Map (num_proc=8):   0%|          | 0/1200 [00:00<?, ? examples/s]
Map (num_proc=8):  12%|#2        | 150/1200 [00:00<00:01, 760.48 examples/s]
Map (num_proc=8): 100%|##########| 1200/1200 [00:00<00:00, 3807.98 examples/s]INFO 2025-05-20 11:35:33,594 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_9_16_5__Configuring_PXE_booting_for_virtual_machines_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 9209 examples [00:00, 107437.07 examples/s]INFO 2025-05-20 11:35:33,699 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:33,699 instructlab.sdg.datamixing:161: Dataset loaded with 9209 samples


Map (num_proc=8):   0%|          | 0/9209 [00:00<?, ? examples/s]
Map (num_proc=8):   2%|1         | 149/9209 [00:00<00:11, 821.09 examples/s]
Map (num_proc=8):  22%|##1       | 2023/9209 [00:00<00:00, 8704.24 examples/s]
Map (num_proc=8):  39%|###9      | 3620/9209 [00:00<00:00, 11452.88 examples/s]
Map (num_proc=8):  57%|#####6    | 5234/9209 [00:00<00:00, 12885.39 examples/s]
Map (num_proc=8):  76%|#######5  | 6997/9209 [00:00<00:00, 13902.87 examples/s]
Map (num_proc=8):  93%|#########2| 8523/9209 [00:00<00:00, 10574.21 examples/s]
Map (num_proc=8): 100%|##########| 9209/9209 [00:00<00:00, 9519.35 examples/s] INFO 2025-05-20 11:35:35,114 instructlab.sdg.datamixing:158: Loading dataset from /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/node_datasets_2025-05-19T13_31_09/knowledge_technology_cloud-computing_6_4_1__Monitoring_OpenShift_Virtualization_upgrade_status_p07.jsonl ...


Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 486 examples [00:00, 75155.10 examples/s]INFO 2025-05-20 11:35:35,138 instructlab.sdg.datamixing:160: Dataset columns: ['messages', 'metadata', 'id', 'unmask']
INFO 2025-05-20 11:35:35,138 instructlab.sdg.datamixing:161: Dataset loaded with 486 samples


Map (num_proc=8):   0%|          | 0/486 [00:00<?, ? examples/s]
Map (num_proc=8):  13%|#2        | 61/486 [00:00<00:01, 410.26 examples/s]
Map (num_proc=8): 100%|##########| 486/486 [00:00<00:00, 1849.19 examples/s]

Map (num_proc=8):   0%|          | 0/217951 [00:00<?, ? examples/s]
Map (num_proc=8):   0%|          | 62/217951 [00:00<12:48, 283.36 examples/s]
Map (num_proc=8):   0%|          | 959/217951 [00:00<00:58, 3709.49 examples/s]
Map (num_proc=8):   1%|          | 2013/217951 [00:00<00:36, 5916.57 examples/s]
Map (num_proc=8):   2%|1         | 3656/217951 [00:00<00:22, 9388.24 examples/s]
Map (num_proc=8):   3%|2         | 5478/217951 [00:00<00:18, 11452.24 examples/s]
Map (num_proc=8):   3%|3         | 6781/217951 [00:00<00:18, 11517.10 examples/s]
Map (num_proc=8):   4%|3         | 8159/217951 [00:00<00:17, 11788.91 examples/s]
Map (num_proc=8):   4%|4         | 9643/217951 [00:00<00:16, 12280.82 examples/s]
Map (num_proc=8):   5%|5         | 11637/217951 [00:01<00:14, 14159.00 examples/s]
Map (num_proc=8):   6%|6         | 13460/217951 [00:01<00:13, 15261.91 examples/s]
Map (num_proc=8):   7%|6         | 15105/217951 [00:01<00:13, 14753.53 examples/s]
Map (num_proc=8):   8%|7         | 16893/217951 [00:01<00:13, 14469.82 examples/s]
Map (num_proc=8):   8%|8         | 18486/217951 [00:01<00:13, 14600.13 examples/s]
Map (num_proc=8):   9%|9         | 20274/217951 [00:01<00:13, 15020.46 examples/s]
Map (num_proc=8):  10%|#         | 21993/217951 [00:01<00:12, 15237.06 examples/s]
Map (num_proc=8):  11%|#         | 23718/217951 [00:01<00:12, 15497.21 examples/s]
Map (num_proc=8):  12%|#1        | 25340/217951 [00:01<00:12, 15099.34 examples/s]
Map (num_proc=8):  12%|#2        | 26892/217951 [00:02<00:12, 14849.38 examples/s]
Map (num_proc=8):  13%|#3        | 28820/217951 [00:02<00:12, 15741.58 examples/s]
Map (num_proc=8):  14%|#3        | 30493/217951 [00:02<00:11, 15711.36 examples/s]
Map (num_proc=8):  15%|#4        | 32437/217951 [00:02<00:11, 15864.73 examples/s]
Map (num_proc=8):  16%|#5        | 34056/217951 [00:02<00:12, 14698.31 examples/s]
Map (num_proc=8):  17%|#6        | 36007/217951 [00:02<00:11, 15683.63 examples/s]
Map (num_proc=8):  17%|#7        | 37682/217951 [00:02<00:11, 15762.16 examples/s]
Map (num_proc=8):  18%|#8        | 39334/217951 [00:02<00:11, 15797.35 examples/s]
Map (num_proc=8):  19%|#8        | 41004/217951 [00:03<00:11, 15278.82 examples/s]
Map (num_proc=8):  20%|#9        | 42564/217951 [00:03<00:12, 14561.08 examples/s]
Map (num_proc=8):  20%|##        | 44509/217951 [00:03<00:11, 15681.44 examples/s]
Map (num_proc=8):  21%|##1       | 46107/217951 [00:03<00:11, 15545.73 examples/s]
Map (num_proc=8):  22%|##1       | 47760/217951 [00:03<00:10, 15819.91 examples/s]
Map (num_proc=8):  23%|##2       | 49469/217951 [00:03<00:11, 14947.04 examples/s]
Map (num_proc=8):  23%|##3       | 51193/217951 [00:03<00:10, 15359.21 examples/s]
Map (num_proc=8):  24%|##4       | 53086/217951 [00:03<00:10, 16088.09 examples/s]
Map (num_proc=8):  25%|##5       | 54715/217951 [00:03<00:10, 15437.66 examples/s]
Map (num_proc=8):  26%|##5       | 56503/217951 [00:03<00:10, 15860.70 examples/s]
Map (num_proc=8):  27%|##6       | 58144/217951 [00:04<00:10, 14703.99 examples/s]
Map (num_proc=8):  28%|##7       | 59940/217951 [00:04<00:10, 15167.16 examples/s]
Map (num_proc=8):  28%|##8       | 61706/217951 [00:04<00:10, 15343.03 examples/s]
Map (num_proc=8):  29%|##9       | 63409/217951 [00:04<00:09, 15624.93 examples/s]
Map (num_proc=8):  30%|##9       | 65335/217951 [00:04<00:09, 15849.75 examples/s]
Map (num_proc=8):  31%|###       | 67043/217951 [00:04<00:09, 15387.04 examples/s]
Map (num_proc=8):  32%|###1      | 68667/217951 [00:04<00:09, 15373.05 examples/s]
Map (num_proc=8):  32%|###2      | 70290/217951 [00:04<00:09, 15224.72 examples/s]
Map (num_proc=8):  33%|###3      | 72052/217951 [00:05<00:09, 15717.09 examples/s]
Map (num_proc=8):  34%|###3      | 73911/217951 [00:05<00:09, 15945.99 examples/s]
Map (num_proc=8):  35%|###4      | 75511/217951 [00:05<00:09, 14534.05 examples/s]
Map (num_proc=8):  35%|###5      | 77345/217951 [00:05<00:09, 15300.07 examples/s]
Map (num_proc=8):  36%|###6      | 78944/217951 [00:05<00:09, 15408.51 examples/s]
Map (num_proc=8):  37%|###7      | 81012/217951 [00:05<00:08, 16378.72 examples/s]
Map (num_proc=8):  38%|###7      | 82710/217951 [00:05<00:08, 15871.09 examples/s]
Map (num_proc=8):  39%|###8      | 84383/217951 [00:05<00:09, 14354.58 examples/s]
Map (num_proc=8):  40%|###9      | 86117/217951 [00:05<00:09, 14617.25 examples/s]
Map (num_proc=8):  40%|####      | 88014/217951 [00:06<00:08, 15763.87 examples/s]
Map (num_proc=8):  41%|####1     | 89806/217951 [00:06<00:07, 16138.06 examples/s]
Map (num_proc=8):  42%|####1     | 91455/217951 [00:06<00:08, 14640.09 examples/s]
Map (num_proc=8):  43%|####2     | 93045/217951 [00:06<00:08, 14742.98 examples/s]
Map (num_proc=8):  43%|####3     | 94792/217951 [00:06<00:08, 15276.37 examples/s]
Map (num_proc=8):  44%|####4     | 96489/217951 [00:06<00:07, 15697.90 examples/s]
Map (num_proc=8):  45%|####5     | 98285/217951 [00:06<00:07, 16115.50 examples/s]
Map (num_proc=8):  46%|####5     | 99920/217951 [00:06<00:07, 14905.11 examples/s]
Map (num_proc=8):  47%|####6     | 101543/217951 [00:06<00:08, 13936.44 examples/s]
Map (num_proc=8):  47%|####7     | 103323/217951 [00:07<00:07, 14530.43 examples/s]
Map (num_proc=8):  48%|####8     | 105543/217951 [00:07<00:06, 16063.95 examples/s]
Map (num_proc=8):  49%|####9     | 107237/217951 [00:07<00:07, 14857.48 examples/s]
Map (num_proc=8):  50%|####9     | 108816/217951 [00:07<00:07, 14960.23 examples/s]
Map (num_proc=8):  51%|#####     | 110429/217951 [00:07<00:07, 14481.37 examples/s]
Map (num_proc=8):  52%|#####1    | 112439/217951 [00:07<00:06, 15728.15 examples/s]
Map (num_proc=8):  52%|#####2    | 114211/217951 [00:07<00:06, 15464.15 examples/s]
Map (num_proc=8):  53%|#####3    | 115877/217951 [00:07<00:06, 15255.00 examples/s]
Map (num_proc=8):  54%|#####3    | 117424/217951 [00:08<00:07, 14335.79 examples/s]
Map (num_proc=8):  55%|#####4    | 119468/217951 [00:08<00:06, 15838.27 examples/s]
Map (num_proc=8):  56%|#####5    | 121387/217951 [00:08<00:05, 16446.99 examples/s]
Map (num_proc=8):  57%|#####6    | 123192/217951 [00:08<00:06, 15392.07 examples/s]
Map (num_proc=8):  57%|#####7    | 124893/217951 [00:08<00:06, 14990.01 examples/s]
Map (num_proc=8):  58%|#####8    | 126563/217951 [00:08<00:05, 15236.61 examples/s]
Map (num_proc=8):  59%|#####8    | 128532/217951 [00:08<00:05, 15929.84 examples/s]
Map (num_proc=8):  60%|#####9    | 130317/217951 [00:08<00:05, 16421.91 examples/s]
Map (num_proc=8):  61%|######    | 131994/217951 [00:08<00:06, 14235.02 examples/s]
Map (num_proc=8):  61%|######1   | 133620/217951 [00:09<00:05, 14367.48 examples/s]
Map (num_proc=8):  62%|######2   | 135617/217951 [00:09<00:05, 15819.35 examples/s]
Map (num_proc=8):  63%|######3   | 137416/217951 [00:09<00:05, 15892.51 examples/s]
Map (num_proc=8):  64%|######3   | 139144/217951 [00:09<00:04, 15909.82 examples/s]
Map (num_proc=8):  65%|######4   | 140775/217951 [00:09<00:05, 15316.97 examples/s]
Map (num_proc=8):  65%|######5   | 142363/217951 [00:09<00:05, 14970.55 examples/s]
Map (num_proc=8):  66%|######6   | 144319/217951 [00:09<00:04, 15720.97 examples/s]
Map (num_proc=8):  67%|######7   | 146338/217951 [00:09<00:04, 16362.92 examples/s]
Map (num_proc=8):  68%|######7   | 148131/217951 [00:09<00:04, 15351.73 examples/s]
Map (num_proc=8):  69%|######8   | 149758/217951 [00:10<00:04, 14662.37 examples/s]
Map (num_proc=8):  70%|######9   | 151598/217951 [00:10<00:04, 15248.68 examples/s]
Map (num_proc=8):  70%|#######   | 153548/217951 [00:10<00:04, 15682.89 examples/s]
Map (num_proc=8):  71%|#######1  | 155255/217951 [00:10<00:03, 15699.04 examples/s]
Map (num_proc=8):  72%|#######1  | 156834/217951 [00:10<00:03, 15567.15 examples/s]
Map (num_proc=8):  73%|#######2  | 158413/217951 [00:10<00:03, 14963.69 examples/s]
Map (num_proc=8):  74%|#######3  | 160288/217951 [00:10<00:03, 15788.28 examples/s]
Map (num_proc=8):  74%|#######4  | 161939/217951 [00:10<00:03, 15722.70 examples/s]
Map (num_proc=8):  75%|#######5  | 163611/217951 [00:10<00:03, 14829.82 examples/s]
Map (num_proc=8):  76%|#######5  | 165130/217951 [00:11<00:03, 14498.54 examples/s]
Map (num_proc=8):  77%|#######6  | 166875/217951 [00:11<00:03, 14804.15 examples/s]
Map (num_proc=8):  78%|#######7  | 168943/217951 [00:11<00:03, 15701.64 examples/s]
Map (num_proc=8):  78%|#######8  | 170932/217951 [00:11<00:02, 16310.68 examples/s]
Map (num_proc=8):  79%|#######9  | 172581/217951 [00:11<00:02, 15235.95 examples/s]
Map (num_proc=8):  80%|#######9  | 174191/217951 [00:11<00:02, 15019.94 examples/s]
Map (num_proc=8):  81%|########  | 176150/217951 [00:11<00:02, 15707.86 examples/s]
Map (num_proc=8):  82%|########1 | 178075/217951 [00:11<00:02, 16098.87 examples/s]
Map (num_proc=8):  82%|########2 | 179726/217951 [00:12<00:02, 15384.51 examples/s]
Map (num_proc=8):  83%|########3 | 181366/217951 [00:12<00:02, 14546.65 examples/s]
Map (num_proc=8):  84%|########4 | 183247/217951 [00:12<00:02, 15535.01 examples/s]
Map (num_proc=8):  85%|########4 | 184905/217951 [00:12<00:02, 15373.45 examples/s]
Map (num_proc=8):  86%|########5 | 186737/217951 [00:12<00:01, 16175.93 examples/s]
Map (num_proc=8):  86%|########6 | 188430/217951 [00:12<00:01, 15565.16 examples/s]
Map (num_proc=8):  87%|########7 | 190073/217951 [00:12<00:01, 14021.85 examples/s]
Map (num_proc=8):  88%|########8 | 191897/217951 [00:12<00:01, 14694.33 examples/s]
Map (num_proc=8):  89%|########8 | 193975/217951 [00:12<00:01, 15886.63 examples/s]
Map (num_proc=8):  90%|########9 | 195684/217951 [00:13<00:01, 15991.70 examples/s]
Map (num_proc=8):  91%|######### | 197349/217951 [00:13<00:01, 14715.70 examples/s]
Map (num_proc=8):  91%|#########1| 198953/217951 [00:13<00:01, 14907.34 examples/s]
Map (num_proc=8):  92%|#########2| 200804/217951 [00:13<00:01, 15406.86 examples/s]
Map (num_proc=8):  93%|#########3| 202825/217951 [00:13<00:00, 16037.87 examples/s]
Map (num_proc=8):  94%|#########3| 204526/217951 [00:13<00:00, 15998.26 examples/s]
Map (num_proc=8):  95%|#########4| 206159/217951 [00:13<00:00, 14417.26 examples/s]
Map (num_proc=8):  96%|#########5| 208164/217951 [00:13<00:00, 15841.87 examples/s]
Map (num_proc=8):  96%|#########6| 210049/217951 [00:13<00:00, 16271.62 examples/s]
Map (num_proc=8):  97%|#########7| 211775/217951 [00:14<00:00, 15968.02 examples/s]
Map (num_proc=8):  98%|#########7| 213423/217951 [00:14<00:00, 15036.58 examples/s]
Map (num_proc=8):  99%|#########8| 215146/217951 [00:14<00:00, 14210.82 examples/s]
Map (num_proc=8):  99%|#########9| 216764/217951 [00:14<00:00, 14114.54 examples/s]
Map (num_proc=8): 100%|##########| 217951/217951 [00:14<00:00, 14864.64 examples/s]

Creating json from Arrow format:   0%|          | 0/218 [00:00<?, ?ba/s]
Creating json from Arrow format:   1%|1         | 3/218 [00:00<00:08, 25.89ba/s]
Creating json from Arrow format:   3%|3         | 7/218 [00:00<00:06, 30.62ba/s]
Creating json from Arrow format:   5%|5         | 11/218 [00:00<00:06, 33.89ba/s]
Creating json from Arrow format:   7%|6         | 15/218 [00:00<00:05, 35.21ba/s]
Creating json from Arrow format:   9%|8         | 19/218 [00:00<00:05, 35.75ba/s]
Creating json from Arrow format:  11%|#         | 23/218 [00:00<00:05, 35.46ba/s]
Creating json from Arrow format:  12%|#2        | 27/218 [00:00<00:05, 35.83ba/s]
Creating json from Arrow format:  14%|#4        | 31/218 [00:00<00:05, 36.50ba/s]
Creating json from Arrow format:  16%|#6        | 35/218 [00:00<00:04, 37.15ba/s]
Creating json from Arrow format:  18%|#8        | 40/218 [00:01<00:04, 38.85ba/s]
Creating json from Arrow format:  20%|##        | 44/218 [00:01<00:04, 38.24ba/s]
Creating json from Arrow format:  22%|##2       | 48/218 [00:01<00:04, 38.70ba/s]
Creating json from Arrow format:  24%|##3       | 52/218 [00:01<00:04, 38.80ba/s]
Creating json from Arrow format:  26%|##6       | 57/218 [00:01<00:04, 39.40ba/s]
Creating json from Arrow format:  28%|##7       | 61/218 [00:01<00:04, 39.18ba/s]
Creating json from Arrow format:  30%|###       | 66/218 [00:01<00:03, 39.98ba/s]
Creating json from Arrow format:  32%|###2      | 70/218 [00:01<00:03, 38.73ba/s]
Creating json from Arrow format:  34%|###3      | 74/218 [00:01<00:03, 38.72ba/s]
Creating json from Arrow format:  36%|###5      | 78/218 [00:02<00:03, 38.90ba/s]
Creating json from Arrow format:  38%|###7      | 82/218 [00:02<00:03, 37.49ba/s]
Creating json from Arrow format:  40%|###9      | 87/218 [00:02<00:03, 38.56ba/s]
Creating json from Arrow format:  42%|####1     | 91/218 [00:02<00:03, 38.39ba/s]
Creating json from Arrow format:  44%|####3     | 95/218 [00:02<00:03, 37.53ba/s]
Creating json from Arrow format:  45%|####5     | 99/218 [00:02<00:03, 37.67ba/s]
Creating json from Arrow format:  47%|####7     | 103/218 [00:02<00:03, 37.96ba/s]
Creating json from Arrow format:  49%|####9     | 107/218 [00:02<00:02, 37.43ba/s]
Creating json from Arrow format:  51%|#####     | 111/218 [00:02<00:02, 37.82ba/s]
Creating json from Arrow format:  53%|#####2    | 115/218 [00:03<00:02, 37.77ba/s]
Creating json from Arrow format:  55%|#####4    | 119/218 [00:03<00:02, 37.59ba/s]
Creating json from Arrow format:  57%|#####6    | 124/218 [00:03<00:02, 38.43ba/s]
Creating json from Arrow format:  59%|#####8    | 128/218 [00:03<00:02, 38.06ba/s]
Creating json from Arrow format:  61%|######    | 132/218 [00:03<00:02, 37.96ba/s]
Creating json from Arrow format:  62%|######2   | 136/218 [00:03<00:02, 38.33ba/s]
Creating json from Arrow format:  65%|######4   | 141/218 [00:03<00:01, 40.28ba/s]
Creating json from Arrow format:  67%|######6   | 146/218 [00:03<00:01, 39.44ba/s]
Creating json from Arrow format:  69%|######8   | 150/218 [00:03<00:01, 37.88ba/s]
Creating json from Arrow format:  71%|#######   | 154/218 [00:04<00:01, 37.81ba/s]
Creating json from Arrow format:  72%|#######2  | 158/218 [00:04<00:01, 36.59ba/s]
Creating json from Arrow format:  74%|#######4  | 162/218 [00:04<00:01, 33.68ba/s]
Creating json from Arrow format:  77%|#######6  | 167/218 [00:04<00:01, 35.68ba/s]
Creating json from Arrow format:  79%|#######8  | 172/218 [00:04<00:01, 37.69ba/s]
Creating json from Arrow format:  81%|########  | 176/218 [00:04<00:01, 35.81ba/s]
Creating json from Arrow format:  83%|########2 | 180/218 [00:04<00:01, 34.88ba/s]
Creating json from Arrow format:  84%|########4 | 184/218 [00:04<00:00, 35.36ba/s]
Creating json from Arrow format:  86%|########6 | 188/218 [00:05<00:00, 34.86ba/s]
Creating json from Arrow format:  88%|########8 | 192/218 [00:05<00:00, 34.93ba/s]
Creating json from Arrow format:  90%|########9 | 196/218 [00:05<00:00, 36.14ba/s]
Creating json from Arrow format:  92%|#########1| 200/218 [00:05<00:00, 34.91ba/s]
Creating json from Arrow format:  94%|#########3| 204/218 [00:05<00:00, 35.25ba/s]
Creating json from Arrow format:  95%|#########5| 208/218 [00:05<00:00, 35.85ba/s]
Creating json from Arrow format:  97%|#########7| 212/218 [00:05<00:00, 35.45ba/s]
Creating json from Arrow format:  99%|#########9| 216/218 [00:05<00:00, 34.92ba/s]
Creating json from Arrow format: 100%|##########| 218/218 [00:05<00:00, 36.91ba/s]INFO 2025-05-20 11:35:57,457 instructlab.sdg.datamixing:235: Mixed Dataset saved to /var/home/vkadlec/.local/share/instructlab/datasets/2025-05-19_132812/knowledge_train_msgs_2025-05-19T13_31_09.jsonl
INFO 2025-05-20 11:35:57,470 instructlab.sdg.generate_data:757: Generation took 79488.08s
INFO 2025-05-20 11:36:03,602 instructlab.model.backends.vllm:512: Waiting for GPU VRAM reclamation...