time="2025-03-10T19:26:51Z" level=warning msg="The input device is not a TTY. The --tty and --interactive flags might not work properly"
LoRA is disabled (rank=0), ignoring all additional LoRA args


~~~~~~~~~~~~~STARTING MULTI-PHASE TRAINING~~~~~~~~~~~~~
No training journal found. Will initialize at: '/var/home/cloud-user/.local/share/instructlab/phased/journalfile.yaml'
Training Phase 1/2...
TrainingArgs for current phase: TrainingArgs(model_path='/var/home/cloud-user/.cache/instructlab/models/granite-3.1-8b-starter-v1', chat_tmpl_path='/opt/app-root/lib64/python3.11/site-packages/instructlab/training/chat_templates/ibm_generic_tmpl.py', use_legacy_tmpl=True, data_path='/var/home/cloud-user/.local/share/instructlab/datasets/2025-03-10_125452/knowledge_train_msgs_2025-03-10T12_58_55.jsonl', ckpt_output_dir='/var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints', data_output_dir='/var/home/cloud-user/.local/share/instructlab/internal', max_seq_len=10000, max_batch_len=60000, num_epochs=7, effective_batch_size=128, save_samples=0, learning_rate=2e-05, warmup_steps=25, random_seed=42, use_dolomite=True, is_padding_free=False, checkpoint_at_epoch=True, accelerate_full_state_at_epoch=True, mock_data=False, mock_data_len=0, deepspeed_options=DeepSpeedOptions(cpu_offload_optimizer=False, cpu_offload_optimizer_ratio=1.0, cpu_offload_optimizer_pin_memory=False, save_samples=None), fsdp_options=FSDPOptions(cpu_offload_params=False, sharding_strategy=<ShardingStrategies.SHARD_GRAD_OP: 'SHARD_GRAD_OP'>), distributed_backend=<DistributedBackend.FSDP: 'fsdp'>, disable_flash_attn=False, lora=LoraOptions(rank=0, alpha=32, dropout=0.1, target_modules=('q_proj', 'k_proj', 'v_proj', 'o_proj'), quantize_data_type=<QuantizeDataType.NONE: None>), process_data=True, keep_last_checkpoint_only=False)
[2025-03-10 19:27:04,366] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
df: /var/home/cloud-user/.triton/autotune: No such file or directory
INFO 2025-03-10 19:27:14,370 numexpr.utils:146: Note: detected 192 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
INFO 2025-03-10 19:27:14,371 numexpr.utils:149: Note: NumExpr detected 192 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
INFO 2025-03-10 19:27:14,371 numexpr.utils:162: NumExpr defaulting to 16 threads.
INFO 2025-03-10 19:27:18,084 datasets:59: PyTorch version 2.5.1 available.
 data arguments are:
{"data_path":"/var/home/cloud-user/.local/share/instructlab/datasets/2025-03-10_125452/knowledge_train_msgs_2025-03-10T12_58_55.jsonl","data_output_path":"/var/home/cloud-user/.local/share/instructlab/internal","max_seq_len":10000,"model_path":"/var/home/cloud-user/.cache/instructlab/models/granite-3.1-8b-starter-v1","chat_tmpl_path":"/opt/app-root/lib64/python3.11/site-packages/instructlab/training/chat_templates/ibm_legacy_tmpl.py","num_cpu_procs":16}
INFO 2025-03-10 19:27:18,797 root:879: Special tokens: eos: [0], pad: [49153], bos: [49152], system: [49154], user: [49155], assistant: [49156]

Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 5090 examples [00:00, 72962.87 examples/s]
tokenizing the dataset with /var/home/cloud-user/.cache/instructlab/models/granite-3.1-8b-starter-v1 tokenizer...

Map (num_proc=16):   0% 0/5090 [00:00<?, ? examples/s]
Map (num_proc=16):   0% 1/5090 [00:00<1:14:09,  1.14 examples/s]
Map (num_proc=16):   0% 11/5090 [00:01<05:59, 14.12 examples/s] 
Map (num_proc=16):   0% 22/5090 [00:01<03:00, 28.04 examples/s]
Map (num_proc=16):   1% 41/5090 [00:01<01:33, 53.73 examples/s]
Map (num_proc=16):   1% 63/5090 [00:01<00:58, 85.65 examples/s]
Map (num_proc=16):   2% 86/5090 [00:01<00:43, 115.25 examples/s]
Map (num_proc=16):   3% 151/5090 [00:01<00:20, 237.49 examples/s]
Map (num_proc=16):   4% 213/5090 [00:01<00:14, 331.67 examples/s]
Map (num_proc=16):   7% 336/5090 [00:01<00:08, 564.73 examples/s]
Map (num_proc=16):   9% 476/5090 [00:01<00:05, 787.50 examples/s]
Map (num_proc=16):  12% 614/5090 [00:02<00:04, 942.56 examples/s]
Map (num_proc=16):  15% 750/5090 [00:02<00:04, 1055.80 examples/s]
Map (num_proc=16):  18% 894/5090 [00:02<00:03, 1156.92 examples/s]
Map (num_proc=16):  20% 1036/5090 [00:02<00:03, 1225.70 examples/s]
Map (num_proc=16):  23% 1180/5090 [00:02<00:03, 1281.00 examples/s]
Map (num_proc=16):  26% 1321/5090 [00:02<00:02, 1300.06 examples/s]
Map (num_proc=16):  29% 1473/5090 [00:02<00:02, 1334.43 examples/s]
Map (num_proc=16):  32% 1609/5090 [00:02<00:02, 1323.45 examples/s]
Map (num_proc=16):  35% 1763/5090 [00:02<00:02, 1341.72 examples/s]
Map (num_proc=16):  37% 1908/5090 [00:02<00:02, 1371.28 examples/s]
Map (num_proc=16):  40% 2048/5090 [00:03<00:02, 1365.43 examples/s]
Map (num_proc=16):  43% 2189/5090 [00:03<00:02, 1353.94 examples/s]
Map (num_proc=16):  46% 2327/5090 [00:03<00:02, 1319.70 examples/s]
Map (num_proc=16):  48% 2463/5090 [00:03<00:01, 1323.41 examples/s]
Map (num_proc=16):  51% 2600/5090 [00:03<00:01, 1284.53 examples/s]
Map (num_proc=16):  54% 2729/5090 [00:03<00:01, 1282.88 examples/s]
Map (num_proc=16):  56% 2860/5090 [00:03<00:01, 1271.53 examples/s]
Map (num_proc=16):  59% 3002/5090 [00:03<00:01, 1304.35 examples/s]
Map (num_proc=16):  62% 3135/5090 [00:03<00:01, 1277.67 examples/s]
Map (num_proc=16):  65% 3284/5090 [00:04<00:01, 1308.40 examples/s]
Map (num_proc=16):  67% 3417/5090 [00:04<00:01, 1313.04 examples/s]
Map (num_proc=16):  70% 3551/5090 [00:04<00:01, 1278.56 examples/s]
Map (num_proc=16):  72% 3681/5090 [00:04<00:01, 1260.43 examples/s]
Map (num_proc=16):  75% 3822/5090 [00:04<00:00, 1300.62 examples/s]
Map (num_proc=16):  78% 3954/5090 [00:04<00:00, 1259.24 examples/s]
Map (num_proc=16):  80% 4085/5090 [00:04<00:00, 1273.39 examples/s]
Map (num_proc=16):  83% 4213/5090 [00:04<00:00, 1223.13 examples/s]
Map (num_proc=16):  85% 4336/5090 [00:04<00:00, 1125.98 examples/s]
Map (num_proc=16):  87% 4451/5090 [00:05<00:00, 1032.67 examples/s]
Map (num_proc=16):  90% 4560/5090 [00:05<00:00, 986.35 examples/s] 
Map (num_proc=16):  92% 4678/5090 [00:05<00:00, 1028.53 examples/s]
Map (num_proc=16):  94% 4790/5090 [00:05<00:00, 1048.34 examples/s]
Map (num_proc=16):  96% 4898/5090 [00:05<00:00, 857.26 examples/s] 
Map (num_proc=16):  98% 4992/5090 [00:05<00:00, 774.61 examples/s]
Map (num_proc=16): 100% 5079/5090 [00:05<00:00, 735.67 examples/s]
Map (num_proc=16): 100% 5090/5090 [00:05<00:00, 858.40 examples/s]
ten largest length percentiles:

Map (num_proc=16):   0% 0/5090 [00:00<?, ? examples/s]
Map (num_proc=16):   2% 105/5090 [00:00<00:14, 339.46 examples/s]
Map (num_proc=16):  14% 720/5090 [00:00<00:02, 2117.87 examples/s]
Map (num_proc=16):  33% 1665/5090 [00:00<00:00, 4167.31 examples/s]
Map (num_proc=16):  50% 2567/5090 [00:00<00:00, 5560.59 examples/s]
Map (num_proc=16):  64% 3267/5090 [00:00<00:00, 5961.42 examples/s]
Map (num_proc=16):  78% 3973/5090 [00:00<00:00, 5425.04 examples/s]
Map (num_proc=16):  90% 4586/5090 [00:01<00:00, 4395.37 examples/s]
Map (num_proc=16): 100% 5090/5090 [00:01<00:00, 3395.12 examples/s]
quantile 90th: 675.0
quantile 91th: 686.0
quantile 92th: 696.0
quantile 93th: 709.0
quantile 94th: 726.0
quantile 95th: 761.6500000000005
quantile 96th: 828.0
quantile 97th: 872.3299999999999
quantile 98th: 1005.0
quantile 99th: 1067.0
quantile 100th: 1484.0

at 10000 max sequence length, the number of samples to be dropped is 0
(0.00% of total)
quantile 0th: 166.0
quantile 1th: 258.0
quantile 2th: 271.0
quantile 3th: 281.66999999999996
quantile 4th: 290.0
quantile 5th: 297.0
quantile 6th: 302.0
quantile 7th: 307.0
quantile 8th: 314.0
quantile 9th: 321.0
quantile 10th: 327.0
at 20 min sequence length, the number of samples to be dropped is 0
checking the validity of the samples...

Filter (num_proc=16):   0% 0/5090 [00:00<?, ? examples/s]
Filter (num_proc=16):   6% 319/5090 [00:01<00:18, 254.47 examples/s]
Filter (num_proc=16):  13% 638/5090 [00:01<00:09, 478.11 examples/s]
Filter (num_proc=16):  19% 956/5090 [00:01<00:05, 720.85 examples/s]
Filter (num_proc=16):  25% 1274/5090 [00:01<00:03, 991.27 examples/s]
Filter (num_proc=16):  38% 1910/5090 [00:01<00:01, 1719.02 examples/s]
Filter (num_proc=16):  63% 3182/5090 [00:02<00:00, 3578.36 examples/s]
Filter (num_proc=16): 100% 5090/5090 [00:02<00:00, 2311.40 examples/s]
INFO 2025-03-10 19:27:30,618 root:879: number of dropped samples: 0 -- out of 5090
Categorizing training data type...

Map (num_proc=16):   0% 0/5090 [00:00<?, ? examples/s]
Map (num_proc=16):   0% 1/5090 [00:00<48:38,  1.74 examples/s]
Map (num_proc=16):   1% 52/5090 [00:00<00:53, 94.86 examples/s]
Map (num_proc=16):   2% 98/5090 [00:00<00:29, 167.86 examples/s]
Map (num_proc=16):   3% 163/5090 [00:00<00:18, 260.36 examples/s]
Map (num_proc=16):   4% 229/5090 [00:01<00:14, 333.15 examples/s]
Map (num_proc=16):   6% 293/5090 [00:01<00:12, 395.71 examples/s]
Map (num_proc=16):   8% 398/5090 [00:01<00:08, 556.27 examples/s]
Map (num_proc=16):  12% 604/5090 [00:01<00:04, 941.73 examples/s]
Map (num_proc=16):  16% 803/5090 [00:01<00:03, 1185.98 examples/s]
Map (num_proc=16):  22% 1128/5090 [00:01<00:02, 1714.36 examples/s]
Map (num_proc=16):  31% 1561/5090 [00:01<00:01, 2435.50 examples/s]
Map (num_proc=16):  38% 1946/5090 [00:01<00:01, 2830.76 examples/s]
Map (num_proc=16):  47% 2379/5090 [00:01<00:00, 3232.95 examples/s]
Map (num_proc=16):  54% 2759/5090 [00:02<00:00, 3376.35 examples/s]
Map (num_proc=16):  62% 3163/5090 [00:02<00:00, 3511.49 examples/s]
Map (num_proc=16):  70% 3585/5090 [00:02<00:00, 3672.47 examples/s]
Map (num_proc=16):  79% 4017/5090 [00:02<00:00, 3835.40 examples/s]
Map (num_proc=16):  87% 4408/5090 [00:02<00:00, 3558.87 examples/s]
Map (num_proc=16):  94% 4770/5090 [00:02<00:00, 3158.80 examples/s]
Map (num_proc=16): 100% 5090/5090 [00:02<00:00, 1698.40 examples/s]
unmasking the appropriate message content...

Map (num_proc=16):   0% 0/5090 [00:00<?, ? examples/s]
Map (num_proc=16):   0% 12/5090 [00:00<02:22, 35.61 examples/s]
Map (num_proc=16):   2% 113/5090 [00:00<00:15, 324.20 examples/s]
Map (num_proc=16):   5% 262/5090 [00:00<00:07, 670.32 examples/s]
Map (num_proc=16):   8% 389/5090 [00:00<00:05, 832.26 examples/s]
Map (num_proc=16):  11% 537/5090 [00:00<00:04, 1018.43 examples/s]
Map (num_proc=16):  13% 659/5090 [00:00<00:04, 969.90 examples/s] 
Map (num_proc=16):  15% 773/5090 [00:01<00:04, 951.64 examples/s]
Map (num_proc=16):  17% 880/5090 [00:01<00:04, 968.12 examples/s]
Map (num_proc=16):  19% 985/5090 [00:01<00:04, 935.34 examples/s]
Map (num_proc=16):  21% 1091/5090 [00:01<00:04, 967.49 examples/s]
Map (num_proc=16):  23% 1195/5090 [00:01<00:04, 919.73 examples/s]
Map (num_proc=16):  26% 1305/5090 [00:01<00:03, 966.60 examples/s]
Map (num_proc=16):  28% 1408/5090 [00:01<00:03, 947.05 examples/s]
Map (num_proc=16):  30% 1508/5090 [00:01<00:03, 944.34 examples/s]
Map (num_proc=16):  32% 1611/5090 [00:01<00:03, 955.65 examples/s]
Map (num_proc=16):  34% 1709/5090 [00:01<00:03, 956.44 examples/s]
Map (num_proc=16):  35% 1806/5090 [00:02<00:03, 900.33 examples/s]
Map (num_proc=16):  37% 1900/5090 [00:02<00:03, 908.21 examples/s]
Map (num_proc=16):  39% 2000/5090 [00:02<00:03, 921.38 examples/s]
Map (num_proc=16):  41% 2098/5090 [00:02<00:03, 928.29 examples/s]
Map (num_proc=16):  43% 2195/5090 [00:02<00:03, 925.47 examples/s]
Map (num_proc=16):  45% 2291/5090 [00:02<00:03, 897.72 examples/s]
Map (num_proc=16):  47% 2384/5090 [00:02<00:03, 899.50 examples/s]
Map (num_proc=16):  49% 2475/5090 [00:02<00:02, 900.86 examples/s]
Map (num_proc=16):  50% 2568/5090 [00:02<00:02, 879.61 examples/s]
Map (num_proc=16):  53% 2677/5090 [00:03<00:02, 921.62 examples/s]
Map (num_proc=16):  54% 2773/5090 [00:03<00:02, 906.55 examples/s]
Map (num_proc=16):  56% 2867/5090 [00:03<00:02, 875.57 examples/s]
Map (num_proc=16):  58% 2971/5090 [00:03<00:02, 900.30 examples/s]
Map (num_proc=16):  60% 3063/5090 [00:03<00:02, 899.75 examples/s]
Map (num_proc=16):  62% 3158/5090 [00:03<00:02, 910.24 examples/s]
Map (num_proc=16):  64% 3254/5090 [00:03<00:01, 923.50 examples/s]
Map (num_proc=16):  66% 3347/5090 [00:03<00:01, 886.07 examples/s]
Map (num_proc=16):  68% 3437/5090 [00:03<00:01, 874.81 examples/s]
Map (num_proc=16):  69% 3529/5090 [00:04<00:01, 883.88 examples/s]
Map (num_proc=16):  71% 3622/5090 [00:04<00:01, 881.34 examples/s]
Map (num_proc=16):  73% 3715/5090 [00:04<00:01, 881.65 examples/s]
Map (num_proc=16):  75% 3805/5090 [00:04<00:01, 864.23 examples/s]
Map (num_proc=16):  76% 3893/5090 [00:04<00:01, 855.34 examples/s]
Map (num_proc=16):  78% 3980/5090 [00:04<00:01, 808.83 examples/s]
Map (num_proc=16):  80% 4064/5090 [00:04<00:01, 813.28 examples/s]
Map (num_proc=16):  81% 4146/5090 [00:04<00:01, 768.04 examples/s]
Map (num_proc=16):  83% 4225/5090 [00:04<00:01, 737.06 examples/s]
Map (num_proc=16):  84% 4300/5090 [00:05<00:01, 718.29 examples/s]
Map (num_proc=16):  86% 4375/5090 [00:05<00:01, 705.40 examples/s]
Map (num_proc=16):  87% 4448/5090 [00:05<00:00, 685.03 examples/s]
Map (num_proc=16):  89% 4517/5090 [00:05<00:00, 655.72 examples/s]
Map (num_proc=16):  90% 4583/5090 [00:05<00:00, 629.31 examples/s]
Map (num_proc=16):  91% 4648/5090 [00:05<00:00, 557.16 examples/s]
Map (num_proc=16):  93% 4710/5090 [00:05<00:00, 548.66 examples/s]
Map (num_proc=16):  94% 4766/5090 [00:05<00:00, 458.39 examples/s]
Map (num_proc=16):  95% 4818/5090 [00:06<00:00, 420.00 examples/s]
Map (num_proc=16):  96% 4864/5090 [00:06<00:00, 350.95 examples/s]
Map (num_proc=16):  96% 4904/5090 [00:06<00:00, 308.65 examples/s]
Map (num_proc=16):  97% 4939/5090 [00:06<00:00, 287.10 examples/s]
Map (num_proc=16):  98% 4970/5090 [00:06<00:00, 226.51 examples/s]
Map (num_proc=16):  98% 4996/5090 [00:07<00:00, 197.53 examples/s]
Map (num_proc=16):  99% 5022/5090 [00:07<00:00, 186.97 examples/s]
Map (num_proc=16):  99% 5042/5090 [00:07<00:00, 160.35 examples/s]
Map (num_proc=16):  99% 5059/5090 [00:07<00:00, 121.10 examples/s]
Map (num_proc=16): 100% 5073/5090 [00:08<00:00, 41.73 examples/s] 
Map (num_proc=16): 100% 5084/5090 [00:09<00:00, 34.18 examples/s]
Map (num_proc=16): 100% 5090/5090 [00:09<00:00, 519.34 examples/s]
 Samples Previews...
 
 

Filter (num_proc=16):   0% 0/5090 [00:00<?, ? examples/s]
Filter (num_proc=16):   6% 319/5090 [00:01<00:18, 253.93 examples/s]
Filter (num_proc=16):  25% 1274/5090 [00:01<00:03, 1170.71 examples/s]
Filter (num_proc=16):  38% 1910/5090 [00:01<00:01, 1796.28 examples/s]
Filter (num_proc=16):  69% 3500/5090 [00:01<00:00, 3875.55 examples/s]
Filter (num_proc=16): 100% 5090/5090 [00:01<00:00, 2938.78 examples/s]
Original Input: <|system|>
I am a Red Hat® Instruct Model, an AI language model developed by Red Hat and IBM Research based on the granite-3.1-8b-base model. My primary role is to serve as a chat assistant.
<|user|>
The document discusses the Simplified Identification method for verifying the identity of specific entities in the Proceeds of Crime (Money Laundering) and Terrorist Financing Regulations (PCMLTFR). This method is applicable for Reporting Entities (REs) mentioned in paragraphs 5(a) to (g) of the PCMLTFA. The Simplified Identification method requires a risk assessment to ensure a low risk of money laundering or terrorist activity financing offenses. It also necessitates obtaining information to confirm the entity's existence and the identities of individuals dealing with the entity.

Additionally, the document provides an annex with examples of acceptable photo identification documents issued by federal, provincial, or territorial authorities. This list is not exhaustive. The purpose of this annex is to assist in identifying valid government-issued photo identification documents for identity verification purposes.

How does the Simplified Identification method affect the risk management strategies of REs in FINTRAC regulations?
<|assistant|>
The Simplified Identification method affects the risk management strategies of REs in FINTRAC regulations by requiring a risk assessment to ensure a low risk of money<|endoftext|>

Pretraining ex sample 1: <|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|>
The document discusses the Simplified Identification method for verifying the identity of specific entities in the Proceeds of Crime (Money Laundering) and Terrorist Financing Regulations (PCMLTFR). This method is applicable for Reporting Entities (REs) mentioned in paragraphs 5(a) to (g) of the PCMLTFA. The Simplified Identification method requires a risk assessment to ensure a low risk of money laundering or terrorist activity financing offenses. It also necessitates obtaining information to confirm the entity's existence and the identities of individuals dealing with the entity.

Additionally, the document provides an annex with examples of acceptable photo identification documents issued by federal, provincial, or territorial authorities. This list is not exhaustive. The purpose of this annex is to assist in identifying valid government-issued photo identification documents for identity verification purposes.

How does the Simplified Identification method affect the risk management strategies of REs in FINTRAC regulations?
<|MASK|>
The Simplified Identification method affects the risk management strategies of REs in FINTRAC regulations by requiring a risk assessment to ensure a low risk of money<|endoftext|>
Original Input: <|system|>
I am a Red Hat® Instruct Model, an AI language model developed by Red Hat and IBM Research based on the granite-3.1-8b-base model. My primary role is to serve as a chat assistant.
<|user|>
1. 'g' is a public service body, as defined in subsection 123(1) of the Excise Tax Act.
2. The Excise Tax Act includes a subsection 123(1).
3. Verifying the identity of a person or entity is required under certain conditions.
4. A risk assessment can determine if the risk of a money laundering offence or terrorist activity financing offence has increased.
5. If the risk increases and is no longer low, the identity of the corporation or other entity must be verified as soon as feasible.
6. Verification of the identity of a corporation or other entity can be done by referring to appropriate records.
7. The Confirmation of existence method is explained in section 7.1.
8. If the simplified identification method is used to verify the identity of a corporation or other entity, a record must be kept.
9. The record must include the grounds for considering there is a low risk of a money laundering offence or terrorist activity financing offence.
10. The record must include information obtained about the corporation or other entity and about the persons representing them.
11. The policies and procedures of a compliance program must describe the processes followed when using the simplified identification method.
12. Verification of the identity of a corporation or other entity is not required for subsequent transactions if previously verified using a method explained in the guidance.
13. If an entity's existence was confirmed in accordance with the PCMLTFR and related record keeping provisions were complied with prior to June 1, 2021, verification is not required for subsequent transactions.

What is the PCMLTFR, and what does it refer to in FINTRAC regulations?
<|assistant|>
The PCMLTFR is an acronym for the Proceeds of Crime (Money Laundering) and Terrorist Financing Regulations. In FINTRAC regulations, it refers to the regulations that were in place prior to June 1, 2021, for confirming the existence of an entity and complying with related record keeping provisions.

<|endoftext|>

Pretraining ex sample 2: <|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|>
1. 'g' is a public service body, as defined in subsection 123(1) of the Excise Tax Act.
2. The Excise Tax Act includes a subsection 123(1).
3. Verifying the identity of a person or entity is required under certain conditions.
4. A risk assessment can determine if the risk of a money laundering offence or terrorist activity financing offence has increased.
5. If the risk increases and is no longer low, the identity of the corporation or other entity must be verified as soon as feasible.
6. Verification of the identity of a corporation or other entity can be done by referring to appropriate records.
7. The Confirmation of existence method is explained in section 7.1.
8. If the simplified identification method is used to verify the identity of a corporation or other entity, a record must be kept.
9. The record must include the grounds for considering there is a low risk of a money laundering offence or terrorist activity financing offence.
10. The record must include information obtained about the corporation or other entity and about the persons representing them.
11. The policies and procedures of a compliance program must describe the processes followed when using the simplified identification method.
12. Verification of the identity of a corporation or other entity is not required for subsequent transactions if previously verified using a method explained in the guidance.
13. If an entity's existence was confirmed in accordance with the PCMLTFR and related record keeping provisions were complied with prior to June 1, 2021, verification is not required for subsequent transactions.

What is the PCMLTFR, and what does it refer to in FINTRAC regulations?
<|MASK|>
The PCMLTFR is an acronym for the Proceeds of Crime (Money Laundering) and Terrorist Financing Regulations. In FINTRAC regulations, it refers to the regulations that were in place prior to June 1, 2021, for confirming the existence of an entity and complying with related record keeping provisions.

<|endoftext|>
Original Input: <|system|>
I am a Red Hat® Instruct Model, an AI language model developed by Red Hat and IBM Research based on the granite-3.1-8b-base model. My primary role is to serve as a chat assistant.
<|user|>
The document provides a table that outlines the identification method for determining the existence of an entity, which can be either an entity other than a corporation or a corporation. For an entity other than a corporation, the documents or information to review include a partnership agreement, articles of association, or any other current, authentic, and valid record. The identification details that must match are the name and address, as well as the names of directors. If an electronic record from a publicly accessible database is consulted, the type of document and source must be confirmed. If a paper record or an electronic document is consulted, the registration number, document consulted, and source of the electronic document must be confirmed.

What information must match when determining the existence of an entity other than a corporation in Canada?
<|assistant|>
When determining the existence of an entity other than a corporation in Canada, the name and address, as well as the names of directors, must match the identification details.

<|endoftext|>

Pretraining ex sample 3: <|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|><|MASK|>
The document provides a table that outlines the identification method for determining the existence of an entity, which can be either an entity other than a corporation or a corporation. For an entity other than a corporation, the documents or information to review include a partnership agreement, articles of association, or any other current, authentic, and valid record. The identification details that must match are the name and address, as well as the names of directors. If an electronic record from a publicly accessible database is consulted, the type of document and source must be confirmed. If a paper record or an electronic document is consulted, the registration number, document consulted, and source of the electronic document must be confirmed.

What information must match when determining the existence of an entity other than a corporation in Canada?
<|MASK|>
When determining the existence of an entity other than a corporation in Canada, the name and address, as well as the names of directors, must match the identification details.

<|endoftext|>

Filter (num_proc=16):   0% 0/5090 [00:00<?, ? examples/s]
Filter (num_proc=16):   6% 319/5090 [00:01<00:17, 267.85 examples/s]
Filter (num_proc=16):  25% 1274/5090 [00:01<00:03, 1228.08 examples/s]
Filter (num_proc=16):  44% 2228/5090 [00:01<00:01, 2279.52 examples/s]
Filter (num_proc=16):  69% 3500/5090 [00:01<00:00, 3812.06 examples/s]
Filter (num_proc=16): 100% 5090/5090 [00:01<00:00, 2988.15 examples/s]

Map (num_proc=16):   0% 0/5090 [00:00<?, ? examples/s]
Map (num_proc=16):   1% 61/5090 [00:00<00:27, 182.52 examples/s]
Map (num_proc=16):  19% 978/5090 [00:00<00:01, 2858.80 examples/s]
Map (num_proc=16):  36% 1843/5090 [00:00<00:00, 4545.31 examples/s]
Map (num_proc=16):  53% 2678/5090 [00:00<00:00, 5591.78 examples/s]
Map (num_proc=16):  67% 3426/5090 [00:00<00:00, 5632.76 examples/s]
Map (num_proc=16):  81% 4109/5090 [00:00<00:00, 5098.26 examples/s]
Map (num_proc=16):  92% 4704/5090 [00:01<00:00, 4180.24 examples/s]
Map (num_proc=16): 100% 5090/5090 [00:01<00:00, 3304.39 examples/s]

Filter (num_proc=16):   0% 0/5090 [00:00<?, ? examples/s]
Filter (num_proc=16):   6% 319/5090 [00:01<00:16, 295.42 examples/s]
Filter (num_proc=16):  13% 637/5090 [00:01<00:08, 551.34 examples/s]
Filter (num_proc=16):  38% 1910/5090 [00:01<00:01, 2021.29 examples/s]
Filter (num_proc=16):  63% 3182/5090 [00:01<00:00, 3554.05 examples/s]
Filter (num_proc=16): 100% 5090/5090 [00:01<00:00, 2986.91 examples/s]

Creating json from Arrow format:   0% 0/6 [00:00<?, ?ba/s]
Creating json from Arrow format:  33% 2/6 [00:00<00:00, 13.41ba/s]
Creating json from Arrow format:  67% 4/6 [00:00<00:00, 13.38ba/s]
Creating json from Arrow format: 100% 6/6 [00:00<00:00, 15.20ba/s]
Running training command as subprocess: torchrun --nnodes=1 --node_rank=0 --nproc_per_node=8 --rdzv_id=123 --rdzv_endpoint=127.0.0.1:12222 /opt/app-root/lib64/python3.11/site-packages/instructlab/training/main_ds.py --model_name_or_path=/var/home/cloud-user/.cache/instructlab/models/granite-3.1-8b-starter-v1 --data_path=/var/home/cloud-user/.local/share/instructlab/internal/data.jsonl --output_dir=/var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints --num_epochs=7 --effective_batch_size=128 --learning_rate=2e-05 --num_warmup_steps=25 --save_samples=0 --log_level=INFO --max_batch_len=60000 --seed=42 --chat-tmpl-path=/opt/app-root/lib64/python3.11/site-packages/instructlab/training/chat_templates/ibm_legacy_tmpl.py --checkpoint_at_epoch --accelerate_full_state_at_epoch --use_dolomite --lora_r=0 --lora_alpha=32 --lora_dropout=0.1 --lora_target_modules q_proj k_proj v_proj o_proj --distributed_training_framework=fsdp --fsdp_sharding_strategy=SHARD_GRAD_OP
W0310 19:27:56.212000 1019 torch/distributed/run.py:793] 
W0310 19:27:56.212000 1019 torch/distributed/run.py:793] *****************************************
W0310 19:27:56.212000 1019 torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0310 19:27:56.212000 1019 torch/distributed/run.py:793] *****************************************
[2025-03-10 19:28:01,301] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-03-10 19:28:01,479] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-03-10 19:28:01,777] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-03-10 19:28:01,859] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-03-10 19:28:02,052] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-03-10 19:28:02,057] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-03-10 19:28:02,077] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-03-10 19:28:02,149] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
model_name_or_path: /var/home/cloud-user/.cache/instructlab/models/granite-3.1-8b-starter-v1
data_path: /var/home/cloud-user/.local/share/instructlab/internal/data.jsonl
output_dir: /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints
num_epochs: 7
current_epoch: 0
last_step: 0
effective_batch_size: 128
learning_rate: 2.0e-05
lr_scheduler: cosine
num_warmup_steps: 25
save_samples: 0
save_samples_ds: null
save_last: false
checkpoint_at_epoch: true
accelerate_full_state_at_epoch: true
log_level: INFO
seed: 42
mock_data: false
mock_len: 2600
distributed_training_framework: fsdp
fsdp_sharding_strategy: SHARD_GRAD_OP
use_dolomite: true
lora_r: 0
lora_alpha: 32
lora_dropout: 0.1
lora_quant_bits: null
lora_target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
max_batch_len: 60000
cpu_offload_optimizer: false
cpu_offload_params_fsdp: false
cpu_offload_optimizer_pin_memory: false
cpu_offload_optimizer_ratio: 1.0
NEFTune_alpha: null
chat_tmpl_path: /opt/app-root/lib64/python3.11/site-packages/instructlab/training/chat_templates/ibm_legacy_tmpl.py
disable_flash_attn: false
keep_last_checkpoint_only: false

{
    "script_params": {
        "model_name_or_path": "/var/home/cloud-user/.cache/instructlab/models/granite-3.1-8b-starter-v1",
        "data_path": "/var/home/cloud-user/.local/share/instructlab/internal/data.jsonl",
        "output_dir": "/var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints",
        "num_epochs": 7,
        "current_epoch": 0,
        "last_step": 0,
        "effective_batch_size": 128,
        "learning_rate": 2e-05,
        "lr_scheduler": "cosine",
        "num_warmup_steps": 25,
        "save_samples": 0,
        "save_samples_ds": null,
        "save_last": false,
        "checkpoint_at_epoch": true,
        "accelerate_full_state_at_epoch": true,
        "log_level": "INFO",
        "seed": 42,
        "mock_data": false,
        "mock_len": 2600,
        "distributed_training_framework": "fsdp",
        "fsdp_sharding_strategy": "SHARD_GRAD_OP",
        "use_dolomite": true,
        "lora_r": 0,
        "lora_alpha": 32,
        "lora_dropout": 0.1,
        "lora_quant_bits": null,
        "lora_target_modules": [
            "q_proj",
            "k_proj",
            "v_proj",
            "o_proj"
        ],
        "max_batch_len": 60000,
        "cpu_offload_optimizer": false,
        "cpu_offload_params_fsdp": false,
        "cpu_offload_optimizer_pin_memory": false,
        "cpu_offload_optimizer_ratio": 1.0,
        "NEFTune_alpha": null,
        "chat_tmpl_path": "/opt/app-root/lib64/python3.11/site-packages/instructlab/training/chat_templates/ibm_legacy_tmpl.py",
        "disable_flash_attn": false,
        "keep_last_checkpoint_only": false
    },
    "timestamp": "2025-03-10T19:28:06.714394"
}

Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 2424 examples [00:00, 15825.25 examples/s]
Generating train split: 4634 examples [00:00, 15312.23 examples/s]
Generating train split: 5090 examples [00:00, 15016.01 examples/s]
You are using a model of type granite to instantiate a model of type gpt_dolomite. This is not supported for all configurations of models and can yield errors.
You are using a model of type granite to instantiate a model of type gpt_dolomite. This is not supported for all configurations of models and can yield errors.
Model saved in /var/home/cloud-user/.cache/instructlab/models/granite-3.1-8b-starter-v1 requires conversion 
{
    "num_gpus": 8,
    "avg_sample_len": 499.860510805501,
    "effective_batch_size": 128,
    "max_batch_len_per_gpu": 60000,
    "packing_max_batch_len": 7997,
    "grad_accum": 1,
    "num_batches": 40,
    "avg_samples_per_batch": 127.25,
    "samples_per_gpu": 16,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:28:23.623489"
}
You are using a model of type granite to instantiate a model of type gpt_dolomite. This is not supported for all configurations of models and can yield errors.
You are using a model of type granite to instantiate a model of type gpt_dolomite. This is not supported for all configurations of models and can yield errors.
You are using a model of type granite to instantiate a model of type gpt_dolomite. This is not supported for all configurations of models and can yield errors.
You are using a model of type granite to instantiate a model of type gpt_dolomite. This is not supported for all configurations of models and can yield errors.
You are using a model of type granite to instantiate a model of type gpt_dolomite. This is not supported for all configurations of models and can yield errors.
You are using a model of type granite to instantiate a model of type gpt_dolomite. This is not supported for all configurations of models and can yield errors.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
CausalLMModelMixin has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
CausalLMModelMixin has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
CausalLMModelMixin has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
CausalLMModelMixin has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
CausalLMModelMixin has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
CausalLMModelMixin has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
CausalLMModelMixin has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`.
CausalLMModelMixin has generative capabilities, as `prepare_inputs_for_generation` is explicitly overwritten. However, it doesn't directly inherit from `GenerationMixin`. From 👉v4.50👈 onwards, `PreTrainedModel` will NOT inherit from `GenerationMixin`, and this model will lose the ability to call `generate` and other related functions.
  - If you're using `trust_remote_code=True`, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
  - If you are the owner of the model architecture code, please modify your model class such that it inherits from `GenerationMixin` (after `PreTrainedModel`, otherwise you'll get an exception).
  - If you are not the owner of the model architecture class, please contact the model code owner to update it.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  4.00it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.99it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.91it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.87it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.73it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:00,  3.46it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:01,  2.97it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:00<00:01,  2.81it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.75it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.62it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.24it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.62it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.54it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.27it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  4.01it/s]
Loading checkpoint shards:  50%|█████     | 2/4 [00:00<00:00,  3.87it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  5.08it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.93it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.95it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.96it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.54it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.72it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.54it/s]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:00<00:00,  4.44it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.31it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.07it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.14it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.92it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.18it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.94it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.24it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.95it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  5.02it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.71it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.75it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.58it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.85it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.47it/s]

Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.82it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:00<00:00,  4.38it/s]
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/wrap.py:492: FutureWarning: Please specify CheckpointImpl.NO_REENTRANT as CheckpointImpl.REENTRANT will soon be removed as the default and eventually deprecated.
  return wrapper_cls(module, **kwargs)
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/wrap.py:492: FutureWarning: Please specify CheckpointImpl.NO_REENTRANT as CheckpointImpl.REENTRANT will soon be removed as the default and eventually deprecated.
  return wrapper_cls(module, **kwargs)
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/wrap.py:492: FutureWarning: Please specify CheckpointImpl.NO_REENTRANT as CheckpointImpl.REENTRANT will soon be removed as the default and eventually deprecated.
  return wrapper_cls(module, **kwargs)
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/wrap.py:492: FutureWarning: Please specify CheckpointImpl.NO_REENTRANT as CheckpointImpl.REENTRANT will soon be removed as the default and eventually deprecated.
  return wrapper_cls(module, **kwargs)
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/wrap.py:492: FutureWarning: Please specify CheckpointImpl.NO_REENTRANT as CheckpointImpl.REENTRANT will soon be removed as the default and eventually deprecated.
  return wrapper_cls(module, **kwargs)
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/wrap.py:492: FutureWarning: Please specify CheckpointImpl.NO_REENTRANT as CheckpointImpl.REENTRANT will soon be removed as the default and eventually deprecated.
  return wrapper_cls(module, **kwargs)
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/wrap.py:492: FutureWarning: Please specify CheckpointImpl.NO_REENTRANT as CheckpointImpl.REENTRANT will soon be removed as the default and eventually deprecated.
  return wrapper_cls(module, **kwargs)
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/wrap.py:492: FutureWarning: Please specify CheckpointImpl.NO_REENTRANT as CheckpointImpl.REENTRANT will soon be removed as the default and eventually deprecated.
  return wrapper_cls(module, **kwargs)

Epoch 0:   0%|          | 0/40 [00:00<?, ?it/s] total length: 7994 dropped: 0 num samples 16 - rank: 4 max len: 522 min len: 476 avg len: 499.625 num_loss_counted_tokens: 7146 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 540 min len: 356 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7996 dropped: 0 num samples 17 - rank: 4 max len: 494 min len: 408 avg len: 470.3529411764706 num_loss_counted_tokens: 7095

 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 542 min len: 310 avg len: 499.75 num_loss_counted_tokens: 7148
 total length: 7835 dropped: 0 num samples 16 - rank: 4 max len: 518 min len: 472 avg len: 489.6875 num_loss_counted_tokens: 6987
 total length: 7935 dropped: 0 num samples 15 - rank: 4 max len: 549 min len: 508 avg len: 529.0 num_loss_counted_tokens: 7140
 total length: 7997 dropped: 0 num samples 17 - rank: 4 max len: 508 min len: 269 avg len: 470.4117647058824 num_loss_counted_tokens: 7096
 total length: 7921 dropped: 0 num samples 16 - rank: 4 max len: 526 min len: 469 avg len: 495.0625 num_loss_counted_tokens: 7073
 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 522 min len: 366 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7996 dropped: 0 num samples 17 - rank: 4 max len: 500 min len: 315 avg len: 470.3529411764706 num_loss_counted_tokens: 7095
 total length: 7918 dropped: 0 num samples 16 - rank: 4 max len: 510 min len: 481 avg len: 494.875 num_loss_counted_tokens: 7070
 total length: 7895 dropped: 0 num samples 16 - rank: 4 max len: 523 min len: 470 avg len: 493.4375 num_loss_counted_tokens: 7047
 total length: 7883 dropped: 0 num samples 17 - rank: 4 max len: 493 min len: 439 avg len: 463.70588235294116 num_loss_counted_tokens: 6982
 total length: 7993 dropped: 0 num samples 17 - rank: 4 max len: 516 min len: 297 avg len: 470.1764705882353 num_loss_counted_tokens: 7092
 total length: 7991 dropped: 0 num samples 16 - rank: 4 max len: 534 min len: 359 avg len: 499.4375 num_loss_counted_tokens: 7143
 total length: 7994 dropped: 0 num samples 16 - rank: 4 max len: 517 min len: 377 avg len: 499.625 num_loss_counted_tokens: 7146
 total length: 7983 dropped: 0 num samples 15 - rank: 4 max len: 574 min len: 325 avg len: 532.2 num_loss_counted_tokens: 7188
 total length: 7890 dropped: 0 num samples 10 - rank: 0 max len: 1029 min len: 675 avg len: 789.0 num_loss_counted_tokens: 7360
 total length: 7967 dropped: 0 num samples 10 - rank: 0 max len: 1143 min len: 172 avg len: 796.7 num_loss_counted_tokens: 7437
 total length: 7979 dropped: 0 num samples 9 - rank: 0 max len: 1140 min len: 645 avg len: 886.5555555555555 num_loss_counted_tokens: 7502
 total length: 7922 dropped: 0 num samples 9 - rank: 0 max len: 1066 min len: 677 avg len: 880.2222222222222 num_loss_counted_tokens: 7445
 total length: 7995 dropped: 0 num samples 9 - rank: 0 max len: 1096 min len: 340 avg len: 888.3333333333334 num_loss_counted_tokens: 7518
 total length: 7841 dropped: 0 num samples 9 - rank: 0 max len: 1060 min len: 744 avg len: 871.2222222222222 num_loss_counted_tokens: 7364
 total length: 7994 dropped: 0 num samples 9 - rank: 0 max len: 1110 min len: 414 avg len: 888.2222222222222 num_loss_counted_tokens: 7517
 total length: 7984 dropped: 0 num samples 10 - rank: 0 max len: 1391 min len: 174 avg len: 798.4 num_loss_counted_tokens: 7454
 total length: 7870 dropped: 0 num samples 9 - rank: 0 max len: 1482 min len: 715 avg len: 874.4444444444445 num_loss_counted_tokens: 7393
 total length: 7996 dropped: 0 num samples 11 - rank: 0 max len: 881 min len: 509 avg len: 726.9090909090909 num_loss_counted_tokens: 7413
 total length: 7867 dropped: 0 num samples 8 - rank: 0 max len: 1164 min len: 735 avg len: 983.375 num_loss_counted_tokens: 7443
 total length: 7985 dropped: 0 num samples 9 - rank: 0 max len: 1106 min len: 716 avg len: 887.2222222222222 num_loss_counted_tokens: 7508
 total length: 7988 dropped: 0 num samples 9 - rank: 0 max len: 1196 min len: 266 avg len: 887.5555555555555 num_loss_counted_tokens: 7511
 total length: 7941 dropped: 0 num samples 9 - rank: 0 max len: 1104 min len: 685 avg len: 882.3333333333334 num_loss_counted_tokens: 7464
 total length: 7994 dropped: 0 num samples 9 - rank: 0 max len: 1079 min len: 292 avg len: 888.2222222222222 num_loss_counted_tokens: 7517
 total length: 7989 dropped: 0 num samples 9 - rank: 0 max len: 1094 min len: 361 avg len: 887.6666666666666 num_loss_counted_tokens: 7512
 total length: 7979 dropped: 0 num samples 9 - rank: 0 max len: 1062 min len: 729 avg len: 886.5555555555555 num_loss_counted_tokens: 7502
 total length: 7781 dropped: 0 num samples 19 - rank: 6 max len: 433 min len: 380 avg len: 409.5263157894737 num_loss_counted_tokens: 6774
 total length: 7988 dropped: 0 num samples 20 - rank: 6 max len: 431 min len: 353 avg len: 399.4 num_loss_counted_tokens: 6928
 total length: 7966 dropped: 0 num samples 20 - rank: 6 max len: 422 min len: 291 avg len: 398.3 num_loss_counted_tokens: 6906
 total length: 7959 dropped: 0 num samples 20 - rank: 6 max len: 414 min len: 382 avg len: 397.95 num_loss_counted_tokens: 6899
 total length: 7853 dropped: 0 num samples 19 - rank: 6 max len: 452 min len: 386 avg len: 413.3157894736842 num_loss_counted_tokens: 6846
 total length: 7987 dropped: 0 num samples 21 - rank: 6 max len: 404 min len: 353 avg len: 380.3333333333333 num_loss_counted_tokens: 6874
 total length: 7978 dropped: 0 num samples 21 - rank: 6 max len: 411 min len: 233 avg len: 379.9047619047619 num_loss_counted_tokens: 6865
 total length: 7913 dropped: 0 num samples 20 - rank: 6 max len: 425 min len: 375 avg len: 395.65 num_loss_counted_tokens: 6853
 total length: 7895 dropped: 0 num samples 20 - rank: 6 max len: 415 min len: 375 avg len: 394.75 num_loss_counted_tokens: 6835
 total length: 7997 dropped: 0 num samples 20 - rank: 6 max len: 423 min len: 292 avg len: 399.85 num_loss_counted_tokens: 6937
 total length: 7850 dropped: 0 num samples 19 - rank: 6 max len: 442 min len: 385 avg len: 413.1578947368421 num_loss_counted_tokens: 6843
 total length: 7963 dropped: 0 num samples 21 - rank: 6 max len: 412 min len: 171 avg len: 379.1904761904762 num_loss_counted_tokens: 6850
 total length: 7996 dropped: 0 num samples 20 - rank: 6 max len: 438 min len: 364 avg len: 399.8 num_loss_counted_tokens: 6936
 total length: 7994 dropped: 0 num samples 21 - rank: 6 max len: 411 min len: 332 avg len: 380.6666666666667 num_loss_counted_tokens: 6881
 total length: 7996 dropped: 0 num samples 22 - rank: 6 max len: 391 min len: 334 avg len: 363.45454545454544 num_loss_counted_tokens: 6830
 total length: 7814 dropped: 0 num samples 20 - rank: 6 max len: 414 min len: 373 avg len: 390.7 num_loss_counted_tokens: 6754
 total length: 7991 dropped: 0 num samples 19 - rank: 6 max len: 477 min len: 300 avg len: 420.57894736842104 num_loss_counted_tokens: 6984
 total length: 7962 dropped: 0 num samples 25 - rank: 7 max len: 371 min len: 255 avg len: 318.48 num_loss_counted_tokens: 6637
 total length: 7287 dropped: 0 num samples 22 - rank: 7 max len: 382 min len: 248 avg len: 331.22727272727275 num_loss_counted_tokens: 6121
 total length: 7765 dropped: 0 num samples 23 - rank: 7 max len: 381 min len: 253 avg len: 337.60869565217394 num_loss_counted_tokens: 6546
 total length: 7467 dropped: 0 num samples 23 - rank: 7 max len: 379 min len: 253 avg len: 324.6521739130435 num_loss_counted_tokens: 6248
 total length: 7885 dropped: 0 num samples 24 - rank: 7 max len: 359 min len: 280 avg len: 328.5416666666667 num_loss_counted_tokens: 6613
 total length: 7810 dropped: 0 num samples 25 - rank: 7 max len: 371 min len: 236 avg len: 312.4 num_loss_counted_tokens: 6485
 total length: 7896 dropped: 0 num samples 25 - rank: 7 max len: 374 min len: 164 avg len: 315.84 num_loss_counted_tokens: 6571
 total length: 7584 dropped: 0 num samples 24 - rank: 7 max len: 353 min len: 261 avg len: 316.0 num_loss_counted_tokens: 6312
 total length: 7963 dropped: 0 num samples 25 - rank: 7 max len: 361 min len: 257 avg len: 318.52 num_loss_counted_tokens: 6638
 total length: 7994 dropped: 0 num samples 25 - rank: 7 max len: 362 min len: 266 avg len: 319.76 num_loss_counted_tokens: 6669
 total length: 7916 dropped: 0 num samples 24 - rank: 7 max len: 385 min len: 250 avg len: 329.8333333333333 num_loss_counted_tokens: 6644
 total length: 7870 dropped: 0 num samples 24 - rank: 7 max len: 371 min len: 253 avg len: 327.9166666666667 num_loss_counted_tokens: 6598
 total length: 7402 dropped: 0 num samples 23 - rank: 7 max len: 359 min len: 257 avg len: 321.82608695652175 num_loss_counted_tokens: 6183
 total length: 7992 dropped: 0 num samples 25 - rank: 7 max len: 372 min len: 263 avg len: 319.68 num_loss_counted_tokens: 6667
 total length: 7830 dropped: 0 num samples 24 - rank: 7 max len: 387 min len: 244 avg len: 326.25 num_loss_counted_tokens: 6558
 total length: 7992 dropped: 0 num samples 26 - rank: 7 max len: 344 min len: 191 avg len: 307.38461538461536 num_loss_counted_tokens: 6614
 total length: 7573 dropped: 0 num samples 21 - rank: 7 max len: 408 min len: 262 avg len: 360.6190476190476 num_loss_counted_tokens: 6460
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 475 min len: 304 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7762 dropped: 0 num samples 17 - rank: 5 max len: 470 min len: 434 avg len: 456.5882352941176 num_loss_counted_tokens: 6861
 total length: 7996 dropped: 0 num samples 18 - rank: 5 max len: 475 min len: 318 avg len: 444.22222222222223 num_loss_counted_tokens: 7042
 total length: 7912 dropped: 0 num samples 18 - rank: 5 max len: 463 min len: 409 avg len: 439.55555555555554 num_loss_counted_tokens: 6958
 total length: 7896 dropped: 0 num samples 18 - rank: 5 max len: 456 min len: 419 avg len: 438.6666666666667 num_loss_counted_tokens: 6942
 total length: 7987 dropped: 0 num samples 17 - rank: 5 max len: 508 min len: 290 avg len: 469.8235294117647 num_loss_counted_tokens: 7086
 total length: 7996 dropped: 0 num samples 18 - rank: 5 max len: 469 min len: 426 avg len: 444.22222222222223 num_loss_counted_tokens: 7042
 total length: 7905 dropped: 0 num samples 18 - rank: 5 max len: 453 min len: 424 avg len: 439.1666666666667 num_loss_counted_tokens: 6951
 total length: 7824 dropped: 0 num samples 18 - rank: 5 max len: 457 min len: 413 avg len: 434.6666666666667 num_loss_counted_tokens: 6870
 total length: 7891 dropped: 0 num samples 19 - rank: 5 max len: 437 min len: 391 avg len: 415.3157894736842 num_loss_counted_tokens: 6884
 total length: 7922 dropped: 0 num samples 17 - rank: 5 max len: 480 min len: 448 avg len: 466.0 num_loss_counted_tokens: 7021
 total length: 7946 dropped: 0 num samples 17 - rank: 5 max len: 492 min len: 440 avg len: 467.4117647058824 num_loss_counted_tokens: 7045
 total length: 7983 dropped: 0 num samples 18 - rank: 5 max len: 465 min len: 417 avg len: 443.5 num_loss_counted_tokens: 7029
 total length: 7997 dropped: 0 num samples 16 - rank: 5 max len: 524 min len: 469 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7985 dropped: 0 num samples 18 - rank: 5 max len: 482 min len: 298 avg len: 443.6111111111111 num_loss_counted_tokens: 7031
 total length: 7990 dropped: 0 num samples 18 - rank: 5 max len: 477 min len: 356 avg len: 443.8888888888889 num_loss_counted_tokens: 7036
 total length: 7996 dropped: 0 num samples 19 - rank: 5 max len: 443 min len: 349 avg len: 420.8421052631579 num_loss_counted_tokens: 6989
 total length: 7916 dropped: 0 num samples 14 - rank: 3 max len: 582 min len: 541 avg len: 565.4285714285714 num_loss_counted_tokens: 7174
 total length: 7794 dropped: 0 num samples 14 - rank: 3 max len: 584 min len: 524 avg len: 556.7142857142857 num_loss_counted_tokens: 7052
 total length: 7956 dropped: 0 num samples 14 - rank: 3 max len: 611 min len: 542 avg len: 568.2857142857143 num_loss_counted_tokens: 7214 total length: 7989 dropped: 0 num samples 14 - rank: 3 max len: 620 min len: 468 avg len: 570.6428571428571 num_loss_counted_tokens: 7247

 total length: 7995 dropped: 0 num samples 15 - rank: 3 max len: 558 min len: 449 avg len: 533.0 num_loss_counted_tokens: 7200
 total length: 7958 dropped: 0 num samples 15 - rank: 3 max len: 578 min len: 496 avg len: 530.5333333333333 num_loss_counted_tokens: 7163
 total length: 7991 dropped: 0 num samples 15 - rank: 3 max len: 571 min len: 474 avg len: 532.7333333333333 num_loss_counted_tokens: 7196
 total length: 7917 dropped: 0 num samples 14 - rank: 3 max len: 609 min len: 540 avg len: 565.5 num_loss_counted_tokens: 7175
 total length: 7995 dropped: 0 num samples 15 - rank: 3 max len: 561 min len: 439 avg len: 533.0 num_loss_counted_tokens: 7200
 total length: 7961 dropped: 0 num samples 15 - rank: 3 max len: 565 min len: 505 avg len: 530.7333333333333 num_loss_counted_tokens: 7166
 total length: 7983 dropped: 0 num samples 15 - rank: 3 max len: 569 min len: 301 avg len: 532.2 num_loss_counted_tokens: 7188
 total length: 7910 dropped: 0 num samples 14 - rank: 3 max len: 600 min len: 528 avg len: 565.0 num_loss_counted_tokens: 7168
 total length: 7995 dropped: 0 num samples 15 - rank: 3 max len: 571 min len: 365 avg len: 533.0 num_loss_counted_tokens: 7200
 total length: 7997 dropped: 0 num samples 16 - rank: 3 max len: 544 min len: 353 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7979 dropped: 0 num samples 15 - rank: 3 max len: 579 min len: 276 avg len: 531.9333333333333 num_loss_counted_tokens: 7184
 total length: 7801 dropped: 0 num samples 14 - rank: 3 max len: 590 min len: 519 avg len: 557.2142857142857 num_loss_counted_tokens: 7059
 total length: 7839 dropped: 0 num samples 13 - rank: 3 max len: 625 min len: 579 avg len: 603.0 num_loss_counted_tokens: 7150
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 742 min len: 515 avg len: 666.3333333333334 num_loss_counted_tokens: 7360 total length: 7805 dropped: 0 num samples 12 - rank: 1 max len: 668 min len: 633 avg len: 650.4166666666666 num_loss_counted_tokens: 7169

 total length: 7796 dropped: 0 num samples 12 - rank: 1 max len: 677 min len: 631 avg len: 649.6666666666666 num_loss_counted_tokens: 7160
 total length: 7994 dropped: 0 num samples 12 - rank: 1 max len: 743 min len: 460 avg len: 666.1666666666666 num_loss_counted_tokens: 7358
 total length: 7992 dropped: 0 num samples 12 - rank: 1 max len: 723 min len: 583 avg len: 666.0 num_loss_counted_tokens: 7356
 total length: 7995 dropped: 0 num samples 13 - rank: 1 max len: 673 min len: 276 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7994 dropped: 0 num samples 12 - rank: 1 max len: 710 min len: 616 avg len: 666.1666666666666 num_loss_counted_tokens: 7358
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 745 min len: 431 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
 total length: 7785 dropped: 0 num samples 12 - rank: 1 max len: 683 min len: 623 avg len: 648.75 num_loss_counted_tokens: 7149
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 833 min len: 433 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7993 dropped: 0 num samples 12 - rank: 1 max len: 714 min len: 583 avg len: 666.0833333333334 num_loss_counted_tokens: 7357
 total length: 7983 dropped: 0 num samples 13 - rank: 1 max len: 661 min len: 241 avg len: 614.0769230769231 num_loss_counted_tokens: 7294
 total length: 7994 dropped: 0 num samples 13 - rank: 1 max len: 684 min len: 243 avg len: 614.9230769230769 num_loss_counted_tokens: 7305
 total length: 7983 dropped: 0 num samples 12 - rank: 1 max len: 751 min len: 463 avg len: 665.25 num_loss_counted_tokens: 7347
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 718 min len: 554 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
 total length: 7850 dropped: 0 num samples 12 - rank: 1 max len: 707 min len: 630 avg len: 654.1666666666666 num_loss_counted_tokens: 7214
 total length: 7866 dropped: 0 num samples 11 - rank: 1 max len: 827 min len: 665 avg len: 715.0909090909091 num_loss_counted_tokens: 7283
 total length: 7958 dropped: 0 num samples 13 - rank: 2 max len: 627 min len: 585 avg len: 612.1538461538462 num_loss_counted_tokens: 7269
 total length: 7894 dropped: 0 num samples 13 - rank: 2 max len: 631 min len: 586 avg len: 607.2307692307693 num_loss_counted_tokens: 7205
 total length: 7910 dropped: 0 num samples 13 - rank: 2 max len: 633 min len: 586 avg len: 608.4615384615385 num_loss_counted_tokens: 7221
 total length: 7994 dropped: 0 num samples 13 - rank: 2 max len: 646 min len: 475 avg len: 614.9230769230769 num_loss_counted_tokens: 7305
 total length: 7995 dropped: 0 num samples 13 - rank: 2 max len: 648 min len: 373 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7795 dropped: 0 num samples 13 - rank: 2 max len: 628 min len: 571 avg len: 599.6153846153846 num_loss_counted_tokens: 7106
 total length: 7983 dropped: 0 num samples 13 - rank: 2 max len: 641 min len: 574 avg len: 614.0769230769231 num_loss_counted_tokens: 7294
 total length: 7984 dropped: 0 num samples 13 - rank: 2 max len: 633 min len: 544 avg len: 614.1538461538462 num_loss_counted_tokens: 7295
 total length: 7995 dropped: 0 num samples 14 - rank: 2 max len: 627 min len: 265 avg len: 571.0714285714286 num_loss_counted_tokens: 7253
 total length: 7995 dropped: 0 num samples 14 - rank: 2 max len: 623 min len: 288 avg len: 571.0714285714286 num_loss_counted_tokens: 7253
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 645 min len: 461 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7992 dropped: 0 num samples 14 - rank: 2 max len: 627 min len: 266 avg len: 570.8571428571429 num_loss_counted_tokens: 7250
 total length: 7884 dropped: 0 num samples 13 - rank: 2 max len: 629 min len: 574 avg len: 606.4615384615385 num_loss_counted_tokens: 7195
 total length: 7961 dropped: 0 num samples 13 - rank: 2 max len: 649 min len: 582 avg len: 612.3846153846154 num_loss_counted_tokens: 7272
 total length: 7994 dropped: 0 num samples 14 - rank: 2 max len: 606 min len: 534 avg len: 571.0 num_loss_counted_tokens: 7252
 total length: 7987 dropped: 0 num samples 13 - rank: 2 max len: 643 min len: 526 avg len: 614.3846153846154 num_loss_counted_tokens: 7298
 total length: 7969 dropped: 0 num samples 13 - rank: 2 max len: 665 min len: 268 avg len: 613.0 num_loss_counted_tokens: 7280
Epoch: 0, Step: 1, Rank: 0, loss = 0.51953125Epoch: 0, Step: 1, Rank: 4, loss = 1.0859375Epoch: 0, Step: 1, Rank: 3, loss = 0.66796875

Epoch: 0, Step: 1, Rank: 6, loss = 1.2421875Epoch: 0, Step: 1, Rank: 5, loss = 1.0859375


Epoch: 0, Step: 1, Rank: 2, loss = 0.703125
Epoch: 0, Step: 1, Rank: 7, loss = 1.4375
Epoch: 0, Step: 1, Rank: 1, loss = 0.68359375

Epoch 0:   2%|▎         | 1/40 [00:04<03:11,  4.91s/it]{
    "epoch": 0,
    "step": 1,
    "rank": 0,
    "overall_throughput": 26.830295458587777,
    "lr": 8.000000000000001e-07,
    "cuda_mem_allocated": 6.578585147857666,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56404,
    "batch_size": 128,
    "total_loss": 0.9284447911495639,
    "samples_seen": 128,
    "gradnorm": 17.375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:28:50.241021"
}
 total length: 7791 dropped: 0 num samples 13 - rank: 2 max len: 630 min len: 570 avg len: 599.3076923076923 num_loss_counted_tokens: 7102
 total length: 7975 dropped: 0 num samples 13 - rank: 1 max len: 676 min len: 172 avg len: 613.4615384615385 num_loss_counted_tokens: 7286 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 563 min len: 457 avg len: 533.1333333333333 num_loss_counted_tokens: 7202

 total length: 7842 dropped: 0 num samples 24 - rank: 7 max len: 370 min len: 267 avg len: 326.75 num_loss_counted_tokens: 6570
 total length: 7817 dropped: 0 num samples 18 - rank: 5 max len: 454 min len: 412 avg len: 434.27777777777777 num_loss_counted_tokens: 6863
 total length: 7996 dropped: 0 num samples 10 - rank: 0 max len: 1068 min len: 565 avg len: 799.6 num_loss_counted_tokens: 7466
 total length: 7824 dropped: 0 num samples 20 - rank: 6 max len: 408 min len: 370 avg len: 391.2 num_loss_counted_tokens: 6764
 total length: 7997 dropped: 0 num samples 17 - rank: 4 max len: 498 min len: 411 avg len: 470.4117647058824 num_loss_counted_tokens: 7096
Epoch: 0, Step: 2, Rank: 5, loss = 0.7890625Epoch: 0, Step: 2, Rank: 4, loss = 0.84765625Epoch: 0, Step: 2, Rank: 6, loss = 1.0703125
Epoch: 0, Step: 2, Rank: 2, loss = 0.859375
Epoch: 0, Step: 2, Rank: 1, loss = 0.65625


Epoch: 0, Step: 2, Rank: 7, loss = 1.453125Epoch: 0, Step: 2, Rank: 3, loss = 0.8984375

Epoch: 0, Step: 2, Rank: 0, loss = 0.451171875

Epoch 0:   5%|▌         | 2/40 [00:06<01:52,  2.96s/it]{
    "epoch": 0,
    "step": 2,
    "rank": 0,
    "overall_throughput": 94.74865837525395,
    "lr": 1.6000000000000001e-06,
    "cuda_mem_allocated": 6.586202621459961,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56726,
    "batch_size": 125,
    "total_loss": 0.8777632831505835,
    "samples_seen": 253,
    "gradnorm": 16.0,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:28:51.838201"
}
 total length: 7995 dropped: 0 num samples 13 - rank: 2 max len: 648 min len: 418 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7982 dropped: 0 num samples 12 - rank: 1 max len: 720 min len: 514 avg len: 665.1666666666666 num_loss_counted_tokens: 7346
 total length: 7956 dropped: 0 num samples 17 - rank: 4 max len: 506 min len: 180 avg len: 468.0 num_loss_counted_tokens: 7055
 total length: 7762 dropped: 0 num samples 25 - rank: 7 max len: 360 min len: 250 avg len: 310.48 num_loss_counted_tokens: 6437
 total length: 7992 dropped: 0 num samples 9 - rank: 0 max len: 1094 min len: 291 avg len: 888.0 num_loss_counted_tokens: 7515
 total length: 7953 dropped: 0 num samples 14 - rank: 3 max len: 600 min len: 510 avg len: 568.0714285714286 num_loss_counted_tokens: 7211
 total length: 7994 dropped: 0 num samples 21 - rank: 6 max len: 405 min len: 279 avg len: 380.6666666666667 num_loss_counted_tokens: 6881
 total length: 7933 dropped: 0 num samples 18 - rank: 5 max len: 462 min len: 412 avg len: 440.72222222222223 num_loss_counted_tokens: 6979
Epoch: 0, Step: 3, Rank: 4, loss = 0.87890625Epoch: 0, Step: 3, Rank: 5, loss = 0.98828125Epoch: 0, Step: 3, Rank: 1, loss = 0.51171875Epoch: 0, Step: 3, Rank: 7, loss = 1.59375
Epoch: 0, Step: 3, Rank: 3, loss = 0.9375

Epoch: 0, Step: 3, Rank: 6, loss = 1.203125


Epoch: 0, Step: 3, Rank: 2, loss = 0.625
Epoch: 0, Step: 3, Rank: 0, loss = 0.67578125

Epoch 0:   8%|▊         | 3/40 [00:08<01:27,  2.36s/it]{
    "epoch": 0,
    "step": 3,
    "rank": 0,
    "overall_throughput": 95.28609331585706,
    "lr": 2.4000000000000003e-06,
    "cuda_mem_allocated": 6.586108207702637,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56447,
    "batch_size": 128,
    "total_loss": 0.9280209754282779,
    "samples_seen": 381,
    "gradnorm": 17.875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:28:53.474998"
}
 total length: 7882 dropped: 0 num samples 11 - rank: 1 max len: 841 min len: 648 avg len: 716.5454545454545 num_loss_counted_tokens: 7299
 total length: 7997 dropped: 0 num samples 14 - rank: 3 max len: 611 min len: 310 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7899 dropped: 0 num samples 19 - rank: 6 max len: 454 min len: 392 avg len: 415.7368421052632 num_loss_counted_tokens: 6892
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 648 min len: 461 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7990 dropped: 0 num samples 9 - rank: 0 max len: 1104 min len: 493 avg len: 887.7777777777778 num_loss_counted_tokens: 7513
 total length: 7771 dropped: 0 num samples 23 - rank: 7 max len: 391 min len: 279 avg len: 337.8695652173913 num_loss_counted_tokens: 6552
 total length: 7992 dropped: 0 num samples 15 - rank: 4 max len: 571 min len: 329 avg len: 532.8 num_loss_counted_tokens: 7197
 total length: 7820 dropped: 0 num samples 16 - rank: 5 max len: 523 min len: 455 avg len: 488.75 num_loss_counted_tokens: 6972
Epoch: 0, Step: 4, Rank: 6, loss = 1.1171875Epoch: 0, Step: 4, Rank: 4, loss = 0.97265625Epoch: 0, Step: 4, Rank: 1, loss = 0.7109375Epoch: 0, Step: 4, Rank: 2, loss = 0.57421875Epoch: 0, Step: 4, Rank: 5, loss = 1.09375

Epoch: 0, Step: 4, Rank: 7, loss = 1.296875
Epoch: 0, Step: 4, Rank: 3, loss = 0.7734375



Epoch: 0, Step: 4, Rank: 0, loss = 0.734375

Epoch 0:  10%|█         | 4/40 [00:09<01:14,  2.08s/it]{
    "epoch": 0,
    "step": 4,
    "rank": 0,
    "overall_throughput": 94.57164106276875,
    "lr": 3.2000000000000003e-06,
    "cuda_mem_allocated": 6.588102340698242,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56612,
    "batch_size": 124,
    "total_loss": 0.9083586518759273,
    "samples_seen": 505,
    "gradnorm": 17.25,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:28:55.119824"
}
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 719 min len: 577 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7991 dropped: 0 num samples 25 - rank: 7 max len: 367 min len: 256 avg len: 319.64 num_loss_counted_tokens: 6666
 total length: 7992 dropped: 0 num samples 9 - rank: 0 max len: 1104 min len: 256 avg len: 888.0 num_loss_counted_tokens: 7515
 total length: 7996 dropped: 0 num samples 15 - rank: 3 max len: 570 min len: 271 avg len: 533.0666666666667 num_loss_counted_tokens: 7201
 total length: 7992 dropped: 0 num samples 19 - rank: 5 max len: 454 min len: 249 avg len: 420.63157894736844 num_loss_counted_tokens: 6985
 total length: 7985 dropped: 0 num samples 16 - rank: 4 max len: 526 min len: 467 avg len: 499.0625 num_loss_counted_tokens: 7137
 total length: 7982 dropped: 0 num samples 21 - rank: 6 max len: 397 min len: 308 avg len: 380.0952380952381 num_loss_counted_tokens: 6869
 total length: 7962 dropped: 0 num samples 13 - rank: 2 max len: 646 min len: 571 avg len: 612.4615384615385 num_loss_counted_tokens: 7273
Epoch: 0, Step: 5, Rank: 0, loss = 0.62890625Epoch: 0, Step: 5, Rank: 5, loss = 1.140625Epoch: 0, Step: 5, Rank: 4, loss = 0.9375Epoch: 0, Step: 5, Rank: 3, loss = 0.8046875Epoch: 0, Step: 5, Rank: 7, loss = 1.296875


Epoch: 0, Step: 5, Rank: 2, loss = 0.72265625
Epoch: 0, Step: 5, Rank: 6, loss = 1.2578125


Epoch: 0, Step: 5, Rank: 1, loss = 0.6484375

Epoch 0:  12%|█▎        | 5/40 [00:11<01:07,  1.92s/it]{
    "epoch": 0,
    "step": 5,
    "rank": 0,
    "overall_throughput": 95.16542859822006,
    "lr": 4.000000000000001e-06,
    "cuda_mem_allocated": 6.574263095855713,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56345,
    "batch_size": 128,
    "total_loss": 0.9291330197888011,
    "samples_seen": 633,
    "gradnorm": 17.875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:28:56.757708"
}
 total length: 7878 dropped: 0 num samples 25 - rank: 7 max len: 370 min len: 236 avg len: 315.12 num_loss_counted_tokens: 6553
 total length: 7997 dropped: 0 num samples 10 - rank: 0 max len: 1071 min len: 339 avg len: 799.7 num_loss_counted_tokens: 7467
 total length: 7846 dropped: 0 num samples 13 - rank: 2 max len: 631 min len: 581 avg len: 603.5384615384615 num_loss_counted_tokens: 7157
 total length: 7975 dropped: 0 num samples 20 - rank: 6 max len: 423 min len: 374 avg len: 398.75 num_loss_counted_tokens: 6915
 total length: 7822 dropped: 0 num samples 15 - rank: 4 max len: 535 min len: 498 avg len: 521.4666666666667 num_loss_counted_tokens: 7027
 total length: 7995 dropped: 0 num samples 13 - rank: 1 max len: 661 min len: 299 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7856 dropped: 0 num samples 17 - rank: 5 max len: 496 min len: 424 avg len: 462.11764705882354 num_loss_counted_tokens: 6955
 total length: 7827 dropped: 0 num samples 14 - rank: 3 max len: 579 min len: 537 avg len: 559.0714285714286 num_loss_counted_tokens: 7085
Epoch: 0, Step: 6, Rank: 5, loss = 0.99609375Epoch: 0, Step: 6, Rank: 4, loss = 0.6875Epoch: 0, Step: 6, Rank: 1, loss = 0.6484375
Epoch: 0, Step: 6, Rank: 6, loss = 1.1328125

Epoch: 0, Step: 6, Rank: 3, loss = 0.57421875Epoch: 0, Step: 6, Rank: 7, loss = 1.2890625
Epoch: 0, Step: 6, Rank: 2, loss = 0.50390625


Epoch: 0, Step: 6, Rank: 0, loss = 0.6328125

Epoch 0:  15%|█▌        | 6/40 [00:13<01:02,  1.83s/it]{
    "epoch": 0,
    "step": 6,
    "rank": 0,
    "overall_throughput": 94.43118040621854,
    "lr": 4.800000000000001e-06,
    "cuda_mem_allocated": 6.588102340698242,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56750,
    "batch_size": 122,
    "total_loss": 0.8080352422907489,
    "samples_seen": 755,
    "gradnorm": 15.0,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:28:58.402407"
}
 total length: 7619 dropped: 0 num samples 24 - rank: 7 max len: 377 min len: 182 avg len: 317.4583333333333 num_loss_counted_tokens: 6347
 total length: 7993 dropped: 0 num samples 12 - rank: 1 max len: 712 min len: 446 avg len: 666.0833333333334 num_loss_counted_tokens: 7357
 total length: 7844 dropped: 0 num samples 14 - rank: 3 max len: 618 min len: 536 avg len: 560.2857142857143 num_loss_counted_tokens: 7102
 total length: 7996 dropped: 0 num samples 9 - rank: 0 max len: 1155 min len: 620 avg len: 888.4444444444445 num_loss_counted_tokens: 7519
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 653 min len: 373 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7995 dropped: 0 num samples 16 - rank: 4 max len: 533 min len: 339 avg len: 499.6875 num_loss_counted_tokens: 7147
 total length: 7991 dropped: 0 num samples 20 - rank: 6 max len: 419 min len: 354 avg len: 399.55 num_loss_counted_tokens: 6931
 total length: 7831 dropped: 0 num samples 17 - rank: 5 max len: 485 min len: 425 avg len: 460.6470588235294 num_loss_counted_tokens: 6930
Epoch: 0, Step: 7, Rank: 1, loss = 0.61328125Epoch: 0, Step: 7, Rank: 3, loss = 0.76953125Epoch: 0, Step: 7, Rank: 7, loss = 1.2421875Epoch: 0, Step: 7, Rank: 5, loss = 0.81640625Epoch: 0, Step: 7, Rank: 6, loss = 1.203125




Epoch: 0, Step: 7, Rank: 4, loss = 0.671875Epoch: 0, Step: 7, Rank: 2, loss = 0.59765625

Epoch: 0, Step: 7, Rank: 0, loss = 0.578125

Epoch 0:  18%|█▊        | 7/40 [00:14<00:58,  1.76s/it]{
    "epoch": 0,
    "step": 7,
    "rank": 0,
    "overall_throughput": 95.17926316373581,
    "lr": 5.600000000000001e-06,
    "cuda_mem_allocated": 6.580931186676025,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56497,
    "batch_size": 130,
    "total_loss": 0.81165371612652,
    "samples_seen": 885,
    "gradnorm": 15.625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:00.036797"
}
 total length: 7994 dropped: 0 num samples 12 - rank: 1 max len: 701 min len: 647 avg len: 666.1666666666666 num_loss_counted_tokens: 7358
 total length: 7995 dropped: 0 num samples 14 - rank: 3 max len: 611 min len: 300 avg len: 571.0714285714286 num_loss_counted_tokens: 7253
 total length: 7879 dropped: 0 num samples 20 - rank: 6 max len: 413 min len: 376 avg len: 393.95 num_loss_counted_tokens: 6819
 total length: 7450 dropped: 0 num samples 23 - rank: 7 max len: 373 min len: 252 avg len: 323.9130434782609 num_loss_counted_tokens: 6231
 total length: 7800 dropped: 0 num samples 15 - rank: 4 max len: 567 min len: 485 avg len: 520.0 num_loss_counted_tokens: 7005
 total length: 7987 dropped: 0 num samples 10 - rank: 0 max len: 1068 min len: 452 avg len: 798.7 num_loss_counted_tokens: 7457
 total length: 7995 dropped: 0 num samples 18 - rank: 5 max len: 484 min len: 418 avg len: 444.1666666666667 num_loss_counted_tokens: 7041
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 651 min len: 490 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
Epoch: 0, Step: 8, Rank: 6, loss = 1.0390625Epoch: 0, Step: 8, Rank: 5, loss = 0.81640625Epoch: 0, Step: 8, Rank: 1, loss = 0.609375Epoch: 0, Step: 8, Rank: 7, loss = 1.1484375Epoch: 0, Step: 8, Rank: 3, loss = 0.47265625

Epoch: 0, Step: 8, Rank: 2, loss = 0.5234375

Epoch: 0, Step: 8, Rank: 4, loss = 0.734375


Epoch: 0, Step: 8, Rank: 0, loss = 0.44921875

Epoch 0:  20%|██        | 8/40 [00:16<00:55,  1.72s/it]{
    "epoch": 0,
    "step": 8,
    "rank": 0,
    "overall_throughput": 94.84537405445398,
    "lr": 6.4000000000000006e-06,
    "cuda_mem_allocated": 6.576383590698242,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56773,
    "batch_size": 127,
    "total_loss": 0.7248516019939055,
    "samples_seen": 1012,
    "gradnorm": 14.0625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:01.679581"
}
 total length: 7921 dropped: 0 num samples 16 - rank: 4 max len: 521 min len: 473 avg len: 495.0625 num_loss_counted_tokens: 7073
 total length: 7737 dropped: 0 num samples 10 - rank: 0 max len: 944 min len: 671 avg len: 773.7 num_loss_counted_tokens: 7207
 total length: 7981 dropped: 0 num samples 18 - rank: 5 max len: 470 min len: 410 avg len: 443.3888888888889 num_loss_counted_tokens: 7027
 total length: 7815 dropped: 0 num samples 12 - rank: 1 max len: 669 min len: 635 avg len: 651.25 num_loss_counted_tokens: 7179
 total length: 7971 dropped: 0 num samples 14 - rank: 3 max len: 598 min len: 523 avg len: 569.3571428571429 num_loss_counted_tokens: 7229
 total length: 7996 dropped: 0 num samples 21 - rank: 6 max len: 409 min len: 275 avg len: 380.76190476190476 num_loss_counted_tokens: 6883
 total length: 7595 dropped: 0 num samples 23 - rank: 7 max len: 367 min len: 280 avg len: 330.2173913043478 num_loss_counted_tokens: 6376
 total length: 7992 dropped: 0 num samples 13 - rank: 2 max len: 633 min len: 535 avg len: 614.7692307692307 num_loss_counted_tokens: 7303
Epoch: 0, Step: 9, Rank: 5, loss = 0.7890625Epoch: 0, Step: 9, Rank: 1, loss = 0.48828125
Epoch: 0, Step: 9, Rank: 4, loss = 0.76171875Epoch: 0, Step: 9, Rank: 6, loss = 0.85546875
Epoch: 0, Step: 9, Rank: 3, loss = 0.57421875
Epoch: 0, Step: 9, Rank: 7, loss = 0.93359375

Epoch: 0, Step: 9, Rank: 2, loss = 0.59375

Epoch: 0, Step: 9, Rank: 0, loss = 0.43359375

Epoch 0:  22%|██▎       | 9/40 [00:17<00:52,  1.70s/it]{
    "epoch": 0,
    "step": 9,
    "rank": 0,
    "overall_throughput": 94.24856214439824,
    "lr": 7.2000000000000005e-06,
    "cuda_mem_allocated": 6.586718559265137,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56563,
    "batch_size": 130,
    "total_loss": 0.6780404151123526,
    "samples_seen": 1142,
    "gradnorm": 11.6875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:03.325494"
}
 total length: 7848 dropped: 0 num samples 12 - rank: 1 max len: 693 min len: 626 avg len: 654.0 num_loss_counted_tokens: 7212
 total length: 7997 dropped: 0 num samples 21 - rank: 6 max len: 410 min len: 251 avg len: 380.8095238095238 num_loss_counted_tokens: 6884
 total length: 7899 dropped: 0 num samples 9 - rank: 0 max len: 1087 min len: 699 avg len: 877.6666666666666 num_loss_counted_tokens: 7422
 total length: 7993 dropped: 0 num samples 14 - rank: 2 max len: 626 min len: 202 avg len: 570.9285714285714 num_loss_counted_tokens: 7251
 total length: 7830 dropped: 0 num samples 16 - rank: 4 max len: 505 min len: 477 avg len: 489.375 num_loss_counted_tokens: 6982
 total length: 7908 dropped: 0 num samples 18 - rank: 5 max len: 469 min len: 411 avg len: 439.3333333333333 num_loss_counted_tokens: 6954
 total length: 7993 dropped: 0 num samples 15 - rank: 3 max len: 571 min len: 499 avg len: 532.8666666666667 num_loss_counted_tokens: 7198
 total length: 7951 dropped: 0 num samples 25 - rank: 7 max len: 358 min len: 259 avg len: 318.04 num_loss_counted_tokens: 6626
Epoch: 0, Step: 10, Rank: 6, loss = 0.88671875Epoch: 0, Step: 10, Rank: 7, loss = 0.9375Epoch: 0, Step: 10, Rank: 2, loss = 0.4140625Epoch: 0, Step: 10, Rank: 3, loss = 0.55859375
Epoch: 0, Step: 10, Rank: 1, loss = 0.337890625



Epoch: 0, Step: 10, Rank: 4, loss = 0.6875Epoch: 0, Step: 10, Rank: 5, loss = 0.6484375

Epoch: 0, Step: 10, Rank: 0, loss = 0.380859375

Epoch 0:  25%|██▌       | 10/40 [00:19<00:50,  1.68s/it]{
    "epoch": 0,
    "step": 10,
    "rank": 0,
    "overall_throughput": 94.6854930931753,
    "lr": 8.000000000000001e-06,
    "cuda_mem_allocated": 6.588008403778076,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56875,
    "batch_size": 129,
    "total_loss": 0.6065230769230769,
    "samples_seen": 1271,
    "gradnorm": 9.75,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:04.968742"
}
 total length: 7893 dropped: 0 num samples 10 - rank: 0 max len: 1053 min len: 685 avg len: 789.3 num_loss_counted_tokens: 7363
 total length: 7595 dropped: 0 num samples 23 - rank: 7 max len: 380 min len: 256 avg len: 330.2173913043478 num_loss_counted_tokens: 6376
 total length: 7997 dropped: 0 num samples 14 - rank: 3 max len: 604 min len: 454 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 645 min len: 430 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7986 dropped: 0 num samples 18 - rank: 5 max len: 468 min len: 328 avg len: 443.6666666666667 num_loss_counted_tokens: 7032
 total length: 7923 dropped: 0 num samples 12 - rank: 1 max len: 677 min len: 645 avg len: 660.25 num_loss_counted_tokens: 7287
 total length: 7984 dropped: 0 num samples 16 - rank: 4 max len: 554 min len: 301 avg len: 499.0 num_loss_counted_tokens: 7136
 total length: 7844 dropped: 0 num samples 19 - rank: 6 max len: 437 min len: 385 avg len: 412.8421052631579 num_loss_counted_tokens: 6837
Epoch: 0, Step: 11, Rank: 5, loss = 0.60546875Epoch: 0, Step: 11, Rank: 1, loss = 0.419921875
Epoch: 0, Step: 11, Rank: 0, loss = 0.4296875
Epoch: 0, Step: 11, Rank: 4, loss = 0.546875Epoch: 0, Step: 11, Rank: 2, loss = 0.4140625Epoch: 0, Step: 11, Rank: 6, loss = 0.765625

Epoch: 0, Step: 11, Rank: 3, loss = 0.53125


Epoch: 0, Step: 11, Rank: 7, loss = 0.87890625

Epoch 0:  28%|██▊       | 11/40 [00:21<00:48,  1.67s/it]{
    "epoch": 0,
    "step": 11,
    "rank": 0,
    "overall_throughput": 94.35570001522005,
    "lr": 8.8e-06,
    "cuda_mem_allocated": 6.5880656242370605,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56969,
    "batch_size": 129,
    "total_loss": 0.5740665976232688,
    "samples_seen": 1400,
    "gradnorm": 7.71875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:06.616907"
}
 total length: 7835 dropped: 0 num samples 18 - rank: 5 max len: 462 min len: 414 avg len: 435.27777777777777 num_loss_counted_tokens: 6881
 total length: 7811 dropped: 0 num samples 24 - rank: 7 max len: 366 min len: 264 avg len: 325.4583333333333 num_loss_counted_tokens: 6539
 total length: 7974 dropped: 0 num samples 13 - rank: 2 max len: 629 min len: 589 avg len: 613.3846153846154 num_loss_counted_tokens: 7285
 total length: 7831 dropped: 0 num samples 14 - rank: 3 max len: 588 min len: 541 avg len: 559.3571428571429 num_loss_counted_tokens: 7089
 total length: 7834 dropped: 0 num samples 12 - rank: 1 max len: 684 min len: 631 avg len: 652.8333333333334 num_loss_counted_tokens: 7198
 total length: 7975 dropped: 0 num samples 10 - rank: 0 max len: 1084 min len: 229 avg len: 797.5 num_loss_counted_tokens: 7445
 total length: 7822 dropped: 0 num samples 20 - rank: 6 max len: 411 min len: 367 avg len: 391.1 num_loss_counted_tokens: 6762
 total length: 7924 dropped: 0 num samples 16 - rank: 4 max len: 534 min len: 466 avg len: 495.25 num_loss_counted_tokens: 7076
Epoch: 0, Step: 12, Rank: 5, loss = 0.482421875Epoch: 0, Step: 12, Rank: 6, loss = 0.61328125Epoch: 0, Step: 12, Rank: 1, loss = 0.390625
Epoch: 0, Step: 12, Rank: 4, loss = 0.5625Epoch: 0, Step: 12, Rank: 2, loss = 0.443359375

Epoch: 0, Step: 12, Rank: 3, loss = 0.435546875

Epoch: 0, Step: 12, Rank: 7, loss = 0.7265625

Epoch: 0, Step: 12, Rank: 0, loss = 0.349609375

Epoch 0:  30%|███       | 12/40 [00:22<00:46,  1.66s/it]{
    "epoch": 0,
    "step": 12,
    "rank": 0,
    "overall_throughput": 95.03132397896418,
    "lr": 9.600000000000001e-06,
    "cuda_mem_allocated": 6.576289653778076,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56828,
    "batch_size": 125,
    "total_loss": 0.4997536425705638,
    "samples_seen": 1525,
    "gradnorm": 4.5625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:08.257510"
}
 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 536 min len: 402 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7994 dropped: 0 num samples 13 - rank: 2 max len: 657 min len: 434 avg len: 614.9230769230769 num_loss_counted_tokens: 7305
 total length: 7587 dropped: 0 num samples 24 - rank: 7 max len: 361 min len: 242 avg len: 316.125 num_loss_counted_tokens: 6315
 total length: 7852 dropped: 0 num samples 14 - rank: 3 max len: 580 min len: 537 avg len: 560.8571428571429 num_loss_counted_tokens: 7110
 total length: 7878 dropped: 0 num samples 8 - rank: 0 max len: 1104 min len: 821 avg len: 984.75 num_loss_counted_tokens: 7454
 total length: 7988 dropped: 0 num samples 12 - rank: 1 max len: 720 min len: 501 avg len: 665.6666666666666 num_loss_counted_tokens: 7352
 total length: 7990 dropped: 0 num samples 18 - rank: 5 max len: 473 min len: 263 avg len: 443.8888888888889 num_loss_counted_tokens: 7036
 total length: 7912 dropped: 0 num samples 20 - rank: 6 max len: 432 min len: 363 avg len: 395.6 num_loss_counted_tokens: 6852
Epoch: 0, Step: 13, Rank: 0, loss = 0.30859375Epoch: 0, Step: 13, Rank: 6, loss = 0.51953125Epoch: 0, Step: 13, Rank: 5, loss = 0.5546875Epoch: 0, Step: 13, Rank: 1, loss = 0.3203125


Epoch: 0, Step: 13, Rank: 2, loss = 0.39453125
Epoch: 0, Step: 13, Rank: 4, loss = 0.37890625

Epoch: 0, Step: 13, Rank: 3, loss = 0.455078125
Epoch: 0, Step: 13, Rank: 7, loss = 0.73046875

Epoch 0:  32%|███▎      | 13/40 [00:24<00:44,  1.66s/it]{
    "epoch": 0,
    "step": 13,
    "rank": 0,
    "overall_throughput": 94.70535266821959,
    "lr": 1.04e-05,
    "cuda_mem_allocated": 6.582769870758057,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56480,
    "batch_size": 136,
    "total_loss": 0.45750708215297453,
    "samples_seen": 1661,
    "gradnorm": 2.1875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:09.901403"
}
 total length: 7992 dropped: 0 num samples 10 - rank: 0 max len: 1072 min len: 388 avg len: 799.2 num_loss_counted_tokens: 7462
 total length: 7878 dropped: 0 num samples 24 - rank: 7 max len: 363 min len: 245 avg len: 328.25 num_loss_counted_tokens: 6606
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 639 min len: 429 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7877 dropped: 0 num samples 17 - rank: 5 max len: 480 min len: 439 avg len: 463.3529411764706 num_loss_counted_tokens: 6976
 total length: 7992 dropped: 0 num samples 20 - rank: 6 max len: 434 min len: 366 avg len: 399.6 num_loss_counted_tokens: 6932
 total length: 7994 dropped: 0 num samples 16 - rank: 4 max len: 550 min len: 291 avg len: 499.625 num_loss_counted_tokens: 7146
 total length: 7787 dropped: 0 num samples 13 - rank: 3 max len: 618 min len: 569 avg len: 599.0 num_loss_counted_tokens: 7098
 total length: 7874 dropped: 0 num samples 12 - rank: 1 max len: 698 min len: 640 avg len: 656.1666666666666 num_loss_counted_tokens: 7238
Epoch: 0, Step: 14, Rank: 5, loss = 0.453125Epoch: 0, Step: 14, Rank: 3, loss = 0.361328125Epoch: 0, Step: 14, Rank: 1, loss = 0.30859375Epoch: 0, Step: 14, Rank: 6, loss = 0.63671875Epoch: 0, Step: 14, Rank: 4, loss = 0.478515625



Epoch: 0, Step: 14, Rank: 2, loss = 0.3203125Epoch: 0, Step: 14, Rank: 7, loss = 0.58203125


Epoch: 0, Step: 14, Rank: 0, loss = 0.314453125

Epoch 0:  35%|███▌      | 14/40 [00:26<00:43,  1.65s/it]{
    "epoch": 0,
    "step": 14,
    "rank": 0,
    "overall_throughput": 94.22838097076814,
    "lr": 1.1200000000000001e-05,
    "cuda_mem_allocated": 6.587028980255127,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56730,
    "batch_size": 128,
    "total_loss": 0.43179975321699277,
    "samples_seen": 1789,
    "gradnorm": 1.34375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:11.551978"
}
 total length: 7993 dropped: 0 num samples 12 - rank: 1 max len: 730 min len: 451 avg len: 666.0833333333334 num_loss_counted_tokens: 7357
 total length: 7987 dropped: 0 num samples 14 - rank: 3 max len: 609 min len: 445 avg len: 570.5 num_loss_counted_tokens: 7245
 total length: 7997 dropped: 0 num samples 10 - rank: 0 max len: 1068 min len: 263 avg len: 799.7 num_loss_counted_tokens: 7467
 total length: 7812 dropped: 0 num samples 25 - rank: 7 max len: 367 min len: 257 avg len: 312.48 num_loss_counted_tokens: 6487
 total length: 7995 dropped: 0 num samples 16 - rank: 4 max len: 544 min len: 296 avg len: 499.6875 num_loss_counted_tokens: 7147
 total length: 7837 dropped: 0 num samples 20 - rank: 6 max len: 422 min len: 367 avg len: 391.85 num_loss_counted_tokens: 6777
 total length: 7922 dropped: 0 num samples 18 - rank: 5 max len: 479 min len: 172 avg len: 440.1111111111111 num_loss_counted_tokens: 6968
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 652 min len: 370 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
Epoch: 0, Step: 15, Rank: 5, loss = 0.470703125Epoch: 0, Step: 15, Rank: 3, loss = 0.369140625Epoch: 0, Step: 15, Rank: 2, loss = 0.328125Epoch: 0, Step: 15, Rank: 4, loss = 0.46484375Epoch: 0, Step: 15, Rank: 6, loss = 0.58984375
Epoch: 0, Step: 15, Rank: 7, loss = 0.478515625
Epoch: 0, Step: 15, Rank: 1, loss = 0.265625




Epoch: 0, Step: 15, Rank: 0, loss = 0.263671875

Epoch 0:  38%|███▊      | 15/40 [00:27<00:41,  1.65s/it]{
    "epoch": 0,
    "step": 15,
    "rank": 0,
    "overall_throughput": 94.62571413766082,
    "lr": 1.2e-05,
    "cuda_mem_allocated": 6.586809158325195,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56456,
    "batch_size": 129,
    "total_loss": 0.4041377355816919,
    "samples_seen": 1918,
    "gradnorm": 1.3828125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:13.193836"
}
 total length: 7962 dropped: 0 num samples 14 - rank: 3 max len: 598 min len: 539 avg len: 568.7142857142857 num_loss_counted_tokens: 7220
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 729 min len: 483 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
 total length: 7018 dropped: 0 num samples 22 - rank: 7 max len: 369 min len: 238 avg len: 319.0 num_loss_counted_tokens: 5852
 total length: 7883 dropped: 0 num samples 8 - rank: 0 max len: 1309 min len: 758 avg len: 985.375 num_loss_counted_tokens: 7459
 total length: 7997 dropped: 0 num samples 17 - rank: 4 max len: 514 min len: 283 avg len: 470.4117647058824 num_loss_counted_tokens: 7096
 total length: 7961 dropped: 0 num samples 18 - rank: 5 max len: 457 min len: 416 avg len: 442.27777777777777 num_loss_counted_tokens: 7007
 total length: 7995 dropped: 0 num samples 20 - rank: 6 max len: 415 min len: 350 avg len: 399.75 num_loss_counted_tokens: 6935
 total length: 7990 dropped: 0 num samples 13 - rank: 2 max len: 647 min len: 519 avg len: 614.6153846153846 num_loss_counted_tokens: 7301
Epoch: 0, Step: 16, Rank: 5, loss = 0.44140625Epoch: 0, Step: 16, Rank: 6, loss = 0.56640625
Epoch: 0, Step: 16, Rank: 1, loss = 0.20703125Epoch: 0, Step: 16, Rank: 4, loss = 0.376953125Epoch: 0, Step: 16, Rank: 7, loss = 0.62890625Epoch: 0, Step: 16, Rank: 3, loss = 0.265625Epoch: 0, Step: 16, Rank: 2, loss = 0.283203125





Epoch: 0, Step: 16, Rank: 0, loss = 0.2216796875

Epoch 0:  40%|████      | 16/40 [00:29<00:39,  1.65s/it]{
    "epoch": 0,
    "step": 16,
    "rank": 0,
    "overall_throughput": 94.23957880051039,
    "lr": 1.2800000000000001e-05,
    "cuda_mem_allocated": 6.586205005645752,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56822,
    "batch_size": 127,
    "total_loss": 0.3739396712540917,
    "samples_seen": 2045,
    "gradnorm": 1.6875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:14.842188"
}
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 558 min len: 372 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7986 dropped: 0 num samples 11 - rank: 0 max len: 904 min len: 571 avg len: 726.0 num_loss_counted_tokens: 7403
 total length: 7812 dropped: 0 num samples 19 - rank: 6 max len: 430 min len: 388 avg len: 411.1578947368421 num_loss_counted_tokens: 6805
 total length: 7813 dropped: 0 num samples 12 - rank: 1 max len: 677 min len: 638 avg len: 651.0833333333334 num_loss_counted_tokens: 7177
 total length: 7995 dropped: 0 num samples 18 - rank: 5 max len: 482 min len: 263 avg len: 444.1666666666667 num_loss_counted_tokens: 7041
 total length: 7994 dropped: 0 num samples 16 - rank: 4 max len: 532 min len: 283 avg len: 499.625 num_loss_counted_tokens: 7146
 total length: 7823 dropped: 0 num samples 13 - rank: 2 max len: 638 min len: 559 avg len: 601.7692307692307 num_loss_counted_tokens: 7134
 total length: 7744 dropped: 0 num samples 23 - rank: 7 max len: 383 min len: 243 avg len: 336.69565217391306 num_loss_counted_tokens: 6525
Epoch: 0, Step: 17, Rank: 5, loss = 0.34765625Epoch: 0, Step: 17, Rank: 6, loss = 0.4765625Epoch: 0, Step: 17, Rank: 4, loss = 0.33984375
Epoch: 0, Step: 17, Rank: 3, loss = 0.1943359375
Epoch: 0, Step: 17, Rank: 1, loss = 0.251953125


Epoch: 0, Step: 17, Rank: 2, loss = 0.33203125
Epoch: 0, Step: 17, Rank: 7, loss = 0.458984375
Epoch: 0, Step: 17, Rank: 0, loss = 0.22265625

Epoch 0:  42%|████▎     | 17/40 [00:31<00:37,  1.65s/it]{
    "epoch": 0,
    "step": 17,
    "rank": 0,
    "overall_throughput": 94.24818159970071,
    "lr": 1.3600000000000002e-05,
    "cuda_mem_allocated": 6.588065147399902,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 57011,
    "batch_size": 117,
    "total_loss": 0.32779639017031803,
    "samples_seen": 2162,
    "gradnorm": 1.8515625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:16.482247"
}
 total length: 7932 dropped: 0 num samples 12 - rank: 1 max len: 711 min len: 631 avg len: 661.0 num_loss_counted_tokens: 7296
 total length: 7947 dropped: 0 num samples 9 - rank: 0 max len: 1118 min len: 719 avg len: 883.0 num_loss_counted_tokens: 7470
 total length: 7993 dropped: 0 num samples 18 - rank: 5 max len: 473 min len: 323 avg len: 444.05555555555554 num_loss_counted_tokens: 7039
 total length: 7990 dropped: 0 num samples 15 - rank: 3 max len: 555 min len: 429 avg len: 532.6666666666666 num_loss_counted_tokens: 7195
 total length: 7546 dropped: 0 num samples 23 - rank: 7 max len: 370 min len: 265 avg len: 328.0869565217391 num_loss_counted_tokens: 6327
 total length: 7903 dropped: 0 num samples 16 - rank: 4 max len: 520 min len: 473 avg len: 493.9375 num_loss_counted_tokens: 7055
 total length: 7992 dropped: 0 num samples 14 - rank: 2 max len: 630 min len: 430 avg len: 570.8571428571429 num_loss_counted_tokens: 7250
 total length: 7987 dropped: 0 num samples 20 - rank: 6 max len: 418 min len: 370 avg len: 399.35 num_loss_counted_tokens: 6927
Epoch: 0, Step: 18, Rank: 1, loss = 0.232421875Epoch: 0, Step: 18, Rank: 6, loss = 0.478515625Epoch: 0, Step: 18, Rank: 5, loss = 0.400390625

Epoch: 0, Step: 18, Rank: 3, loss = 0.349609375Epoch: 0, Step: 18, Rank: 4, loss = 0.33203125Epoch: 0, Step: 18, Rank: 7, loss = 0.50390625



Epoch: 0, Step: 18, Rank: 2, loss = 0.2255859375Epoch: 0, Step: 18, Rank: 0, loss = 0.1796875


Epoch 0:  45%|████▌     | 18/40 [00:32<00:36,  1.65s/it]{
    "epoch": 0,
    "step": 18,
    "rank": 0,
    "overall_throughput": 94.42601508189513,
    "lr": 1.4400000000000001e-05,
    "cuda_mem_allocated": 6.5880961418151855,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56349,
    "batch_size": 130,
    "total_loss": 0.3378941951055032,
    "samples_seen": 2292,
    "gradnorm": 1.875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:18.125068"
}
 total length: 7799 dropped: 0 num samples 10 - rank: 0 max len: 1065 min len: 689 avg len: 779.9 num_loss_counted_tokens: 7269
 total length: 7991 dropped: 0 num samples 20 - rank: 6 max len: 430 min len: 375 avg len: 399.55 num_loss_counted_tokens: 6931
 total length: 7994 dropped: 0 num samples 13 - rank: 2 max len: 637 min len: 483 avg len: 614.9230769230769 num_loss_counted_tokens: 7305
 total length: 7982 dropped: 0 num samples 25 - rank: 7 max len: 369 min len: 257 avg len: 319.28 num_loss_counted_tokens: 6657
 total length: 7959 dropped: 0 num samples 17 - rank: 5 max len: 503 min len: 434 avg len: 468.1764705882353 num_loss_counted_tokens: 7058
 total length: 7993 dropped: 0 num samples 14 - rank: 3 max len: 605 min len: 523 avg len: 570.9285714285714 num_loss_counted_tokens: 7251
 total length: 7857 dropped: 0 num samples 12 - rank: 1 max len: 688 min len: 638 avg len: 654.75 num_loss_counted_tokens: 7221
 total length: 7842 dropped: 0 num samples 15 - rank: 4 max len: 545 min len: 506 avg len: 522.8 num_loss_counted_tokens: 7047
Epoch: 0, Step: 19, Rank: 6, loss = 0.416015625Epoch: 0, Step: 19, Rank: 7, loss = 0.419921875Epoch: 0, Step: 19, Rank: 1, loss = 0.2265625Epoch: 0, Step: 19, Rank: 5, loss = 0.392578125

Epoch: 0, Step: 19, Rank: 4, loss = 0.33203125Epoch: 0, Step: 19, Rank: 2, loss = 0.2158203125Epoch: 0, Step: 19, Rank: 3, loss = 0.232421875




Epoch: 0, Step: 19, Rank: 0, loss = 0.205078125

Epoch 0:  48%|████▊     | 19/40 [00:34<00:34,  1.65s/it]{
    "epoch": 0,
    "step": 19,
    "rank": 0,
    "overall_throughput": 93.38516012187172,
    "lr": 1.5200000000000002e-05,
    "cuda_mem_allocated": 6.587976932525635,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56730,
    "batch_size": 129,
    "total_loss": 0.30488277807156705,
    "samples_seen": 2421,
    "gradnorm": 1.6328125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:19.785277"
}
 total length: 7511 dropped: 0 num samples 22 - rank: 7 max len: 376 min len: 253 avg len: 341.40909090909093 num_loss_counted_tokens: 6345
 total length: 7996 dropped: 0 num samples 14 - rank: 3 max len: 612 min len: 404 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7990 dropped: 0 num samples 8 - rank: 0 max len: 1100 min len: 703 avg len: 998.75 num_loss_counted_tokens: 7566
 total length: 7997 dropped: 0 num samples 11 - rank: 1 max len: 895 min len: 501 avg len: 727.0 num_loss_counted_tokens: 7414
 total length: 7881 dropped: 0 num samples 17 - rank: 5 max len: 485 min len: 443 avg len: 463.5882352941176 num_loss_counted_tokens: 6980
 total length: 7980 dropped: 0 num samples 13 - rank: 2 max len: 685 min len: 268 avg len: 613.8461538461538 num_loss_counted_tokens: 7291
 total length: 7995 dropped: 0 num samples 20 - rank: 6 max len: 441 min len: 306 avg len: 399.75 num_loss_counted_tokens: 6935
 total length: 7995 dropped: 0 num samples 16 - rank: 4 max len: 530 min len: 310 avg len: 499.6875 num_loss_counted_tokens: 7147
Epoch: 0, Step: 20, Rank: 5, loss = 0.271484375Epoch: 0, Step: 20, Rank: 7, loss = 0.349609375Epoch: 0, Step: 20, Rank: 2, loss = 0.185546875Epoch: 0, Step: 20, Rank: 4, loss = 0.265625Epoch: 0, Step: 20, Rank: 3, loss = 0.302734375

Epoch: 0, Step: 20, Rank: 6, loss = 0.390625
Epoch: 0, Step: 20, Rank: 1, loss = 0.208984375



Epoch: 0, Step: 20, Rank: 0, loss = 0.1904296875

Epoch 0:  50%|█████     | 20/40 [00:36<00:33,  1.65s/it]{
    "epoch": 0,
    "step": 20,
    "rank": 0,
    "overall_throughput": 94.21244064680722,
    "lr": 1.6000000000000003e-05,
    "cuda_mem_allocated": 6.5880961418151855,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56988,
    "batch_size": 120,
    "total_loss": 0.2707938513371236,
    "samples_seen": 2541,
    "gradnorm": 1.359375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:21.436575"
}
 total length: 6906 dropped: 0 num samples 22 - rank: 7 max len: 362 min len: 255 avg len: 313.90909090909093 num_loss_counted_tokens: 5740
 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 525 min len: 383 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 749 min len: 459 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7893 dropped: 0 num samples 18 - rank: 5 max len: 475 min len: 406 avg len: 438.5 num_loss_counted_tokens: 6939
 total length: 7996 dropped: 0 num samples 21 - rank: 6 max len: 401 min len: 259 avg len: 380.76190476190476 num_loss_counted_tokens: 6883
 total length: 7992 dropped: 0 num samples 15 - rank: 3 max len: 590 min len: 304 avg len: 532.8 num_loss_counted_tokens: 7197
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 650 min len: 453 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7994 dropped: 0 num samples 9 - rank: 0 max len: 1064 min len: 627 avg len: 888.2222222222222 num_loss_counted_tokens: 7517
Epoch: 0, Step: 21, Rank: 1, loss = 0.1982421875Epoch: 0, Step: 21, Rank: 7, loss = 0.408203125Epoch: 0, Step: 21, Rank: 5, loss = 0.3046875Epoch: 0, Step: 21, Rank: 6, loss = 0.345703125

Epoch: 0, Step: 21, Rank: 3, loss = 0.197265625
Epoch: 0, Step: 21, Rank: 2, loss = 0.1435546875Epoch: 0, Step: 21, Rank: 4, loss = 0.271484375



Epoch: 0, Step: 21, Rank: 0, loss = 0.162109375

Epoch 0:  52%|█████▎    | 21/40 [00:37<00:31,  1.65s/it]{
    "epoch": 0,
    "step": 21,
    "rank": 0,
    "overall_throughput": 94.10561118236461,
    "lr": 1.6800000000000002e-05,
    "cuda_mem_allocated": 6.587976932525635,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 57005,
    "batch_size": 130,
    "total_loss": 0.25415314446101217,
    "samples_seen": 2671,
    "gradnorm": 1.0859375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:23.088530"
}
 total length: 7715 dropped: 0 num samples 24 - rank: 7 max len: 365 min len: 269 avg len: 321.4583333333333 num_loss_counted_tokens: 6443
 total length: 7843 dropped: 0 num samples 17 - rank: 5 max len: 490 min len: 427 avg len: 461.3529411764706 num_loss_counted_tokens: 6942
 total length: 7997 dropped: 0 num samples 9 - rank: 0 max len: 1247 min len: 392 avg len: 888.5555555555555 num_loss_counted_tokens: 7520
 total length: 7814 dropped: 0 num samples 14 - rank: 3 max len: 596 min len: 539 avg len: 558.1428571428571 num_loss_counted_tokens: 7072
 total length: 7989 dropped: 0 num samples 13 - rank: 2 max len: 642 min len: 517 avg len: 614.5384615384615 num_loss_counted_tokens: 7300
 total length: 7994 dropped: 0 num samples 16 - rank: 4 max len: 537 min len: 355 avg len: 499.625 num_loss_counted_tokens: 7146
 total length: 7989 dropped: 0 num samples 12 - rank: 1 max len: 705 min len: 600 avg len: 665.75 num_loss_counted_tokens: 7353
 total length: 7959 dropped: 0 num samples 20 - rank: 6 max len: 426 min len: 367 avg len: 397.95 num_loss_counted_tokens: 6899
Epoch: 0, Step: 22, Rank: 6, loss = 0.296875Epoch: 0, Step: 22, Rank: 1, loss = 0.16796875Epoch: 0, Step: 22, Rank: 5, loss = 0.27734375Epoch: 0, Step: 22, Rank: 2, loss = 0.1630859375
Epoch: 0, Step: 22, Rank: 7, loss = 0.32421875

Epoch: 0, Step: 22, Rank: 3, loss = 0.208984375Epoch: 0, Step: 22, Rank: 4, loss = 0.2470703125



Epoch: 0, Step: 22, Rank: 0, loss = 0.1396484375

Epoch 0:  55%|█████▌    | 22/40 [00:39<00:29,  1.65s/it]{
    "epoch": 0,
    "step": 22,
    "rank": 0,
    "overall_throughput": 94.37260124774053,
    "lr": 1.76e-05,
    "cuda_mem_allocated": 6.5880961418151855,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56465,
    "batch_size": 127,
    "total_loss": 0.22803506597006995,
    "samples_seen": 2798,
    "gradnorm": 0.88671875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:24.736286"
}
 total length: 7746 dropped: 0 num samples 10 - rank: 0 max len: 1098 min len: 710 avg len: 774.6 num_loss_counted_tokens: 7216 total length: 7989 dropped: 0 num samples 12 - rank: 1 max len: 688 min len: 636 avg len: 665.75 num_loss_counted_tokens: 7353

 total length: 7991 dropped: 0 num samples 15 - rank: 3 max len: 573 min len: 373 avg len: 532.7333333333333 num_loss_counted_tokens: 7196
 total length: 7884 dropped: 0 num samples 24 - rank: 7 max len: 358 min len: 265 avg len: 328.5 num_loss_counted_tokens: 6612
 total length: 7943 dropped: 0 num samples 16 - rank: 4 max len: 516 min len: 475 avg len: 496.4375 num_loss_counted_tokens: 7095
 total length: 7868 dropped: 0 num samples 20 - rank: 6 max len: 418 min len: 363 avg len: 393.4 num_loss_counted_tokens: 6808
 total length: 7996 dropped: 0 num samples 18 - rank: 5 max len: 474 min len: 359 avg len: 444.22222222222223 num_loss_counted_tokens: 7042
 total length: 7914 dropped: 0 num samples 13 - rank: 2 max len: 634 min len: 573 avg len: 608.7692307692307 num_loss_counted_tokens: 7225
Epoch: 0, Step: 23, Rank: 6, loss = 0.30078125Epoch: 0, Step: 23, Rank: 7, loss = 0.322265625Epoch: 0, Step: 23, Rank: 1, loss = 0.169921875Epoch: 0, Step: 23, Rank: 2, loss = 0.1435546875Epoch: 0, Step: 23, Rank: 5, loss = 0.33203125


Epoch: 0, Step: 23, Rank: 4, loss = 0.302734375
Epoch: 0, Step: 23, Rank: 3, loss = 0.2216796875


Epoch: 0, Step: 23, Rank: 0, loss = 0.1171875

Epoch 0:  57%|█████▊    | 23/40 [00:41<00:28,  1.65s/it]{
    "epoch": 0,
    "step": 23,
    "rank": 0,
    "overall_throughput": 94.06857723774532,
    "lr": 1.8400000000000003e-05,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56640,
    "batch_size": 125,
    "total_loss": 0.23877118644067796,
    "samples_seen": 2923,
    "gradnorm": 0.734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:26.384005"
}
 total length: 7760 dropped: 0 num samples 10 - rank: 0 max len: 1104 min len: 694 avg len: 776.0 num_loss_counted_tokens: 7230
 total length: 7785 dropped: 0 num samples 14 - rank: 3 max len: 584 min len: 532 avg len: 556.0714285714286 num_loss_counted_tokens: 7043
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 693 min len: 604 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
 total length: 7989 dropped: 0 num samples 25 - rank: 7 max len: 362 min len: 272 avg len: 319.56 num_loss_counted_tokens: 6664
 total length: 7784 dropped: 0 num samples 17 - rank: 5 max len: 481 min len: 436 avg len: 457.88235294117646 num_loss_counted_tokens: 6883
 total length: 7996 dropped: 0 num samples 20 - rank: 6 max len: 434 min len: 295 avg len: 399.8 num_loss_counted_tokens: 6936
 total length: 7994 dropped: 0 num samples 16 - rank: 4 max len: 531 min len: 368 avg len: 499.625 num_loss_counted_tokens: 7146
 total length: 7992 dropped: 0 num samples 13 - rank: 2 max len: 648 min len: 517 avg len: 614.7692307692307 num_loss_counted_tokens: 7303
Epoch: 0, Step: 24, Rank: 5, loss = 0.251953125Epoch: 0, Step: 24, Rank: 6, loss = 0.27734375Epoch: 0, Step: 24, Rank: 4, loss = 0.19921875
Epoch: 0, Step: 24, Rank: 3, loss = 0.1865234375Epoch: 0, Step: 24, Rank: 7, loss = 0.294921875
Epoch: 0, Step: 24, Rank: 1, loss = 0.150390625
Epoch: 0, Step: 24, Rank: 2, loss = 0.1611328125



Epoch: 0, Step: 24, Rank: 0, loss = 0.1142578125

Epoch 0:  60%|██████    | 24/40 [00:42<00:26,  1.65s/it]{
    "epoch": 0,
    "step": 24,
    "rank": 0,
    "overall_throughput": 93.44952964611537,
    "lr": 1.9200000000000003e-05,
    "cuda_mem_allocated": 6.586936950683594,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56472,
    "batch_size": 125,
    "total_loss": 0.2047032157529395,
    "samples_seen": 3048,
    "gradnorm": 0.63671875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:28.036966"
}
Epoch: 0, Step: 25, Rank: 0, loss = 0.11767578125Epoch: 0, Step: 25, Rank: 4, loss = 0.1708984375Epoch: 0, Step: 25, Rank: 5, loss = 0.2294921875Epoch: 0, Step: 25, Rank: 1, loss = 0.1220703125Epoch: 0, Step: 25, Rank: 3, loss = 0.1533203125

Epoch: 0, Step: 25, Rank: 7, loss = 0.251953125



Epoch: 0, Step: 25, Rank: 2, loss = 0.1513671875
Epoch: 0, Step: 25, Rank: 6, loss = 0.28515625

Epoch 0:  62%|██████▎   | 25/40 [00:44<00:24,  1.65s/it]{
    "epoch": 0,
    "step": 25,
    "rank": 0,
    "overall_throughput": 95.33155693443696,
    "lr": 2e-05,
    "cuda_mem_allocated": 6.5645670890808105,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56277,
    "batch_size": 127,
    "total_loss": 0.1852266467651083,
    "samples_seen": 3175,
    "gradnorm": 0.5859375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:29.669760"
}
Epoch: 0, Step: 26, Rank: 5, loss = 0.234375Epoch: 0, Step: 26, Rank: 7, loss = 0.2490234375Epoch: 0, Step: 26, Rank: 3, loss = 0.14453125Epoch: 0, Step: 26, Rank: 2, loss = 0.1240234375Epoch: 0, Step: 26, Rank: 1, loss = 0.126953125

Epoch: 0, Step: 26, Rank: 4, loss = 0.203125


Epoch: 0, Step: 26, Rank: 6, loss = 0.212890625

Epoch: 0, Step: 26, Rank: 0, loss = 0.154296875

Epoch 0:  65%|██████▌   | 26/40 [00:45<00:23,  1.64s/it]{
    "epoch": 0,
    "step": 26,
    "rank": 0,
    "overall_throughput": 94.75044761378832,
    "lr": 1.999924110114831e-05,
    "cuda_mem_allocated": 6.578878879547119,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56529,
    "batch_size": 130,
    "total_loss": 0.1810752003396487,
    "samples_seen": 3305,
    "gradnorm": 0.578125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:31.312212"
}
Epoch: 0, Step: 27, Rank: 0, loss = 0.07958984375Epoch: 0, Step: 27, Rank: 1, loss = 0.107421875Epoch: 0, Step: 27, Rank: 5, loss = 0.18359375
Epoch: 0, Step: 27, Rank: 6, loss = 0.2001953125

Epoch: 0, Step: 27, Rank: 4, loss = 0.14453125Epoch: 0, Step: 27, Rank: 3, loss = 0.138671875
Epoch: 0, Step: 27, Rank: 7, loss = 0.1875


Epoch: 0, Step: 27, Rank: 2, loss = 0.1279296875

Epoch 0:  68%|██████▊   | 27/40 [00:47<00:21,  1.64s/it]{
    "epoch": 0,
    "step": 27,
    "rank": 0,
    "overall_throughput": 95.0973941122862,
    "lr": 1.999696451977872e-05,
    "cuda_mem_allocated": 6.578363418579102,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56593,
    "batch_size": 125,
    "total_loss": 0.14616648702136306,
    "samples_seen": 3430,
    "gradnorm": 0.5234375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:32.949798"
}
Epoch: 0, Step: 28, Rank: 5, loss = 0.1669921875Epoch: 0, Step: 28, Rank: 6, loss = 0.1923828125Epoch: 0, Step: 28, Rank: 3, loss = 0.1220703125
Epoch: 0, Step: 28, Rank: 4, loss = 0.158203125Epoch: 0, Step: 28, Rank: 7, loss = 0.201171875

Epoch: 0, Step: 28, Rank: 2, loss = 0.11962890625Epoch: 0, Step: 28, Rank: 1, loss = 0.1025390625



Epoch: 0, Step: 28, Rank: 0, loss = 0.08837890625

Epoch 0:  70%|███████   | 28/40 [00:49<00:19,  1.64s/it]{
    "epoch": 0,
    "step": 28,
    "rank": 0,
    "overall_throughput": 94.3122887893697,
    "lr": 1.9993170601430233e-05,
    "cuda_mem_allocated": 6.586054801940918,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56275,
    "batch_size": 127,
    "total_loss": 0.14386494891159485,
    "samples_seen": 3557,
    "gradnorm": 0.5078125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:34.597037"
}
Epoch: 0, Step: 29, Rank: 0, loss = 0.06884765625Epoch: 0, Step: 29, Rank: 5, loss = 0.1953125

Epoch: 0, Step: 29, Rank: 6, loss = 0.1923828125Epoch: 0, Step: 29, Rank: 7, loss = 0.1845703125
Epoch: 0, Step: 29, Rank: 2, loss = 0.1044921875Epoch: 0, Step: 29, Rank: 3, loss = 0.1357421875


Epoch: 0, Step: 29, Rank: 4, loss = 0.1806640625
Epoch: 0, Step: 29, Rank: 1, loss = 0.13671875

Epoch 0:  72%|███████▎  | 29/40 [00:50<00:18,  1.64s/it]{
    "epoch": 0,
    "step": 29,
    "rank": 0,
    "overall_throughput": 94.82421633315076,
    "lr": 1.99878599219429e-05,
    "cuda_mem_allocated": 6.576986312866211,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56573,
    "batch_size": 125,
    "total_loss": 0.14989482615381897,
    "samples_seen": 3682,
    "gradnorm": 0.5546875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:36.239422"
}
Epoch: 0, Step: 30, Rank: 6, loss = 0.181640625Epoch: 0, Step: 30, Rank: 3, loss = 0.099609375Epoch: 0, Step: 30, Rank: 1, loss = 0.103515625

Epoch: 0, Step: 30, Rank: 5, loss = 0.1513671875
Epoch: 0, Step: 30, Rank: 7, loss = 0.1884765625Epoch: 0, Step: 30, Rank: 4, loss = 0.1279296875
Epoch: 0, Step: 30, Rank: 2, loss = 0.11181640625


Epoch: 0, Step: 30, Rank: 0, loss = 0.0712890625

Epoch 0:  75%|███████▌  | 30/40 [00:52<00:16,  1.65s/it]{
    "epoch": 0,
    "step": 30,
    "rank": 0,
    "overall_throughput": 94.32408658889706,
    "lr": 1.9981033287370443e-05,
    "cuda_mem_allocated": 6.588123798370361,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56766,
    "batch_size": 125,
    "total_loss": 0.12940844871930382,
    "samples_seen": 3807,
    "gradnorm": 0.484375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:37.891035"
}
Epoch: 0, Step: 31, Rank: 5, loss = 0.171875Epoch: 0, Step: 31, Rank: 0, loss = 0.05810546875Epoch: 0, Step: 31, Rank: 6, loss = 0.1865234375
Epoch: 0, Step: 31, Rank: 7, loss = 0.2021484375
Epoch: 0, Step: 31, Rank: 3, loss = 0.1416015625Epoch: 0, Step: 31, Rank: 2, loss = 0.0908203125
Epoch: 0, Step: 31, Rank: 4, loss = 0.11767578125



Epoch: 0, Step: 31, Rank: 1, loss = 0.1044921875

Epoch 0:  78%|███████▊  | 31/40 [00:54<00:14,  1.65s/it]{
    "epoch": 0,
    "step": 31,
    "rank": 0,
    "overall_throughput": 94.44679607640172,
    "lr": 1.997269173385788e-05,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56755,
    "batch_size": 128,
    "total_loss": 0.13404986344815434,
    "samples_seen": 3935,
    "gradnorm": 0.56640625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:39.538489"
}
Epoch: 0, Step: 32, Rank: 5, loss = 0.169921875Epoch: 0, Step: 32, Rank: 6, loss = 0.1357421875
Epoch: 0, Step: 32, Rank: 1, loss = 0.10791015625
Epoch: 0, Step: 32, Rank: 3, loss = 0.11328125Epoch: 0, Step: 32, Rank: 7, loss = 0.130859375
Epoch: 0, Step: 32, Rank: 4, loss = 0.138671875
Epoch: 0, Step: 32, Rank: 2, loss = 0.07666015625


Epoch: 0, Step: 32, Rank: 0, loss = 0.07763671875

Epoch 0:  80%|████████  | 32/40 [00:55<00:13,  1.65s/it]{
    "epoch": 0,
    "step": 32,
    "rank": 0,
    "overall_throughput": 94.74044881065089,
    "lr": 1.9962836527484296e-05,
    "cuda_mem_allocated": 6.578217506408691,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56230,
    "batch_size": 124,
    "total_loss": 0.11872665836741952,
    "samples_seen": 4059,
    "gradnorm": 0.462890625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:41.181197"
}
Epoch: 0, Step: 33, Rank: 6, loss = 0.1318359375Epoch: 0, Step: 33, Rank: 5, loss = 0.1572265625Epoch: 0, Step: 33, Rank: 3, loss = 0.142578125
Epoch: 0, Step: 33, Rank: 1, loss = 0.09130859375
Epoch: 0, Step: 33, Rank: 2, loss = 0.083984375
Epoch: 0, Step: 33, Rank: 7, loss = 0.1533203125Epoch: 0, Step: 33, Rank: 4, loss = 0.154296875



Epoch: 0, Step: 33, Rank: 0, loss = 0.109375

Epoch 0:  82%|████████▎ | 33/40 [00:57<00:11,  1.64s/it]{
    "epoch": 0,
    "step": 33,
    "rank": 0,
    "overall_throughput": 94.77908462347149,
    "lr": 1.9951469164070647e-05,
    "cuda_mem_allocated": 6.586925983428955,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56433,
    "batch_size": 127,
    "total_loss": 0.12801020679389719,
    "samples_seen": 4186,
    "gradnorm": 0.546875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:42.821179"
}
Epoch: 0, Step: 34, Rank: 5, loss = 0.126953125Epoch: 0, Step: 34, Rank: 1, loss = 0.08837890625Epoch: 0, Step: 34, Rank: 4, loss = 0.09033203125Epoch: 0, Step: 34, Rank: 2, loss = 0.08203125
Epoch: 0, Step: 34, Rank: 7, loss = 0.1640625
Epoch: 0, Step: 34, Rank: 3, loss = 0.0859375



Epoch: 0, Step: 34, Rank: 6, loss = 0.12158203125
Epoch: 0, Step: 34, Rank: 0, loss = 0.05224609375

Epoch 0:  85%|████████▌ | 34/40 [00:59<00:09,  1.64s/it]{
    "epoch": 0,
    "step": 34,
    "rank": 0,
    "overall_throughput": 94.88773460535857,
    "lr": 1.993859136895274e-05,
    "cuda_mem_allocated": 6.584191799163818,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56559,
    "batch_size": 127,
    "total_loss": 0.10152230414257678,
    "samples_seen": 4313,
    "gradnorm": 0.419921875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:44.462269"
}
Epoch: 0, Step: 35, Rank: 0, loss = 0.091796875Epoch: 0, Step: 35, Rank: 3, loss = 0.0859375
Epoch: 0, Step: 35, Rank: 5, loss = 0.11767578125Epoch: 0, Step: 35, Rank: 6, loss = 0.138671875Epoch: 0, Step: 35, Rank: 2, loss = 0.09326171875


Epoch: 0, Step: 35, Rank: 1, loss = 0.08642578125Epoch: 0, Step: 35, Rank: 4, loss = 0.08447265625


Epoch: 0, Step: 35, Rank: 7, loss = 0.1474609375

Epoch 0:  88%|████████▊ | 35/40 [01:00<00:08,  1.64s/it]{
    "epoch": 0,
    "step": 35,
    "rank": 0,
    "overall_throughput": 94.65267374352302,
    "lr": 1.992420509671936e-05,
    "cuda_mem_allocated": 6.5703511238098145,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56739,
    "batch_size": 126,
    "total_loss": 0.10574736953418283,
    "samples_seen": 4439,
    "gradnorm": 0.40234375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:46.106147"
}
Epoch: 0, Step: 36, Rank: 6, loss = 0.1201171875Epoch: 0, Step: 36, Rank: 5, loss = 0.1396484375Epoch: 0, Step: 36, Rank: 4, loss = 0.10546875

Epoch: 0, Step: 36, Rank: 7, loss = 0.1533203125

Epoch: 0, Step: 36, Rank: 2, loss = 0.109375Epoch: 0, Step: 36, Rank: 3, loss = 0.08837890625Epoch: 0, Step: 36, Rank: 1, loss = 0.0888671875


Epoch: 0, Step: 36, Rank: 0, loss = 0.03173828125

Epoch 0:  90%|█████████ | 36/40 [01:02<00:06,  1.64s/it]{
    "epoch": 0,
    "step": 36,
    "rank": 0,
    "overall_throughput": 94.35687743187755,
    "lr": 1.9908312530915603e-05,
    "cuda_mem_allocated": 6.588098049163818,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56932,
    "batch_size": 121,
    "total_loss": 0.10456333871987634,
    "samples_seen": 4560,
    "gradnorm": 0.4140625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:47.754365"
}
Epoch: 0, Step: 37, Rank: 6, loss = 0.1474609375Epoch: 0, Step: 37, Rank: 5, loss = 0.1337890625Epoch: 0, Step: 37, Rank: 1, loss = 0.10498046875Epoch: 0, Step: 37, Rank: 4, loss = 0.10595703125Epoch: 0, Step: 37, Rank: 3, loss = 0.10546875
Epoch: 0, Step: 37, Rank: 2, loss = 0.08447265625
Epoch: 0, Step: 37, Rank: 7, loss = 0.1142578125




Epoch: 0, Step: 37, Rank: 0, loss = 0.099609375

Epoch 0:  92%|█████████▎| 37/40 [01:04<00:04,  1.64s/it]{
    "epoch": 0,
    "step": 37,
    "rank": 0,
    "overall_throughput": 94.49754942556494,
    "lr": 1.9890916083711463e-05,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56093,
    "batch_size": 126,
    "total_loss": 0.11202823881767779,
    "samples_seen": 4686,
    "gradnorm": 0.423828125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:49.399524"
}
Epoch: 0, Step: 38, Rank: 6, loss = 0.1083984375Epoch: 0, Step: 38, Rank: 5, loss = 0.08642578125
Epoch: 0, Step: 38, Rank: 1, loss = 0.0908203125Epoch: 0, Step: 38, Rank: 7, loss = 0.142578125Epoch: 0, Step: 38, Rank: 4, loss = 0.1220703125

Epoch: 0, Step: 38, Rank: 3, loss = 0.10400390625Epoch: 0, Step: 38, Rank: 2, loss = 0.09375



Epoch: 0, Step: 38, Rank: 0, loss = 0.05908203125

Epoch 0:  95%|█████████▌| 38/40 [01:05<00:03,  1.72s/it]{
    "epoch": 0,
    "step": 38,
    "rank": 0,
    "overall_throughput": 79.62464371640166,
    "lr": 1.9872018395535694e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56675,
    "batch_size": 125,
    "total_loss": 0.10082046757829731,
    "samples_seen": 4811,
    "gradnorm": 0.39453125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:51.297770"
}
Epoch: 0, Step: 39, Rank: 5, loss = 0.1259765625Epoch: 0, Step: 39, Rank: 3, loss = 0.09814453125Epoch: 0, Step: 39, Rank: 0, loss = 0.09228515625Epoch: 0, Step: 39, Rank: 7, loss = 0.158203125Epoch: 0, Step: 39, Rank: 4, loss = 0.109375Epoch: 0, Step: 39, Rank: 6, loss = 0.11376953125
Epoch: 0, Step: 39, Rank: 2, loss = 0.07080078125





Epoch: 0, Step: 39, Rank: 1, loss = 0.08447265625

Epoch 0:  98%|█████████▊| 39/40 [01:07<00:01,  1.70s/it]{
    "epoch": 0,
    "step": 39,
    "rank": 0,
    "overall_throughput": 94.75282222161431,
    "lr": 1.9851622334675065e-05,
    "cuda_mem_allocated": 6.564865589141846,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56547,
    "batch_size": 128,
    "total_loss": 0.10656621925124234,
    "samples_seen": 4939,
    "gradnorm": 0.40234375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:52.936868"
}
Epoch: 0, Step: 40, Rank: 0, loss = 0.099609375Epoch: 0, Step: 40, Rank: 5, loss = 0.10009765625Epoch: 0, Step: 40, Rank: 6, loss = 0.10595703125


Epoch: 0, Step: 40, Rank: 4, loss = 0.08642578125Epoch: 0, Step: 40, Rank: 3, loss = 0.0830078125Epoch: 0, Step: 40, Rank: 1, loss = 0.0703125

Epoch: 0, Step: 40, Rank: 2, loss = 0.078125

Epoch: 0, Step: 40, Rank: 7, loss = 0.11767578125

Epoch 0: 100%|██████████| 40/40 [01:09<00:00,  1.68s/it]{
    "epoch": 0,
    "step": 40,
    "rank": 0,
    "overall_throughput": 94.85653466011287,
    "lr": 1.982973099683902e-05,
    "cuda_mem_allocated": 6.566525936126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56565,
    "batch_size": 127,
    "total_loss": 0.09267214708742155,
    "samples_seen": 5066,
    "gradnorm": 0.38671875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:29:54.580127"
}
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
Saving model in huggingface format at: samples_5066
/opt/app-root/lib64/python3.11/site-packages/instructlab/training/utils.py:990: UserWarning: Adding architectures to ckpt: ['GraniteForCausalLM']
  warnings.warn(
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
[19:30:21] INFO     The model is bigger than the maximum     accelerator.py:2924
                    size per checkpoint (5GB) and is going                      
                    to be split in 4 checkpoint shards. You                     
                    can find where each parameters has been                     
                    saved in the index located at                               
                    /tmp/tmpkwucrg5gw/model.safetensors.inde                    
                    x.json.                                                     
Model saved in /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/hf_format/samples_5066
[19:30:36] INFO     saving took 41.79482436180115 seconds           utils.py:879
Saving full model state in /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/full_state/epoch_0
           INFO     Saving current state to                  accelerator.py:3030
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_0                                                  
           INFO     Saving FSDP model                        accelerator.py:3040
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
[19:30:45] INFO     Saving model to                             fsdp_utils.py:88
                    /var/home/cloud-user/.local/share/instructl                 
                    ab/phased/phase1/checkpoints/full_state/epo                 
                    ch_0/pytorch_model_fsdp.bin                                 
[19:30:56] INFO     Model saved to                              fsdp_utils.py:90
                    /var/home/cloud-user/.local/share/instructl                 
                    ab/phased/phase1/checkpoints/full_state/epo                 
                    ch_0/pytorch_model_fsdp.bin                                 
           INFO     FSDP Model saved to output dir           accelerator.py:3042
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_0                                                  
           INFO     Saving FSDP Optimizer                    accelerator.py:3059
 total length: 7880 dropped: 0 num samples 14 - rank: 3 max len: 581 min len: 538 avg len: 562.8571428571429 num_loss_counted_tokens: 7138
 total length: 7761 dropped: 0 num samples 13 - rank: 3 max len: 617 min len: 583 avg len: 597.0 num_loss_counted_tokens: 7072
 total length: 7994 dropped: 0 num samples 15 - rank: 3 max len: 589 min len: 278 avg len: 532.9333333333333 num_loss_counted_tokens: 7199
 total length: 7846 dropped: 0 num samples 14 - rank: 3 max len: 580 min len: 542 avg len: 560.4285714285714 num_loss_counted_tokens: 7104
 total length: 7992 dropped: 0 num samples 15 - rank: 3 max len: 573 min len: 297 avg len: 532.8 num_loss_counted_tokens: 7197 total length: 7992 dropped: 0 num samples 14 - rank: 3 max len: 623 min len: 421 avg len: 570.8571428571429 num_loss_counted_tokens: 7250

 total length: 7993 dropped: 0 num samples 15 - rank: 3 max len: 571 min len: 412 avg len: 532.8666666666667 num_loss_counted_tokens: 7198
 total length: 7770 dropped: 0 num samples 14 - rank: 3 max len: 596 min len: 515 avg len: 555.0 num_loss_counted_tokens: 7028
 total length: 7994 dropped: 0 num samples 16 - rank: 3 max len: 539 min len: 313 avg len: 499.625 num_loss_counted_tokens: 7146
 total length: 7989 dropped: 0 num samples 14 - rank: 3 max len: 603 min len: 427 avg len: 570.6428571428571 num_loss_counted_tokens: 7247
 total length: 7993 dropped: 0 num samples 13 - rank: 2 max len: 660 min len: 382 avg len: 614.8461538461538 num_loss_counted_tokens: 7304
 total length: 7991 dropped: 0 num samples 13 - rank: 2 max len: 649 min len: 362 avg len: 614.6923076923077 num_loss_counted_tokens: 7302
 total length: 7899 dropped: 0 num samples 14 - rank: 3 max len: 602 min len: 521 avg len: 564.2142857142857 num_loss_counted_tokens: 7157
 total length: 7996 dropped: 0 num samples 14 - rank: 3 max len: 614 min len: 457 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7983 dropped: 0 num samples 15 - rank: 3 max len: 587 min len: 257 avg len: 532.2 num_loss_counted_tokens: 7188
 total length: 7991 dropped: 0 num samples 14 - rank: 3 max len: 614 min len: 301 avg len: 570.7857142857143 num_loss_counted_tokens: 7249
 total length: 7902 dropped: 0 num samples 15 - rank: 3 max len: 547 min len: 505 avg len: 526.8 num_loss_counted_tokens: 7107
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 845 min len: 310 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7806 dropped: 0 num samples 12 - rank: 1 max len: 678 min len: 633 avg len: 650.5 num_loss_counted_tokens: 7170
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 783 min len: 323 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7984 dropped: 0 num samples 12 - rank: 1 max len: 809 min len: 313 avg len: 665.3333333333334 num_loss_counted_tokens: 7348
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 633 min len: 587 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7885 dropped: 0 num samples 13 - rank: 2 max len: 633 min len: 574 avg len: 606.5384615384615 num_loss_counted_tokens: 7196
 total length: 7995 dropped: 0 num samples 13 - rank: 1 max len: 689 min len: 271 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 656 min len: 298 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 647 min len: 441 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7810 dropped: 0 num samples 13 - rank: 2 max len: 623 min len: 582 avg len: 600.7692307692307 num_loss_counted_tokens: 7121
 total length: 7824 dropped: 0 num samples 13 - rank: 2 max len: 630 min len: 572 avg len: 601.8461538461538 num_loss_counted_tokens: 7135
 total length: 7846 dropped: 0 num samples 12 - rank: 1 max len: 691 min len: 634 avg len: 653.8333333333334 num_loss_counted_tokens: 7210
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 642 min len: 469 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 732 min len: 544 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
 total length: 7994 dropped: 0 num samples 15 - rank: 3 max len: 571 min len: 372 avg len: 532.9333333333333 num_loss_counted_tokens: 7199
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 809 min len: 327 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7997 dropped: 0 num samples 13 - rank: 1 max len: 676 min len: 491 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7807 dropped: 0 num samples 12 - rank: 1 max len: 670 min len: 644 avg len: 650.5833333333334 num_loss_counted_tokens: 7171
 total length: 7986 dropped: 0 num samples 13 - rank: 1 max len: 695 min len: 172 avg len: 614.3076923076923 num_loss_counted_tokens: 7297
 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 530 min len: 418 avg len: 499.75 num_loss_counted_tokens: 7148
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 640 min len: 437 avg len: 615.0769230769231 num_loss_counted_tokens: 7307 total length: 7786 dropped: 0 num samples 13 - rank: 2 max len: 627 min len: 576 avg len: 598.9230769230769 num_loss_counted_tokens: 7097

 total length: 7936 dropped: 0 num samples 13 - rank: 2 max len: 627 min len: 588 avg len: 610.4615384615385 num_loss_counted_tokens: 7247
 total length: 7994 dropped: 0 num samples 13 - rank: 2 max len: 665 min len: 411 avg len: 614.9230769230769 num_loss_counted_tokens: 7305
 total length: 7849 dropped: 0 num samples 14 - rank: 2 max len: 588 min len: 540 avg len: 560.6428571428571 num_loss_counted_tokens: 7107
 total length: 7876 dropped: 0 num samples 13 - rank: 2 max len: 631 min len: 585 avg len: 605.8461538461538 num_loss_counted_tokens: 7187
 total length: 7848 dropped: 0 num samples 14 - rank: 3 max len: 582 min len: 536 avg len: 560.5714285714286 num_loss_counted_tokens: 7106
 total length: 7988 dropped: 0 num samples 13 - rank: 1 max len: 653 min len: 426 avg len: 614.4615384615385 num_loss_counted_tokens: 7299
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 725 min len: 430 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7995 dropped: 0 num samples 13 - rank: 2 max len: 655 min len: 432 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 769 min len: 442 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 724 min len: 523 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
[19:31:05] INFO     Saving Optimizer state to                  fsdp_utils.py:192
                    /var/home/cloud-user/.local/share/instruct                  
                    lab/phased/phase1/checkpoints/full_state/e                  
                    poch_0/optimizer.bin                                        
 total length: 7808 dropped: 0 num samples 17 - rank: 5 max len: 484 min len: 432 avg len: 459.29411764705884 num_loss_counted_tokens: 6907
 total length: 7996 dropped: 0 num samples 21 - rank: 6 max len: 427 min len: 264 avg len: 380.76190476190476 num_loss_counted_tokens: 6883
 total length: 7997 dropped: 0 num samples 14 - rank: 2 max len: 610 min len: 381 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7989 dropped: 0 num samples 13 - rank: 1 max len: 656 min len: 299 avg len: 614.5384615384615 num_loss_counted_tokens: 7300
 total length: 7778 dropped: 0 num samples 15 - rank: 4 max len: 537 min len: 493 avg len: 518.5333333333333 num_loss_counted_tokens: 6983
 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 541 min len: 450 avg len: 499.75 num_loss_counted_tokens: 7148 total length: 7749 dropped: 0 num samples 16 - rank: 5 max len: 516 min len: 454 avg len: 484.3125 num_loss_counted_tokens: 6901

 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 487 min len: 307 avg len: 444.27777777777777 num_loss_counted_tokens: 7043 total length: 7996 dropped: 0 num samples 18 - rank: 5 max len: 471 min len: 338 avg len: 444.22222222222223 num_loss_counted_tokens: 7042

 total length: 7994 dropped: 0 num samples 21 - rank: 6 max len: 412 min len: 302 avg len: 380.6666666666667 num_loss_counted_tokens: 6881 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 524 min len: 392 avg len: 499.8125 num_loss_counted_tokens: 7149

 total length: 7924 dropped: 0 num samples 17 - rank: 5 max len: 491 min len: 434 avg len: 466.11764705882354 num_loss_counted_tokens: 7023
 total length: 7991 dropped: 0 num samples 20 - rank: 6 max len: 418 min len: 388 avg len: 399.55 num_loss_counted_tokens: 6931
 total length: 7816 dropped: 0 num samples 16 - rank: 4 max len: 512 min len: 469 avg len: 488.5 num_loss_counted_tokens: 6968
 total length: 7876 dropped: 0 num samples 20 - rank: 6 max len: 421 min len: 365 avg len: 393.8 num_loss_counted_tokens: 6816
 total length: 7463 dropped: 0 num samples 21 - rank: 7 max len: 400 min len: 270 avg len: 355.3809523809524 num_loss_counted_tokens: 6350
 total length: 7590 dropped: 0 num samples 23 - rank: 7 max len: 382 min len: 274 avg len: 330.0 num_loss_counted_tokens: 6371
 total length: 7648 dropped: 0 num samples 24 - rank: 7 max len: 365 min len: 248 avg len: 318.6666666666667 num_loss_counted_tokens: 6376
 total length: 7810 dropped: 0 num samples 20 - rank: 6 max len: 423 min len: 366 avg len: 390.5 num_loss_counted_tokens: 6750
 total length: 7538 dropped: 0 num samples 24 - rank: 7 max len: 351 min len: 257 avg len: 314.0833333333333 num_loss_counted_tokens: 6266
 total length: 7844 dropped: 0 num samples 16 - rank: 4 max len: 517 min len: 473 avg len: 490.25 num_loss_counted_tokens: 6996
 total length: 7995 dropped: 0 num samples 19 - rank: 6 max len: 452 min len: 383 avg len: 420.7894736842105 num_loss_counted_tokens: 6988
 total length: 7938 dropped: 0 num samples 25 - rank: 7 max len: 357 min len: 263 avg len: 317.52 num_loss_counted_tokens: 6613
 total length: 7997 dropped: 0 num samples 19 - rank: 6 max len: 449 min len: 294 avg len: 420.89473684210526 num_loss_counted_tokens: 6990
 total length: 7997 dropped: 0 num samples 15 - rank: 4 max len: 573 min len: 417 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7961 dropped: 0 num samples 16 - rank: 4 max len: 525 min len: 475 avg len: 497.5625 num_loss_counted_tokens: 7113
 total length: 7467 dropped: 0 num samples 22 - rank: 7 max len: 388 min len: 255 avg len: 339.40909090909093 num_loss_counted_tokens: 6301
 total length: 7896 dropped: 0 num samples 20 - rank: 6 max len: 432 min len: 372 avg len: 394.8 num_loss_counted_tokens: 6836
 total length: 7994 dropped: 0 num samples 17 - rank: 5 max len: 504 min len: 273 avg len: 470.2352941176471 num_loss_counted_tokens: 7093
 total length: 7888 dropped: 0 num samples 18 - rank: 5 max len: 460 min len: 412 avg len: 438.22222222222223 num_loss_counted_tokens: 6934
 total length: 7997 dropped: 0 num samples 13 - rank: 1 max len: 705 min len: 180 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7996 dropped: 0 num samples 18 - rank: 5 max len: 469 min len: 287 avg len: 444.22222222222223 num_loss_counted_tokens: 7042
 total length: 7797 dropped: 0 num samples 16 - rank: 4 max len: 519 min len: 461 avg len: 487.3125 num_loss_counted_tokens: 6949
 total length: 7901 dropped: 0 num samples 20 - rank: 6 max len: 411 min len: 375 avg len: 395.05 num_loss_counted_tokens: 6841
 total length: 7796 dropped: 0 num samples 20 - rank: 6 max len: 408 min len: 358 avg len: 389.8 num_loss_counted_tokens: 6736
 total length: 7984 dropped: 0 num samples 20 - rank: 6 max len: 433 min len: 371 avg len: 399.2 num_loss_counted_tokens: 6924
 total length: 7958 dropped: 0 num samples 18 - rank: 5 max len: 462 min len: 418 avg len: 442.1111111111111 num_loss_counted_tokens: 7004
 total length: 7869 dropped: 0 num samples 18 - rank: 5 max len: 454 min len: 413 avg len: 437.1666666666667 num_loss_counted_tokens: 6915
 total length: 7713 dropped: 0 num samples 24 - rank: 7 max len: 366 min len: 243 avg len: 321.375 num_loss_counted_tokens: 6441
 total length: 7996 dropped: 0 num samples 17 - rank: 4 max len: 504 min len: 328 avg len: 470.3529411764706 num_loss_counted_tokens: 7095
 total length: 7977 dropped: 0 num samples 18 - rank: 5 max len: 479 min len: 425 avg len: 443.1666666666667 num_loss_counted_tokens: 7023
 total length: 7805 dropped: 0 num samples 18 - rank: 5 max len: 456 min len: 413 avg len: 433.6111111111111 num_loss_counted_tokens: 6851
 total length: 7733 dropped: 0 num samples 16 - rank: 4 max len: 523 min len: 456 avg len: 483.3125 num_loss_counted_tokens: 6885
 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 537 min len: 364 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7997 dropped: 0 num samples 15 - rank: 4 max len: 561 min len: 501 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7861 dropped: 0 num samples 20 - rank: 6 max len: 412 min len: 365 avg len: 393.05 num_loss_counted_tokens: 6801
 total length: 7583 dropped: 0 num samples 23 - rank: 7 max len: 371 min len: 279 avg len: 329.69565217391306 num_loss_counted_tokens: 6364
 total length: 7947 dropped: 0 num samples 25 - rank: 7 max len: 370 min len: 236 avg len: 317.88 num_loss_counted_tokens: 6622
 total length: 7888 dropped: 0 num samples 25 - rank: 7 max len: 366 min len: 263 avg len: 315.52 num_loss_counted_tokens: 6563
 total length: 7997 dropped: 0 num samples 19 - rank: 6 max len: 450 min len: 386 avg len: 420.89473684210526 num_loss_counted_tokens: 6990
 total length: 7692 dropped: 0 num samples 24 - rank: 7 max len: 356 min len: 263 avg len: 320.5 num_loss_counted_tokens: 6420
 total length: 7884 dropped: 0 num samples 19 - rank: 6 max len: 453 min len: 386 avg len: 414.94736842105266 num_loss_counted_tokens: 6877
 total length: 7481 dropped: 0 num samples 23 - rank: 7 max len: 370 min len: 261 avg len: 325.2608695652174 num_loss_counted_tokens: 6262
 total length: 7813 dropped: 0 num samples 25 - rank: 7 max len: 365 min len: 253 avg len: 312.52 num_loss_counted_tokens: 6488
 total length: 7883 dropped: 0 num samples 16 - rank: 4 max len: 512 min len: 480 avg len: 492.6875 num_loss_counted_tokens: 7035
 total length: 7993 dropped: 0 num samples 16 - rank: 4 max len: 531 min len: 259 avg len: 499.5625 num_loss_counted_tokens: 7145
 total length: 7943 dropped: 0 num samples 20 - rank: 6 max len: 422 min len: 366 avg len: 397.15 num_loss_counted_tokens: 6883
 total length: 7997 dropped: 0 num samples 17 - rank: 4 max len: 490 min len: 418 avg len: 470.4117647058824 num_loss_counted_tokens: 7096
 total length: 7792 dropped: 0 num samples 23 - rank: 7 max len: 382 min len: 229 avg len: 338.7826086956522 num_loss_counted_tokens: 6573
 total length: 7779 dropped: 0 num samples 20 - rank: 6 max len: 415 min len: 357 avg len: 388.95 num_loss_counted_tokens: 6719
 total length: 7994 dropped: 0 num samples 17 - rank: 5 max len: 512 min len: 320 avg len: 470.2352941176471 num_loss_counted_tokens: 7093
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 473 min len: 385 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 477 min len: 362 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7582 dropped: 0 num samples 23 - rank: 7 max len: 399 min len: 171 avg len: 329.6521739130435 num_loss_counted_tokens: 6363
 total length: 7200 dropped: 0 num samples 22 - rank: 7 max len: 377 min len: 266 avg len: 327.27272727272725 num_loss_counted_tokens: 6034
 total length: 7951 dropped: 0 num samples 15 - rank: 4 max len: 563 min len: 515 avg len: 530.0666666666667 num_loss_counted_tokens: 7156
 total length: 7864 dropped: 0 num samples 18 - rank: 5 max len: 455 min len: 416 avg len: 436.8888888888889 num_loss_counted_tokens: 6910
 total length: 7997 dropped: 0 num samples 20 - rank: 6 max len: 421 min len: 379 avg len: 399.85 num_loss_counted_tokens: 6937
 total length: 7993 dropped: 0 num samples 17 - rank: 5 max len: 491 min len: 432 avg len: 470.1764705882353 num_loss_counted_tokens: 7092
 total length: 7869 dropped: 0 num samples 25 - rank: 7 max len: 357 min len: 245 avg len: 314.76 num_loss_counted_tokens: 6544
[19:31:28] INFO     Optimizer state saved in                   fsdp_utils.py:194
                    /var/home/cloud-user/.local/share/instruct                  
                    lab/phased/phase1/checkpoints/full_state/e                  
                    poch_0/optimizer.bin                                        
           INFO     FSDP Optimizer saved to output dir       accelerator.py:3061
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_0                                                  
           INFO     Scheduler state saved in                checkpointing.py:118
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_0/scheduler.bin                                  
           INFO     Sampler state for dataloader 0 saved in checkpointing.py:135
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_0/sampler.bin                                    
           INFO     Random states saved in                  checkpointing.py:160
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_0/random_states_0.pkl                            
Saving training state: {'current_epoch': 0, 'samples_seen': 5066}
Model state saved in: /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/full_state/epoch_0


Epoch 1:   0%|          | 0/40 [00:00<?, ?it/s]
Epoch 0: 100%|██████████| 40/40 [02:43<00:00,  4.09s/it]
 total length: 7995 dropped: 0 num samples 9 - rank: 0 max len: 1185 min len: 383 avg len: 888.3333333333334 num_loss_counted_tokens: 7518
 total length: 7997 dropped: 0 num samples 8 - rank: 0 max len: 1155 min len: 627 avg len: 999.625 num_loss_counted_tokens: 7573
 total length: 7993 dropped: 0 num samples 9 - rank: 0 max len: 1098 min len: 476 avg len: 888.1111111111111 num_loss_counted_tokens: 7516
 total length: 7996 dropped: 0 num samples 11 - rank: 0 max len: 1104 min len: 283 avg len: 726.9090909090909 num_loss_counted_tokens: 7413
 total length: 7983 dropped: 0 num samples 10 - rank: 0 max len: 1143 min len: 182 avg len: 798.3 num_loss_counted_tokens: 7453
 total length: 7970 dropped: 0 num samples 10 - rank: 0 max len: 1103 min len: 661 avg len: 797.0 num_loss_counted_tokens: 7440
 total length: 7996 dropped: 0 num samples 11 - rank: 0 max len: 1094 min len: 407 avg len: 726.9090909090909 num_loss_counted_tokens: 7413
 total length: 7995 dropped: 0 num samples 9 - rank: 0 max len: 1094 min len: 485 avg len: 888.3333333333334 num_loss_counted_tokens: 7518
 total length: 7989 dropped: 0 num samples 10 - rank: 0 max len: 1118 min len: 659 avg len: 798.9 num_loss_counted_tokens: 7459
 total length: 7982 dropped: 0 num samples 10 - rank: 0 max len: 968 min len: 678 avg len: 798.2 num_loss_counted_tokens: 7452
 total length: 7828 dropped: 0 num samples 8 - rank: 0 max len: 1110 min len: 814 avg len: 978.5 num_loss_counted_tokens: 7404
 total length: 7995 dropped: 0 num samples 11 - rank: 0 max len: 1015 min len: 648 avg len: 726.8181818181819 num_loss_counted_tokens: 7412
 total length: 7995 dropped: 0 num samples 9 - rank: 0 max len: 1247 min len: 539 avg len: 888.3333333333334 num_loss_counted_tokens: 7518
 total length: 7784 dropped: 0 num samples 8 - rank: 0 max len: 1084 min len: 795 avg len: 973.0 num_loss_counted_tokens: 7360
 total length: 7996 dropped: 0 num samples 9 - rank: 0 max len: 1072 min len: 402 avg len: 888.4444444444445 num_loss_counted_tokens: 7519
 total length: 7907 dropped: 0 num samples 10 - rank: 0 max len: 1062 min len: 660 avg len: 790.7 num_loss_counted_tokens: 7377
 total length: 7890 dropped: 0 num samples 8 - rank: 0 max len: 1309 min len: 825 avg len: 986.25 num_loss_counted_tokens: 7466
Epoch: 1, Step: 41, Rank: 3, loss = 0.0859375Epoch: 1, Step: 41, Rank: 6, loss = 0.09375
Epoch: 1, Step: 41, Rank: 4, loss = 0.08642578125Epoch: 1, Step: 41, Rank: 5, loss = 0.09716796875


Epoch: 1, Step: 41, Rank: 7, loss = 0.103515625Epoch: 1, Step: 41, Rank: 2, loss = 0.099609375

Epoch: 1, Step: 41, Rank: 1, loss = 0.0927734375
Epoch: 1, Step: 41, Rank: 0, loss = 0.06591796875


Epoch 1:   2%|▎         | 1/40 [00:01<01:02,  1.60s/it]{
    "epoch": 1,
    "step": 41,
    "rank": 0,
    "overall_throughput": 94.22234484017399,
    "lr": 1.9806347704689778e-05,
    "cuda_mem_allocated": 6.588063716888428,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56870,
    "batch_size": 127,
    "total_loss": 0.09076841920168806,
    "samples_seen": 5193,
    "gradnorm": 1.359375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:30.544867"
}
 total length: 7988 dropped: 0 num samples 24 - rank: 7 max len: 390 min len: 265 avg len: 332.8333333333333 num_loss_counted_tokens: 6716
 total length: 7978 dropped: 0 num samples 9 - rank: 0 max len: 1062 min len: 673 avg len: 886.4444444444445 num_loss_counted_tokens: 7501
 total length: 7951 dropped: 0 num samples 17 - rank: 5 max len: 485 min len: 453 avg len: 467.70588235294116 num_loss_counted_tokens: 7050
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 699 min len: 643 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 647 min len: 381 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7736 dropped: 0 num samples 13 - rank: 3 max len: 620 min len: 565 avg len: 595.0769230769231 num_loss_counted_tokens: 7047
 total length: 7929 dropped: 0 num samples 15 - rank: 4 max len: 557 min len: 485 avg len: 528.6 num_loss_counted_tokens: 7134
 total length: 7935 dropped: 0 num samples 19 - rank: 6 max len: 452 min len: 391 avg len: 417.63157894736844 num_loss_counted_tokens: 6928
Epoch: 1, Step: 42, Rank: 3, loss = 0.076171875Epoch: 1, Step: 42, Rank: 6, loss = 0.11279296875Epoch: 1, Step: 42, Rank: 2, loss = 0.1064453125Epoch: 1, Step: 42, Rank: 1, loss = 0.08251953125Epoch: 1, Step: 42, Rank: 4, loss = 0.08984375

Epoch: 1, Step: 42, Rank: 5, loss = 0.09326171875
Epoch: 1, Step: 42, Rank: 7, loss = 0.09375



Epoch: 1, Step: 42, Rank: 0, loss = 0.036865234375


Epoch 1:   5%|▌         | 2/40 [00:03<01:01,  1.63s/it]{
    "epoch": 1,
    "step": 42,
    "rank": 0,
    "overall_throughput": 93.95825850159173,
    "lr": 1.9781476007338058e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56751,
    "batch_size": 117,
    "total_loss": 0.08637733255801659,
    "samples_seen": 5310,
    "gradnorm": 0.380859375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:32.192071"
}
 total length: 7993 dropped: 0 num samples 9 - rank: 0 max len: 1098 min len: 375 avg len: 888.1111111111111 num_loss_counted_tokens: 7516
 total length: 7910 dropped: 0 num samples 25 - rank: 7 max len: 370 min len: 242 avg len: 316.4 num_loss_counted_tokens: 6585
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 744 min len: 374 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
 total length: 7817 dropped: 0 num samples 18 - rank: 5 max len: 463 min len: 416 avg len: 434.27777777777777 num_loss_counted_tokens: 6863
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 649 min len: 407 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7992 dropped: 0 num samples 14 - rank: 3 max len: 603 min len: 537 avg len: 570.8571428571429 num_loss_counted_tokens: 7250
 total length: 7939 dropped: 0 num samples 20 - rank: 6 max len: 415 min len: 372 avg len: 396.95 num_loss_counted_tokens: 6879
 total length: 7892 dropped: 0 num samples 16 - rank: 4 max len: 528 min len: 468 avg len: 493.25 num_loss_counted_tokens: 7044
Epoch: 1, Step: 43, Rank: 5, loss = 0.08935546875Epoch: 1, Step: 43, Rank: 6, loss = 0.1123046875Epoch: 1, Step: 43, Rank: 3, loss = 0.076171875Epoch: 1, Step: 43, Rank: 4, loss = 0.07177734375Epoch: 1, Step: 43, Rank: 1, loss = 0.044677734375Epoch: 1, Step: 43, Rank: 7, loss = 0.111328125
Epoch: 1, Step: 43, Rank: 2, loss = 0.07275390625





Epoch: 1, Step: 43, Rank: 0, loss = 0.056396484375


Epoch 1:   8%|▊         | 3/40 [00:04<01:00,  1.63s/it]{
    "epoch": 1,
    "step": 43,
    "rank": 0,
    "overall_throughput": 94.36342838501662,
    "lr": 1.975511967980437e-05,
    "cuda_mem_allocated": 6.588063716888428,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56436,
    "batch_size": 129,
    "total_loss": 0.07931107803529662,
    "samples_seen": 5439,
    "gradnorm": 0.33203125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:33.832929"
}
 total length: 7991 dropped: 0 num samples 9 - rank: 0 max len: 1482 min len: 191 avg len: 887.8888888888889 num_loss_counted_tokens: 7514
 total length: 7867 dropped: 0 num samples 17 - rank: 5 max len: 481 min len: 438 avg len: 462.7647058823529 num_loss_counted_tokens: 6966
 total length: 7877 dropped: 0 num samples 13 - rank: 2 max len: 648 min len: 575 avg len: 605.9230769230769 num_loss_counted_tokens: 7188
 total length: 7997 dropped: 0 num samples 20 - rank: 6 max len: 434 min len: 275 avg len: 399.85 num_loss_counted_tokens: 6937
 total length: 7993 dropped: 0 num samples 15 - rank: 3 max len: 573 min len: 395 avg len: 532.8666666666667 num_loss_counted_tokens: 7198
 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 514 min len: 479 avg len: 499.75 num_loss_counted_tokens: 7148
 total length: 7740 dropped: 0 num samples 24 - rank: 7 max len: 371 min len: 257 avg len: 322.5 num_loss_counted_tokens: 6468
 total length: 7993 dropped: 0 num samples 12 - rank: 1 max len: 698 min len: 544 avg len: 666.0833333333334 num_loss_counted_tokens: 7357
Epoch: 1, Step: 44, Rank: 5, loss = 0.11376953125Epoch: 1, Step: 44, Rank: 1, loss = 0.07958984375Epoch: 1, Step: 44, Rank: 6, loss = 0.1103515625Epoch: 1, Step: 44, Rank: 4, loss = 0.08984375


Epoch: 1, Step: 44, Rank: 2, loss = 0.06787109375
Epoch: 1, Step: 44, Rank: 3, loss = 0.06982421875

Epoch: 1, Step: 44, Rank: 7, loss = 0.10791015625
Epoch: 1, Step: 44, Rank: 0, loss = 0.046630859375


Epoch 1:  10%|█         | 4/40 [00:06<00:59,  1.64s/it]{
    "epoch": 1,
    "step": 44,
    "rank": 0,
    "overall_throughput": 93.65679597203368,
    "lr": 1.972728272244605e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56749,
    "batch_size": 124,
    "total_loss": 0.08567551851133941,
    "samples_seen": 5563,
    "gradnorm": 0.32421875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:35.480858"
}
 total length: 7108 dropped: 0 num samples 21 - rank: 7 max len: 378 min len: 257 avg len: 338.4761904761905 num_loss_counted_tokens: 5995
 total length: 7972 dropped: 0 num samples 18 - rank: 5 max len: 480 min len: 172 avg len: 442.8888888888889 num_loss_counted_tokens: 7018
 total length: 7995 dropped: 0 num samples 13 - rank: 2 max len: 636 min len: 532 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7992 dropped: 0 num samples 14 - rank: 3 max len: 587 min len: 526 avg len: 570.8571428571429 num_loss_counted_tokens: 7250
 total length: 7834 dropped: 0 num samples 9 - rank: 0 max len: 1064 min len: 716 avg len: 870.4444444444445 num_loss_counted_tokens: 7357
 total length: 7985 dropped: 0 num samples 12 - rank: 1 max len: 700 min len: 599 avg len: 665.4166666666666 num_loss_counted_tokens: 7349
 total length: 7995 dropped: 0 num samples 20 - rank: 6 max len: 432 min len: 328 avg len: 399.75 num_loss_counted_tokens: 6935
 total length: 7995 dropped: 0 num samples 16 - rank: 4 max len: 546 min len: 347 avg len: 499.6875 num_loss_counted_tokens: 7147
Epoch: 1, Step: 45, Rank: 1, loss = 0.061767578125Epoch: 1, Step: 45, Rank: 6, loss = 0.1298828125Epoch: 1, Step: 45, Rank: 3, loss = 0.0751953125Epoch: 1, Step: 45, Rank: 4, loss = 0.11181640625Epoch: 1, Step: 45, Rank: 5, loss = 0.1025390625Epoch: 1, Step: 45, Rank: 7, loss = 0.10009765625
Epoch: 1, Step: 45, Rank: 2, loss = 0.0546875





Epoch: 1, Step: 45, Rank: 0, loss = 0.07861328125


Epoch 1:  12%|█▎        | 5/40 [00:08<00:57,  1.64s/it]{
    "epoch": 1,
    "step": 45,
    "rank": 0,
    "overall_throughput": 95.1525424509851,
    "lr": 1.9697969360350098e-05,
    "cuda_mem_allocated": 6.586104393005371,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56200,
    "batch_size": 129,
    "total_loss": 0.089288256227758,
    "samples_seen": 5692,
    "gradnorm": 0.337890625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:37.117442"
}
 total length: 7992 dropped: 0 num samples 20 - rank: 6 max len: 423 min len: 273 avg len: 399.6 num_loss_counted_tokens: 6932
 total length: 7981 dropped: 0 num samples 18 - rank: 5 max len: 471 min len: 424 avg len: 443.3888888888889 num_loss_counted_tokens: 7027
 total length: 7879 dropped: 0 num samples 14 - rank: 3 max len: 589 min len: 537 avg len: 562.7857142857143 num_loss_counted_tokens: 7137
 total length: 7995 dropped: 0 num samples 8 - rank: 0 max len: 1391 min len: 630 avg len: 999.375 num_loss_counted_tokens: 7571
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 739 min len: 361 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7994 dropped: 0 num samples 13 - rank: 2 max len: 665 min len: 412 avg len: 614.9230769230769 num_loss_counted_tokens: 7305
 total length: 7965 dropped: 0 num samples 16 - rank: 4 max len: 532 min len: 473 avg len: 497.8125 num_loss_counted_tokens: 7117
 total length: 7532 dropped: 0 num samples 24 - rank: 7 max len: 373 min len: 255 avg len: 313.8333333333333 num_loss_counted_tokens: 6260
Epoch: 1, Step: 46, Rank: 6, loss = 0.119140625Epoch: 1, Step: 46, Rank: 3, loss = 0.0634765625Epoch: 1, Step: 46, Rank: 4, loss = 0.0859375Epoch: 1, Step: 46, Rank: 5, loss = 0.1064453125
Epoch: 1, Step: 46, Rank: 7, loss = 0.10400390625
Epoch: 1, Step: 46, Rank: 1, loss = 0.06201171875

Epoch: 1, Step: 46, Rank: 2, loss = 0.06884765625


Epoch: 1, Step: 46, Rank: 0, loss = 0.07861328125


Epoch 1:  15%|█▌        | 6/40 [00:10<00:59,  1.74s/it]{
    "epoch": 1,
    "step": 46,
    "rank": 0,
    "overall_throughput": 77.82207451639056,
    "lr": 1.9667184042691877e-05,
    "cuda_mem_allocated": 6.586573123931885,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56587,
    "batch_size": 128,
    "total_loss": 0.08602682594942301,
    "samples_seen": 5820,
    "gradnorm": 0.33203125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:39.049852"
}
 total length: 7771 dropped: 0 num samples 12 - rank: 1 max len: 657 min len: 638 avg len: 647.5833333333334 num_loss_counted_tokens: 7135
 total length: 7698 dropped: 0 num samples 24 - rank: 7 max len: 361 min len: 276 avg len: 320.75 num_loss_counted_tokens: 6426
 total length: 7915 dropped: 0 num samples 13 - rank: 2 max len: 635 min len: 579 avg len: 608.8461538461538 num_loss_counted_tokens: 7226
 total length: 7993 dropped: 0 num samples 15 - rank: 3 max len: 573 min len: 298 avg len: 532.8666666666667 num_loss_counted_tokens: 7198
 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 517 min len: 438 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7993 dropped: 0 num samples 11 - rank: 0 max len: 1052 min len: 521 avg len: 726.6363636363636 num_loss_counted_tokens: 7410
 total length: 7963 dropped: 0 num samples 18 - rank: 5 max len: 477 min len: 414 avg len: 442.3888888888889 num_loss_counted_tokens: 7009
 total length: 7990 dropped: 0 num samples 21 - rank: 6 max len: 414 min len: 253 avg len: 380.4761904761905 num_loss_counted_tokens: 6877
Epoch: 1, Step: 47, Rank: 5, loss = 0.0810546875Epoch: 1, Step: 47, Rank: 6, loss = 0.0947265625Epoch: 1, Step: 47, Rank: 7, loss = 0.10205078125Epoch: 1, Step: 47, Rank: 4, loss = 0.087890625Epoch: 1, Step: 47, Rank: 3, loss = 0.08203125Epoch: 1, Step: 47, Rank: 1, loss = 0.058349609375Epoch: 1, Step: 47, Rank: 2, loss = 0.05859375






Epoch: 1, Step: 47, Rank: 0, loss = 0.0859375


Epoch 1:  18%|█▊        | 7/40 [00:11<00:56,  1.71s/it]{
    "epoch": 1,
    "step": 47,
    "rank": 0,
    "overall_throughput": 94.5612136054345,
    "lr": 1.9634931442059833e-05,
    "cuda_mem_allocated": 6.588063716888428,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56723,
    "batch_size": 123,
    "total_loss": 0.08134266523279798,
    "samples_seen": 5943,
    "gradnorm": 0.31640625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:40.689732"
}
 total length: 7994 dropped: 0 num samples 14 - rank: 3 max len: 617 min len: 387 avg len: 571.0 num_loss_counted_tokens: 7252 total length: 7779 dropped: 0 num samples 23 - rank: 7 max len: 366 min len: 269 avg len: 338.2173913043478 num_loss_counted_tokens: 6560

 total length: 7736 dropped: 0 num samples 8 - rank: 0 max len: 1101 min len: 834 avg len: 967.0 num_loss_counted_tokens: 7312
 total length: 7995 dropped: 0 num samples 17 - rank: 5 max len: 498 min len: 282 avg len: 470.29411764705884 num_loss_counted_tokens: 7094
 total length: 7990 dropped: 0 num samples 13 - rank: 2 max len: 667 min len: 335 avg len: 614.6153846153846 num_loss_counted_tokens: 7301
 total length: 7852 dropped: 0 num samples 15 - rank: 4 max len: 556 min len: 500 avg len: 523.4666666666667 num_loss_counted_tokens: 7057
 total length: 7977 dropped: 0 num samples 11 - rank: 1 max len: 806 min len: 676 avg len: 725.1818181818181 num_loss_counted_tokens: 7394
 total length: 7819 dropped: 0 num samples 19 - rank: 6 max len: 447 min len: 366 avg len: 411.5263157894737 num_loss_counted_tokens: 6812
Epoch: 1, Step: 48, Rank: 0, loss = 0.06640625Epoch: 1, Step: 48, Rank: 5, loss = 0.095703125
Epoch: 1, Step: 48, Rank: 2, loss = 0.06396484375Epoch: 1, Step: 48, Rank: 3, loss = 0.0732421875
Epoch: 1, Step: 48, Rank: 7, loss = 0.0849609375

Epoch: 1, Step: 48, Rank: 6, loss = 0.09375

Epoch: 1, Step: 48, Rank: 4, loss = 0.08203125
Epoch: 1, Step: 48, Rank: 1, loss = 0.052001953125


Epoch 1:  20%|██        | 8/40 [00:13<00:53,  1.68s/it]{
    "epoch": 1,
    "step": 48,
    "rank": 0,
    "overall_throughput": 95.18503437856974,
    "lr": 1.9601216453746285e-05,
    "cuda_mem_allocated": 6.580197811126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56241,
    "batch_size": 129,
    "total_loss": 0.07652780000355612,
    "samples_seen": 6072,
    "gradnorm": 0.35546875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:42.325044"
}
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 461 min len: 427 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7480 dropped: 0 num samples 22 - rank: 7 max len: 381 min len: 253 avg len: 340.0 num_loss_counted_tokens: 6314
 total length: 7993 dropped: 0 num samples 20 - rank: 6 max len: 426 min len: 356 avg len: 399.65 num_loss_counted_tokens: 6933
 total length: 7805 dropped: 0 num samples 12 - rank: 1 max len: 724 min len: 612 avg len: 650.4166666666666 num_loss_counted_tokens: 7169
 total length: 7994 dropped: 0 num samples 17 - rank: 4 max len: 501 min len: 383 avg len: 470.2352941176471 num_loss_counted_tokens: 7093
 total length: 7994 dropped: 0 num samples 9 - rank: 0 max len: 1068 min len: 487 avg len: 888.2222222222222 num_loss_counted_tokens: 7517
 total length: 7966 dropped: 0 num samples 15 - rank: 3 max len: 571 min len: 507 avg len: 531.0666666666667 num_loss_counted_tokens: 7171
 total length: 7754 dropped: 0 num samples 13 - rank: 2 max len: 612 min len: 573 avg len: 596.4615384615385 num_loss_counted_tokens: 7065
Epoch: 1, Step: 49, Rank: 5, loss = 0.09228515625Epoch: 1, Step: 49, Rank: 6, loss = 0.10791015625Epoch: 1, Step: 49, Rank: 4, loss = 0.068359375Epoch: 1, Step: 49, Rank: 7, loss = 0.10546875Epoch: 1, Step: 49, Rank: 3, loss = 0.061767578125


Epoch: 1, Step: 49, Rank: 2, loss = 0.08056640625
Epoch: 1, Step: 49, Rank: 1, loss = 0.052734375


Epoch: 1, Step: 49, Rank: 0, loss = 0.054931640625


Epoch 1:  22%|██▎       | 9/40 [00:15<00:51,  1.67s/it]{
    "epoch": 1,
    "step": 49,
    "rank": 0,
    "overall_throughput": 94.36893520621318,
    "lr": 1.956604419500441e-05,
    "cuda_mem_allocated": 6.588063716888428,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56467,
    "batch_size": 126,
    "total_loss": 0.07799245577062709,
    "samples_seen": 6198,
    "gradnorm": 0.310546875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:43.971765"
}
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 683 min len: 652 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
 total length: 7973 dropped: 0 num samples 26 - rank: 7 max len: 349 min len: 164 avg len: 306.65384615384613 num_loss_counted_tokens: 6595
 total length: 7838 dropped: 0 num samples 17 - rank: 5 max len: 491 min len: 429 avg len: 461.05882352941177 num_loss_counted_tokens: 6937
 total length: 7996 dropped: 0 num samples 10 - rank: 0 max len: 1164 min len: 532 avg len: 799.6 num_loss_counted_tokens: 7466
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 655 min len: 397 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7986 dropped: 0 num samples 16 - rank: 4 max len: 546 min len: 233 avg len: 499.125 num_loss_counted_tokens: 7138
 total length: 7996 dropped: 0 num samples 14 - rank: 3 max len: 614 min len: 400 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7991 dropped: 0 num samples 21 - rank: 6 max len: 426 min len: 259 avg len: 380.5238095238095 num_loss_counted_tokens: 6878
Epoch: 1, Step: 50, Rank: 6, loss = 0.08837890625Epoch: 1, Step: 50, Rank: 7, loss = 0.1064453125Epoch: 1, Step: 50, Rank: 3, loss = 0.07275390625
Epoch: 1, Step: 50, Rank: 1, loss = 0.0703125
Epoch: 1, Step: 50, Rank: 2, loss = 0.0615234375


Epoch: 1, Step: 50, Rank: 4, loss = 0.068359375Epoch: 1, Step: 50, Rank: 5, loss = 0.07958984375

Epoch: 1, Step: 50, Rank: 0, loss = 0.031494140625


Epoch 1:  25%|██▌       | 10/40 [00:17<00:56,  1.87s/it]{
    "epoch": 1,
    "step": 50,
    "rank": 0,
    "overall_throughput": 63.54496076190855,
    "lr": 1.9529420004271568e-05,
    "cuda_mem_allocated": 6.572385311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56604,
    "batch_size": 126,
    "total_loss": 0.07232704402515723,
    "samples_seen": 6324,
    "gradnorm": 0.306640625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:46.280179"
}
 total length: 7896 dropped: 0 num samples 12 - rank: 1 max len: 677 min len: 639 avg len: 658.0 num_loss_counted_tokens: 7260
 total length: 7977 dropped: 0 num samples 21 - rank: 6 max len: 410 min len: 245 avg len: 379.85714285714283 num_loss_counted_tokens: 6864
 total length: 7970 dropped: 0 num samples 19 - rank: 5 max len: 446 min len: 253 avg len: 419.4736842105263 num_loss_counted_tokens: 6963
 total length: 7851 dropped: 0 num samples 16 - rank: 4 max len: 526 min len: 448 avg len: 490.6875 num_loss_counted_tokens: 7003
 total length: 7797 dropped: 0 num samples 14 - rank: 3 max len: 596 min len: 533 avg len: 556.9285714285714 num_loss_counted_tokens: 7055
 total length: 7684 dropped: 0 num samples 24 - rank: 7 max len: 361 min len: 202 avg len: 320.1666666666667 num_loss_counted_tokens: 6412
 total length: 7974 dropped: 0 num samples 10 - rank: 0 max len: 1076 min len: 256 avg len: 797.4 num_loss_counted_tokens: 7444
 total length: 7991 dropped: 0 num samples 13 - rank: 2 max len: 636 min len: 492 avg len: 614.6923076923077 num_loss_counted_tokens: 7302
Epoch: 1, Step: 51, Rank: 4, loss = 0.09521484375Epoch: 1, Step: 51, Rank: 5, loss = 0.111328125Epoch: 1, Step: 51, Rank: 3, loss = 0.06689453125Epoch: 1, Step: 51, Rank: 2, loss = 0.09033203125Epoch: 1, Step: 51, Rank: 6, loss = 0.09326171875Epoch: 1, Step: 51, Rank: 7, loss = 0.09228515625Epoch: 1, Step: 51, Rank: 1, loss = 0.06884765625






Epoch: 1, Step: 51, Rank: 0, loss = 0.04638671875


Epoch 1:  28%|██▊       | 11/40 [00:18<00:52,  1.80s/it]{
    "epoch": 1,
    "step": 51,
    "rank": 0,
    "overall_throughput": 94.46210112056878,
    "lr": 1.9491349440359014e-05,
    "cuda_mem_allocated": 6.588062286376953,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56256,
    "batch_size": 133,
    "total_loss": 0.08304891922639362,
    "samples_seen": 6457,
    "gradnorm": 0.330078125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:47.928027"
}
 total length: 7829 dropped: 0 num samples 19 - rank: 6 max len: 436 min len: 379 avg len: 412.05263157894734 num_loss_counted_tokens: 6822
 total length: 7905 dropped: 0 num samples 24 - rank: 7 max len: 379 min len: 236 avg len: 329.375 num_loss_counted_tokens: 6633
 total length: 7997 dropped: 0 num samples 9 - rank: 0 max len: 1058 min len: 645 avg len: 888.5555555555555 num_loss_counted_tokens: 7520
 total length: 7984 dropped: 0 num samples 16 - rank: 4 max len: 539 min len: 310 avg len: 499.0 num_loss_counted_tokens: 7136
 total length: 7767 dropped: 0 num samples 14 - rank: 3 max len: 570 min len: 542 avg len: 554.7857142857143 num_loss_counted_tokens: 7025
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 729 min len: 600 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
 total length: 7994 dropped: 0 num samples 18 - rank: 5 max len: 473 min len: 243 avg len: 444.1111111111111 num_loss_counted_tokens: 7040
 total length: 7842 dropped: 0 num samples 13 - rank: 2 max len: 627 min len: 580 avg len: 603.2307692307693 num_loss_counted_tokens: 7153
Epoch: 1, Step: 52, Rank: 6, loss = 0.125Epoch: 1, Step: 52, Rank: 3, loss = 0.0693359375
Epoch: 1, Step: 52, Rank: 1, loss = 0.07568359375Epoch: 1, Step: 52, Rank: 5, loss = 0.0703125Epoch: 1, Step: 52, Rank: 7, loss = 0.07275390625


Epoch: 1, Step: 52, Rank: 2, loss = 0.06103515625

Epoch: 1, Step: 52, Rank: 0, loss = 0.06396484375
Epoch: 1, Step: 52, Rank: 4, loss = 0.08349609375


Epoch 1:  30%|███       | 12/40 [00:20<00:49,  1.76s/it]{
    "epoch": 1,
    "step": 52,
    "rank": 0,
    "overall_throughput": 94.0445356271003,
    "lr": 1.94518382816082e-05,
    "cuda_mem_allocated": 6.58648157119751,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56189,
    "batch_size": 128,
    "total_loss": 0.07773763548025414,
    "samples_seen": 6585,
    "gradnorm": 0.34765625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:49.579915"
}
 total length: 7748 dropped: 0 num samples 23 - rank: 7 max len: 386 min len: 253 avg len: 336.8695652173913 num_loss_counted_tokens: 6529
 total length: 7942 dropped: 0 num samples 12 - rank: 1 max len: 687 min len: 635 avg len: 661.8333333333334 num_loss_counted_tokens: 7306
 total length: 7873 dropped: 0 num samples 16 - rank: 4 max len: 506 min len: 477 avg len: 492.0625 num_loss_counted_tokens: 7025
 total length: 7777 dropped: 0 num samples 17 - rank: 5 max len: 477 min len: 428 avg len: 457.47058823529414 num_loss_counted_tokens: 6876
 total length: 7849 dropped: 0 num samples 13 - rank: 2 max len: 628 min len: 574 avg len: 603.7692307692307 num_loss_counted_tokens: 7160
 total length: 7763 dropped: 0 num samples 10 - rank: 0 max len: 1003 min len: 688 avg len: 776.3 num_loss_counted_tokens: 7233
 total length: 7787 dropped: 0 num samples 19 - rank: 6 max len: 426 min len: 390 avg len: 409.8421052631579 num_loss_counted_tokens: 6780
 total length: 7994 dropped: 0 num samples 15 - rank: 3 max len: 573 min len: 457 avg len: 532.9333333333333 num_loss_counted_tokens: 7199
Epoch: 1, Step: 53, Rank: 2, loss = 0.0771484375Epoch: 1, Step: 53, Rank: 5, loss = 0.09375Epoch: 1, Step: 53, Rank: 7, loss = 0.0859375

Epoch: 1, Step: 53, Rank: 3, loss = 0.06298828125Epoch: 1, Step: 53, Rank: 6, loss = 0.09423828125
Epoch: 1, Step: 53, Rank: 4, loss = 0.06689453125

Epoch: 1, Step: 53, Rank: 0, loss = 0.0859375

Epoch: 1, Step: 53, Rank: 1, loss = 0.07177734375


Epoch 1:  32%|███▎      | 13/40 [00:22<00:48,  1.81s/it]{
    "epoch": 1,
    "step": 53,
    "rank": 0,
    "overall_throughput": 77.31588483240525,
    "lr": 1.9410892525013717e-05,
    "cuda_mem_allocated": 6.588063716888428,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56356,
    "batch_size": 126,
    "total_loss": 0.07974306196323373,
    "samples_seen": 6711,
    "gradnorm": 0.34765625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:51.526772"
}
 total length: 7992 dropped: 0 num samples 15 - rank: 3 max len: 571 min len: 318 avg len: 532.8 num_loss_counted_tokens: 7197
 total length: 7908 dropped: 0 num samples 12 - rank: 1 max len: 681 min len: 629 avg len: 659.0 num_loss_counted_tokens: 7272
 total length: 7838 dropped: 0 num samples 13 - rank: 2 max len: 624 min len: 574 avg len: 602.9230769230769 num_loss_counted_tokens: 7149
 total length: 7879 dropped: 0 num samples 16 - rank: 4 max len: 516 min len: 476 avg len: 492.4375 num_loss_counted_tokens: 7031
 total length: 7995 dropped: 0 num samples 11 - rank: 0 max len: 871 min len: 438 avg len: 726.8181818181819 num_loss_counted_tokens: 7412
 total length: 7921 dropped: 0 num samples 19 - rank: 5 max len: 465 min len: 172 avg len: 416.89473684210526 num_loss_counted_tokens: 6914
 total length: 7947 dropped: 0 num samples 25 - rank: 7 max len: 366 min len: 250 avg len: 317.88 num_loss_counted_tokens: 6622
 total length: 7793 dropped: 0 num samples 20 - rank: 6 max len: 406 min len: 369 avg len: 389.65 num_loss_counted_tokens: 6733
Epoch: 1, Step: 54, Rank: 5, loss = 0.11279296875Epoch: 1, Step: 54, Rank: 7, loss = 0.09130859375Epoch: 1, Step: 54, Rank: 2, loss = 0.06884765625Epoch: 1, Step: 54, Rank: 1, loss = 0.07080078125Epoch: 1, Step: 54, Rank: 6, loss = 0.06494140625
Epoch: 1, Step: 54, Rank: 4, loss = 0.0751953125

Epoch: 1, Step: 54, Rank: 3, loss = 0.0654296875



Epoch: 1, Step: 54, Rank: 0, loss = 0.0380859375


Epoch 1:  35%|███▌      | 14/40 [00:24<00:48,  1.85s/it]{
    "epoch": 1,
    "step": 54,
    "rank": 0,
    "overall_throughput": 78.02629318983507,
    "lr": 1.9368518385313108e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56435,
    "batch_size": 133,
    "total_loss": 0.07346504828563835,
    "samples_seen": 6844,
    "gradnorm": 0.310546875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:53.459531"
}
 total length: 7962 dropped: 0 num samples 9 - rank: 0 max len: 1140 min len: 719 avg len: 884.6666666666666 num_loss_counted_tokens: 7485
 total length: 7993 dropped: 0 num samples 15 - rank: 3 max len: 592 min len: 248 avg len: 532.8666666666667 num_loss_counted_tokens: 7198
 total length: 7992 dropped: 0 num samples 13 - rank: 2 max len: 632 min len: 545 avg len: 614.7692307692307 num_loss_counted_tokens: 7303
 total length: 7993 dropped: 0 num samples 12 - rank: 1 max len: 708 min len: 583 avg len: 666.0833333333334 num_loss_counted_tokens: 7357
 total length: 7793 dropped: 0 num samples 24 - rank: 7 max len: 364 min len: 267 avg len: 324.7083333333333 num_loss_counted_tokens: 6521
 total length: 7781 dropped: 0 num samples 20 - rank: 6 max len: 422 min len: 366 avg len: 389.05 num_loss_counted_tokens: 6721
 total length: 7981 dropped: 0 num samples 16 - rank: 4 max len: 524 min len: 461 avg len: 498.8125 num_loss_counted_tokens: 7133
 total length: 7967 dropped: 0 num samples 18 - rank: 5 max len: 480 min len: 427 avg len: 442.6111111111111 num_loss_counted_tokens: 7013
Epoch: 1, Step: 55, Rank: 0, loss = 0.026123046875Epoch: 1, Step: 55, Rank: 6, loss = 0.0859375Epoch: 1, Step: 55, Rank: 5, loss = 0.0654296875

Epoch: 1, Step: 55, Rank: 7, loss = 0.09619140625
Epoch: 1, Step: 55, Rank: 1, loss = 0.055908203125
Epoch: 1, Step: 55, Rank: 2, loss = 0.0595703125

Epoch: 1, Step: 55, Rank: 3, loss = 0.052978515625
Epoch: 1, Step: 55, Rank: 4, loss = 0.07861328125


Epoch 1:  38%|███▊      | 15/40 [00:26<00:46,  1.88s/it]{
    "epoch": 1,
    "step": 55,
    "rank": 0,
    "overall_throughput": 77.41583394593547,
    "lr": 1.932472229404356e-05,
    "cuda_mem_allocated": 6.568481922149658,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56975,
    "batch_size": 121,
    "total_loss": 0.06508117595436595,
    "samples_seen": 6965,
    "gradnorm": 0.30859375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:55.403613"
}
 total length: 7919 dropped: 0 num samples 12 - rank: 1 max len: 677 min len: 638 avg len: 659.9166666666666 num_loss_counted_tokens: 7283
 total length: 7996 dropped: 0 num samples 15 - rank: 3 max len: 574 min len: 352 avg len: 533.0666666666667 num_loss_counted_tokens: 7201
 total length: 7996 dropped: 0 num samples 10 - rank: 0 max len: 1104 min len: 583 avg len: 799.6 num_loss_counted_tokens: 7466
 total length: 7901 dropped: 0 num samples 13 - rank: 2 max len: 626 min len: 585 avg len: 607.7692307692307 num_loss_counted_tokens: 7212
 total length: 7967 dropped: 0 num samples 18 - rank: 5 max len: 470 min len: 407 avg len: 442.6111111111111 num_loss_counted_tokens: 7013
 total length: 7996 dropped: 0 num samples 21 - rank: 6 max len: 404 min len: 355 avg len: 380.76190476190476 num_loss_counted_tokens: 6883
 total length: 7797 dropped: 0 num samples 25 - rank: 7 max len: 358 min len: 243 avg len: 311.88 num_loss_counted_tokens: 6472
 total length: 7898 dropped: 0 num samples 16 - rank: 4 max len: 522 min len: 472 avg len: 493.625 num_loss_counted_tokens: 7050
Epoch: 1, Step: 56, Rank: 4, loss = 0.08740234375Epoch: 1, Step: 56, Rank: 1, loss = 0.0908203125Epoch: 1, Step: 56, Rank: 6, loss = 0.09326171875

Epoch: 1, Step: 56, Rank: 2, loss = 0.060791015625Epoch: 1, Step: 56, Rank: 7, loss = 0.07958984375Epoch: 1, Step: 56, Rank: 3, loss = 0.053955078125



Epoch: 1, Step: 56, Rank: 5, loss = 0.0625
Epoch: 1, Step: 56, Rank: 0, loss = 0.04296875


Epoch 1:  40%|████      | 16/40 [00:28<00:43,  1.81s/it]{
    "epoch": 1,
    "step": 56,
    "rank": 0,
    "overall_throughput": 94.78867319035459,
    "lr": 1.927951089856575e-05,
    "cuda_mem_allocated": 6.578244686126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56657,
    "batch_size": 123,
    "total_loss": 0.07141218207811921,
    "samples_seen": 7088,
    "gradnorm": 0.33203125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:57.045623"
}
 total length: 7868 dropped: 0 num samples 12 - rank: 1 max len: 720 min len: 626 avg len: 655.6666666666666 num_loss_counted_tokens: 7232
 total length: 7935 dropped: 0 num samples 13 - rank: 2 max len: 624 min len: 579 avg len: 610.3846153846154 num_loss_counted_tokens: 7246
 total length: 7964 dropped: 0 num samples 21 - rank: 6 max len: 406 min len: 355 avg len: 379.23809523809524 num_loss_counted_tokens: 6851
 total length: 7851 dropped: 0 num samples 14 - rank: 3 max len: 579 min len: 543 avg len: 560.7857142857143 num_loss_counted_tokens: 7109
 total length: 7950 dropped: 0 num samples 16 - rank: 4 max len: 540 min len: 458 avg len: 496.875 num_loss_counted_tokens: 7102
 total length: 7780 dropped: 0 num samples 18 - rank: 5 max len: 457 min len: 406 avg len: 432.22222222222223 num_loss_counted_tokens: 6826
 total length: 7834 dropped: 0 num samples 9 - rank: 0 max len: 1068 min len: 738 avg len: 870.4444444444445 num_loss_counted_tokens: 7357
 total length: 7868 dropped: 0 num samples 25 - rank: 7 max len: 354 min len: 262 avg len: 314.72 num_loss_counted_tokens: 6543
Epoch: 1, Step: 57, Rank: 6, loss = 0.09326171875Epoch: 1, Step: 57, Rank: 5, loss = 0.078125Epoch: 1, Step: 57, Rank: 4, loss = 0.07763671875Epoch: 1, Step: 57, Rank: 1, loss = 0.06201171875
Epoch: 1, Step: 57, Rank: 7, loss = 0.0908203125Epoch: 1, Step: 57, Rank: 3, loss = 0.06591796875
Epoch: 1, Step: 57, Rank: 2, loss = 0.0537109375




Epoch: 1, Step: 57, Rank: 0, loss = 0.0291748046875


Epoch 1:  42%|████▎     | 17/40 [00:29<00:40,  1.76s/it]{
    "epoch": 1,
    "step": 57,
    "rank": 0,
    "overall_throughput": 93.8895411632371,
    "lr": 1.9232891061054896e-05,
    "cuda_mem_allocated": 6.588063716888428,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56604,
    "batch_size": 124,
    "total_loss": 0.0687937248251007,
    "samples_seen": 7212,
    "gradnorm": 0.318359375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:31:58.696952"
}
 total length: 7757 dropped: 0 num samples 24 - rank: 7 max len: 376 min len: 255 avg len: 323.2083333333333 num_loss_counted_tokens: 6485
 total length: 7994 dropped: 0 num samples 11 - rank: 1 max len: 825 min len: 441 avg len: 726.7272727272727 num_loss_counted_tokens: 7411
 total length: 7848 dropped: 0 num samples 14 - rank: 3 max len: 605 min len: 536 avg len: 560.5714285714286 num_loss_counted_tokens: 7106
 total length: 7960 dropped: 0 num samples 8 - rank: 0 max len: 1096 min len: 825 avg len: 995.0 num_loss_counted_tokens: 7536
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 477 min len: 316 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 534 min len: 475 avg len: 499.75 num_loss_counted_tokens: 7148
 total length: 7898 dropped: 0 num samples 20 - rank: 6 max len: 420 min len: 377 avg len: 394.9 num_loss_counted_tokens: 6838
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 673 min len: 350 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
Epoch: 1, Step: 58, Rank: 6, loss = 0.0908203125Epoch: 1, Step: 58, Rank: 4, loss = 0.06201171875Epoch: 1, Step: 58, Rank: 5, loss = 0.06494140625Epoch: 1, Step: 58, Rank: 1, loss = 0.064453125



Epoch: 1, Step: 58, Rank: 7, loss = 0.09375Epoch: 1, Step: 58, Rank: 2, loss = 0.05078125Epoch: 1, Step: 58, Rank: 3, loss = 0.044677734375


Epoch: 1, Step: 58, Rank: 0, loss = 0.057861328125


Epoch 1:  45%|████▌     | 18/40 [00:31<00:38,  1.73s/it]{
    "epoch": 1,
    "step": 58,
    "rank": 0,
    "overall_throughput": 93.98418095237095,
    "lr": 1.9184869857459233e-05,
    "cuda_mem_allocated": 6.58611536026001,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 57044,
    "batch_size": 122,
    "total_loss": 0.06615945585863543,
    "samples_seen": 7334,
    "gradnorm": 0.302734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:00.349343"
}
 total length: 7938 dropped: 0 num samples 12 - rank: 1 max len: 722 min len: 627 avg len: 661.5 num_loss_counted_tokens: 7302
 total length: 7995 dropped: 0 num samples 10 - rank: 0 max len: 1196 min len: 588 avg len: 799.5 num_loss_counted_tokens: 7465
 total length: 7809 dropped: 0 num samples 20 - rank: 6 max len: 415 min len: 362 avg len: 390.45 num_loss_counted_tokens: 6749
 total length: 7808 dropped: 0 num samples 18 - rank: 5 max len: 451 min len: 415 avg len: 433.77777777777777 num_loss_counted_tokens: 6854
 total length: 7838 dropped: 0 num samples 13 - rank: 2 max len: 626 min len: 565 avg len: 602.9230769230769 num_loss_counted_tokens: 7149
 total length: 7994 dropped: 0 num samples 15 - rank: 3 max len: 565 min len: 516 avg len: 532.9333333333333 num_loss_counted_tokens: 7199
 total length: 7765 dropped: 0 num samples 16 - rank: 4 max len: 515 min len: 452 avg len: 485.3125 num_loss_counted_tokens: 6917
 total length: 7814 dropped: 0 num samples 25 - rank: 7 max len: 354 min len: 250 avg len: 312.56 num_loss_counted_tokens: 6489
Epoch: 1, Step: 59, Rank: 4, loss = 0.0654296875Epoch: 1, Step: 59, Rank: 6, loss = 0.08837890625Epoch: 1, Step: 59, Rank: 1, loss = 0.05712890625Epoch: 1, Step: 59, Rank: 5, loss = 0.07421875Epoch: 1, Step: 59, Rank: 7, loss = 0.1083984375


Epoch: 1, Step: 59, Rank: 2, loss = 0.06884765625


Epoch: 1, Step: 59, Rank: 3, loss = 0.060546875
Epoch: 1, Step: 59, Rank: 0, loss = 0.07470703125


Epoch 1:  48%|████▊     | 19/40 [00:33<00:35,  1.71s/it]{
    "epoch": 1,
    "step": 59,
    "rank": 0,
    "overall_throughput": 93.02867670324133,
    "lr": 1.913545457642601e-05,
    "cuda_mem_allocated": 6.5880632400512695,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56805,
    "batch_size": 127,
    "total_loss": 0.07474694129037937,
    "samples_seen": 7461,
    "gradnorm": 0.318359375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:02.007489"
}
 total length: 7945 dropped: 0 num samples 12 - rank: 1 max len: 701 min len: 638 avg len: 662.0833333333334 num_loss_counted_tokens: 7309
 total length: 7905 dropped: 0 num samples 16 - rank: 4 max len: 519 min len: 464 avg len: 494.0625 num_loss_counted_tokens: 7057
 total length: 7866 dropped: 0 num samples 18 - rank: 5 max len: 461 min len: 411 avg len: 437.0 num_loss_counted_tokens: 6912
 total length: 7942 dropped: 0 num samples 10 - rank: 0 max len: 1066 min len: 174 avg len: 794.2 num_loss_counted_tokens: 7412
 total length: 7760 dropped: 0 num samples 14 - rank: 3 max len: 586 min len: 523 avg len: 554.2857142857143 num_loss_counted_tokens: 7018
 total length: 7671 dropped: 0 num samples 24 - rank: 7 max len: 356 min len: 250 avg len: 319.625 num_loss_counted_tokens: 6399
 total length: 7990 dropped: 0 num samples 13 - rank: 2 max len: 638 min len: 490 avg len: 614.6153846153846 num_loss_counted_tokens: 7301
 total length: 7839 dropped: 0 num samples 20 - rank: 6 max len: 411 min len: 373 avg len: 391.95 num_loss_counted_tokens: 6779
Epoch: 1, Step: 60, Rank: 6, loss = 0.0810546875Epoch: 1, Step: 60, Rank: 1, loss = 0.091796875Epoch: 1, Step: 60, Rank: 4, loss = 0.07421875Epoch: 1, Step: 60, Rank: 5, loss = 0.07421875Epoch: 1, Step: 60, Rank: 3, loss = 0.052734375


Epoch: 1, Step: 60, Rank: 7, loss = 0.0986328125

Epoch: 1, Step: 60, Rank: 2, loss = 0.05712890625

Epoch: 1, Step: 60, Rank: 0, loss = 0.040771484375


Epoch 1:  50%|█████     | 20/40 [00:34<00:33,  1.69s/it]{
    "epoch": 1,
    "step": 60,
    "rank": 0,
    "overall_throughput": 93.52824265213013,
    "lr": 1.9084652718195237e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56776,
    "batch_size": 126,
    "total_loss": 0.07140340989150346,
    "samples_seen": 7587,
    "gradnorm": 0.32421875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:03.667629"
}
 total length: 7720 dropped: 0 num samples 25 - rank: 7 max len: 350 min len: 241 avg len: 308.8 num_loss_counted_tokens: 6395
 total length: 7839 dropped: 0 num samples 10 - rank: 1 max len: 876 min len: 699 avg len: 783.9 num_loss_counted_tokens: 7309
 total length: 7961 dropped: 0 num samples 8 - rank: 0 max len: 1104 min len: 745 avg len: 995.125 num_loss_counted_tokens: 7537
 total length: 7907 dropped: 0 num samples 14 - rank: 3 max len: 596 min len: 536 avg len: 564.7857142857143 num_loss_counted_tokens: 7165
 total length: 7991 dropped: 0 num samples 13 - rank: 2 max len: 679 min len: 369 avg len: 614.6923076923077 num_loss_counted_tokens: 7302
 total length: 7986 dropped: 0 num samples 21 - rank: 6 max len: 407 min len: 313 avg len: 380.2857142857143 num_loss_counted_tokens: 6873
 total length: 7995 dropped: 0 num samples 16 - rank: 4 max len: 534 min len: 243 avg len: 499.6875 num_loss_counted_tokens: 7147
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 486 min len: 344 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
Epoch: 1, Step: 61, Rank: 0, loss = 0.053955078125Epoch: 1, Step: 61, Rank: 6, loss = 0.0849609375Epoch: 1, Step: 61, Rank: 5, loss = 0.0966796875Epoch: 1, Step: 61, Rank: 2, loss = 0.0576171875

Epoch: 1, Step: 61, Rank: 3, loss = 0.07275390625
Epoch: 1, Step: 61, Rank: 7, loss = 0.07177734375


Epoch: 1, Step: 61, Rank: 4, loss = 0.07958984375
Epoch: 1, Step: 61, Rank: 1, loss = 0.064453125


Epoch 1:  52%|█████▎    | 21/40 [00:36<00:31,  1.68s/it]{
    "epoch": 1,
    "step": 61,
    "rank": 0,
    "overall_throughput": 94.34812213515622,
    "lr": 1.903247199346129e-05,
    "cuda_mem_allocated": 6.572944164276123,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56357,
    "batch_size": 123,
    "total_loss": 0.07271501321929841,
    "samples_seen": 7710,
    "gradnorm": 0.345703125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:05.316577"
}
 total length: 7991 dropped: 0 num samples 15 - rank: 4 max len: 555 min len: 502 avg len: 532.7333333333333 num_loss_counted_tokens: 7196
 total length: 7888 dropped: 0 num samples 24 - rank: 7 max len: 377 min len: 243 avg len: 328.6666666666667 num_loss_counted_tokens: 6616
 total length: 7992 dropped: 0 num samples 9 - rank: 0 max len: 1065 min len: 344 avg len: 888.0 num_loss_counted_tokens: 7515
 total length: 7996 dropped: 0 num samples 14 - rank: 3 max len: 618 min len: 366 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7993 dropped: 0 num samples 13 - rank: 2 max len: 645 min len: 343 avg len: 614.8461538461538 num_loss_counted_tokens: 7304
 total length: 7773 dropped: 0 num samples 19 - rank: 6 max len: 439 min len: 381 avg len: 409.10526315789474 num_loss_counted_tokens: 6766
 total length: 7994 dropped: 0 num samples 17 - rank: 5 max len: 511 min len: 338 avg len: 470.2352941176471 num_loss_counted_tokens: 7093
 total length: 7990 dropped: 0 num samples 12 - rank: 1 max len: 756 min len: 582 avg len: 665.8333333333334 num_loss_counted_tokens: 7354
Epoch: 1, Step: 62, Rank: 6, loss = 0.09130859375Epoch: 1, Step: 62, Rank: 5, loss = 0.05859375Epoch: 1, Step: 62, Rank: 3, loss = 0.054443359375Epoch: 1, Step: 62, Rank: 1, loss = 0.061767578125Epoch: 1, Step: 62, Rank: 2, loss = 0.0654296875

Epoch: 1, Step: 62, Rank: 7, loss = 0.09423828125



Epoch: 1, Step: 62, Rank: 4, loss = 0.07763671875
Epoch: 1, Step: 62, Rank: 0, loss = 0.0262451171875


Epoch 1:  55%|█████▌    | 22/40 [00:38<00:30,  1.67s/it]{
    "epoch": 1,
    "step": 62,
    "rank": 0,
    "overall_throughput": 93.53490718827442,
    "lr": 1.8978920322202582e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56708,
    "batch_size": 125,
    "total_loss": 0.0662693094448755,
    "samples_seen": 7835,
    "gradnorm": 0.294921875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:06.973712"
}
 total length: 7996 dropped: 0 num samples 13 - rank: 1 max len: 648 min len: 368 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7753 dropped: 0 num samples 15 - rank: 3 max len: 543 min len: 490 avg len: 516.8666666666667 num_loss_counted_tokens: 6958
 total length: 7573 dropped: 0 num samples 24 - rank: 7 max len: 344 min len: 261 avg len: 315.5416666666667 num_loss_counted_tokens: 6301
 total length: 7883 dropped: 0 num samples 17 - rank: 4 max len: 485 min len: 436 avg len: 463.70588235294116 num_loss_counted_tokens: 6982
 total length: 7989 dropped: 0 num samples 14 - rank: 2 max len: 619 min len: 353 avg len: 570.6428571428571 num_loss_counted_tokens: 7247
 total length: 7996 dropped: 0 num samples 22 - rank: 6 max len: 385 min len: 283 avg len: 363.45454545454544 num_loss_counted_tokens: 6830
 total length: 7899 dropped: 0 num samples 11 - rank: 0 max len: 883 min len: 653 avg len: 718.0909090909091 num_loss_counted_tokens: 7316
 total length: 7990 dropped: 0 num samples 20 - rank: 5 max len: 431 min len: 251 avg len: 399.5 num_loss_counted_tokens: 6930
Epoch: 1, Step: 63, Rank: 5, loss = 0.080078125Epoch: 1, Step: 63, Rank: 6, loss = 0.083984375
Epoch: 1, Step: 63, Rank: 1, loss = 0.07080078125Epoch: 1, Step: 63, Rank: 7, loss = 0.0947265625Epoch: 1, Step: 63, Rank: 3, loss = 0.072265625
Epoch: 1, Step: 63, Rank: 2, loss = 0.0703125



Epoch: 1, Step: 63, Rank: 4, loss = 0.06591796875
Epoch: 1, Step: 63, Rank: 0, loss = 0.048828125


Epoch 1:  57%|█████▊    | 23/40 [00:39<00:28,  1.67s/it]{
    "epoch": 1,
    "step": 63,
    "rank": 0,
    "overall_throughput": 94.38708572654215,
    "lr": 1.892400583247948e-05,
    "cuda_mem_allocated": 6.5880632400512695,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56430,
    "batch_size": 130,
    "total_loss": 0.0733297891192628,
    "samples_seen": 7965,
    "gradnorm": 0.333984375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:08.622290"
}
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 725 min len: 490 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
 total length: 7626 dropped: 0 num samples 25 - rank: 7 max len: 343 min len: 261 avg len: 305.04 num_loss_counted_tokens: 6301
 total length: 7757 dropped: 0 num samples 9 - rank: 0 max len: 1084 min len: 741 avg len: 861.8888888888889 num_loss_counted_tokens: 7280
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 638 min len: 512 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 528 min len: 440 avg len: 499.75 num_loss_counted_tokens: 7148
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 482 min len: 414 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7904 dropped: 0 num samples 14 - rank: 3 max len: 595 min len: 528 avg len: 564.5714285714286 num_loss_counted_tokens: 7162
 total length: 7995 dropped: 0 num samples 21 - rank: 6 max len: 416 min len: 267 avg len: 380.7142857142857 num_loss_counted_tokens: 6882
Epoch: 1, Step: 64, Rank: 0, loss = 0.041259765625Epoch: 1, Step: 64, Rank: 5, loss = 0.06103515625Epoch: 1, Step: 64, Rank: 6, loss = 0.0771484375Epoch: 1, Step: 64, Rank: 3, loss = 0.0869140625

Epoch: 1, Step: 64, Rank: 7, loss = 0.09375Epoch: 1, Step: 64, Rank: 2, loss = 0.0732421875



Epoch: 1, Step: 64, Rank: 4, loss = 0.060302734375Epoch: 1, Step: 64, Rank: 1, loss = 0.058349609375



Epoch 1:  60%|██████    | 24/40 [00:41<00:26,  1.66s/it]{
    "epoch": 1,
    "step": 64,
    "rank": 0,
    "overall_throughput": 94.35347792863124,
    "lr": 1.886773685920062e-05,
    "cuda_mem_allocated": 6.564572811126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56782,
    "batch_size": 120,
    "total_loss": 0.06896551724137931,
    "samples_seen": 8085,
    "gradnorm": 0.310546875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:10.269055"
}
Epoch: 1, Step: 65, Rank: 5, loss = 0.09130859375Epoch: 1, Step: 65, Rank: 3, loss = 0.0595703125Epoch: 1, Step: 65, Rank: 1, loss = 0.057861328125Epoch: 1, Step: 65, Rank: 6, loss = 0.09375

Epoch: 1, Step: 65, Rank: 7, loss = 0.07568359375Epoch: 1, Step: 65, Rank: 2, loss = 0.05712890625
Epoch: 1, Step: 65, Rank: 4, loss = 0.0830078125



Epoch: 1, Step: 65, Rank: 0, loss = 0.0235595703125


Epoch 1:  62%|██████▎   | 25/40 [00:42<00:24,  1.66s/it]{
    "epoch": 1,
    "step": 65,
    "rank": 0,
    "overall_throughput": 94.21883927824003,
    "lr": 1.8810121942857848e-05,
    "cuda_mem_allocated": 6.5880632400512695,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56305,
    "batch_size": 126,
    "total_loss": 0.06777373235059053,
    "samples_seen": 8211,
    "gradnorm": 0.30078125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:11.921550"
}
Epoch: 1, Step: 66, Rank: 5, loss = 0.059326171875Epoch: 1, Step: 66, Rank: 6, loss = 0.08056640625Epoch: 1, Step: 66, Rank: 1, loss = 0.095703125
Epoch: 1, Step: 66, Rank: 4, loss = 0.0830078125Epoch: 1, Step: 66, Rank: 3, loss = 0.0654296875Epoch: 1, Step: 66, Rank: 2, loss = 0.07958984375Epoch: 1, Step: 66, Rank: 7, loss = 0.09814453125





Epoch: 1, Step: 66, Rank: 0, loss = 0.04443359375


Epoch 1:  65%|██████▌   | 26/40 [00:44<00:23,  1.65s/it]{
    "epoch": 1,
    "step": 66,
    "rank": 0,
    "overall_throughput": 94.50185757522097,
    "lr": 1.8751169828229927e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56936,
    "batch_size": 129,
    "total_loss": 0.07580441197133624,
    "samples_seen": 8340,
    "gradnorm": 0.298828125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:13.567567"
}
Epoch: 1, Step: 67, Rank: 3, loss = 0.05029296875Epoch: 1, Step: 67, Rank: 1, loss = 0.058349609375Epoch: 1, Step: 67, Rank: 5, loss = 0.087890625Epoch: 1, Step: 67, Rank: 6, loss = 0.08447265625
Epoch: 1, Step: 67, Rank: 2, loss = 0.05908203125

Epoch: 1, Step: 67, Rank: 7, loss = 0.099609375Epoch: 1, Step: 67, Rank: 4, loss = 0.08544921875



Epoch: 1, Step: 67, Rank: 0, loss = 0.033203125


Epoch 1:  68%|██████▊   | 27/40 [00:46<00:21,  1.65s/it]{
    "epoch": 1,
    "step": 67,
    "rank": 0,
    "overall_throughput": 94.52450261026064,
    "lr": 1.8690889463055285e-05,
    "cuda_mem_allocated": 6.5861053466796875,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56303,
    "batch_size": 129,
    "total_loss": 0.06980089870877218,
    "samples_seen": 8469,
    "gradnorm": 0.3203125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:15.214692"
}
Epoch: 1, Step: 68, Rank: 4, loss = 0.0791015625Epoch: 1, Step: 68, Rank: 5, loss = 0.06298828125Epoch: 1, Step: 68, Rank: 7, loss = 0.0830078125Epoch: 1, Step: 68, Rank: 1, loss = 0.054931640625


Epoch: 1, Step: 68, Rank: 3, loss = 0.072265625Epoch: 1, Step: 68, Rank: 2, loss = 0.06640625
Epoch: 1, Step: 68, Rank: 6, loss = 0.08349609375


Epoch: 1, Step: 68, Rank: 0, loss = 0.029541015625


Epoch 1:  70%|███████   | 28/40 [00:47<00:19,  1.65s/it]{
    "epoch": 1,
    "step": 68,
    "rank": 0,
    "overall_throughput": 93.65805404339835,
    "lr": 1.86292899966739e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56689,
    "batch_size": 125,
    "total_loss": 0.0664855615727919,
    "samples_seen": 8594,
    "gradnorm": 0.298828125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:16.865620"
}
Epoch: 1, Step: 69, Rank: 0, loss = 0.03955078125Epoch: 1, Step: 69, Rank: 4, loss = 0.058837890625
Epoch: 1, Step: 69, Rank: 6, loss = 0.09521484375Epoch: 1, Step: 69, Rank: 5, loss = 0.08154296875Epoch: 1, Step: 69, Rank: 3, loss = 0.08349609375
Epoch: 1, Step: 69, Rank: 7, loss = 0.07470703125
Epoch: 1, Step: 69, Rank: 2, loss = 0.05419921875



Epoch: 1, Step: 69, Rank: 1, loss = 0.060546875


Epoch 1:  72%|███████▎  | 29/40 [00:49<00:18,  1.65s/it]{
    "epoch": 1,
    "step": 69,
    "rank": 0,
    "overall_throughput": 95.39684180075781,
    "lr": 1.856638077863863e-05,
    "cuda_mem_allocated": 6.566524028778076,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56108,
    "batch_size": 125,
    "total_loss": 0.06847508376702074,
    "samples_seen": 8719,
    "gradnorm": 0.306640625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:18.504636"
}
Epoch: 1, Step: 70, Rank: 5, loss = 0.08642578125Epoch: 1, Step: 70, Rank: 6, loss = 0.0751953125Epoch: 1, Step: 70, Rank: 4, loss = 0.080078125

Epoch: 1, Step: 70, Rank: 7, loss = 0.0927734375Epoch: 1, Step: 70, Rank: 1, loss = 0.055908203125Epoch: 1, Step: 70, Rank: 3, loss = 0.0634765625
Epoch: 1, Step: 70, Rank: 2, loss = 0.050048828125



Epoch: 1, Step: 70, Rank: 0, loss = 0.06689453125


Epoch 1:  75%|███████▌  | 30/40 [00:51<00:16,  1.64s/it]{
    "epoch": 1,
    "step": 70,
    "rank": 0,
    "overall_throughput": 94.81510617694987,
    "lr": 1.8502171357296144e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56330,
    "batch_size": 131,
    "total_loss": 0.07125865435824605,
    "samples_seen": 8850,
    "gradnorm": 0.306640625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:20.142156"
}
Epoch: 1, Step: 71, Rank: 5, loss = 0.076171875Epoch: 1, Step: 71, Rank: 6, loss = 0.08349609375Epoch: 1, Step: 71, Rank: 7, loss = 0.087890625Epoch: 1, Step: 71, Rank: 3, loss = 0.05810546875Epoch: 1, Step: 71, Rank: 1, loss = 0.08349609375Epoch: 1, Step: 71, Rank: 2, loss = 0.0595703125





Epoch: 1, Step: 71, Rank: 4, loss = 0.07080078125Epoch: 1, Step: 71, Rank: 0, loss = 0.051513671875



Epoch 1:  78%|███████▊  | 31/40 [00:52<00:14,  1.65s/it]{
    "epoch": 1,
    "step": 71,
    "rank": 0,
    "overall_throughput": 94.21138255894716,
    "lr": 1.8436671478337666e-05,
    "cuda_mem_allocated": 6.584695339202881,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56731,
    "batch_size": 127,
    "total_loss": 0.07138954011034532,
    "samples_seen": 8977,
    "gradnorm": 0.314453125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:21.789104"
}
Epoch: 1, Step: 72, Rank: 5, loss = 0.09423828125Epoch: 1, Step: 72, Rank: 6, loss = 0.0693359375Epoch: 1, Step: 72, Rank: 2, loss = 0.043212890625Epoch: 1, Step: 72, Rank: 3, loss = 0.0673828125Epoch: 1, Step: 72, Rank: 1, loss = 0.060791015625




Epoch: 1, Step: 72, Rank: 4, loss = 0.07177734375Epoch: 1, Step: 72, Rank: 7, loss = 0.08349609375

Epoch: 1, Step: 72, Rank: 0, loss = 0.052978515625


Epoch 1:  80%|████████  | 32/40 [00:54<00:13,  1.65s/it]{
    "epoch": 1,
    "step": 72,
    "rank": 0,
    "overall_throughput": 94.56762638602628,
    "lr": 1.836989108331978e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56580,
    "batch_size": 130,
    "total_loss": 0.06793920113114174,
    "samples_seen": 9107,
    "gradnorm": 0.298828125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:23.438302"
}
Epoch: 1, Step: 73, Rank: 6, loss = 0.111328125Epoch: 1, Step: 73, Rank: 4, loss = 0.06689453125Epoch: 1, Step: 73, Rank: 5, loss = 0.06787109375Epoch: 1, Step: 73, Rank: 1, loss = 0.044189453125
Epoch: 1, Step: 73, Rank: 3, loss = 0.06494140625
Epoch: 1, Step: 73, Rank: 7, loss = 0.1044921875


Epoch: 1, Step: 73, Rank: 2, loss = 0.057373046875

Epoch: 1, Step: 73, Rank: 0, loss = 0.0208740234375


Epoch 1:  82%|████████▎ | 33/40 [00:56<00:11,  1.64s/it]{
    "epoch": 1,
    "step": 73,
    "rank": 0,
    "overall_throughput": 95.84471493712539,
    "lr": 1.8301840308155507e-05,
    "cuda_mem_allocated": 6.572944164276123,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56266,
    "batch_size": 128,
    "total_loss": 0.06726975438097607,
    "samples_seen": 9235,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:25.062959"
}
Epoch: 1, Step: 74, Rank: 4, loss = 0.052490234375Epoch: 1, Step: 74, Rank: 6, loss = 0.091796875Epoch: 1, Step: 74, Rank: 7, loss = 0.095703125

Epoch: 1, Step: 74, Rank: 1, loss = 0.0771484375Epoch: 1, Step: 74, Rank: 3, loss = 0.04052734375Epoch: 1, Step: 74, Rank: 2, loss = 0.042724609375Epoch: 1, Step: 74, Rank: 5, loss = 0.08837890625




Epoch: 1, Step: 74, Rank: 0, loss = 0.017333984375


Epoch 1:  85%|████████▌ | 34/40 [00:57<00:09,  1.64s/it]{
    "epoch": 1,
    "step": 74,
    "rank": 0,
    "overall_throughput": 93.7198064124649,
    "lr": 1.8232529481575874e-05,
    "cuda_mem_allocated": 6.584466934204102,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56875,
    "batch_size": 124,
    "total_loss": 0.06324395604395604,
    "samples_seen": 9359,
    "gradnorm": 0.2890625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:26.715989"
}
Epoch: 1, Step: 75, Rank: 6, loss = 0.072265625Epoch: 1, Step: 75, Rank: 5, loss = 0.06884765625Epoch: 1, Step: 75, Rank: 3, loss = 0.05712890625Epoch: 1, Step: 75, Rank: 1, loss = 0.06298828125


Epoch: 1, Step: 75, Rank: 7, loss = 0.09130859375

Epoch: 1, Step: 75, Rank: 2, loss = 0.05419921875
Epoch: 1, Step: 75, Rank: 0, loss = 0.04638671875
Epoch: 1, Step: 75, Rank: 4, loss = 0.0888671875


Epoch 1:  88%|████████▊ | 35/40 [00:59<00:08,  1.65s/it]{
    "epoch": 1,
    "step": 75,
    "rank": 0,
    "overall_throughput": 94.31780621304998,
    "lr": 1.816196912356222e-05,
    "cuda_mem_allocated": 6.588063716888428,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56124,
    "batch_size": 129,
    "total_loss": 0.06770721972774571,
    "samples_seen": 9488,
    "gradnorm": 0.306640625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:28.365062"
}
Epoch: 1, Step: 76, Rank: 6, loss = 0.09228515625Epoch: 1, Step: 76, Rank: 5, loss = 0.08203125Epoch: 1, Step: 76, Rank: 3, loss = 0.05810546875
Epoch: 1, Step: 76, Rank: 7, loss = 0.1064453125Epoch: 1, Step: 76, Rank: 2, loss = 0.051025390625Epoch: 1, Step: 76, Rank: 4, loss = 0.059814453125

Epoch: 1, Step: 76, Rank: 1, loss = 0.048583984375



Epoch: 1, Step: 76, Rank: 0, loss = 0.058349609375


Epoch 1:  90%|█████████ | 36/40 [01:01<00:06,  1.65s/it]{
    "epoch": 1,
    "step": 76,
    "rank": 0,
    "overall_throughput": 94.73305974717307,
    "lr": 1.8090169943749477e-05,
    "cuda_mem_allocated": 6.582818984985352,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56187,
    "batch_size": 127,
    "total_loss": 0.06958905084806094,
    "samples_seen": 9615,
    "gradnorm": 0.294921875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:30.010485"
}
Epoch: 1, Step: 77, Rank: 5, loss = 0.06494140625Epoch: 1, Step: 77, Rank: 4, loss = 0.062255859375Epoch: 1, Step: 77, Rank: 1, loss = 0.0703125Epoch: 1, Step: 77, Rank: 2, loss = 0.044189453125Epoch: 1, Step: 77, Rank: 6, loss = 0.0810546875
Epoch: 1, Step: 77, Rank: 3, loss = 0.059814453125
Epoch: 1, Step: 77, Rank: 7, loss = 0.09130859375




Epoch: 1, Step: 77, Rank: 0, loss = 0.019775390625


Epoch 1:  92%|█████████▎| 37/40 [01:03<00:05,  1.73s/it]{
    "epoch": 1,
    "step": 77,
    "rank": 0,
    "overall_throughput": 77.64949125184589,
    "lr": 1.8017142839800667e-05,
    "cuda_mem_allocated": 6.584603786468506,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56771,
    "batch_size": 125,
    "total_loss": 0.06168642440682743,
    "samples_seen": 9740,
    "gradnorm": 0.29296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:31.947772"
}
Epoch: 1, Step: 78, Rank: 1, loss = 0.0830078125Epoch: 1, Step: 78, Rank: 0, loss = 0.0341796875Epoch: 1, Step: 78, Rank: 7, loss = 0.08447265625Epoch: 1, Step: 78, Rank: 6, loss = 0.07421875

Epoch: 1, Step: 78, Rank: 3, loss = 0.05859375Epoch: 1, Step: 78, Rank: 5, loss = 0.0849609375Epoch: 1, Step: 78, Rank: 2, loss = 0.0458984375




Epoch: 1, Step: 78, Rank: 4, loss = 0.0654296875


Epoch 1:  95%|█████████▌| 38/40 [01:04<00:03,  1.71s/it]{
    "epoch": 1,
    "step": 78,
    "rank": 0,
    "overall_throughput": 94.16881424357244,
    "lr": 1.794289889575286e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 57098,
    "batch_size": 123,
    "total_loss": 0.06637710602823216,
    "samples_seen": 9863,
    "gradnorm": 0.27734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:33.600807"
}
Epoch: 1, Step: 79, Rank: 5, loss = 0.07666015625Epoch: 1, Step: 79, Rank: 0, loss = 0.0517578125
Epoch: 1, Step: 79, Rank: 6, loss = 0.08154296875Epoch: 1, Step: 79, Rank: 3, loss = 0.05029296875Epoch: 1, Step: 79, Rank: 2, loss = 0.0595703125Epoch: 1, Step: 79, Rank: 7, loss = 0.0751953125




Epoch: 1, Step: 79, Rank: 4, loss = 0.07080078125Epoch: 1, Step: 79, Rank: 1, loss = 0.072265625



Epoch 1:  98%|█████████▊| 39/40 [01:06<00:01,  1.69s/it]{
    "epoch": 1,
    "step": 79,
    "rank": 0,
    "overall_throughput": 94.81800315076332,
    "lr": 1.7867449380334834e-05,
    "cuda_mem_allocated": 6.578911781311035,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 55871,
    "batch_size": 136,
    "total_loss": 0.06726208587639383,
    "samples_seen": 9999,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:35.243564"
}
Epoch: 1, Step: 80, Rank: 0, loss = 0.03857421875Epoch: 1, Step: 80, Rank: 4, loss = 0.058349609375Epoch: 1, Step: 80, Rank: 5, loss = 0.08349609375Epoch: 1, Step: 80, Rank: 3, loss = 0.05908203125Epoch: 1, Step: 80, Rank: 7, loss = 0.09326171875
Epoch: 1, Step: 80, Rank: 2, loss = 0.053466796875




Epoch: 1, Step: 80, Rank: 6, loss = 0.06787109375
Epoch: 1, Step: 80, Rank: 1, loss = 0.054931640625


Epoch 1: 100%|██████████| 40/40 [01:07<00:00,  1.67s/it]{
    "epoch": 1,
    "step": 80,
    "rank": 0,
    "overall_throughput": 95.1514462775496,
    "lr": 1.7790805745256703e-05,
    "cuda_mem_allocated": 6.566525936126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56483,
    "batch_size": 128,
    "total_loss": 0.06366517359205424,
    "samples_seen": 10127,
    "gradnorm": 0.271484375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:32:36.884268"
}
Saving model in huggingface format at: samples_10127
[19:33:03] INFO     The model is bigger than the maximum     accelerator.py:2924
                    size per checkpoint (5GB) and is going                      
                    to be split in 4 checkpoint shards. You                     
                    can find where each parameters has been                     
                    saved in the index located at                               
                    /tmp/tmpq8hzoasjw/model.safetensors.inde                    
                    x.json.                                                     
Model saved in /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/hf_format/samples_10127
[19:33:19] INFO     saving took 42.134588956832886 seconds          utils.py:879
Saving full model state in /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/full_state/epoch_1
           INFO     Saving current state to                  accelerator.py:3030
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_1                                                  
           INFO     Saving FSDP model                        accelerator.py:3040
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
[19:33:27] INFO     Saving model to                             fsdp_utils.py:88
                    /var/home/cloud-user/.local/share/instructl                 
                    ab/phased/phase1/checkpoints/full_state/epo                 
                    ch_1/pytorch_model_fsdp.bin                                 
[19:33:39] INFO     Model saved to                              fsdp_utils.py:90
                    /var/home/cloud-user/.local/share/instructl                 
                    ab/phased/phase1/checkpoints/full_state/epo                 
                    ch_1/pytorch_model_fsdp.bin                                 
           INFO     FSDP Model saved to output dir           accelerator.py:3042
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_1                                                  
           INFO     Saving FSDP Optimizer                    accelerator.py:3059
 total length: 7996 dropped: 0 num samples 17 - rank: 4 max len: 508 min len: 323 avg len: 470.3529411764706 num_loss_counted_tokens: 7095
 total length: 7996 dropped: 0 num samples 21 - rank: 6 max len: 412 min len: 354 avg len: 380.76190476190476 num_loss_counted_tokens: 6883
 total length: 7995 dropped: 0 num samples 17 - rank: 5 max len: 503 min len: 331 avg len: 470.29411764705884 num_loss_counted_tokens: 7094 total length: 7996 dropped: 0 num samples 18 - rank: 5 max len: 483 min len: 276 avg len: 444.22222222222223 num_loss_counted_tokens: 7042
 total length: 7774 dropped: 0 num samples 15 - rank: 4 max len: 531 min len: 505 avg len: 518.2666666666667 num_loss_counted_tokens: 6979 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 557 min len: 339 avg len: 499.75 num_loss_counted_tokens: 7148
 total length: 7989 dropped: 0 num samples 18 - rank: 5 max len: 487 min len: 304 avg len: 443.8333333333333 num_loss_counted_tokens: 7035

 total length: 7985 dropped: 0 num samples 19 - rank: 5 max len: 442 min len: 256 avg len: 420.2631578947368 num_loss_counted_tokens: 6978
 total length: 7836 dropped: 0 num samples 18 - rank: 5 max len: 458 min len: 413 avg len: 435.3333333333333 num_loss_counted_tokens: 6882

 total length: 7996 dropped: 0 num samples 20 - rank: 6 max len: 441 min len: 279 avg len: 399.8 num_loss_counted_tokens: 6936
 total length: 7997 dropped: 0 num samples 20 - rank: 6 max len: 424 min len: 271 avg len: 399.85 num_loss_counted_tokens: 6937
 total length: 7865 dropped: 0 num samples 17 - rank: 4 max len: 482 min len: 443 avg len: 462.6470588235294 num_loss_counted_tokens: 6964
 total length: 7989 dropped: 0 num samples 16 - rank: 4 max len: 517 min len: 470 avg len: 499.3125 num_loss_counted_tokens: 7141
 total length: 7996 dropped: 0 num samples 21 - rank: 6 max len: 417 min len: 304 avg len: 380.76190476190476 num_loss_counted_tokens: 6883
 total length: 7864 dropped: 0 num samples 20 - rank: 6 max len: 424 min len: 370 avg len: 393.2 num_loss_counted_tokens: 6804
 total length: 7953 dropped: 0 num samples 15 - rank: 3 max len: 555 min len: 514 avg len: 530.2 num_loss_counted_tokens: 7158
 total length: 7995 dropped: 0 num samples 14 - rank: 3 max len: 622 min len: 299 avg len: 571.0714285714286 num_loss_counted_tokens: 7253 total length: 7996 dropped: 0 num samples 14 - rank: 3 max len: 611 min len: 477 avg len: 571.1428571428571 num_loss_counted_tokens: 7254

 total length: 7996 dropped: 0 num samples 16 - rank: 3 max len: 542 min len: 388 avg len: 499.75 num_loss_counted_tokens: 7148
 total length: 7930 dropped: 0 num samples 15 - rank: 4 max len: 559 min len: 497 avg len: 528.6666666666666 num_loss_counted_tokens: 7135
 total length: 7829 dropped: 0 num samples 15 - rank: 4 max len: 539 min len: 507 avg len: 521.9333333333333 num_loss_counted_tokens: 7034
 total length: 7986 dropped: 0 num samples 16 - rank: 4 max len: 537 min len: 375 avg len: 499.125 num_loss_counted_tokens: 7138
 total length: 7904 dropped: 0 num samples 16 - rank: 4 max len: 539 min len: 461 avg len: 494.0 num_loss_counted_tokens: 7056
 total length: 7790 dropped: 0 num samples 20 - rank: 6 max len: 421 min len: 365 avg len: 389.5 num_loss_counted_tokens: 6730
 total length: 7995 dropped: 0 num samples 15 - rank: 4 max len: 557 min len: 467 avg len: 533.0 num_loss_counted_tokens: 7200
 total length: 7829 dropped: 0 num samples 16 - rank: 4 max len: 518 min len: 473 avg len: 489.3125 num_loss_counted_tokens: 6981
 total length: 7995 dropped: 0 num samples 16 - rank: 4 max len: 531 min len: 400 avg len: 499.6875 num_loss_counted_tokens: 7147
 total length: 7936 dropped: 0 num samples 18 - rank: 5 max len: 461 min len: 423 avg len: 440.8888888888889 num_loss_counted_tokens: 6982
 total length: 7846 dropped: 0 num samples 17 - rank: 5 max len: 510 min len: 428 avg len: 461.52941176470586 num_loss_counted_tokens: 6945
 total length: 7989 dropped: 0 num samples 18 - rank: 5 max len: 494 min len: 253 avg len: 443.8333333333333 num_loss_counted_tokens: 7035
 total length: 7756 dropped: 0 num samples 20 - rank: 6 max len: 418 min len: 366 avg len: 387.8 num_loss_counted_tokens: 6696
 total length: 7989 dropped: 0 num samples 18 - rank: 5 max len: 469 min len: 304 avg len: 443.8333333333333 num_loss_counted_tokens: 7035
 total length: 7990 dropped: 0 num samples 20 - rank: 6 max len: 423 min len: 284 avg len: 399.5 num_loss_counted_tokens: 6930
 total length: 7860 dropped: 0 num samples 19 - rank: 6 max len: 442 min len: 393 avg len: 413.6842105263158 num_loss_counted_tokens: 6853
 total length: 7901 dropped: 0 num samples 16 - rank: 4 max len: 514 min len: 468 avg len: 493.8125 num_loss_counted_tokens: 7053
 total length: 7758 dropped: 0 num samples 20 - rank: 6 max len: 423 min len: 360 avg len: 387.9 num_loss_counted_tokens: 6698
 total length: 7754 dropped: 0 num samples 20 - rank: 6 max len: 414 min len: 356 avg len: 387.7 num_loss_counted_tokens: 6694
 total length: 7875 dropped: 0 num samples 14 - rank: 3 max len: 591 min len: 542 avg len: 562.5 num_loss_counted_tokens: 7133
 total length: 7903 dropped: 0 num samples 14 - rank: 3 max len: 578 min len: 545 avg len: 564.5 num_loss_counted_tokens: 7161
 total length: 7987 dropped: 0 num samples 14 - rank: 3 max len: 630 min len: 259 avg len: 570.5 num_loss_counted_tokens: 7245
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 554 min len: 471 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7835 dropped: 0 num samples 16 - rank: 4 max len: 514 min len: 469 avg len: 489.6875 num_loss_counted_tokens: 6987
 total length: 7995 dropped: 0 num samples 20 - rank: 6 max len: 418 min len: 344 avg len: 399.75 num_loss_counted_tokens: 6935
 total length: 7982 dropped: 0 num samples 18 - rank: 5 max len: 467 min len: 372 avg len: 443.44444444444446 num_loss_counted_tokens: 7028
 total length: 7909 dropped: 0 num samples 20 - rank: 6 max len: 415 min len: 367 avg len: 395.45 num_loss_counted_tokens: 6849
 total length: 7982 dropped: 0 num samples 17 - rank: 5 max len: 496 min len: 447 avg len: 469.52941176470586 num_loss_counted_tokens: 7081
 total length: 7941 dropped: 0 num samples 18 - rank: 5 max len: 470 min len: 422 avg len: 441.1666666666667 num_loss_counted_tokens: 6987
 total length: 7971 dropped: 0 num samples 14 - rank: 3 max len: 593 min len: 551 avg len: 569.3571428571429 num_loss_counted_tokens: 7229
 total length: 7809 dropped: 0 num samples 18 - rank: 6 max len: 454 min len: 395 avg len: 433.8333333333333 num_loss_counted_tokens: 6855
 total length: 7793 dropped: 0 num samples 14 - rank: 3 max len: 586 min len: 533 avg len: 556.6428571428571 num_loss_counted_tokens: 7051
 total length: 7848 dropped: 0 num samples 18 - rank: 5 max len: 467 min len: 418 avg len: 436.0 num_loss_counted_tokens: 6894
 total length: 7928 dropped: 0 num samples 15 - rank: 4 max len: 561 min len: 490 avg len: 528.5333333333333 num_loss_counted_tokens: 7133
 total length: 7993 dropped: 0 num samples 16 - rank: 4 max len: 519 min len: 456 avg len: 499.5625 num_loss_counted_tokens: 7145
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 571 min len: 298 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7994 dropped: 0 num samples 16 - rank: 4 max len: 535 min len: 382 avg len: 499.625 num_loss_counted_tokens: 7146
 total length: 7249 dropped: 0 num samples 24 - rank: 7 max len: 351 min len: 243 avg len: 302.0416666666667 num_loss_counted_tokens: 5977
 total length: 7856 dropped: 0 num samples 14 - rank: 3 max len: 607 min len: 519 avg len: 561.1428571428571 num_loss_counted_tokens: 7114
 total length: 7725 dropped: 0 num samples 19 - rank: 6 max len: 427 min len: 384 avg len: 406.57894736842104 num_loss_counted_tokens: 6718
 total length: 7973 dropped: 0 num samples 17 - rank: 5 max len: 490 min len: 454 avg len: 469.0 num_loss_counted_tokens: 7072
 total length: 7997 dropped: 0 num samples 14 - rank: 2 max len: 629 min len: 299 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7995 dropped: 0 num samples 18 - rank: 5 max len: 468 min len: 334 avg len: 444.1666666666667 num_loss_counted_tokens: 7041
 total length: 7997 dropped: 0 num samples 14 - rank: 3 max len: 617 min len: 321 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7996 dropped: 0 num samples 20 - rank: 6 max len: 424 min len: 329 avg len: 399.8 num_loss_counted_tokens: 6936
 total length: 7992 dropped: 0 num samples 13 - rank: 2 max len: 646 min len: 442 avg len: 614.7692307692307 num_loss_counted_tokens: 7303
 total length: 7769 dropped: 0 num samples 20 - rank: 6 max len: 408 min len: 368 avg len: 388.45 num_loss_counted_tokens: 6709
 total length: 7762 dropped: 0 num samples 14 - rank: 3 max len: 585 min len: 523 avg len: 554.4285714285714 num_loss_counted_tokens: 7020
 total length: 7762 dropped: 0 num samples 14 - rank: 3 max len: 590 min len: 516 avg len: 554.4285714285714 num_loss_counted_tokens: 7020
 total length: 7992 dropped: 0 num samples 18 - rank: 5 max len: 470 min len: 337 avg len: 444.0 num_loss_counted_tokens: 7038
 total length: 7896 dropped: 0 num samples 18 - rank: 5 max len: 469 min len: 409 avg len: 438.6666666666667 num_loss_counted_tokens: 6942
 total length: 7391 dropped: 0 num samples 23 - rank: 7 max len: 371 min len: 243 avg len: 321.3478260869565 num_loss_counted_tokens: 6172
 total length: 7149 dropped: 0 num samples 22 - rank: 7 max len: 364 min len: 272 avg len: 324.95454545454544 num_loss_counted_tokens: 5983
 total length: 7751 dropped: 0 num samples 25 - rank: 7 max len: 352 min len: 240 avg len: 310.04 num_loss_counted_tokens: 6426
 total length: 7992 dropped: 0 num samples 14 - rank: 3 max len: 608 min len: 440 avg len: 570.8571428571429 num_loss_counted_tokens: 7250
 total length: 7946 dropped: 0 num samples 13 - rank: 1 max len: 666 min len: 191 avg len: 611.2307692307693 num_loss_counted_tokens: 7257
 total length: 7984 dropped: 0 num samples 15 - rank: 3 max len: 569 min len: 248 avg len: 532.2666666666667 num_loss_counted_tokens: 7189
 total length: 7746 dropped: 0 num samples 22 - rank: 7 max len: 392 min len: 255 avg len: 352.09090909090907 num_loss_counted_tokens: 6580
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 645 min len: 428 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 653 min len: 365 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7205 dropped: 0 num samples 21 - rank: 7 max len: 390 min len: 250 avg len: 343.0952380952381 num_loss_counted_tokens: 6092
 total length: 7669 dropped: 0 num samples 22 - rank: 7 max len: 385 min len: 262 avg len: 348.59090909090907 num_loss_counted_tokens: 6503
 total length: 7932 dropped: 0 num samples 25 - rank: 7 max len: 359 min len: 266 avg len: 317.28 num_loss_counted_tokens: 6607
 total length: 7846 dropped: 0 num samples 12 - rank: 2 max len: 665 min len: 642 avg len: 653.8333333333334 num_loss_counted_tokens: 7210
 total length: 7994 dropped: 0 num samples 12 - rank: 1 max len: 698 min len: 593 avg len: 666.1666666666666 num_loss_counted_tokens: 7358
 total length: 7940 dropped: 0 num samples 12 - rank: 1 max len: 707 min len: 643 avg len: 661.6666666666666 num_loss_counted_tokens: 7304
 total length: 7996 dropped: 0 num samples 13 - rank: 1 max len: 677 min len: 261 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7971 dropped: 0 num samples 13 - rank: 2 max len: 632 min len: 590 avg len: 613.1538461538462 num_loss_counted_tokens: 7282 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 642 min len: 476 avg len: 615.0769230769231 num_loss_counted_tokens: 7307 total length: 7984 dropped: 0 num samples 14 - rank: 2 max len: 616 min len: 263 avg len: 570.2857142857143 num_loss_counted_tokens: 7242


 total length: 7976 dropped: 0 num samples 12 - rank: 1 max len: 719 min len: 645 avg len: 664.6666666666666 num_loss_counted_tokens: 7340
 total length: 7864 dropped: 0 num samples 11 - rank: 1 max len: 826 min len: 670 avg len: 714.9090909090909 num_loss_counted_tokens: 7281
 total length: 7591 dropped: 0 num samples 24 - rank: 7 max len: 352 min len: 279 avg len: 316.2916666666667 num_loss_counted_tokens: 6319
 total length: 7867 dropped: 0 num samples 24 - rank: 7 max len: 364 min len: 260 avg len: 327.7916666666667 num_loss_counted_tokens: 6595
 total length: 7732 dropped: 0 num samples 23 - rank: 7 max len: 385 min len: 257 avg len: 336.17391304347825 num_loss_counted_tokens: 6513
 total length: 7698 dropped: 0 num samples 23 - rank: 7 max len: 383 min len: 279 avg len: 334.69565217391306 num_loss_counted_tokens: 6479
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 699 min len: 544 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7992 dropped: 0 num samples 13 - rank: 1 max len: 655 min len: 385 avg len: 614.7692307692307 num_loss_counted_tokens: 7303
 total length: 7949 dropped: 0 num samples 12 - rank: 1 max len: 710 min len: 636 avg len: 662.4166666666666 num_loss_counted_tokens: 7313
 total length: 7984 dropped: 0 num samples 13 - rank: 2 max len: 635 min len: 578 avg len: 614.1538461538462 num_loss_counted_tokens: 7295
 total length: 7918 dropped: 0 num samples 14 - rank: 2 max len: 600 min len: 544 avg len: 565.5714285714286 num_loss_counted_tokens: 7176
[19:33:48] INFO     Saving Optimizer state to                  fsdp_utils.py:192
                    /var/home/cloud-user/.local/share/instruct                  
                    lab/phased/phase1/checkpoints/full_state/e                  
                    poch_1/optimizer.bin                                        
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 732 min len: 436 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7963 dropped: 0 num samples 12 - rank: 1 max len: 712 min len: 636 avg len: 663.5833333333334 num_loss_counted_tokens: 7327 total length: 7804 dropped: 0 num samples 23 - rank: 7 max len: 380 min len: 249 avg len: 339.30434782608694 num_loss_counted_tokens: 6585

 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 641 min len: 519 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7952 dropped: 0 num samples 13 - rank: 2 max len: 628 min len: 586 avg len: 611.6923076923077 num_loss_counted_tokens: 7263
 total length: 7991 dropped: 0 num samples 13 - rank: 2 max len: 656 min len: 341 avg len: 614.6923076923077 num_loss_counted_tokens: 7302
 total length: 7995 dropped: 0 num samples 13 - rank: 2 max len: 640 min len: 500 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7910 dropped: 0 num samples 25 - rank: 7 max len: 365 min len: 248 avg len: 316.4 num_loss_counted_tokens: 6585
 total length: 7977 dropped: 0 num samples 12 - rank: 1 max len: 745 min len: 480 avg len: 664.75 num_loss_counted_tokens: 7341
 total length: 7992 dropped: 0 num samples 12 - rank: 1 max len: 834 min len: 271 avg len: 666.0 num_loss_counted_tokens: 7356
 total length: 7967 dropped: 0 num samples 13 - rank: 2 max len: 634 min len: 591 avg len: 612.8461538461538 num_loss_counted_tokens: 7278
 total length: 7909 dropped: 0 num samples 12 - rank: 1 max len: 686 min len: 633 avg len: 659.0833333333334 num_loss_counted_tokens: 7273
 total length: 7681 dropped: 0 num samples 22 - rank: 7 max len: 391 min len: 268 avg len: 349.1363636363636 num_loss_counted_tokens: 6515
 total length: 7985 dropped: 0 num samples 13 - rank: 1 max len: 660 min len: 273 avg len: 614.2307692307693 num_loss_counted_tokens: 7296
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 706 min len: 600 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7993 dropped: 0 num samples 13 - rank: 2 max len: 636 min len: 544 avg len: 614.8461538461538 num_loss_counted_tokens: 7304
 total length: 7830 dropped: 0 num samples 25 - rank: 7 max len: 363 min len: 248 avg len: 313.2 num_loss_counted_tokens: 6505
 total length: 7793 dropped: 0 num samples 13 - rank: 2 max len: 629 min len: 582 avg len: 599.4615384615385 num_loss_counted_tokens: 7104
 total length: 7923 dropped: 0 num samples 12 - rank: 1 max len: 679 min len: 644 avg len: 660.25 num_loss_counted_tokens: 7287
 total length: 7647 dropped: 0 num samples 23 - rank: 7 max len: 367 min len: 263 avg len: 332.4782608695652 num_loss_counted_tokens: 6428
[19:34:11] INFO     Optimizer state saved in                   fsdp_utils.py:194
                    /var/home/cloud-user/.local/share/instruct                  
                    lab/phased/phase1/checkpoints/full_state/e                  
                    poch_1/optimizer.bin                                        
           INFO     FSDP Optimizer saved to output dir       accelerator.py:3061
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_1                                                  
           INFO     Scheduler state saved in                checkpointing.py:118
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_1/scheduler.bin                                  
           INFO     Sampler state for dataloader 0 saved in checkpointing.py:135
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_1/sampler.bin                                    
           INFO     Random states saved in                  checkpointing.py:160
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_1/random_states_0.pkl                            
Saving training state: {'current_epoch': 1, 'samples_seen': 10127}
Model state saved in: /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/full_state/epoch_1

Epoch 2:   0%|          | 0/40 [00:00<?, ?it/s]
Epoch 1: 100%|██████████| 40/40 [02:42<00:00,  4.07s/it]
 total length: 7842 dropped: 0 num samples 9 - rank: 0 max len: 1094 min len: 672 avg len: 871.3333333333334 num_loss_counted_tokens: 7365 total length: 7909 dropped: 0 num samples 9 - rank: 0 max len: 1079 min len: 700 avg len: 878.7777777777778 num_loss_counted_tokens: 7432

 total length: 7992 dropped: 0 num samples 10 - rank: 0 max len: 1066 min len: 490 avg len: 799.2 num_loss_counted_tokens: 7462
 total length: 7993 dropped: 0 num samples 9 - rank: 0 max len: 1155 min len: 596 avg len: 888.1111111111111 num_loss_counted_tokens: 7516
 total length: 7908 dropped: 0 num samples 10 - rank: 0 max len: 888 min len: 656 avg len: 790.8 num_loss_counted_tokens: 7378
 total length: 7983 dropped: 0 num samples 10 - rank: 0 max len: 1052 min len: 600 avg len: 798.3 num_loss_counted_tokens: 7453
 total length: 7995 dropped: 0 num samples 10 - rank: 0 max len: 1057 min len: 326 avg len: 799.5 num_loss_counted_tokens: 7465
 total length: 7997 dropped: 0 num samples 10 - rank: 0 max len: 1098 min len: 438 avg len: 799.7 num_loss_counted_tokens: 7467
 total length: 7941 dropped: 0 num samples 8 - rank: 0 max len: 1076 min len: 751 avg len: 992.625 num_loss_counted_tokens: 7517
 total length: 7996 dropped: 0 num samples 9 - rank: 0 max len: 1057 min len: 539 avg len: 888.4444444444445 num_loss_counted_tokens: 7519
 total length: 7997 dropped: 0 num samples 9 - rank: 0 max len: 1068 min len: 431 avg len: 888.5555555555555 num_loss_counted_tokens: 7520
 total length: 7994 dropped: 0 num samples 9 - rank: 0 max len: 1164 min len: 318 avg len: 888.2222222222222 num_loss_counted_tokens: 7517
 total length: 7993 dropped: 0 num samples 11 - rank: 0 max len: 1041 min len: 278 avg len: 726.6363636363636 num_loss_counted_tokens: 7410
 total length: 7990 dropped: 0 num samples 9 - rank: 0 max len: 1104 min len: 287 avg len: 887.7777777777778 num_loss_counted_tokens: 7513
 total length: 7990 dropped: 0 num samples 9 - rank: 0 max len: 1104 min len: 722 avg len: 887.7777777777778 num_loss_counted_tokens: 7513
 total length: 7826 dropped: 0 num samples 9 - rank: 0 max len: 1074 min len: 694 avg len: 869.5555555555555 num_loss_counted_tokens: 7349
 total length: 7994 dropped: 0 num samples 11 - rank: 0 max len: 1068 min len: 328 avg len: 726.7272727272727 num_loss_counted_tokens: 7411
Epoch: 2, Step: 81, Rank: 6, loss = 0.06494140625Epoch: 2, Step: 81, Rank: 5, loss = 0.080078125Epoch: 2, Step: 81, Rank: 3, loss = 0.048583984375Epoch: 2, Step: 81, Rank: 4, loss = 0.06591796875Epoch: 2, Step: 81, Rank: 0, loss = 0.03662109375Epoch: 2, Step: 81, Rank: 1, loss = 0.050048828125

Epoch: 2, Step: 81, Rank: 7, loss = 0.0673828125




Epoch: 2, Step: 81, Rank: 2, loss = 0.0546875

Epoch 2:   2%|▎         | 1/40 [00:01<01:03,  1.63s/it]{
    "epoch": 2,
    "step": 81,
    "rank": 0,
    "overall_throughput": 95.22021679602526,
    "lr": 1.771297962347181e-05,
    "cuda_mem_allocated": 6.574355125427246,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 55872,
    "batch_size": 131,
    "total_loss": 0.05856242840778923,
    "samples_seen": 10258,
    "gradnorm": 0.2734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:13.318171"
}
 total length: 7892 dropped: 0 num samples 16 - rank: 4 max len: 518 min len: 468 avg len: 493.25 num_loss_counted_tokens: 7044
 total length: 7914 dropped: 0 num samples 12 - rank: 1 max len: 738 min len: 629 avg len: 659.5 num_loss_counted_tokens: 7278
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 558 min len: 505 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7997 dropped: 0 num samples 14 - rank: 2 max len: 628 min len: 303 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7962 dropped: 0 num samples 25 - rank: 7 max len: 366 min len: 271 avg len: 318.48 num_loss_counted_tokens: 6637
 total length: 7934 dropped: 0 num samples 18 - rank: 5 max len: 467 min len: 409 avg len: 440.77777777777777 num_loss_counted_tokens: 6980
 total length: 7995 dropped: 0 num samples 10 - rank: 0 max len: 953 min len: 534 avg len: 799.5 num_loss_counted_tokens: 7465
 total length: 7969 dropped: 0 num samples 21 - rank: 6 max len: 408 min len: 172 avg len: 379.4761904761905 num_loss_counted_tokens: 6856
Epoch: 2, Step: 82, Rank: 4, loss = 0.04736328125Epoch: 2, Step: 82, Rank: 2, loss = 0.053466796875Epoch: 2, Step: 82, Rank: 7, loss = 0.0634765625Epoch: 2, Step: 82, Rank: 6, loss = 0.068359375Epoch: 2, Step: 82, Rank: 5, loss = 0.0595703125
Epoch: 2, Step: 82, Rank: 3, loss = 0.042236328125




Epoch: 2, Step: 82, Rank: 1, loss = 0.050537109375
Epoch: 2, Step: 82, Rank: 0, loss = 0.06103515625

Epoch 2:   5%|▌         | 2/40 [00:03<01:02,  1.63s/it]{
    "epoch": 2,
    "step": 82,
    "rank": 0,
    "overall_throughput": 94.87636545691127,
    "lr": 1.763398282741103e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56558,
    "batch_size": 124,
    "total_loss": 0.055801124509353234,
    "samples_seen": 10382,
    "gradnorm": 0.28125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:14.953180"
}
 total length: 7656 dropped: 0 num samples 23 - rank: 7 max len: 399 min len: 268 avg len: 332.8695652173913 num_loss_counted_tokens: 6437
 total length: 7996 dropped: 0 num samples 15 - rank: 4 max len: 571 min len: 378 avg len: 533.0666666666667 num_loss_counted_tokens: 7201
 total length: 7984 dropped: 0 num samples 13 - rank: 2 max len: 661 min len: 289 avg len: 614.1538461538462 num_loss_counted_tokens: 7295
 total length: 7974 dropped: 0 num samples 17 - rank: 5 max len: 506 min len: 233 avg len: 469.05882352941177 num_loss_counted_tokens: 7073
 total length: 7990 dropped: 0 num samples 12 - rank: 1 max len: 744 min len: 413 avg len: 665.8333333333334 num_loss_counted_tokens: 7354
 total length: 7976 dropped: 0 num samples 8 - rank: 0 max len: 1482 min len: 701 avg len: 997.0 num_loss_counted_tokens: 7552
 total length: 7772 dropped: 0 num samples 13 - rank: 3 max len: 614 min len: 574 avg len: 597.8461538461538 num_loss_counted_tokens: 7083
 total length: 7879 dropped: 0 num samples 18 - rank: 6 max len: 463 min len: 400 avg len: 437.72222222222223 num_loss_counted_tokens: 6925
Epoch: 2, Step: 83, Rank: 5, loss = 0.06982421875Epoch: 2, Step: 83, Rank: 0, loss = 0.032470703125Epoch: 2, Step: 83, Rank: 7, loss = 0.06884765625Epoch: 2, Step: 83, Rank: 6, loss = 0.08740234375
Epoch: 2, Step: 83, Rank: 4, loss = 0.052490234375
Epoch: 2, Step: 83, Rank: 2, loss = 0.0498046875Epoch: 2, Step: 83, Rank: 3, loss = 0.048583984375




Epoch: 2, Step: 83, Rank: 1, loss = 0.045166015625

Epoch 2:   8%|▊         | 3/40 [00:04<01:00,  1.64s/it]{
    "epoch": 2,
    "step": 83,
    "rank": 0,
    "overall_throughput": 95.11629780012377,
    "lr": 1.7553827347189937e-05,
    "cuda_mem_allocated": 6.580230712890625,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56329,
    "batch_size": 124,
    "total_loss": 0.05680910365886133,
    "samples_seen": 10506,
    "gradnorm": 0.26953125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:16.593789"
}
 total length: 7990 dropped: 0 num samples 15 - rank: 3 max len: 583 min len: 336 avg len: 532.6666666666666 num_loss_counted_tokens: 7195
 total length: 7996 dropped: 0 num samples 10 - rank: 0 max len: 1068 min len: 542 avg len: 799.6 num_loss_counted_tokens: 7466
 total length: 7110 dropped: 0 num samples 22 - rank: 7 max len: 367 min len: 257 avg len: 323.1818181818182 num_loss_counted_tokens: 5944
 total length: 7894 dropped: 0 num samples 12 - rank: 1 max len: 675 min len: 638 avg len: 657.8333333333334 num_loss_counted_tokens: 7258
 total length: 7958 dropped: 0 num samples 17 - rank: 5 max len: 481 min len: 445 avg len: 468.11764705882354 num_loss_counted_tokens: 7057
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 637 min len: 586 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7980 dropped: 0 num samples 16 - rank: 4 max len: 525 min len: 402 avg len: 498.75 num_loss_counted_tokens: 7132
 total length: 7993 dropped: 0 num samples 20 - rank: 6 max len: 444 min len: 274 avg len: 399.65 num_loss_counted_tokens: 6933
Epoch: 2, Step: 84, Rank: 4, loss = 0.05322265625Epoch: 2, Step: 84, Rank: 5, loss = 0.06494140625Epoch: 2, Step: 84, Rank: 1, loss = 0.048095703125
Epoch: 2, Step: 84, Rank: 6, loss = 0.06103515625Epoch: 2, Step: 84, Rank: 7, loss = 0.0732421875Epoch: 2, Step: 84, Rank: 3, loss = 0.05419921875
Epoch: 2, Step: 84, Rank: 2, loss = 0.043212890625




Epoch: 2, Step: 84, Rank: 0, loss = 0.051513671875

Epoch 2:  10%|█         | 4/40 [00:06<00:58,  1.64s/it]{
    "epoch": 2,
    "step": 84,
    "rank": 0,
    "overall_throughput": 94.70252939483058,
    "lr": 1.747252534878891e-05,
    "cuda_mem_allocated": 6.586573123931885,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56335,
    "batch_size": 135,
    "total_loss": 0.05616401881601136,
    "samples_seen": 10641,
    "gradnorm": 0.28125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:18.234027"
}
 total length: 7986 dropped: 0 num samples 12 - rank: 1 max len: 721 min len: 545 avg len: 665.5 num_loss_counted_tokens: 7350
 total length: 7877 dropped: 0 num samples 14 - rank: 3 max len: 583 min len: 530 avg len: 562.6428571428571 num_loss_counted_tokens: 7135
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 643 min len: 541 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7824 dropped: 0 num samples 18 - rank: 5 max len: 460 min len: 417 avg len: 434.6666666666667 num_loss_counted_tokens: 6870
 total length: 7860 dropped: 0 num samples 16 - rank: 4 max len: 523 min len: 463 avg len: 491.25 num_loss_counted_tokens: 7012
 total length: 7757 dropped: 0 num samples 20 - rank: 6 max len: 416 min len: 354 avg len: 387.85 num_loss_counted_tokens: 6697
 total length: 7745 dropped: 0 num samples 25 - rank: 7 max len: 351 min len: 250 avg len: 309.8 num_loss_counted_tokens: 6420
 total length: 7921 dropped: 0 num samples 9 - rank: 0 max len: 1058 min len: 724 avg len: 880.1111111111111 num_loss_counted_tokens: 7444
Epoch: 2, Step: 85, Rank: 6, loss = 0.0703125Epoch: 2, Step: 85, Rank: 1, loss = 0.04345703125Epoch: 2, Step: 85, Rank: 7, loss = 0.06396484375Epoch: 2, Step: 85, Rank: 2, loss = 0.0537109375Epoch: 2, Step: 85, Rank: 5, loss = 0.05322265625
Epoch: 2, Step: 85, Rank: 4, loss = 0.058349609375

Epoch: 2, Step: 85, Rank: 3, loss = 0.036865234375



Epoch: 2, Step: 85, Rank: 0, loss = 0.04052734375

Epoch 2:  12%|█▎        | 5/40 [00:08<00:57,  1.64s/it]{
    "epoch": 2,
    "step": 85,
    "rank": 0,
    "overall_throughput": 94.64378004714993,
    "lr": 1.7390089172206594e-05,
    "cuda_mem_allocated": 6.588063716888428,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56483,
    "batch_size": 125,
    "total_loss": 0.052511375104013595,
    "samples_seen": 10766,
    "gradnorm": 0.2470703125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:19.876579"
}
 total length: 7994 dropped: 0 num samples 12 - rank: 1 max len: 709 min len: 511 avg len: 666.1666666666666 num_loss_counted_tokens: 7358
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 656 min len: 319 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7995 dropped: 0 num samples 14 - rank: 3 max len: 618 min len: 317 avg len: 571.0714285714286 num_loss_counted_tokens: 7253
 total length: 7738 dropped: 0 num samples 23 - rank: 7 max len: 367 min len: 256 avg len: 336.4347826086956 num_loss_counted_tokens: 6519
 total length: 7771 dropped: 0 num samples 17 - rank: 5 max len: 482 min len: 438 avg len: 457.11764705882354 num_loss_counted_tokens: 6870
 total length: 7997 dropped: 0 num samples 9 - rank: 0 max len: 1096 min len: 608 avg len: 888.5555555555555 num_loss_counted_tokens: 7520
 total length: 7987 dropped: 0 num samples 20 - rank: 6 max len: 438 min len: 299 avg len: 399.35 num_loss_counted_tokens: 6927
 total length: 7969 dropped: 0 num samples 16 - rank: 4 max len: 540 min len: 269 avg len: 498.0625 num_loss_counted_tokens: 7121
Epoch: 2, Step: 86, Rank: 0, loss = 0.0296630859375Epoch: 2, Step: 86, Rank: 2, loss = 0.078125Epoch: 2, Step: 86, Rank: 6, loss = 0.083984375Epoch: 2, Step: 86, Rank: 5, loss = 0.072265625


Epoch: 2, Step: 86, Rank: 3, loss = 0.04931640625

Epoch: 2, Step: 86, Rank: 7, loss = 0.06640625
Epoch: 2, Step: 86, Rank: 4, loss = 0.052978515625
Epoch: 2, Step: 86, Rank: 1, loss = 0.060302734375

Epoch 2:  15%|█▌        | 6/40 [00:09<00:55,  1.64s/it]{
    "epoch": 2,
    "step": 86,
    "rank": 0,
    "overall_throughput": 94.47267297187292,
    "lr": 1.7306531329586933e-05,
    "cuda_mem_allocated": 6.580197811126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56531,
    "batch_size": 128,
    "total_loss": 0.06162990217756629,
    "samples_seen": 10894,
    "gradnorm": 0.271484375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:21.523785"
}
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 633 min len: 587 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7976 dropped: 0 num samples 14 - rank: 3 max len: 598 min len: 543 avg len: 569.7142857142857 num_loss_counted_tokens: 7234
 total length: 7938 dropped: 0 num samples 12 - rank: 1 max len: 724 min len: 634 avg len: 661.5 num_loss_counted_tokens: 7302
 total length: 7776 dropped: 0 num samples 18 - rank: 5 max len: 470 min len: 403 avg len: 432.0 num_loss_counted_tokens: 6822
 total length: 7836 dropped: 0 num samples 21 - rank: 6 max len: 402 min len: 349 avg len: 373.14285714285717 num_loss_counted_tokens: 6723
 total length: 7787 dropped: 0 num samples 27 - rank: 7 max len: 342 min len: 236 avg len: 288.4074074074074 num_loss_counted_tokens: 6356
 total length: 7991 dropped: 0 num samples 10 - rank: 0 max len: 1048 min len: 437 avg len: 799.1 num_loss_counted_tokens: 7461
 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 532 min len: 469 avg len: 499.8125 num_loss_counted_tokens: 7149
Epoch: 2, Step: 87, Rank: 1, loss = 0.048828125Epoch: 2, Step: 87, Rank: 6, loss = 0.06201171875Epoch: 2, Step: 87, Rank: 5, loss = 0.06591796875Epoch: 2, Step: 87, Rank: 7, loss = 0.06982421875Epoch: 2, Step: 87, Rank: 2, loss = 0.049560546875Epoch: 2, Step: 87, Rank: 4, loss = 0.0556640625

Epoch: 2, Step: 87, Rank: 3, loss = 0.04833984375




Epoch: 2, Step: 87, Rank: 0, loss = 0.03369140625

Epoch 2:  18%|█▊        | 7/40 [00:11<00:54,  1.65s/it]{
    "epoch": 2,
    "step": 87,
    "rank": 0,
    "overall_throughput": 93.64862749968515,
    "lr": 1.7221864503320093e-05,
    "cuda_mem_allocated": 6.5880632400512695,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56696,
    "batch_size": 126,
    "total_loss": 0.054218992521518274,
    "samples_seen": 11020,
    "gradnorm": 0.2734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:23.182035"
}
 total length: 7994 dropped: 0 num samples 15 - rank: 3 max len: 579 min len: 457 avg len: 532.9333333333333 num_loss_counted_tokens: 7199
 total length: 7997 dropped: 0 num samples 10 - rank: 0 max len: 1185 min len: 330 avg len: 799.7 num_loss_counted_tokens: 7467
 total length: 7869 dropped: 0 num samples 18 - rank: 5 max len: 455 min len: 408 avg len: 437.1666666666667 num_loss_counted_tokens: 6915
 total length: 7842 dropped: 0 num samples 12 - rank: 1 max len: 685 min len: 627 avg len: 653.5 num_loss_counted_tokens: 7206
 total length: 7842 dropped: 0 num samples 20 - rank: 6 max len: 407 min len: 364 avg len: 392.1 num_loss_counted_tokens: 6782
 total length: 7995 dropped: 0 num samples 17 - rank: 4 max len: 500 min len: 291 avg len: 470.29411764705884 num_loss_counted_tokens: 7094
 total length: 7895 dropped: 0 num samples 13 - rank: 2 max len: 626 min len: 579 avg len: 607.3076923076923 num_loss_counted_tokens: 7206
 total length: 7908 dropped: 0 num samples 25 - rank: 7 max len: 364 min len: 174 avg len: 316.32 num_loss_counted_tokens: 6583
Epoch: 2, Step: 88, Rank: 1, loss = 0.061767578125Epoch: 2, Step: 88, Rank: 6, loss = 0.08056640625Epoch: 2, Step: 88, Rank: 2, loss = 0.05517578125Epoch: 2, Step: 88, Rank: 5, loss = 0.06884765625
Epoch: 2, Step: 88, Rank: 4, loss = 0.053466796875


Epoch: 2, Step: 88, Rank: 7, loss = 0.07373046875Epoch: 2, Step: 88, Rank: 3, loss = 0.04248046875


Epoch: 2, Step: 88, Rank: 0, loss = 0.032470703125

Epoch 2:  20%|██        | 8/40 [00:13<00:52,  1.65s/it]{
    "epoch": 2,
    "step": 88,
    "rank": 0,
    "overall_throughput": 94.25352605235577,
    "lr": 1.7136101544117526e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56379,
    "batch_size": 128,
    "total_loss": 0.058550169389311624,
    "samples_seen": 11148,
    "gradnorm": 0.28125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:24.831704"
}
 total length: 7983 dropped: 0 num samples 12 - rank: 1 max len: 719 min len: 539 avg len: 665.25 num_loss_counted_tokens: 7347
 total length: 7992 dropped: 0 num samples 18 - rank: 5 max len: 483 min len: 418 avg len: 444.0 num_loss_counted_tokens: 7038
 total length: 7992 dropped: 0 num samples 10 - rank: 0 max len: 1196 min len: 545 avg len: 799.2 num_loss_counted_tokens: 7462
 total length: 7886 dropped: 0 num samples 25 - rank: 7 max len: 359 min len: 258 avg len: 315.44 num_loss_counted_tokens: 6561
 total length: 7955 dropped: 0 num samples 14 - rank: 3 max len: 596 min len: 536 avg len: 568.2142857142857 num_loss_counted_tokens: 7213
 total length: 7986 dropped: 0 num samples 21 - rank: 6 max len: 417 min len: 180 avg len: 380.2857142857143 num_loss_counted_tokens: 6873
 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 528 min len: 404 avg len: 499.75 num_loss_counted_tokens: 7148
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 651 min len: 526 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
Epoch: 2, Step: 89, Rank: 6, loss = 0.06787109375Epoch: 2, Step: 89, Rank: 3, loss = 0.042724609375Epoch: 2, Step: 89, Rank: 4, loss = 0.055419921875Epoch: 2, Step: 89, Rank: 5, loss = 0.060791015625Epoch: 2, Step: 89, Rank: 2, loss = 0.038330078125
Epoch: 2, Step: 89, Rank: 1, loss = 0.05224609375

Epoch: 2, Step: 89, Rank: 7, loss = 0.06591796875



Epoch: 2, Step: 89, Rank: 0, loss = 0.0225830078125

Epoch 2:  22%|██▎       | 9/40 [00:14<00:51,  1.65s/it]{
    "epoch": 2,
    "step": 89,
    "rank": 0,
    "overall_throughput": 93.91145012102972,
    "lr": 1.7049255469061476e-05,
    "cuda_mem_allocated": 6.588063716888428,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56904,
    "batch_size": 119,
    "total_loss": 0.05073457050470968,
    "samples_seen": 11267,
    "gradnorm": 0.271484375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:26.479632"
}
 total length: 7787 dropped: 0 num samples 20 - rank: 6 max len: 414 min len: 367 avg len: 389.35 num_loss_counted_tokens: 6727
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 730 min len: 427 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7899 dropped: 0 num samples 13 - rank: 2 max len: 647 min len: 569 avg len: 607.6153846153846 num_loss_counted_tokens: 7210
 total length: 7757 dropped: 0 num samples 24 - rank: 7 max len: 357 min len: 264 avg len: 323.2083333333333 num_loss_counted_tokens: 6485
 total length: 7997 dropped: 0 num samples 8 - rank: 0 max len: 1143 min len: 597 avg len: 999.625 num_loss_counted_tokens: 7573
 total length: 7870 dropped: 0 num samples 15 - rank: 3 max len: 564 min len: 496 avg len: 524.6666666666666 num_loss_counted_tokens: 7075
 total length: 7848 dropped: 0 num samples 18 - rank: 5 max len: 453 min len: 414 avg len: 436.0 num_loss_counted_tokens: 6894
 total length: 7995 dropped: 0 num samples 17 - rank: 4 max len: 495 min len: 404 avg len: 470.29411764705884 num_loss_counted_tokens: 7094
Epoch: 2, Step: 90, Rank: 5, loss = 0.07470703125Epoch: 2, Step: 90, Rank: 7, loss = 0.06689453125Epoch: 2, Step: 90, Rank: 6, loss = 0.06494140625Epoch: 2, Step: 90, Rank: 2, loss = 0.03955078125Epoch: 2, Step: 90, Rank: 1, loss = 0.04736328125Epoch: 2, Step: 90, Rank: 3, loss = 0.052734375



Epoch: 2, Step: 90, Rank: 4, loss = 0.06396484375


Epoch: 2, Step: 90, Rank: 0, loss = 0.044677734375

Epoch 2:  25%|██▌       | 10/40 [00:16<00:49,  1.65s/it]{
    "epoch": 2,
    "step": 90,
    "rank": 0,
    "overall_throughput": 94.8002389469805,
    "lr": 1.696133945962927e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56787,
    "batch_size": 127,
    "total_loss": 0.05687921531336398,
    "samples_seen": 11394,
    "gradnorm": 0.28125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:28.122139"
}
 total length: 7993 dropped: 0 num samples 13 - rank: 1 max len: 666 min len: 312 avg len: 614.8461538461538 num_loss_counted_tokens: 7304
 total length: 7996 dropped: 0 num samples 17 - rank: 4 max len: 509 min len: 271 avg len: 470.3529411764706 num_loss_counted_tokens: 7095
 total length: 7943 dropped: 0 num samples 14 - rank: 2 max len: 621 min len: 172 avg len: 567.3571428571429 num_loss_counted_tokens: 7201
 total length: 7847 dropped: 0 num samples 10 - rank: 0 max len: 1072 min len: 668 avg len: 784.7 num_loss_counted_tokens: 7317
 total length: 7995 dropped: 0 num samples 19 - rank: 5 max len: 445 min len: 295 avg len: 420.7894736842105 num_loss_counted_tokens: 6988
 total length: 7996 dropped: 0 num samples 15 - rank: 3 max len: 561 min len: 391 avg len: 533.0666666666667 num_loss_counted_tokens: 7201
 total length: 7389 dropped: 0 num samples 23 - rank: 7 max len: 364 min len: 229 avg len: 321.2608695652174 num_loss_counted_tokens: 6170
 total length: 7993 dropped: 0 num samples 21 - rank: 6 max len: 407 min len: 277 avg len: 380.6190476190476 num_loss_counted_tokens: 6880
Epoch: 2, Step: 91, Rank: 4, loss = 0.045166015625Epoch: 2, Step: 91, Rank: 6, loss = 0.05615234375Epoch: 2, Step: 91, Rank: 5, loss = 0.0634765625Epoch: 2, Step: 91, Rank: 1, loss = 0.0625Epoch: 2, Step: 91, Rank: 3, loss = 0.056884765625

Epoch: 2, Step: 91, Rank: 2, loss = 0.060791015625

Epoch: 2, Step: 91, Rank: 7, loss = 0.07763671875


Epoch: 2, Step: 91, Rank: 0, loss = 0.0185546875

Epoch 2:  28%|██▊       | 11/40 [00:18<00:47,  1.65s/it]{
    "epoch": 2,
    "step": 91,
    "rank": 0,
    "overall_throughput": 94.36996365970693,
    "lr": 1.687236685969263e-05,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56417,
    "batch_size": 125,
    "total_loss": 0.05514295336512044,
    "samples_seen": 11519,
    "gradnorm": 0.275390625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:29.772325"
}
 total length: 7983 dropped: 0 num samples 9 - rank: 0 max len: 1068 min len: 449 avg len: 887.0 num_loss_counted_tokens: 7506
 total length: 7896 dropped: 0 num samples 17 - rank: 5 max len: 500 min len: 430 avg len: 464.47058823529414 num_loss_counted_tokens: 6995
 total length: 7571 dropped: 0 num samples 23 - rank: 7 max len: 379 min len: 243 avg len: 329.17391304347825 num_loss_counted_tokens: 6352
 total length: 7978 dropped: 0 num samples 14 - rank: 3 max len: 608 min len: 543 avg len: 569.8571428571429 num_loss_counted_tokens: 7236
 total length: 7993 dropped: 0 num samples 12 - rank: 1 max len: 740 min len: 565 avg len: 666.0833333333334 num_loss_counted_tokens: 7357
 total length: 7996 dropped: 0 num samples 20 - rank: 6 max len: 427 min len: 315 avg len: 399.8 num_loss_counted_tokens: 6936
 total length: 7994 dropped: 0 num samples 13 - rank: 2 max len: 652 min len: 417 avg len: 614.9230769230769 num_loss_counted_tokens: 7305
 total length: 7786 dropped: 0 num samples 15 - rank: 4 max len: 540 min len: 501 avg len: 519.0666666666667 num_loss_counted_tokens: 6991
Epoch: 2, Step: 92, Rank: 5, loss = 0.0595703125Epoch: 2, Step: 92, Rank: 2, loss = 0.0400390625Epoch: 2, Step: 92, Rank: 4, loss = 0.040771484375Epoch: 2, Step: 92, Rank: 6, loss = 0.0576171875
Epoch: 2, Step: 92, Rank: 1, loss = 0.041015625Epoch: 2, Step: 92, Rank: 7, loss = 0.061279296875
Epoch: 2, Step: 92, Rank: 3, loss = 0.048583984375




Epoch: 2, Step: 92, Rank: 0, loss = 0.0250244140625

Epoch 2:  30%|███       | 12/40 [00:19<00:46,  1.65s/it]{
    "epoch": 2,
    "step": 92,
    "rank": 0,
    "overall_throughput": 93.59555082370814,
    "lr": 1.678235117349234e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56854,
    "batch_size": 125,
    "total_loss": 0.04678650578675203,
    "samples_seen": 11644,
    "gradnorm": 0.27734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:31.426500"
}
 total length: 7793 dropped: 0 num samples 12 - rank: 1 max len: 676 min len: 633 avg len: 649.4166666666666 num_loss_counted_tokens: 7157
 total length: 7915 dropped: 0 num samples 10 - rank: 0 max len: 1064 min len: 676 avg len: 791.5 num_loss_counted_tokens: 7385
 total length: 7739 dropped: 0 num samples 13 - rank: 2 max len: 633 min len: 568 avg len: 595.3076923076923 num_loss_counted_tokens: 7050
 total length: 7994 dropped: 0 num samples 15 - rank: 3 max len: 566 min len: 466 avg len: 532.9333333333333 num_loss_counted_tokens: 7199
 total length: 7418 dropped: 0 num samples 23 - rank: 7 max len: 359 min len: 267 avg len: 322.5217391304348 num_loss_counted_tokens: 6199
 total length: 7991 dropped: 0 num samples 21 - rank: 6 max len: 404 min len: 344 avg len: 380.5238095238095 num_loss_counted_tokens: 6878
 total length: 7823 dropped: 0 num samples 18 - rank: 5 max len: 455 min len: 406 avg len: 434.6111111111111 num_loss_counted_tokens: 6869
 total length: 7994 dropped: 0 num samples 17 - rank: 4 max len: 506 min len: 346 avg len: 470.2352941176471 num_loss_counted_tokens: 7093
Epoch: 2, Step: 93, Rank: 2, loss = 0.0556640625Epoch: 2, Step: 93, Rank: 7, loss = 0.07568359375Epoch: 2, Step: 93, Rank: 6, loss = 0.06689453125
Epoch: 2, Step: 93, Rank: 3, loss = 0.052734375Epoch: 2, Step: 93, Rank: 5, loss = 0.05029296875



Epoch: 2, Step: 93, Rank: 1, loss = 0.040283203125
Epoch: 2, Step: 93, Rank: 0, loss = 0.0341796875
Epoch: 2, Step: 93, Rank: 4, loss = 0.046630859375

Epoch 2:  32%|███▎      | 13/40 [00:21<00:44,  1.65s/it]{
    "epoch": 2,
    "step": 93,
    "rank": 0,
    "overall_throughput": 93.57150573232776,
    "lr": 1.6691306063588583e-05,
    "cuda_mem_allocated": 6.588062286376953,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56837,
    "batch_size": 125,
    "total_loss": 0.052800112602705986,
    "samples_seen": 11769,
    "gradnorm": 0.27734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:33.083602"
}
 total length: 7946 dropped: 0 num samples 12 - rank: 1 max len: 687 min len: 634 avg len: 662.1666666666666 num_loss_counted_tokens: 7310
 total length: 7931 dropped: 0 num samples 16 - rank: 4 max len: 519 min len: 474 avg len: 495.6875 num_loss_counted_tokens: 7083
 total length: 7992 dropped: 0 num samples 15 - rank: 3 max len: 578 min len: 345 avg len: 532.8 num_loss_counted_tokens: 7197
 total length: 7853 dropped: 0 num samples 20 - rank: 6 max len: 418 min len: 366 avg len: 392.65 num_loss_counted_tokens: 6793
 total length: 7805 dropped: 0 num samples 13 - rank: 2 max len: 628 min len: 578 avg len: 600.3846153846154 num_loss_counted_tokens: 7116
 total length: 7860 dropped: 0 num samples 25 - rank: 7 max len: 361 min len: 262 avg len: 314.4 num_loss_counted_tokens: 6535
 total length: 7987 dropped: 0 num samples 18 - rank: 5 max len: 469 min len: 333 avg len: 443.72222222222223 num_loss_counted_tokens: 7033
 total length: 7996 dropped: 0 num samples 10 - rank: 0 max len: 1052 min len: 431 avg len: 799.6 num_loss_counted_tokens: 7466
Epoch: 2, Step: 94, Rank: 6, loss = 0.07421875Epoch: 2, Step: 94, Rank: 2, loss = 0.048095703125Epoch: 2, Step: 94, Rank: 5, loss = 0.058349609375Epoch: 2, Step: 94, Rank: 3, loss = 0.042236328125Epoch: 2, Step: 94, Rank: 1, loss = 0.036865234375


Epoch: 2, Step: 94, Rank: 4, loss = 0.04638671875

Epoch: 2, Step: 94, Rank: 7, loss = 0.061767578125

Epoch: 2, Step: 94, Rank: 0, loss = 0.04541015625

Epoch 2:  35%|███▌      | 14/40 [00:23<00:45,  1.74s/it]{
    "epoch": 2,
    "step": 94,
    "rank": 0,
    "overall_throughput": 77.35176544187122,
    "lr": 1.659924534878723e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56821,
    "batch_size": 127,
    "total_loss": 0.051671037116559015,
    "samples_seen": 11896,
    "gradnorm": 0.28125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:35.033453"
}
 total length: 7994 dropped: 0 num samples 9 - rank: 0 max len: 1391 min len: 164 avg len: 888.2222222222222 num_loss_counted_tokens: 7517
 total length: 7868 dropped: 0 num samples 12 - rank: 1 max len: 745 min len: 622 avg len: 655.6666666666666 num_loss_counted_tokens: 7232
 total length: 7996 dropped: 0 num samples 14 - rank: 2 max len: 621 min len: 419 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7911 dropped: 0 num samples 17 - rank: 4 max len: 485 min len: 448 avg len: 465.3529411764706 num_loss_counted_tokens: 7010
 total length: 7997 dropped: 0 num samples 21 - rank: 6 max len: 411 min len: 271 avg len: 380.8095238095238 num_loss_counted_tokens: 6884
 total length: 7773 dropped: 0 num samples 15 - rank: 3 max len: 539 min len: 486 avg len: 518.2 num_loss_counted_tokens: 6978
 total length: 7190 dropped: 0 num samples 23 - rank: 7 max len: 361 min len: 245 avg len: 312.60869565217394 num_loss_counted_tokens: 5971
 total length: 7992 dropped: 0 num samples 19 - rank: 5 max len: 448 min len: 265 avg len: 420.63157894736844 num_loss_counted_tokens: 6985
Epoch: 2, Step: 95, Rank: 6, loss = 0.068359375Epoch: 2, Step: 95, Rank: 1, loss = 0.0439453125Epoch: 2, Step: 95, Rank: 5, loss = 0.048095703125Epoch: 2, Step: 95, Rank: 7, loss = 0.0791015625
Epoch: 2, Step: 95, Rank: 4, loss = 0.04443359375Epoch: 2, Step: 95, Rank: 3, loss = 0.042724609375



Epoch: 2, Step: 95, Rank: 2, loss = 0.04443359375

Epoch: 2, Step: 95, Rank: 0, loss = 0.01953125

Epoch 2:  38%|███▊      | 15/40 [00:24<00:42,  1.71s/it]{
    "epoch": 2,
    "step": 95,
    "rank": 0,
    "overall_throughput": 93.97630072505183,
    "lr": 1.650618300204242e-05,
    "cuda_mem_allocated": 6.582767963409424,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 57005,
    "batch_size": 119,
    "total_loss": 0.048785194281203405,
    "samples_seen": 12015,
    "gradnorm": 0.275390625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:36.686457"
}
 total length: 7995 dropped: 0 num samples 14 - rank: 3 max len: 595 min len: 541 avg len: 571.0714285714286 num_loss_counted_tokens: 7253
 total length: 7997 dropped: 0 num samples 10 - rank: 0 max len: 1077 min len: 491 avg len: 799.7 num_loss_counted_tokens: 7467
 total length: 7927 dropped: 0 num samples 12 - rank: 1 max len: 682 min len: 645 avg len: 660.5833333333334 num_loss_counted_tokens: 7291
 total length: 7967 dropped: 0 num samples 18 - rank: 5 max len: 474 min len: 418 avg len: 442.6111111111111 num_loss_counted_tokens: 7013
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 645 min len: 528 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7991 dropped: 0 num samples 16 - rank: 4 max len: 532 min len: 318 avg len: 499.4375 num_loss_counted_tokens: 7143
 total length: 7863 dropped: 0 num samples 25 - rank: 7 max len: 357 min len: 202 avg len: 314.52 num_loss_counted_tokens: 6538
 total length: 7844 dropped: 0 num samples 20 - rank: 6 max len: 417 min len: 363 avg len: 392.2 num_loss_counted_tokens: 6784
Epoch: 2, Step: 96, Rank: 2, loss = 0.039794921875Epoch: 2, Step: 96, Rank: 1, loss = 0.0439453125Epoch: 2, Step: 96, Rank: 5, loss = 0.07958984375Epoch: 2, Step: 96, Rank: 6, loss = 0.068359375Epoch: 2, Step: 96, Rank: 3, loss = 0.035888671875


Epoch: 2, Step: 96, Rank: 7, loss = 0.06201171875


Epoch: 2, Step: 96, Rank: 4, loss = 0.050537109375
Epoch: 2, Step: 96, Rank: 0, loss = 0.044921875

Epoch 2:  40%|████      | 16/40 [00:26<00:40,  1.70s/it]{
    "epoch": 2,
    "step": 96,
    "rank": 0,
    "overall_throughput": 94.38328581682885,
    "lr": 1.6412133148335786e-05,
    "cuda_mem_allocated": 6.572385311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56163,
    "batch_size": 127,
    "total_loss": 0.053166675569325,
    "samples_seen": 12142,
    "gradnorm": 0.28125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:38.336661"
}
 total length: 7819 dropped: 0 num samples 18 - rank: 5 max len: 466 min len: 411 avg len: 434.3888888888889 num_loss_counted_tokens: 6865
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 640 min len: 478 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7792 dropped: 0 num samples 14 - rank: 3 max len: 593 min len: 524 avg len: 556.5714285714286 num_loss_counted_tokens: 7050
 total length: 7992 dropped: 0 num samples 10 - rank: 0 max len: 1066 min len: 551 avg len: 799.2 num_loss_counted_tokens: 7462
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 704 min len: 609 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7966 dropped: 0 num samples 21 - rank: 6 max len: 410 min len: 353 avg len: 379.3333333333333 num_loss_counted_tokens: 6853
 total length: 7778 dropped: 0 num samples 16 - rank: 4 max len: 517 min len: 467 avg len: 486.125 num_loss_counted_tokens: 6930
 total length: 7971 dropped: 0 num samples 25 - rank: 7 max len: 352 min len: 253 avg len: 318.84 num_loss_counted_tokens: 6646
Epoch: 2, Step: 97, Rank: 5, loss = 0.052734375Epoch: 2, Step: 97, Rank: 2, loss = 0.048583984375
Epoch: 2, Step: 97, Rank: 1, loss = 0.040771484375Epoch: 2, Step: 97, Rank: 4, loss = 0.040771484375
Epoch: 2, Step: 97, Rank: 3, loss = 0.052001953125
Epoch: 2, Step: 97, Rank: 6, loss = 0.06494140625

Epoch: 2, Step: 97, Rank: 7, loss = 0.0625

Epoch: 2, Step: 97, Rank: 0, loss = 0.04931640625

Epoch 2:  42%|████▎     | 17/40 [00:28<00:38,  1.68s/it]{
    "epoch": 2,
    "step": 97,
    "rank": 0,
    "overall_throughput": 94.72809534513843,
    "lr": 1.631711006253251e-05,
    "cuda_mem_allocated": 6.5880632400512695,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56192,
    "batch_size": 126,
    "total_loss": 0.05146640091116173,
    "samples_seen": 12268,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:39.979514"
}
 total length: 7991 dropped: 0 num samples 12 - rank: 1 max len: 732 min len: 426 avg len: 665.9166666666666 num_loss_counted_tokens: 7355
 total length: 7937 dropped: 0 num samples 20 - rank: 6 max len: 423 min len: 371 avg len: 396.85 num_loss_counted_tokens: 6877
 total length: 7985 dropped: 0 num samples 9 - rank: 0 max len: 1101 min len: 686 avg len: 887.2222222222222 num_loss_counted_tokens: 7508
 total length: 7850 dropped: 0 num samples 25 - rank: 7 max len: 368 min len: 243 avg len: 314.0 num_loss_counted_tokens: 6525
 total length: 7847 dropped: 0 num samples 17 - rank: 5 max len: 497 min len: 423 avg len: 461.5882352941176 num_loss_counted_tokens: 6946
 total length: 7993 dropped: 0 num samples 15 - rank: 4 max len: 558 min len: 490 avg len: 532.8666666666667 num_loss_counted_tokens: 7198
 total length: 7995 dropped: 0 num samples 13 - rank: 2 max len: 658 min len: 328 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7981 dropped: 0 num samples 14 - rank: 3 max len: 615 min len: 259 avg len: 570.0714285714286 num_loss_counted_tokens: 7239
Epoch: 2, Step: 98, Rank: 6, loss = 0.05517578125Epoch: 2, Step: 98, Rank: 3, loss = 0.05224609375Epoch: 2, Step: 98, Rank: 1, loss = 0.03955078125Epoch: 2, Step: 98, Rank: 4, loss = 0.072265625Epoch: 2, Step: 98, Rank: 2, loss = 0.059326171875Epoch: 2, Step: 98, Rank: 7, loss = 0.0703125





Epoch: 2, Step: 98, Rank: 5, loss = 0.05322265625
Epoch: 2, Step: 98, Rank: 0, loss = 0.02978515625

Epoch 2:  45%|████▌     | 18/40 [00:29<00:36,  1.67s/it]{
    "epoch": 2,
    "step": 98,
    "rank": 0,
    "overall_throughput": 94.46112051844001,
    "lr": 1.6221128167214742e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56717,
    "batch_size": 131,
    "total_loss": 0.05396970925824709,
    "samples_seen": 12399,
    "gradnorm": 0.287109375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:41.625212"
}
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 564 min len: 431 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7989 dropped: 0 num samples 18 - rank: 5 max len: 479 min len: 259 avg len: 443.8333333333333 num_loss_counted_tokens: 7035
 total length: 7833 dropped: 0 num samples 12 - rank: 1 max len: 708 min len: 627 avg len: 652.75 num_loss_counted_tokens: 7197
 total length: 7862 dropped: 0 num samples 9 - rank: 0 max len: 1309 min len: 719 avg len: 873.5555555555555 num_loss_counted_tokens: 7385
 total length: 7994 dropped: 0 num samples 16 - rank: 4 max len: 522 min len: 390 avg len: 499.625 num_loss_counted_tokens: 7146
 total length: 7690 dropped: 0 num samples 25 - rank: 7 max len: 363 min len: 171 avg len: 307.6 num_loss_counted_tokens: 6365
 total length: 7943 dropped: 0 num samples 20 - rank: 6 max len: 419 min len: 364 avg len: 397.15 num_loss_counted_tokens: 6883
 total length: 7981 dropped: 0 num samples 14 - rank: 2 max len: 624 min len: 172 avg len: 570.0714285714286 num_loss_counted_tokens: 7239
Epoch: 2, Step: 99, Rank: 6, loss = 0.0654296875Epoch: 2, Step: 99, Rank: 5, loss = 0.078125
Epoch: 2, Step: 99, Rank: 3, loss = 0.043212890625Epoch: 2, Step: 99, Rank: 2, loss = 0.05224609375Epoch: 2, Step: 99, Rank: 1, loss = 0.037109375
Epoch: 2, Step: 99, Rank: 4, loss = 0.046142578125Epoch: 2, Step: 99, Rank: 7, loss = 0.0732421875




Epoch: 2, Step: 99, Rank: 0, loss = 0.018310546875

Epoch 2:  48%|████▊     | 19/40 [00:31<00:34,  1.67s/it]{
    "epoch": 2,
    "step": 99,
    "rank": 0,
    "overall_throughput": 93.87405996270363,
    "lr": 1.61242020304925e-05,
    "cuda_mem_allocated": 6.586105823516846,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56920,
    "batch_size": 119,
    "total_loss": 0.051721714687280394,
    "samples_seen": 12518,
    "gradnorm": 0.294921875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:43.283462"
}
 total length: 7996 dropped: 0 num samples 15 - rank: 3 max len: 571 min len: 402 avg len: 533.0666666666667 num_loss_counted_tokens: 7201
 total length: 7932 dropped: 0 num samples 10 - rank: 0 max len: 1066 min len: 691 avg len: 793.2 num_loss_counted_tokens: 7402
 total length: 7979 dropped: 0 num samples 17 - rank: 5 max len: 492 min len: 452 avg len: 469.3529411764706 num_loss_counted_tokens: 7078
 total length: 7983 dropped: 0 num samples 19 - rank: 6 max len: 450 min len: 305 avg len: 420.1578947368421 num_loss_counted_tokens: 6976
 total length: 7818 dropped: 0 num samples 13 - rank: 2 max len: 630 min len: 573 avg len: 601.3846153846154 num_loss_counted_tokens: 7129
 total length: 7890 dropped: 0 num samples 12 - rank: 1 max len: 691 min len: 631 avg len: 657.5 num_loss_counted_tokens: 7254
 total length: 7679 dropped: 0 num samples 23 - rank: 7 max len: 376 min len: 257 avg len: 333.8695652173913 num_loss_counted_tokens: 6460
 total length: 7995 dropped: 0 num samples 16 - rank: 4 max len: 522 min len: 398 avg len: 499.6875 num_loss_counted_tokens: 7147
Epoch: 2, Step: 100, Rank: 6, loss = 0.059326171875Epoch: 2, Step: 100, Rank: 5, loss = 0.0703125Epoch: 2, Step: 100, Rank: 3, loss = 0.03857421875Epoch: 2, Step: 100, Rank: 4, loss = 0.051513671875Epoch: 2, Step: 100, Rank: 1, loss = 0.0625Epoch: 2, Step: 100, Rank: 7, loss = 0.07470703125


Epoch: 2, Step: 100, Rank: 2, loss = 0.0439453125



Epoch: 2, Step: 100, Rank: 0, loss = 0.03369140625

Epoch 2:  50%|█████     | 20/40 [00:33<00:33,  1.66s/it]{
    "epoch": 2,
    "step": 100,
    "rank": 0,
    "overall_throughput": 94.08460081626187,
    "lr": 1.6026346363792565e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56293,
    "batch_size": 125,
    "total_loss": 0.05430515339385004,
    "samples_seen": 12643,
    "gradnorm": 0.279296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:44.934126"
}
 total length: 7994 dropped: 0 num samples 13 - rank: 1 max len: 649 min len: 404 avg len: 614.9230769230769 num_loss_counted_tokens: 7305
 total length: 7917 dropped: 0 num samples 10 - rank: 0 max len: 1064 min len: 664 avg len: 791.7 num_loss_counted_tokens: 7387
 total length: 7840 dropped: 0 num samples 18 - rank: 5 max len: 473 min len: 400 avg len: 435.55555555555554 num_loss_counted_tokens: 6886
 total length: 7994 dropped: 0 num samples 15 - rank: 3 max len: 572 min len: 305 avg len: 532.9333333333333 num_loss_counted_tokens: 7199
 total length: 7997 dropped: 0 num samples 14 - rank: 2 max len: 612 min len: 269 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7917 dropped: 0 num samples 16 - rank: 4 max len: 528 min len: 475 avg len: 494.8125 num_loss_counted_tokens: 7069
 total length: 7645 dropped: 0 num samples 24 - rank: 7 max len: 361 min len: 262 avg len: 318.5416666666667 num_loss_counted_tokens: 6373
 total length: 7987 dropped: 0 num samples 21 - rank: 6 max len: 399 min len: 362 avg len: 380.3333333333333 num_loss_counted_tokens: 6874
Epoch: 2, Step: 101, Rank: 3, loss = 0.05029296875Epoch: 2, Step: 101, Rank: 0, loss = 0.0341796875Epoch: 2, Step: 101, Rank: 5, loss = 0.0673828125Epoch: 2, Step: 101, Rank: 6, loss = 0.05322265625Epoch: 2, Step: 101, Rank: 7, loss = 0.064453125

Epoch: 2, Step: 101, Rank: 4, loss = 0.0419921875Epoch: 2, Step: 101, Rank: 2, loss = 0.047119140625




Epoch: 2, Step: 101, Rank: 1, loss = 0.062255859375

Epoch 2:  52%|█████▎    | 21/40 [00:34<00:31,  1.66s/it]{
    "epoch": 2,
    "step": 101,
    "rank": 0,
    "overall_throughput": 94.73506571377096,
    "lr": 1.592757601962555e-05,
    "cuda_mem_allocated": 6.580931663513184,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56236,
    "batch_size": 127,
    "total_loss": 0.05259975816203144,
    "samples_seen": 12770,
    "gradnorm": 0.279296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:46.575064"
}
 total length: 7959 dropped: 0 num samples 8 - rank: 0 max len: 1247 min len: 846 avg len: 994.875 num_loss_counted_tokens: 7535
 total length: 7995 dropped: 0 num samples 13 - rank: 2 max len: 683 min len: 278 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7748 dropped: 0 num samples 15 - rank: 4 max len: 551 min len: 485 avg len: 516.5333333333333 num_loss_counted_tokens: 6953
 total length: 7997 dropped: 0 num samples 14 - rank: 3 max len: 612 min len: 532 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7894 dropped: 0 num samples 17 - rank: 5 max len: 480 min len: 444 avg len: 464.3529411764706 num_loss_counted_tokens: 6993
 total length: 7994 dropped: 0 num samples 11 - rank: 1 max len: 832 min len: 588 avg len: 726.7272727272727 num_loss_counted_tokens: 7411
 total length: 7740 dropped: 0 num samples 19 - rank: 6 max len: 439 min len: 382 avg len: 407.36842105263156 num_loss_counted_tokens: 6733
 total length: 7733 dropped: 0 num samples 24 - rank: 7 max len: 382 min len: 261 avg len: 322.2083333333333 num_loss_counted_tokens: 6461
Epoch: 2, Step: 102, Rank: 3, loss = 0.045654296875Epoch: 2, Step: 102, Rank: 5, loss = 0.0634765625Epoch: 2, Step: 102, Rank: 2, loss = 0.05224609375Epoch: 2, Step: 102, Rank: 1, loss = 0.05712890625Epoch: 2, Step: 102, Rank: 6, loss = 0.06298828125



Epoch: 2, Step: 102, Rank: 7, loss = 0.06689453125

Epoch: 2, Step: 102, Rank: 4, loss = 0.060302734375
Epoch: 2, Step: 102, Rank: 0, loss = 0.03955078125

Epoch 2:  55%|█████▌    | 22/40 [00:36<00:31,  1.76s/it]{
    "epoch": 2,
    "step": 102,
    "rank": 0,
    "overall_throughput": 74.8645924699626,
    "lr": 1.582790598933161e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56876,
    "batch_size": 124,
    "total_loss": 0.056051761727266336,
    "samples_seen": 12894,
    "gradnorm": 0.30078125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:48.573867"
}
 total length: 7995 dropped: 0 num samples 11 - rank: 1 max len: 888 min len: 464 avg len: 726.8181818181819 num_loss_counted_tokens: 7412
 total length: 7859 dropped: 0 num samples 23 - rank: 7 max len: 386 min len: 263 avg len: 341.69565217391306 num_loss_counted_tokens: 6640
 total length: 7997 dropped: 0 num samples 8 - rank: 0 max len: 1106 min len: 406 avg len: 999.625 num_loss_counted_tokens: 7573
 total length: 7997 dropped: 0 num samples 14 - rank: 3 max len: 605 min len: 512 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7994 dropped: 0 num samples 13 - rank: 2 max len: 649 min len: 410 avg len: 614.9230769230769 num_loss_counted_tokens: 7305
 total length: 7872 dropped: 0 num samples 19 - rank: 6 max len: 440 min len: 387 avg len: 414.3157894736842 num_loss_counted_tokens: 6865
 total length: 7952 dropped: 0 num samples 17 - rank: 5 max len: 488 min len: 441 avg len: 467.7647058823529 num_loss_counted_tokens: 7051
 total length: 7985 dropped: 0 num samples 16 - rank: 4 max len: 542 min len: 300 avg len: 499.0625 num_loss_counted_tokens: 7137
Epoch: 2, Step: 103, Rank: 5, loss = 0.0673828125Epoch: 2, Step: 103, Rank: 1, loss = 0.044189453125Epoch: 2, Step: 103, Rank: 2, loss = 0.0419921875Epoch: 2, Step: 103, Rank: 7, loss = 0.08154296875

Epoch: 2, Step: 103, Rank: 6, loss = 0.052001953125Epoch: 2, Step: 103, Rank: 3, loss = 0.0615234375


Epoch: 2, Step: 103, Rank: 4, loss = 0.05615234375

Epoch: 2, Step: 103, Rank: 0, loss = 0.0260009765625

Epoch 2:  57%|█████▊    | 23/40 [00:38<00:29,  1.73s/it]{
    "epoch": 2,
    "step": 103,
    "rank": 0,
    "overall_throughput": 94.2426723186077,
    "lr": 1.5727351400805054e-05,
    "cuda_mem_allocated": 6.588062763214111,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56354,
    "batch_size": 131,
    "total_loss": 0.05392696170635625,
    "samples_seen": 13025,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:50.226442"
}
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 644 min len: 476 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7758 dropped: 0 num samples 20 - rank: 6 max len: 409 min len: 366 avg len: 387.9 num_loss_counted_tokens: 6698
 total length: 7919 dropped: 0 num samples 14 - rank: 3 max len: 600 min len: 538 avg len: 565.6428571428571 num_loss_counted_tokens: 7177
 total length: 7986 dropped: 0 num samples 18 - rank: 5 max len: 473 min len: 412 avg len: 443.6666666666667 num_loss_counted_tokens: 7032
 total length: 7525 dropped: 0 num samples 23 - rank: 7 max len: 365 min len: 252 avg len: 327.17391304347825 num_loss_counted_tokens: 6306
 total length: 7911 dropped: 0 num samples 8 - rank: 0 max len: 1104 min len: 809 avg len: 988.875 num_loss_counted_tokens: 7487
 total length: 7993 dropped: 0 num samples 16 - rank: 4 max len: 536 min len: 351 avg len: 499.5625 num_loss_counted_tokens: 7145
 total length: 7994 dropped: 0 num samples 12 - rank: 1 max len: 800 min len: 384 avg len: 666.1666666666666 num_loss_counted_tokens: 7358
Epoch: 2, Step: 104, Rank: 6, loss = 0.064453125Epoch: 2, Step: 104, Rank: 3, loss = 0.0556640625Epoch: 2, Step: 104, Rank: 5, loss = 0.072265625

Epoch: 2, Step: 104, Rank: 4, loss = 0.0625Epoch: 2, Step: 104, Rank: 1, loss = 0.045166015625Epoch: 2, Step: 104, Rank: 7, loss = 0.0751953125Epoch: 2, Step: 104, Rank: 2, loss = 0.05419921875




Epoch: 2, Step: 104, Rank: 0, loss = 0.04248046875

Epoch 2:  60%|██████    | 24/40 [00:40<00:27,  1.70s/it]{
    "epoch": 2,
    "step": 104,
    "rank": 0,
    "overall_throughput": 94.73600186058954,
    "lr": 1.5625927516198235e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56452,
    "batch_size": 130,
    "total_loss": 0.059023595266775315,
    "samples_seen": 13155,
    "gradnorm": 0.345703125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:51.869350"
}
Epoch: 2, Step: 105, Rank: 1, loss = 0.048828125Epoch: 2, Step: 105, Rank: 3, loss = 0.0361328125Epoch: 2, Step: 105, Rank: 4, loss = 0.044921875Epoch: 2, Step: 105, Rank: 2, loss = 0.05029296875Epoch: 2, Step: 105, Rank: 5, loss = 0.0751953125Epoch: 2, Step: 105, Rank: 6, loss = 0.05517578125



Epoch: 2, Step: 105, Rank: 7, loss = 0.07177734375


Epoch: 2, Step: 105, Rank: 0, loss = 0.040283203125

Epoch 2:  62%|██████▎   | 25/40 [00:41<00:25,  1.68s/it]{
    "epoch": 2,
    "step": 105,
    "rank": 0,
    "overall_throughput": 94.5513213069888,
    "lr": 1.552364972960506e-05,
    "cuda_mem_allocated": 6.588062763214111,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56949,
    "batch_size": 129,
    "total_loss": 0.052854308240706596,
    "samples_seen": 13284,
    "gradnorm": 0.2890625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:53.513324"
}
Epoch: 2, Step: 106, Rank: 6, loss = 0.07080078125Epoch: 2, Step: 106, Rank: 5, loss = 0.056396484375Epoch: 2, Step: 106, Rank: 1, loss = 0.076171875
Epoch: 2, Step: 106, Rank: 3, loss = 0.052978515625
Epoch: 2, Step: 106, Rank: 2, loss = 0.045166015625


Epoch: 2, Step: 106, Rank: 7, loss = 0.076171875
Epoch: 2, Step: 106, Rank: 4, loss = 0.0576171875Epoch: 2, Step: 106, Rank: 0, loss = 0.017333984375


Epoch 2:  65%|██████▌   | 26/40 [00:43<00:23,  1.68s/it]{
    "epoch": 2,
    "step": 106,
    "rank": 0,
    "overall_throughput": 93.58744197316516,
    "lr": 1.5420533564724495e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56419,
    "batch_size": 127,
    "total_loss": 0.05655009837111611,
    "samples_seen": 13411,
    "gradnorm": 0.31640625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:55.171040"
}
Epoch: 2, Step: 107, Rank: 6, loss = 0.0654296875Epoch: 2, Step: 107, Rank: 5, loss = 0.0654296875Epoch: 2, Step: 107, Rank: 1, loss = 0.050048828125Epoch: 2, Step: 107, Rank: 2, loss = 0.052978515625Epoch: 2, Step: 107, Rank: 3, loss = 0.060791015625
Epoch: 2, Step: 107, Rank: 7, loss = 0.083984375




Epoch: 2, Step: 107, Rank: 4, loss = 0.06982421875
Epoch: 2, Step: 107, Rank: 0, loss = 0.04052734375

Epoch 2:  68%|██████▊   | 27/40 [00:45<00:22,  1.75s/it]{
    "epoch": 2,
    "step": 107,
    "rank": 0,
    "overall_throughput": 78.15643197212361,
    "lr": 1.5316594672504362e-05,
    "cuda_mem_allocated": 6.574356555938721,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56156,
    "batch_size": 132,
    "total_loss": 0.061079848992093455,
    "samples_seen": 13543,
    "gradnorm": 0.322265625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:57.099941"
}
Epoch: 2, Step: 108, Rank: 1, loss = 0.051025390625Epoch: 2, Step: 108, Rank: 3, loss = 0.058837890625Epoch: 2, Step: 108, Rank: 2, loss = 0.058837890625Epoch: 2, Step: 108, Rank: 7, loss = 0.0859375Epoch: 2, Step: 108, Rank: 5, loss = 0.0634765625Epoch: 2, Step: 108, Rank: 6, loss = 0.0556640625Epoch: 2, Step: 108, Rank: 4, loss = 0.03662109375






Epoch: 2, Step: 108, Rank: 0, loss = 0.0233154296875

Epoch 2:  70%|███████   | 28/40 [00:47<00:20,  1.72s/it]{
    "epoch": 2,
    "step": 108,
    "rank": 0,
    "overall_throughput": 94.15071454250516,
    "lr": 1.5211848828765852e-05,
    "cuda_mem_allocated": 6.586573123931885,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56678,
    "batch_size": 123,
    "total_loss": 0.05414799393062564,
    "samples_seen": 13666,
    "gradnorm": 0.30078125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:34:58.756066"
}
Epoch: 2, Step: 109, Rank: 3, loss = 0.055419921875Epoch: 2, Step: 109, Rank: 0, loss = 0.031494140625Epoch: 2, Step: 109, Rank: 5, loss = 0.0634765625
Epoch: 2, Step: 109, Rank: 1, loss = 0.03662109375
Epoch: 2, Step: 109, Rank: 7, loss = 0.061279296875
Epoch: 2, Step: 109, Rank: 2, loss = 0.03857421875


Epoch: 2, Step: 109, Rank: 4, loss = 0.048828125
Epoch: 2, Step: 109, Rank: 6, loss = 0.06689453125

Epoch 2:  72%|███████▎  | 29/40 [00:48<00:18,  1.70s/it]{
    "epoch": 2,
    "step": 109,
    "rank": 0,
    "overall_throughput": 95.07733611774431,
    "lr": 1.510631193180907e-05,
    "cuda_mem_allocated": 6.580380916595459,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 55830,
    "batch_size": 129,
    "total_loss": 0.05029554003224073,
    "samples_seen": 13795,
    "gradnorm": 0.26171875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:35:00.395091"
}
Epoch: 2, Step: 110, Rank: 1, loss = 0.0380859375Epoch: 2, Step: 110, Rank: 5, loss = 0.05810546875Epoch: 2, Step: 110, Rank: 4, loss = 0.068359375Epoch: 2, Step: 110, Rank: 7, loss = 0.07861328125
Epoch: 2, Step: 110, Rank: 2, loss = 0.049560546875Epoch: 2, Step: 110, Rank: 6, loss = 0.0673828125
Epoch: 2, Step: 110, Rank: 3, loss = 0.0537109375




Epoch: 2, Step: 110, Rank: 0, loss = 0.0274658203125

Epoch 2:  75%|███████▌  | 30/40 [00:50<00:17,  1.77s/it]{
    "epoch": 2,
    "step": 110,
    "rank": 0,
    "overall_throughput": 78.19618357479479,
    "lr": 1.5000000000000002e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56533,
    "batch_size": 129,
    "total_loss": 0.05515362708506536,
    "samples_seen": 13924,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:35:02.323830"
}
Epoch: 2, Step: 111, Rank: 5, loss = 0.0576171875Epoch: 2, Step: 111, Rank: 1, loss = 0.055419921875Epoch: 2, Step: 111, Rank: 4, loss = 0.0537109375Epoch: 2, Step: 111, Rank: 3, loss = 0.05224609375Epoch: 2, Step: 111, Rank: 6, loss = 0.07470703125

Epoch: 2, Step: 111, Rank: 2, loss = 0.041259765625Epoch: 2, Step: 111, Rank: 7, loss = 0.080078125




Epoch: 2, Step: 111, Rank: 0, loss = 0.0240478515625

Epoch 2:  78%|███████▊  | 31/40 [00:52<00:15,  1.73s/it]{
    "epoch": 2,
    "step": 111,
    "rank": 0,
    "overall_throughput": 94.04280589609748,
    "lr": 1.4892929169339237e-05,
    "cuda_mem_allocated": 6.5880632400512695,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 55831,
    "batch_size": 130,
    "total_loss": 0.054844083036305995,
    "samples_seen": 14054,
    "gradnorm": 0.29296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:35:03.978718"
}
Epoch: 2, Step: 112, Rank: 5, loss = 0.072265625Epoch: 2, Step: 112, Rank: 3, loss = 0.04931640625Epoch: 2, Step: 112, Rank: 4, loss = 0.0634765625Epoch: 2, Step: 112, Rank: 6, loss = 0.07373046875



Epoch: 2, Step: 112, Rank: 7, loss = 0.0810546875Epoch: 2, Step: 112, Rank: 1, loss = 0.0419921875Epoch: 2, Step: 112, Rank: 2, loss = 0.051025390625


Epoch: 2, Step: 112, Rank: 0, loss = 0.0302734375

Epoch 2:  80%|████████  | 32/40 [00:54<00:14,  1.80s/it]{
    "epoch": 2,
    "step": 112,
    "rank": 0,
    "overall_throughput": 77.54835580344897,
    "lr": 1.4785115691012866e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56796,
    "batch_size": 128,
    "total_loss": 0.05787379392914994,
    "samples_seen": 14182,
    "gradnorm": 0.298828125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:35:05.920121"
}
Epoch: 2, Step: 113, Rank: 5, loss = 0.07568359375Epoch: 2, Step: 113, Rank: 4, loss = 0.052734375Epoch: 2, Step: 113, Rank: 0, loss = 0.040283203125Epoch: 2, Step: 113, Rank: 3, loss = 0.045166015625



Epoch: 2, Step: 113, Rank: 2, loss = 0.03955078125
Epoch: 2, Step: 113, Rank: 6, loss = 0.07177734375
Epoch: 2, Step: 113, Rank: 7, loss = 0.0869140625
Epoch: 2, Step: 113, Rank: 1, loss = 0.04931640625

Epoch 2:  82%|████████▎ | 33/40 [00:56<00:12,  1.84s/it]{
    "epoch": 2,
    "step": 113,
    "rank": 0,
    "overall_throughput": 76.8858184451181,
    "lr": 1.4676575928925869e-05,
    "cuda_mem_allocated": 6.588062763214111,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56475,
    "batch_size": 129,
    "total_loss": 0.05765382912793272,
    "samples_seen": 14311,
    "gradnorm": 0.298828125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:35:07.875068"
}
Epoch: 2, Step: 114, Rank: 6, loss = 0.07373046875Epoch: 2, Step: 114, Rank: 1, loss = 0.046875Epoch: 2, Step: 114, Rank: 5, loss = 0.05419921875Epoch: 2, Step: 114, Rank: 4, loss = 0.042236328125Epoch: 2, Step: 114, Rank: 3, loss = 0.041748046875
Epoch: 2, Step: 114, Rank: 7, loss = 0.06640625

Epoch: 2, Step: 114, Rank: 2, loss = 0.041015625



Epoch: 2, Step: 114, Rank: 0, loss = 0.0296630859375

Epoch 2:  85%|████████▌ | 34/40 [00:57<00:10,  1.79s/it]{
    "epoch": 2,
    "step": 114,
    "rank": 0,
    "overall_throughput": 94.39050424736328,
    "lr": 1.4567326357218408e-05,
    "cuda_mem_allocated": 6.586756229400635,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56954,
    "batch_size": 125,
    "total_loss": 0.04946096850089546,
    "samples_seen": 14436,
    "gradnorm": 0.2734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:35:09.523643"
}
Epoch: 2, Step: 115, Rank: 6, loss = 0.0654296875Epoch: 2, Step: 115, Rank: 5, loss = 0.07080078125Epoch: 2, Step: 115, Rank: 3, loss = 0.052490234375Epoch: 2, Step: 115, Rank: 1, loss = 0.0517578125Epoch: 2, Step: 115, Rank: 7, loss = 0.083984375Epoch: 2, Step: 115, Rank: 4, loss = 0.056640625


Epoch: 2, Step: 115, Rank: 2, loss = 0.035888671875



Epoch: 2, Step: 115, Rank: 0, loss = 0.046875

Epoch 2:  88%|████████▊ | 35/40 [00:59<00:08,  1.74s/it]{
    "epoch": 2,
    "step": 115,
    "rank": 0,
    "overall_throughput": 95.43126496915781,
    "lr": 1.4457383557765385e-05,
    "cuda_mem_allocated": 6.576313018798828,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56452,
    "batch_size": 129,
    "total_loss": 0.05796074541203146,
    "samples_seen": 14565,
    "gradnorm": 0.296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:35:11.159738"
}
Epoch: 2, Step: 116, Rank: 6, loss = 0.05810546875Epoch: 2, Step: 116, Rank: 1, loss = 0.0625Epoch: 2, Step: 116, Rank: 5, loss = 0.061279296875Epoch: 2, Step: 116, Rank: 2, loss = 0.03759765625Epoch: 2, Step: 116, Rank: 7, loss = 0.06591796875Epoch: 2, Step: 116, Rank: 0, loss = 0.052734375


Epoch: 2, Step: 116, Rank: 4, loss = 0.056396484375



Epoch: 2, Step: 116, Rank: 3, loss = 0.059814453125

Epoch 2:  90%|█████████ | 36/40 [01:01<00:07,  1.79s/it]{
    "epoch": 2,
    "step": 116,
    "rank": 0,
    "overall_throughput": 78.62761269006242,
    "lr": 1.4346764217659652e-05,
    "cuda_mem_allocated": 6.582150936126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56647,
    "batch_size": 125,
    "total_loss": 0.056843257365791655,
    "samples_seen": 14690,
    "gradnorm": 0.302734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:35:13.064459"
}
Epoch: 2, Step: 117, Rank: 6, loss = 0.07861328125Epoch: 2, Step: 117, Rank: 5, loss = 0.0693359375
Epoch: 2, Step: 117, Rank: 7, loss = 0.060546875Epoch: 2, Step: 117, Rank: 3, loss = 0.068359375Epoch: 2, Step: 117, Rank: 2, loss = 0.05615234375Epoch: 2, Step: 117, Rank: 1, loss = 0.043212890625




Epoch: 2, Step: 117, Rank: 0, loss = 0.033447265625
Epoch: 2, Step: 117, Rank: 4, loss = 0.04541015625

Epoch 2:  92%|█████████▎| 37/40 [01:03<00:05,  1.75s/it]{
    "epoch": 2,
    "step": 117,
    "rank": 0,
    "overall_throughput": 94.96759658819843,
    "lr": 1.4235485126679244e-05,
    "cuda_mem_allocated": 6.580564498901367,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56348,
    "batch_size": 131,
    "total_loss": 0.05689642933200823,
    "samples_seen": 14821,
    "gradnorm": 0.296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:35:14.705411"
}
Epoch: 2, Step: 118, Rank: 5, loss = 0.052490234375Epoch: 2, Step: 118, Rank: 6, loss = 0.0673828125Epoch: 2, Step: 118, Rank: 3, loss = 0.04736328125Epoch: 2, Step: 118, Rank: 7, loss = 0.0712890625Epoch: 2, Step: 118, Rank: 1, loss = 0.043212890625

Epoch: 2, Step: 118, Rank: 2, loss = 0.04248046875



Epoch: 2, Step: 118, Rank: 4, loss = 0.05712890625
Epoch: 2, Step: 118, Rank: 0, loss = 0.0390625

Epoch 2:  95%|█████████▌| 38/40 [01:04<00:03,  1.72s/it]{
    "epoch": 2,
    "step": 118,
    "rank": 0,
    "overall_throughput": 94.21434195815351,
    "lr": 1.4123563174739036e-05,
    "cuda_mem_allocated": 6.584375381469727,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56647,
    "batch_size": 121,
    "total_loss": 0.05253588009956397,
    "samples_seen": 14942,
    "gradnorm": 0.267578125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:35:16.353917"
}
Epoch: 2, Step: 119, Rank: 2, loss = 0.04931640625Epoch: 2, Step: 119, Rank: 6, loss = 0.046875Epoch: 2, Step: 119, Rank: 1, loss = 0.059326171875Epoch: 2, Step: 119, Rank: 5, loss = 0.053955078125Epoch: 2, Step: 119, Rank: 4, loss = 0.0537109375
Epoch: 2, Step: 119, Rank: 7, loss = 0.072265625

Epoch: 2, Step: 119, Rank: 3, loss = 0.05126953125



Epoch: 2, Step: 119, Rank: 0, loss = 0.0238037109375

Epoch 2:  98%|█████████▊| 39/40 [01:06<00:01,  1.70s/it]{
    "epoch": 2,
    "step": 119,
    "rank": 0,
    "overall_throughput": 93.91384857377561,
    "lr": 1.4011015349327188e-05,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 57238,
    "batch_size": 121,
    "total_loss": 0.05125965267829065,
    "samples_seen": 15063,
    "gradnorm": 0.275390625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:35:18.008789"
}
Epoch: 2, Step: 120, Rank: 4, loss = 0.04736328125Epoch: 2, Step: 120, Rank: 2, loss = 0.0400390625Epoch: 2, Step: 120, Rank: 0, loss = 0.0179443359375Epoch: 2, Step: 120, Rank: 3, loss = 0.0419921875


Epoch: 2, Step: 120, Rank: 6, loss = 0.0830078125Epoch: 2, Step: 120, Rank: 5, loss = 0.0703125
Epoch: 2, Step: 120, Rank: 7, loss = 0.0732421875


Epoch: 2, Step: 120, Rank: 1, loss = 0.041259765625

Epoch 2: 100%|██████████| 40/40 [01:07<00:00,  1.68s/it]{
    "epoch": 2,
    "step": 120,
    "rank": 0,
    "overall_throughput": 94.67981568796752,
    "lr": 1.3897858732926794e-05,
    "cuda_mem_allocated": 6.580197811126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56510,
    "batch_size": 124,
    "total_loss": 0.05185807821624491,
    "samples_seen": 15187,
    "gradnorm": 0.28125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:35:19.653999"
}
Saving model in huggingface format at: samples_15187
[19:35:46] INFO     The model is bigger than the maximum     accelerator.py:2924
                    size per checkpoint (5GB) and is going                      
                    to be split in 4 checkpoint shards. You                     
                    can find where each parameters has been                     
                    saved in the index located at                               
                    /tmp/tmp9rsqb4r5w/model.safetensors.inde                    
                    x.json.                                                     
Model saved in /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/hf_format/samples_15187
[19:36:02] INFO     saving took 42.124101400375366 seconds          utils.py:879
Saving full model state in /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/full_state/epoch_2
           INFO     Saving current state to                  accelerator.py:3030
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_2                                                  
           INFO     Saving FSDP model                        accelerator.py:3040
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
[19:36:10] INFO     Saving model to                             fsdp_utils.py:88
                    /var/home/cloud-user/.local/share/instructl                 
                    ab/phased/phase1/checkpoints/full_state/epo                 
                    ch_2/pytorch_model_fsdp.bin                                 
[19:36:21] INFO     Model saved to                              fsdp_utils.py:90
                    /var/home/cloud-user/.local/share/instructl                 
                    ab/phased/phase1/checkpoints/full_state/epo                 
                    ch_2/pytorch_model_fsdp.bin                                 
           INFO     FSDP Model saved to output dir           accelerator.py:3042
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_2                                                  
           INFO     Saving FSDP Optimizer                    accelerator.py:3059
 total length: 7993 dropped: 0 num samples 21 - rank: 6 max len: 412 min len: 286 avg len: 380.6190476190476 num_loss_counted_tokens: 6880
 total length: 7997 dropped: 0 num samples 20 - rank: 6 max len: 436 min len: 259 avg len: 399.85 num_loss_counted_tokens: 6937 total length: 7883 dropped: 0 num samples 20 - rank: 6 max len: 421 min len: 372 avg len: 394.15 num_loss_counted_tokens: 6823
 total length: 7865 dropped: 0 num samples 20 - rank: 6 max len: 418 min len: 368 avg len: 393.25 num_loss_counted_tokens: 6805
 total length: 7910 dropped: 0 num samples 20 - rank: 6 max len: 422 min len: 367 avg len: 395.5 num_loss_counted_tokens: 6850
 total length: 7878 dropped: 0 num samples 20 - rank: 6 max len: 432 min len: 367 avg len: 393.9 num_loss_counted_tokens: 6818

 total length: 7975 dropped: 0 num samples 21 - rank: 6 max len: 413 min len: 272 avg len: 379.76190476190476 num_loss_counted_tokens: 6862
 total length: 7996 dropped: 0 num samples 20 - rank: 6 max len: 422 min len: 316 avg len: 399.8 num_loss_counted_tokens: 6936
 total length: 7788 dropped: 0 num samples 19 - rank: 6 max len: 430 min len: 394 avg len: 409.89473684210526 num_loss_counted_tokens: 6781
 total length: 7997 dropped: 0 num samples 20 - rank: 6 max len: 424 min len: 243 avg len: 399.85 num_loss_counted_tokens: 6937
 total length: 7996 dropped: 0 num samples 20 - rank: 6 max len: 427 min len: 323 avg len: 399.8 num_loss_counted_tokens: 6936
 total length: 7986 dropped: 0 num samples 19 - rank: 6 max len: 469 min len: 264 avg len: 420.3157894736842 num_loss_counted_tokens: 6979
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 639 min len: 501 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7799 dropped: 0 num samples 21 - rank: 6 max len: 397 min len: 344 avg len: 371.3809523809524 num_loss_counted_tokens: 6686
 total length: 7811 dropped: 0 num samples 20 - rank: 6 max len: 404 min len: 372 avg len: 390.55 num_loss_counted_tokens: 6751
 total length: 7926 dropped: 0 num samples 13 - rank: 2 max len: 637 min len: 569 avg len: 609.6923076923077 num_loss_counted_tokens: 7237 total length: 7261 dropped: 0 num samples 23 - rank: 7 max len: 363 min len: 243 avg len: 315.69565217391306 num_loss_counted_tokens: 6042
 total length: 7981 dropped: 0 num samples 20 - rank: 6 max len: 430 min len: 374 avg len: 399.05 num_loss_counted_tokens: 6921
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 646 min len: 444 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 660 min len: 418 avg len: 615.1538461538462 num_loss_counted_tokens: 7308

 total length: 7981 dropped: 0 num samples 13 - rank: 2 max len: 627 min len: 594 avg len: 613.9230769230769 num_loss_counted_tokens: 7292
 total length: 7907 dropped: 0 num samples 24 - rank: 7 max len: 367 min len: 256 avg len: 329.4583333333333 num_loss_counted_tokens: 6635
 total length: 7864 dropped: 0 num samples 25 - rank: 7 max len: 373 min len: 180 avg len: 314.56 num_loss_counted_tokens: 6539
 total length: 7994 dropped: 0 num samples 25 - rank: 7 max len: 366 min len: 236 avg len: 319.76 num_loss_counted_tokens: 6669
 total length: 7968 dropped: 0 num samples 14 - rank: 3 max len: 589 min len: 548 avg len: 569.1428571428571 num_loss_counted_tokens: 7226 total length: 7879 dropped: 0 num samples 14 - rank: 3 max len: 586 min len: 542 avg len: 562.7857142857143 num_loss_counted_tokens: 7137

 total length: 7732 dropped: 0 num samples 24 - rank: 7 max len: 359 min len: 248 avg len: 322.1666666666667 num_loss_counted_tokens: 6460
 total length: 7663 dropped: 0 num samples 25 - rank: 7 max len: 366 min len: 250 avg len: 306.52 num_loss_counted_tokens: 6338
 total length: 7977 dropped: 0 num samples 15 - rank: 3 max len: 598 min len: 291 avg len: 531.8 num_loss_counted_tokens: 7182
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 664 min len: 391 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 656 min len: 261 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 633 min len: 585 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7970 dropped: 0 num samples 25 - rank: 7 max len: 362 min len: 253 avg len: 318.8 num_loss_counted_tokens: 6645
 total length: 7470 dropped: 0 num samples 22 - rank: 7 max len: 393 min len: 290 avg len: 339.54545454545456 num_loss_counted_tokens: 6304
 total length: 7933 dropped: 0 num samples 20 - rank: 6 max len: 425 min len: 372 avg len: 396.65 num_loss_counted_tokens: 6873
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 818 min len: 292 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7995 dropped: 0 num samples 14 - rank: 3 max len: 599 min len: 453 avg len: 571.0714285714286 num_loss_counted_tokens: 7253
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 639 min len: 574 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7990 dropped: 0 num samples 15 - rank: 3 max len: 567 min len: 300 avg len: 532.6666666666666 num_loss_counted_tokens: 7195
 total length: 7995 dropped: 0 num samples 12 - rank: 2 max len: 706 min len: 523 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7788 dropped: 0 num samples 14 - rank: 3 max len: 600 min len: 521 avg len: 556.2857142857143 num_loss_counted_tokens: 7046
 total length: 7976 dropped: 0 num samples 14 - rank: 2 max len: 623 min len: 229 avg len: 569.7142857142857 num_loss_counted_tokens: 7234
 total length: 7884 dropped: 0 num samples 14 - rank: 3 max len: 591 min len: 545 avg len: 563.1428571428571 num_loss_counted_tokens: 7142
 total length: 7825 dropped: 0 num samples 15 - rank: 3 max len: 551 min len: 503 avg len: 521.6666666666666 num_loss_counted_tokens: 7030
 total length: 7483 dropped: 0 num samples 23 - rank: 7 max len: 379 min len: 171 avg len: 325.3478260869565 num_loss_counted_tokens: 6264
 total length: 7171 dropped: 0 num samples 23 - rank: 7 max len: 343 min len: 261 avg len: 311.7826086956522 num_loss_counted_tokens: 5952
 total length: 7854 dropped: 0 num samples 14 - rank: 3 max len: 595 min len: 538 avg len: 561.0 num_loss_counted_tokens: 7112
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 648 min len: 394 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7997 dropped: 0 num samples 14 - rank: 3 max len: 595 min len: 485 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 694 min len: 474 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7682 dropped: 0 num samples 23 - rank: 7 max len: 367 min len: 236 avg len: 334.0 num_loss_counted_tokens: 6463
 total length: 7492 dropped: 0 num samples 23 - rank: 7 max len: 382 min len: 244 avg len: 325.7391304347826 num_loss_counted_tokens: 6273
 total length: 7809 dropped: 0 num samples 24 - rank: 7 max len: 378 min len: 242 avg len: 325.375 num_loss_counted_tokens: 6537
 total length: 7757 dropped: 0 num samples 13 - rank: 2 max len: 617 min len: 568 avg len: 596.6923076923077 num_loss_counted_tokens: 7068
 total length: 7630 dropped: 0 num samples 23 - rank: 7 max len: 373 min len: 279 avg len: 331.7391304347826 num_loss_counted_tokens: 6411
 total length: 7994 dropped: 0 num samples 12 - rank: 1 max len: 715 min len: 491 avg len: 666.1666666666666 num_loss_counted_tokens: 7358
 total length: 7993 dropped: 0 num samples 13 - rank: 2 max len: 633 min len: 491 avg len: 614.8461538461538 num_loss_counted_tokens: 7304
 total length: 7942 dropped: 0 num samples 12 - rank: 1 max len: 694 min len: 645 avg len: 661.8333333333334 num_loss_counted_tokens: 7306
 total length: 7867 dropped: 0 num samples 12 - rank: 1 max len: 684 min len: 629 avg len: 655.5833333333334 num_loss_counted_tokens: 7231
 total length: 7936 dropped: 0 num samples 12 - rank: 1 max len: 684 min len: 642 avg len: 661.3333333333334 num_loss_counted_tokens: 7300
 total length: 7997 dropped: 0 num samples 21 - rank: 6 max len: 405 min len: 344 avg len: 380.8095238095238 num_loss_counted_tokens: 6884
 total length: 7409 dropped: 0 num samples 22 - rank: 7 max len: 395 min len: 259 avg len: 336.77272727272725 num_loss_counted_tokens: 6243
 total length: 7993 dropped: 0 num samples 13 - rank: 2 max len: 647 min len: 496 avg len: 614.8461538461538 num_loss_counted_tokens: 7304
 total length: 7950 dropped: 0 num samples 12 - rank: 1 max len: 693 min len: 637 avg len: 662.5 num_loss_counted_tokens: 7314
 total length: 7941 dropped: 0 num samples 12 - rank: 1 max len: 701 min len: 647 avg len: 661.75 num_loss_counted_tokens: 7305
 total length: 7993 dropped: 0 num samples 15 - rank: 3 max len: 565 min len: 354 avg len: 532.8666666666667 num_loss_counted_tokens: 7198
 total length: 7860 dropped: 0 num samples 13 - rank: 3 max len: 627 min len: 573 avg len: 604.6153846153846 num_loss_counted_tokens: 7171
 total length: 7997 dropped: 0 num samples 14 - rank: 2 max len: 603 min len: 473 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7993 dropped: 0 num samples 14 - rank: 3 max len: 596 min len: 550 avg len: 570.9285714285714 num_loss_counted_tokens: 7251
 total length: 7997 dropped: 0 num samples 13 - rank: 3 max len: 647 min len: 477 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7993 dropped: 0 num samples 10 - rank: 1 max len: 1044 min len: 457 avg len: 799.3 num_loss_counted_tokens: 7463
 total length: 7973 dropped: 0 num samples 11 - rank: 1 max len: 819 min len: 665 avg len: 724.8181818181819 num_loss_counted_tokens: 7390
 total length: 7458 dropped: 0 num samples 23 - rank: 7 max len: 368 min len: 252 avg len: 324.2608695652174 num_loss_counted_tokens: 6239
 total length: 7995 dropped: 0 num samples 15 - rank: 3 max len: 564 min len: 448 avg len: 533.0 num_loss_counted_tokens: 7200
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 710 min len: 477 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7938 dropped: 0 num samples 12 - rank: 1 max len: 690 min len: 648 avg len: 661.5 num_loss_counted_tokens: 7302
 total length: 7990 dropped: 0 num samples 13 - rank: 1 max len: 695 min len: 253 avg len: 614.6153846153846 num_loss_counted_tokens: 7301
 total length: 7992 dropped: 0 num samples 13 - rank: 2 max len: 665 min len: 347 avg len: 614.7692307692307 num_loss_counted_tokens: 7303
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 747 min len: 339 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7988 dropped: 0 num samples 14 - rank: 3 max len: 614 min len: 288 avg len: 570.5714285714286 num_loss_counted_tokens: 7246
 total length: 7992 dropped: 0 num samples 13 - rank: 1 max len: 661 min len: 264 avg len: 614.7692307692307 num_loss_counted_tokens: 7303
 total length: 7976 dropped: 0 num samples 12 - rank: 1 max len: 707 min len: 633 avg len: 664.6666666666666 num_loss_counted_tokens: 7340
 total length: 7997 dropped: 0 num samples 13 - rank: 1 max len: 646 min len: 400 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7588 dropped: 0 num samples 24 - rank: 7 max len: 350 min len: 266 avg len: 316.1666666666667 num_loss_counted_tokens: 6316
 total length: 7981 dropped: 0 num samples 18 - rank: 5 max len: 473 min len: 256 avg len: 443.3888888888889 num_loss_counted_tokens: 7027
 total length: 7993 dropped: 0 num samples 17 - rank: 5 max len: 510 min len: 374 avg len: 470.1764705882353 num_loss_counted_tokens: 7092
 total length: 7996 dropped: 0 num samples 18 - rank: 5 max len: 469 min len: 414 avg len: 444.22222222222223 num_loss_counted_tokens: 7042
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 468 min len: 353 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 495 min len: 352 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7816 dropped: 0 num samples 17 - rank: 5 max len: 484 min len: 433 avg len: 459.7647058823529 num_loss_counted_tokens: 6915
 total length: 7996 dropped: 0 num samples 14 - rank: 3 max len: 617 min len: 334 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7973 dropped: 0 num samples 17 - rank: 5 max len: 485 min len: 445 avg len: 469.0 num_loss_counted_tokens: 7072
 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 521 min len: 466 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7859 dropped: 0 num samples 16 - rank: 4 max len: 517 min len: 471 avg len: 491.1875 num_loss_counted_tokens: 7011 total length: 7785 dropped: 0 num samples 15 - rank: 4 max len: 541 min len: 497 avg len: 519.0 num_loss_counted_tokens: 6990

 total length: 7986 dropped: 0 num samples 16 - rank: 4 max len: 545 min len: 262 avg len: 499.125 num_loss_counted_tokens: 7138
 total length: 7981 dropped: 0 num samples 18 - rank: 5 max len: 501 min len: 164 avg len: 443.3888888888889 num_loss_counted_tokens: 7027
 total length: 7969 dropped: 0 num samples 16 - rank: 5 max len: 522 min len: 470 avg len: 498.0625 num_loss_counted_tokens: 7121
 total length: 7989 dropped: 0 num samples 18 - rank: 5 max len: 479 min len: 402 avg len: 443.8333333333333 num_loss_counted_tokens: 7035
 total length: 7844 dropped: 0 num samples 18 - rank: 5 max len: 465 min len: 397 avg len: 435.77777777777777 num_loss_counted_tokens: 6890
 total length: 7770 dropped: 0 num samples 17 - rank: 5 max len: 495 min len: 436 avg len: 457.05882352941177 num_loss_counted_tokens: 6869
 total length: 7959 dropped: 0 num samples 15 - rank: 4 max len: 553 min len: 510 avg len: 530.6 num_loss_counted_tokens: 7164 total length: 7821 dropped: 0 num samples 15 - rank: 4 max len: 543 min len: 495 avg len: 521.4 num_loss_counted_tokens: 7026

 total length: 7861 dropped: 0 num samples 16 - rank: 4 max len: 525 min len: 468 avg len: 491.3125 num_loss_counted_tokens: 7013
 total length: 7995 dropped: 0 num samples 19 - rank: 5 max len: 464 min len: 355 avg len: 420.7894736842105 num_loss_counted_tokens: 6988
 total length: 7995 dropped: 0 num samples 16 - rank: 4 max len: 544 min len: 295 avg len: 499.6875 num_loss_counted_tokens: 7147
 total length: 7993 dropped: 0 num samples 18 - rank: 5 max len: 467 min len: 402 avg len: 444.05555555555554 num_loss_counted_tokens: 7039
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 488 min len: 172 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7913 dropped: 0 num samples 15 - rank: 4 max len: 546 min len: 501 avg len: 527.5333333333333 num_loss_counted_tokens: 7118
 total length: 7943 dropped: 0 num samples 17 - rank: 5 max len: 508 min len: 433 avg len: 467.2352941176471 num_loss_counted_tokens: 7042
 total length: 7849 dropped: 0 num samples 16 - rank: 4 max len: 528 min len: 465 avg len: 490.5625 num_loss_counted_tokens: 7001
 total length: 7871 dropped: 0 num samples 15 - rank: 4 max len: 557 min len: 496 avg len: 524.7333333333333 num_loss_counted_tokens: 7076
 total length: 7774 dropped: 0 num samples 15 - rank: 4 max len: 540 min len: 492 avg len: 518.2666666666667 num_loss_counted_tokens: 6979
 total length: 7809 dropped: 0 num samples 16 - rank: 4 max len: 524 min len: 467 avg len: 488.0625 num_loss_counted_tokens: 6961
 total length: 7776 dropped: 0 num samples 16 - rank: 4 max len: 501 min len: 468 avg len: 486.0 num_loss_counted_tokens: 6928
 total length: 7935 dropped: 0 num samples 14 - rank: 4 max len: 595 min len: 530 avg len: 566.7857142857143 num_loss_counted_tokens: 7193
 total length: 7749 dropped: 0 num samples 16 - rank: 4 max len: 516 min len: 457 avg len: 484.3125 num_loss_counted_tokens: 6901
 total length: 7995 dropped: 0 num samples 15 - rank: 4 max len: 555 min len: 430 avg len: 533.0 num_loss_counted_tokens: 7200
 total length: 7754 dropped: 0 num samples 18 - rank: 5 max len: 452 min len: 409 avg len: 430.77777777777777 num_loss_counted_tokens: 6800
[19:36:31] INFO     Saving Optimizer state to                  fsdp_utils.py:192
                    /var/home/cloud-user/.local/share/instruct                  
                    lab/phased/phase1/checkpoints/full_state/e                  
                    poch_2/optimizer.bin                                        
[19:36:54] INFO     Optimizer state saved in                   fsdp_utils.py:194
                    /var/home/cloud-user/.local/share/instruct                  
                    lab/phased/phase1/checkpoints/full_state/e                  
                    poch_2/optimizer.bin                                        
           INFO     FSDP Optimizer saved to output dir       accelerator.py:3061
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_2                                                  
           INFO     Scheduler state saved in                checkpointing.py:118
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_2/scheduler.bin                                  
           INFO     Sampler state for dataloader 0 saved in checkpointing.py:135
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_2/sampler.bin                                    
           INFO     Random states saved in                  checkpointing.py:160
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_2/random_states_0.pkl                            
Saving training state: {'current_epoch': 2, 'samples_seen': 15187}
Model state saved in: /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/full_state/epoch_2


Epoch 3:   0%|          | 0/40 [00:00<?, ?it/s]
Epoch 2: 100%|██████████| 40/40 [02:42<00:00,  4.07s/it]
 total length: 7997 dropped: 0 num samples 8 - rank: 0 max len: 1482 min len: 431 avg len: 999.625 num_loss_counted_tokens: 7573
 total length: 7883 dropped: 0 num samples 10 - rank: 0 max len: 1056 min len: 689 avg len: 788.3 num_loss_counted_tokens: 7353
 total length: 7994 dropped: 0 num samples 10 - rank: 0 max len: 1098 min len: 465 avg len: 799.4 num_loss_counted_tokens: 7464
 total length: 7996 dropped: 0 num samples 10 - rank: 0 max len: 1072 min len: 406 avg len: 799.6 num_loss_counted_tokens: 7466
 total length: 7758 dropped: 0 num samples 9 - rank: 0 max len: 1143 min len: 699 avg len: 862.0 num_loss_counted_tokens: 7281
 total length: 7996 dropped: 0 num samples 10 - rank: 0 max len: 1077 min len: 317 avg len: 799.6 num_loss_counted_tokens: 7466
 total length: 7990 dropped: 0 num samples 10 - rank: 0 max len: 991 min len: 655 avg len: 799.0 num_loss_counted_tokens: 7460
 total length: 7884 dropped: 0 num samples 8 - rank: 0 max len: 1104 min len: 831 avg len: 985.5 num_loss_counted_tokens: 7460
 total length: 7992 dropped: 0 num samples 8 - rank: 0 max len: 1196 min len: 446 avg len: 999.0 num_loss_counted_tokens: 7568
 total length: 7990 dropped: 0 num samples 11 - rank: 0 max len: 1068 min len: 427 avg len: 726.3636363636364 num_loss_counted_tokens: 7407
 total length: 7997 dropped: 0 num samples 8 - rank: 0 max len: 1155 min len: 359 avg len: 999.625 num_loss_counted_tokens: 7573
 total length: 7997 dropped: 0 num samples 10 - rank: 0 max len: 1068 min len: 525 avg len: 799.7 num_loss_counted_tokens: 7467
 total length: 7782 dropped: 0 num samples 9 - rank: 0 max len: 1064 min len: 697 avg len: 864.6666666666666 num_loss_counted_tokens: 7305
 total length: 7991 dropped: 0 num samples 9 - rank: 0 max len: 1100 min len: 654 avg len: 887.8888888888889 num_loss_counted_tokens: 7514
 total length: 7991 dropped: 0 num samples 10 - rank: 0 max len: 1104 min len: 446 avg len: 799.1 num_loss_counted_tokens: 7461
 total length: 7997 dropped: 0 num samples 10 - rank: 0 max len: 1063 min len: 561 avg len: 799.7 num_loss_counted_tokens: 7467
 total length: 7826 dropped: 0 num samples 10 - rank: 0 max len: 1066 min len: 657 avg len: 782.6 num_loss_counted_tokens: 7296
Epoch: 3, Step: 121, Rank: 6, loss = 0.0546875Epoch: 3, Step: 121, Rank: 3, loss = 0.03955078125Epoch: 3, Step: 121, Rank: 1, loss = 0.052490234375
Epoch: 3, Step: 121, Rank: 7, loss = 0.062255859375
Epoch: 3, Step: 121, Rank: 5, loss = 0.053955078125
Epoch: 3, Step: 121, Rank: 2, loss = 0.043212890625Epoch: 3, Step: 121, Rank: 4, loss = 0.0478515625



Epoch: 3, Step: 121, Rank: 0, loss = 0.025390625


Epoch 3:   2%|▎         | 1/40 [00:01<01:04,  1.65s/it]{
    "epoch": 3,
    "step": 121,
    "rank": 0,
    "overall_throughput": 93.84719750687505,
    "lr": 1.3784110500423104e-05,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56384,
    "batch_size": 125,
    "total_loss": 0.04740706583427923,
    "samples_seen": 15312,
    "gradnorm": 0.26171875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:36:56.169659"
}
 total length: 7647 dropped: 0 num samples 25 - rank: 7 max len: 357 min len: 233 avg len: 305.88 num_loss_counted_tokens: 6322
 total length: 7863 dropped: 0 num samples 11 - rank: 0 max len: 1057 min len: 633 avg len: 714.8181818181819 num_loss_counted_tokens: 7280
 total length: 7997 dropped: 0 num samples 16 - rank: 3 max len: 523 min len: 418 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7814 dropped: 0 num samples 14 - rank: 2 max len: 601 min len: 526 avg len: 558.1428571428571 num_loss_counted_tokens: 7072
 total length: 7969 dropped: 0 num samples 17 - rank: 4 max len: 485 min len: 448 avg len: 468.7647058823529 num_loss_counted_tokens: 7068
 total length: 7983 dropped: 0 num samples 13 - rank: 1 max len: 631 min len: 574 avg len: 614.0769230769231 num_loss_counted_tokens: 7294
 total length: 7921 dropped: 0 num samples 20 - rank: 6 max len: 418 min len: 359 avg len: 396.05 num_loss_counted_tokens: 6861
 total length: 7799 dropped: 0 num samples 18 - rank: 5 max len: 446 min len: 418 avg len: 433.27777777777777 num_loss_counted_tokens: 6845
Epoch: 3, Step: 122, Rank: 5, loss = 0.043701171875Epoch: 3, Step: 122, Rank: 3, loss = 0.037353515625Epoch: 3, Step: 122, Rank: 7, loss = 0.06396484375
Epoch: 3, Step: 122, Rank: 1, loss = 0.04345703125
Epoch: 3, Step: 122, Rank: 6, loss = 0.0654296875
Epoch: 3, Step: 122, Rank: 4, loss = 0.04931640625

Epoch: 3, Step: 122, Rank: 2, loss = 0.038330078125

Epoch: 3, Step: 122, Rank: 0, loss = 0.030029296875


Epoch 3:   5%|▌         | 2/40 [00:03<01:02,  1.65s/it]{
    "epoch": 3,
    "step": 122,
    "rank": 0,
    "overall_throughput": 94.41442422458239,
    "lr": 1.3669787916496722e-05,
    "cuda_mem_allocated": 6.578244686126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56660,
    "batch_size": 129,
    "total_loss": 0.04643487469114013,
    "samples_seen": 15441,
    "gradnorm": 0.26953125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:36:57.814751"
}
 total length: 7871 dropped: 0 num samples 19 - rank: 6 max len: 442 min len: 388 avg len: 414.2631578947368 num_loss_counted_tokens: 6864
 total length: 7818 dropped: 0 num samples 23 - rank: 7 max len: 388 min len: 287 avg len: 339.9130434782609 num_loss_counted_tokens: 6599
 total length: 7936 dropped: 0 num samples 15 - rank: 3 max len: 585 min len: 172 avg len: 529.0666666666667 num_loss_counted_tokens: 7141
 total length: 7884 dropped: 0 num samples 13 - rank: 2 max len: 621 min len: 586 avg len: 606.4615384615385 num_loss_counted_tokens: 7195
 total length: 7995 dropped: 0 num samples 16 - rank: 4 max len: 528 min len: 402 avg len: 499.6875 num_loss_counted_tokens: 7147
 total length: 7963 dropped: 0 num samples 13 - rank: 1 max len: 667 min len: 249 avg len: 612.5384615384615 num_loss_counted_tokens: 7274
 total length: 7997 dropped: 0 num samples 10 - rank: 0 max len: 1052 min len: 604 avg len: 799.7 num_loss_counted_tokens: 7467
 total length: 7936 dropped: 0 num samples 17 - rank: 5 max len: 489 min len: 443 avg len: 466.8235294117647 num_loss_counted_tokens: 7035
Epoch: 3, Step: 123, Rank: 5, loss = 0.052001953125Epoch: 3, Step: 123, Rank: 2, loss = 0.0419921875Epoch: 3, Step: 123, Rank: 1, loss = 0.037353515625Epoch: 3, Step: 123, Rank: 3, loss = 0.042236328125
Epoch: 3, Step: 123, Rank: 7, loss = 0.06298828125

Epoch: 3, Step: 123, Rank: 4, loss = 0.039306640625
Epoch: 3, Step: 123, Rank: 6, loss = 0.048828125


Epoch: 3, Step: 123, Rank: 0, loss = 0.0220947265625


Epoch 3:   8%|▊         | 3/40 [00:04<01:00,  1.65s/it]{
    "epoch": 3,
    "step": 123,
    "rank": 0,
    "overall_throughput": 94.50446927030113,
    "lr": 1.355490833300318e-05,
    "cuda_mem_allocated": 6.5880632400512695,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 57116,
    "batch_size": 126,
    "total_loss": 0.04331535821836263,
    "samples_seen": 15567,
    "gradnorm": 0.259765625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:36:59.461853"
}
 total length: 7779 dropped: 0 num samples 25 - rank: 7 max len: 378 min len: 238 avg len: 311.16 num_loss_counted_tokens: 6454
 total length: 7996 dropped: 0 num samples 11 - rank: 0 max len: 846 min len: 438 avg len: 726.9090909090909 num_loss_counted_tokens: 7413
 total length: 7995 dropped: 0 num samples 15 - rank: 3 max len: 576 min len: 390 avg len: 533.0 num_loss_counted_tokens: 7200
 total length: 7945 dropped: 0 num samples 16 - rank: 4 max len: 516 min len: 477 avg len: 496.5625 num_loss_counted_tokens: 7097
 total length: 7760 dropped: 0 num samples 17 - rank: 5 max len: 476 min len: 430 avg len: 456.47058823529414 num_loss_counted_tokens: 6859
 total length: 7997 dropped: 0 num samples 20 - rank: 6 max len: 427 min len: 356 avg len: 399.85 num_loss_counted_tokens: 6937
 total length: 7995 dropped: 0 num samples 13 - rank: 1 max len: 664 min len: 284 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7916 dropped: 0 num samples 13 - rank: 2 max len: 629 min len: 578 avg len: 608.9230769230769 num_loss_counted_tokens: 7227
Epoch: 3, Step: 124, Rank: 5, loss = 0.04736328125Epoch: 3, Step: 124, Rank: 1, loss = 0.037109375Epoch: 3, Step: 124, Rank: 6, loss = 0.054931640625Epoch: 3, Step: 124, Rank: 3, loss = 0.038818359375Epoch: 3, Step: 124, Rank: 2, loss = 0.0458984375
Epoch: 3, Step: 124, Rank: 4, loss = 0.03955078125

Epoch: 3, Step: 124, Rank: 7, loss = 0.064453125



Epoch: 3, Step: 124, Rank: 0, loss = 0.0380859375


Epoch 3:  10%|█         | 4/40 [00:06<00:59,  1.65s/it]{
    "epoch": 3,
    "step": 124,
    "rank": 0,
    "overall_throughput": 94.5732736844108,
    "lr": 1.3439489186339283e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56771,
    "batch_size": 128,
    "total_loss": 0.04579803068468056,
    "samples_seen": 15695,
    "gradnorm": 0.26171875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:01.107435"
}
 total length: 7802 dropped: 0 num samples 14 - rank: 3 max len: 578 min len: 534 avg len: 557.2857142857143 num_loss_counted_tokens: 7060
 total length: 7997 dropped: 0 num samples 9 - rank: 0 max len: 1110 min len: 454 avg len: 888.5555555555555 num_loss_counted_tokens: 7520
 total length: 7995 dropped: 0 num samples 16 - rank: 4 max len: 530 min len: 372 avg len: 499.6875 num_loss_counted_tokens: 7147
 total length: 7981 dropped: 0 num samples 24 - rank: 7 max len: 377 min len: 263 avg len: 332.5416666666667 num_loss_counted_tokens: 6709
 total length: 7814 dropped: 0 num samples 17 - rank: 5 max len: 481 min len: 442 avg len: 459.6470588235294 num_loss_counted_tokens: 6913
 total length: 7797 dropped: 0 num samples 19 - rank: 6 max len: 442 min len: 381 avg len: 410.36842105263156 num_loss_counted_tokens: 6790
 total length: 7846 dropped: 0 num samples 13 - rank: 2 max len: 624 min len: 578 avg len: 603.5384615384615 num_loss_counted_tokens: 7157
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 731 min len: 584 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
Epoch: 3, Step: 125, Rank: 0, loss = 0.0296630859375Epoch: 3, Step: 125, Rank: 6, loss = 0.053955078125Epoch: 3, Step: 125, Rank: 5, loss = 0.038818359375

Epoch: 3, Step: 125, Rank: 2, loss = 0.041259765625Epoch: 3, Step: 125, Rank: 7, loss = 0.053955078125
Epoch: 3, Step: 125, Rank: 1, loss = 0.03466796875
Epoch: 3, Step: 125, Rank: 4, loss = 0.048583984375


Epoch: 3, Step: 125, Rank: 3, loss = 0.051025390625


Epoch 3:  12%|█▎        | 5/40 [00:08<00:57,  1.65s/it]{
    "epoch": 3,
    "step": 125,
    "rank": 0,
    "overall_throughput": 94.8656360167562,
    "lr": 1.3323547994796597e-05,
    "cuda_mem_allocated": 6.566522598266602,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56375,
    "batch_size": 125,
    "total_loss": 0.04399113082039911,
    "samples_seen": 15820,
    "gradnorm": 0.275390625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:02.752036"
}
 total length: 7334 dropped: 0 num samples 23 - rank: 7 max len: 377 min len: 243 avg len: 318.8695652173913 num_loss_counted_tokens: 6115
 total length: 7993 dropped: 0 num samples 11 - rank: 1 max len: 830 min len: 611 avg len: 726.6363636363636 num_loss_counted_tokens: 7410
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 575 min len: 453 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7953 dropped: 0 num samples 8 - rank: 0 max len: 1118 min len: 853 avg len: 994.125 num_loss_counted_tokens: 7529
 total length: 7977 dropped: 0 num samples 20 - rank: 6 max len: 427 min len: 308 avg len: 398.85 num_loss_counted_tokens: 6917
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 656 min len: 522 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7902 dropped: 0 num samples 16 - rank: 4 max len: 507 min len: 476 avg len: 493.875 num_loss_counted_tokens: 7054
 total length: 7993 dropped: 0 num samples 18 - rank: 5 max len: 473 min len: 356 avg len: 444.05555555555554 num_loss_counted_tokens: 7039
Epoch: 3, Step: 126, Rank: 5, loss = 0.047607421875Epoch: 3, Step: 126, Rank: 7, loss = 0.060791015625Epoch: 3, Step: 126, Rank: 6, loss = 0.07275390625Epoch: 3, Step: 126, Rank: 1, loss = 0.044921875



Epoch: 3, Step: 126, Rank: 2, loss = 0.0400390625Epoch: 3, Step: 126, Rank: 3, loss = 0.047119140625Epoch: 3, Step: 126, Rank: 4, loss = 0.039306640625


Epoch: 3, Step: 126, Rank: 0, loss = 0.037841796875


Epoch 3:  15%|█▌        | 6/40 [00:09<00:55,  1.64s/it]{
    "epoch": 3,
    "step": 126,
    "rank": 0,
    "overall_throughput": 94.84963019183665,
    "lr": 1.3207102355902553e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56660,
    "batch_size": 128,
    "total_loss": 0.04881750794211084,
    "samples_seen": 15948,
    "gradnorm": 0.275390625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:04.389281"
}
 total length: 7777 dropped: 0 num samples 18 - rank: 5 max len: 448 min len: 415 avg len: 432.05555555555554 num_loss_counted_tokens: 6823
 total length: 7995 dropped: 0 num samples 13 - rank: 1 max len: 653 min len: 348 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7997 dropped: 0 num samples 11 - rank: 0 max len: 1309 min len: 458 avg len: 727.0 num_loss_counted_tokens: 7414
 total length: 7996 dropped: 0 num samples 17 - rank: 4 max len: 493 min len: 442 avg len: 470.3529411764706 num_loss_counted_tokens: 7095
 total length: 7654 dropped: 0 num samples 24 - rank: 7 max len: 362 min len: 271 avg len: 318.9166666666667 num_loss_counted_tokens: 6382
 total length: 7980 dropped: 0 num samples 21 - rank: 6 max len: 411 min len: 253 avg len: 380.0 num_loss_counted_tokens: 6867
 total length: 7747 dropped: 0 num samples 15 - rank: 3 max len: 540 min len: 495 avg len: 516.4666666666667 num_loss_counted_tokens: 6952
 total length: 7995 dropped: 0 num samples 14 - rank: 2 max len: 596 min len: 530 avg len: 571.0714285714286 num_loss_counted_tokens: 7253
Epoch: 3, Step: 127, Rank: 6, loss = 0.05419921875Epoch: 3, Step: 127, Rank: 1, loss = 0.042236328125Epoch: 3, Step: 127, Rank: 5, loss = 0.046142578125Epoch: 3, Step: 127, Rank: 7, loss = 0.0576171875


Epoch: 3, Step: 127, Rank: 3, loss = 0.0439453125
Epoch: 3, Step: 127, Rank: 2, loss = 0.044677734375Epoch: 3, Step: 127, Rank: 4, loss = 0.044189453125


Epoch: 3, Step: 127, Rank: 0, loss = 0.05615234375


Epoch 3:  18%|█▊        | 7/40 [00:11<00:54,  1.65s/it]{
    "epoch": 3,
    "step": 127,
    "rank": 0,
    "overall_throughput": 94.26277686252814,
    "lr": 1.3090169943749475e-05,
    "cuda_mem_allocated": 6.588063716888428,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56473,
    "batch_size": 123,
    "total_loss": 0.0486250066403414,
    "samples_seen": 16071,
    "gradnorm": 0.2734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:06.039235"
}
 total length: 7964 dropped: 0 num samples 18 - rank: 5 max len: 458 min len: 424 avg len: 442.44444444444446 num_loss_counted_tokens: 7010
 total length: 7997 dropped: 0 num samples 9 - rank: 0 max len: 1391 min len: 326 avg len: 888.5555555555555 num_loss_counted_tokens: 7520
 total length: 7980 dropped: 0 num samples 13 - rank: 2 max len: 661 min len: 253 avg len: 613.8461538461538 num_loss_counted_tokens: 7291
 total length: 7989 dropped: 0 num samples 14 - rank: 3 max len: 599 min len: 536 avg len: 570.6428571428571 num_loss_counted_tokens: 7247
 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 529 min len: 452 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7630 dropped: 0 num samples 23 - rank: 7 max len: 390 min len: 251 avg len: 331.7391304347826 num_loss_counted_tokens: 6411
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 750 min len: 411 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7984 dropped: 0 num samples 20 - rank: 6 max len: 424 min len: 307 avg len: 399.2 num_loss_counted_tokens: 6924
Epoch: 3, Step: 128, Rank: 0, loss = 0.01513671875Epoch: 3, Step: 128, Rank: 3, loss = 0.040283203125Epoch: 3, Step: 128, Rank: 2, loss = 0.044677734375
Epoch: 3, Step: 128, Rank: 6, loss = 0.0693359375
Epoch: 3, Step: 128, Rank: 5, loss = 0.056640625Epoch: 3, Step: 128, Rank: 4, loss = 0.04833984375
Epoch: 3, Step: 128, Rank: 7, loss = 0.052978515625



Epoch: 3, Step: 128, Rank: 1, loss = 0.037353515625


Epoch 3:  20%|██        | 8/40 [00:13<00:52,  1.65s/it]{
    "epoch": 3,
    "step": 128,
    "rank": 0,
    "overall_throughput": 94.25440306377642,
    "lr": 1.2972768506312028e-05,
    "cuda_mem_allocated": 6.578244686126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56714,
    "batch_size": 123,
    "total_loss": 0.04555312621222273,
    "samples_seen": 16194,
    "gradnorm": 0.27734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:07.691727"
}
 total length: 7835 dropped: 0 num samples 25 - rank: 7 max len: 365 min len: 255 avg len: 313.4 num_loss_counted_tokens: 6510
 total length: 7989 dropped: 0 num samples 15 - rank: 3 max len: 565 min len: 352 avg len: 532.6 num_loss_counted_tokens: 7194
 total length: 7991 dropped: 0 num samples 10 - rank: 0 max len: 1070 min len: 525 avg len: 799.1 num_loss_counted_tokens: 7461
 total length: 7873 dropped: 0 num samples 13 - rank: 2 max len: 635 min len: 566 avg len: 605.6153846153846 num_loss_counted_tokens: 7184
 total length: 7774 dropped: 0 num samples 17 - rank: 5 max len: 477 min len: 439 avg len: 457.29411764705884 num_loss_counted_tokens: 6873
 total length: 7867 dropped: 0 num samples 20 - rank: 6 max len: 430 min len: 369 avg len: 393.35 num_loss_counted_tokens: 6807
 total length: 7904 dropped: 0 num samples 16 - rank: 4 max len: 512 min len: 479 avg len: 494.0 num_loss_counted_tokens: 7056
 total length: 7905 dropped: 0 num samples 12 - rank: 1 max len: 685 min len: 639 avg len: 658.75 num_loss_counted_tokens: 7269
Epoch: 3, Step: 129, Rank: 5, loss = 0.055419921875Epoch: 3, Step: 129, Rank: 6, loss = 0.064453125Epoch: 3, Step: 129, Rank: 1, loss = 0.04150390625Epoch: 3, Step: 129, Rank: 7, loss = 0.061279296875Epoch: 3, Step: 129, Rank: 2, loss = 0.033935546875Epoch: 3, Step: 129, Rank: 3, loss = 0.036865234375





Epoch: 3, Step: 129, Rank: 4, loss = 0.038818359375
Epoch: 3, Step: 129, Rank: 0, loss = 0.02001953125


Epoch 3:  22%|██▎       | 9/40 [00:14<00:51,  1.65s/it]{
    "epoch": 3,
    "step": 129,
    "rank": 0,
    "overall_throughput": 93.60062568844384,
    "lr": 1.2854915862753424e-05,
    "cuda_mem_allocated": 6.588062763214111,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56815,
    "batch_size": 123,
    "total_loss": 0.044055267094957315,
    "samples_seen": 16317,
    "gradnorm": 0.271484375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:09.348770"
}
 total length: 7842 dropped: 0 num samples 13 - rank: 2 max len: 638 min len: 557 avg len: 603.2307692307693 num_loss_counted_tokens: 7153
 total length: 7959 dropped: 0 num samples 12 - rank: 1 max len: 688 min len: 639 avg len: 663.25 num_loss_counted_tokens: 7323
 total length: 7981 dropped: 0 num samples 10 - rank: 0 max len: 1060 min len: 611 avg len: 798.1 num_loss_counted_tokens: 7451
 total length: 7483 dropped: 0 num samples 23 - rank: 7 max len: 368 min len: 267 avg len: 325.3478260869565 num_loss_counted_tokens: 6264
 total length: 7949 dropped: 0 num samples 19 - rank: 5 max len: 453 min len: 174 avg len: 418.36842105263156 num_loss_counted_tokens: 6942
 total length: 7895 dropped: 0 num samples 15 - rank: 3 max len: 553 min len: 499 avg len: 526.3333333333334 num_loss_counted_tokens: 7100
 total length: 7970 dropped: 0 num samples 17 - rank: 4 max len: 496 min len: 454 avg len: 468.8235294117647 num_loss_counted_tokens: 7069
 total length: 7940 dropped: 0 num samples 20 - rank: 6 max len: 414 min len: 372 avg len: 397.0 num_loss_counted_tokens: 6880
Epoch: 3, Step: 130, Rank: 5, loss = 0.04736328125Epoch: 3, Step: 130, Rank: 2, loss = 0.053955078125Epoch: 3, Step: 130, Rank: 6, loss = 0.0595703125Epoch: 3, Step: 130, Rank: 1, loss = 0.03564453125Epoch: 3, Step: 130, Rank: 4, loss = 0.045654296875
Epoch: 3, Step: 130, Rank: 3, loss = 0.044189453125
Epoch: 3, Step: 130, Rank: 7, loss = 0.0537109375




Epoch: 3, Step: 130, Rank: 0, loss = 0.034423828125


Epoch 3:  25%|██▌       | 10/40 [00:16<00:49,  1.65s/it]{
    "epoch": 3,
    "step": 130,
    "rank": 0,
    "overall_throughput": 94.25979787034002,
    "lr": 1.2736629900720832e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 55631,
    "batch_size": 131,
    "total_loss": 0.04686236091387895,
    "samples_seen": 16448,
    "gradnorm": 0.30078125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:10.998279"
}
 total length: 7682 dropped: 0 num samples 23 - rank: 7 max len: 371 min len: 253 avg len: 334.0 num_loss_counted_tokens: 6463
 total length: 7964 dropped: 0 num samples 14 - rank: 3 max len: 611 min len: 531 avg len: 568.8571428571429 num_loss_counted_tokens: 7222
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 639 min len: 495 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7789 dropped: 0 num samples 17 - rank: 5 max len: 472 min len: 441 avg len: 458.1764705882353 num_loss_counted_tokens: 6888
 total length: 7760 dropped: 0 num samples 19 - rank: 6 max len: 439 min len: 372 avg len: 408.42105263157896 num_loss_counted_tokens: 6753
 total length: 7964 dropped: 0 num samples 16 - rank: 4 max len: 525 min len: 473 avg len: 497.75 num_loss_counted_tokens: 7116
 total length: 7993 dropped: 0 num samples 12 - rank: 1 max len: 707 min len: 640 avg len: 666.0833333333334 num_loss_counted_tokens: 7357
 total length: 7854 dropped: 0 num samples 9 - rank: 0 max len: 1104 min len: 714 avg len: 872.6666666666666 num_loss_counted_tokens: 7377
Epoch: 3, Step: 131, Rank: 6, loss = 0.052734375Epoch: 3, Step: 131, Rank: 3, loss = 0.05322265625Epoch: 3, Step: 131, Rank: 2, loss = 0.039794921875Epoch: 3, Step: 131, Rank: 5, loss = 0.0498046875Epoch: 3, Step: 131, Rank: 1, loss = 0.0341796875Epoch: 3, Step: 131, Rank: 7, loss = 0.06103515625


Epoch: 3, Step: 131, Rank: 4, loss = 0.0380859375



Epoch: 3, Step: 131, Rank: 0, loss = 0.0185546875


Epoch 3:  28%|██▊       | 11/40 [00:18<00:47,  1.65s/it]{
    "epoch": 3,
    "step": 131,
    "rank": 0,
    "overall_throughput": 93.6556032862355,
    "lr": 1.2617928573630405e-05,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 57239,
    "batch_size": 114,
    "total_loss": 0.043379513967749264,
    "samples_seen": 16562,
    "gradnorm": 0.275390625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:12.653738"
}
 total length: 7987 dropped: 0 num samples 19 - rank: 5 max len: 461 min len: 172 avg len: 420.36842105263156 num_loss_counted_tokens: 6980
 total length: 7831 dropped: 0 num samples 20 - rank: 6 max len: 407 min len: 372 avg len: 391.55 num_loss_counted_tokens: 6771
 total length: 7995 dropped: 0 num samples 9 - rank: 0 max len: 1074 min len: 552 avg len: 888.3333333333334 num_loss_counted_tokens: 7518
 total length: 7968 dropped: 0 num samples 12 - rank: 1 max len: 724 min len: 631 avg len: 664.0 num_loss_counted_tokens: 7332
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 570 min len: 392 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7912 dropped: 0 num samples 24 - rank: 7 max len: 371 min len: 258 avg len: 329.6666666666667 num_loss_counted_tokens: 6640
 total length: 7845 dropped: 0 num samples 13 - rank: 2 max len: 630 min len: 579 avg len: 603.4615384615385 num_loss_counted_tokens: 7156
 total length: 7850 dropped: 0 num samples 16 - rank: 4 max len: 516 min len: 462 avg len: 490.625 num_loss_counted_tokens: 7002
Epoch: 3, Step: 132, Rank: 5, loss = 0.05419921875Epoch: 3, Step: 132, Rank: 6, loss = 0.06884765625Epoch: 3, Step: 132, Rank: 2, loss = 0.042236328125Epoch: 3, Step: 132, Rank: 1, loss = 0.038330078125


Epoch: 3, Step: 132, Rank: 3, loss = 0.037841796875
Epoch: 3, Step: 132, Rank: 4, loss = 0.03955078125Epoch: 3, Step: 132, Rank: 7, loss = 0.0595703125


Epoch: 3, Step: 132, Rank: 0, loss = 0.0322265625


Epoch 3:  30%|███       | 12/40 [00:19<00:46,  1.65s/it]{
    "epoch": 3,
    "step": 132,
    "rank": 0,
    "overall_throughput": 94.6346044388837,
    "lr": 1.2498829897942308e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56479,
    "batch_size": 125,
    "total_loss": 0.0466013916676995,
    "samples_seen": 16687,
    "gradnorm": 0.283203125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:14.300410"
}
 total length: 7990 dropped: 0 num samples 9 - rank: 0 max len: 1076 min len: 257 avg len: 887.7777777777778 num_loss_counted_tokens: 7513
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 729 min len: 617 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7917 dropped: 0 num samples 14 - rank: 3 max len: 587 min len: 531 avg len: 565.5 num_loss_counted_tokens: 7175
 total length: 7959 dropped: 0 num samples 16 - rank: 4 max len: 530 min len: 473 avg len: 497.4375 num_loss_counted_tokens: 7111
 total length: 7745 dropped: 0 num samples 19 - rank: 6 max len: 430 min len: 382 avg len: 407.63157894736844 num_loss_counted_tokens: 6738
 total length: 7834 dropped: 0 num samples 24 - rank: 7 max len: 378 min len: 266 avg len: 326.4166666666667 num_loss_counted_tokens: 6562
 total length: 7980 dropped: 0 num samples 18 - rank: 5 max len: 472 min len: 277 avg len: 443.3333333333333 num_loss_counted_tokens: 7026
 total length: 7935 dropped: 0 num samples 13 - rank: 2 max len: 631 min len: 590 avg len: 610.3846153846154 num_loss_counted_tokens: 7246
Epoch: 3, Step: 133, Rank: 1, loss = 0.0400390625Epoch: 3, Step: 133, Rank: 6, loss = 0.04931640625Epoch: 3, Step: 133, Rank: 5, loss = 0.05029296875

Epoch: 3, Step: 133, Rank: 2, loss = 0.039794921875Epoch: 3, Step: 133, Rank: 3, loss = 0.037841796875Epoch: 3, Step: 133, Rank: 7, loss = 0.0625
Epoch: 3, Step: 133, Rank: 4, loss = 0.041748046875



Epoch: 3, Step: 133, Rank: 0, loss = 0.018310546875


Epoch 3:  32%|███▎      | 13/40 [00:21<00:44,  1.65s/it]{
    "epoch": 3,
    "step": 133,
    "rank": 0,
    "overall_throughput": 94.65959963643384,
    "lr": 1.2379351950426188e-05,
    "cuda_mem_allocated": 6.5684814453125,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56332,
    "batch_size": 129,
    "total_loss": 0.0424802953916069,
    "samples_seen": 16816,
    "gradnorm": 0.271484375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:15.941826"
}
 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 539 min len: 328 avg len: 499.75 num_loss_counted_tokens: 7148
 total length: 7994 dropped: 0 num samples 12 - rank: 1 max len: 738 min len: 448 avg len: 666.1666666666666 num_loss_counted_tokens: 7358
 total length: 7754 dropped: 0 num samples 8 - rank: 0 max len: 1094 min len: 828 avg len: 969.25 num_loss_counted_tokens: 7330
 total length: 7819 dropped: 0 num samples 20 - rank: 6 max len: 415 min len: 364 avg len: 390.95 num_loss_counted_tokens: 6759
 total length: 7992 dropped: 0 num samples 13 - rank: 2 max len: 644 min len: 401 avg len: 614.7692307692307 num_loss_counted_tokens: 7303
 total length: 7996 dropped: 0 num samples 18 - rank: 5 max len: 480 min len: 382 avg len: 444.22222222222223 num_loss_counted_tokens: 7042
 total length: 7992 dropped: 0 num samples 25 - rank: 7 max len: 364 min len: 245 avg len: 319.68 num_loss_counted_tokens: 6667
 total length: 7997 dropped: 0 num samples 14 - rank: 3 max len: 615 min len: 337 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
Epoch: 3, Step: 134, Rank: 5, loss = 0.061767578125Epoch: 3, Step: 134, Rank: 6, loss = 0.052734375Epoch: 3, Step: 134, Rank: 1, loss = 0.028564453125Epoch: 3, Step: 134, Rank: 2, loss = 0.032958984375Epoch: 3, Step: 134, Rank: 3, loss = 0.0537109375Epoch: 3, Step: 134, Rank: 4, loss = 0.046875

Epoch: 3, Step: 134, Rank: 7, loss = 0.0556640625




Epoch: 3, Step: 134, Rank: 0, loss = 0.032958984375


Epoch 3:  35%|███▌      | 14/40 [00:23<00:42,  1.65s/it]{
    "epoch": 3,
    "step": 134,
    "rank": 0,
    "overall_throughput": 94.34218670291777,
    "lr": 1.2259512865417478e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56382,
    "batch_size": 127,
    "total_loss": 0.04567060409350502,
    "samples_seen": 16943,
    "gradnorm": 0.296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:17.591707"
}
 total length: 7843 dropped: 0 num samples 12 - rank: 1 max len: 691 min len: 631 avg len: 653.5833333333334 num_loss_counted_tokens: 7207
 total length: 7995 dropped: 0 num samples 10 - rank: 0 max len: 1068 min len: 556 avg len: 799.5 num_loss_counted_tokens: 7465
 total length: 7796 dropped: 0 num samples 17 - rank: 5 max len: 480 min len: 435 avg len: 458.5882352941176 num_loss_counted_tokens: 6895
 total length: 7807 dropped: 0 num samples 24 - rank: 7 max len: 366 min len: 243 avg len: 325.2916666666667 num_loss_counted_tokens: 6535
 total length: 7990 dropped: 0 num samples 20 - rank: 6 max len: 433 min len: 368 avg len: 399.5 num_loss_counted_tokens: 6930
 total length: 7975 dropped: 0 num samples 16 - rank: 4 max len: 519 min len: 483 avg len: 498.4375 num_loss_counted_tokens: 7127
 total length: 7814 dropped: 0 num samples 13 - rank: 2 max len: 629 min len: 573 avg len: 601.0769230769231 num_loss_counted_tokens: 7125
 total length: 7996 dropped: 0 num samples 15 - rank: 3 max len: 570 min len: 331 avg len: 533.0666666666667 num_loss_counted_tokens: 7201
Epoch: 3, Step: 135, Rank: 1, loss = 0.035888671875Epoch: 3, Step: 135, Rank: 5, loss = 0.04833984375Epoch: 3, Step: 135, Rank: 6, loss = 0.06494140625Epoch: 3, Step: 135, Rank: 7, loss = 0.0576171875



Epoch: 3, Step: 135, Rank: 3, loss = 0.041748046875Epoch: 3, Step: 135, Rank: 2, loss = 0.041748046875Epoch: 3, Step: 135, Rank: 4, loss = 0.0311279296875


Epoch: 3, Step: 135, Rank: 0, loss = 0.022705078125


Epoch 3:  38%|███▊      | 15/40 [00:24<00:41,  1.65s/it]{
    "epoch": 3,
    "step": 135,
    "rank": 0,
    "overall_throughput": 93.6364264561916,
    "lr": 1.2139330832064975e-05,
    "cuda_mem_allocated": 6.588062763214111,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56707,
    "batch_size": 123,
    "total_loss": 0.04301056306981502,
    "samples_seen": 17066,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:19.247915"
}
 total length: 7997 dropped: 0 num samples 9 - rank: 0 max len: 1104 min len: 441 avg len: 888.5555555555555 num_loss_counted_tokens: 7520
 total length: 7842 dropped: 0 num samples 25 - rank: 7 max len: 367 min len: 276 avg len: 313.68 num_loss_counted_tokens: 6517
 total length: 7887 dropped: 0 num samples 12 - rank: 1 max len: 719 min len: 627 avg len: 657.25 num_loss_counted_tokens: 7251
 total length: 7846 dropped: 0 num samples 13 - rank: 2 max len: 626 min len: 582 avg len: 603.5384615384615 num_loss_counted_tokens: 7157
 total length: 7992 dropped: 0 num samples 15 - rank: 3 max len: 575 min len: 345 avg len: 532.8 num_loss_counted_tokens: 7197
 total length: 7798 dropped: 0 num samples 17 - rank: 5 max len: 494 min len: 431 avg len: 458.70588235294116 num_loss_counted_tokens: 6897
 total length: 7991 dropped: 0 num samples 16 - rank: 4 max len: 521 min len: 338 avg len: 499.4375 num_loss_counted_tokens: 7143
 total length: 7994 dropped: 0 num samples 20 - rank: 6 max len: 430 min len: 368 avg len: 399.7 num_loss_counted_tokens: 6934
Epoch: 3, Step: 136, Rank: 0, loss = 0.02734375Epoch: 3, Step: 136, Rank: 1, loss = 0.046142578125Epoch: 3, Step: 136, Rank: 5, loss = 0.04150390625
Epoch: 3, Step: 136, Rank: 4, loss = 0.05078125

Epoch: 3, Step: 136, Rank: 2, loss = 0.040771484375Epoch: 3, Step: 136, Rank: 7, loss = 0.062255859375
Epoch: 3, Step: 136, Rank: 6, loss = 0.0556640625


Epoch: 3, Step: 136, Rank: 3, loss = 0.048095703125


Epoch 3:  40%|████      | 16/40 [00:26<00:39,  1.65s/it]{
    "epoch": 3,
    "step": 136,
    "rank": 0,
    "overall_throughput": 94.62708176758653,
    "lr": 1.2018824091570103e-05,
    "cuda_mem_allocated": 6.572385311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 55773,
    "batch_size": 130,
    "total_loss": 0.04659960912986571,
    "samples_seen": 17196,
    "gradnorm": 0.302734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:20.894259"
}
 total length: 7678 dropped: 0 num samples 24 - rank: 7 max len: 366 min len: 191 avg len: 319.9166666666667 num_loss_counted_tokens: 6406
 total length: 7944 dropped: 0 num samples 12 - rank: 1 max len: 711 min len: 643 avg len: 662.0 num_loss_counted_tokens: 7308
 total length: 7994 dropped: 0 num samples 10 - rank: 0 max len: 1066 min len: 382 avg len: 799.4 num_loss_counted_tokens: 7464
 total length: 7948 dropped: 0 num samples 13 - rank: 2 max len: 639 min len: 590 avg len: 611.3846153846154 num_loss_counted_tokens: 7259
 total length: 7921 dropped: 0 num samples 14 - rank: 3 max len: 585 min len: 543 avg len: 565.7857142857143 num_loss_counted_tokens: 7179
 total length: 7936 dropped: 0 num samples 20 - rank: 6 max len: 422 min len: 371 avg len: 396.8 num_loss_counted_tokens: 6876
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 473 min len: 377 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7993 dropped: 0 num samples 16 - rank: 4 max len: 539 min len: 430 avg len: 499.5625 num_loss_counted_tokens: 7145
Epoch: 3, Step: 137, Rank: 5, loss = 0.053955078125Epoch: 3, Step: 137, Rank: 1, loss = 0.04833984375Epoch: 3, Step: 137, Rank: 6, loss = 0.056640625Epoch: 3, Step: 137, Rank: 2, loss = 0.03662109375


Epoch: 3, Step: 137, Rank: 7, loss = 0.059326171875Epoch: 3, Step: 137, Rank: 3, loss = 0.037109375

Epoch: 3, Step: 137, Rank: 4, loss = 0.044189453125

Epoch: 3, Step: 137, Rank: 0, loss = 0.0301513671875


Epoch 3:  42%|████▎     | 17/40 [00:28<00:37,  1.65s/it]{
    "epoch": 3,
    "step": 137,
    "rank": 0,
    "overall_throughput": 94.16199300999182,
    "lr": 1.1898010934418261e-05,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56824,
    "batch_size": 124,
    "total_loss": 0.04582570744755737,
    "samples_seen": 17320,
    "gradnorm": 0.30859375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:22.540443"
}
 total length: 7377 dropped: 0 num samples 24 - rank: 7 max len: 353 min len: 260 avg len: 307.375 num_loss_counted_tokens: 6105
 total length: 7997 dropped: 0 num samples 9 - rank: 0 max len: 1087 min len: 467 avg len: 888.5555555555555 num_loss_counted_tokens: 7520
 total length: 7871 dropped: 0 num samples 12 - rank: 1 max len: 677 min len: 634 avg len: 655.9166666666666 num_loss_counted_tokens: 7235
 total length: 7960 dropped: 0 num samples 13 - rank: 2 max len: 632 min len: 584 avg len: 612.3076923076923 num_loss_counted_tokens: 7271
 total length: 7755 dropped: 0 num samples 14 - rank: 3 max len: 582 min len: 530 avg len: 553.9285714285714 num_loss_counted_tokens: 7013
 total length: 7991 dropped: 0 num samples 18 - rank: 5 max len: 473 min len: 412 avg len: 443.94444444444446 num_loss_counted_tokens: 7037
 total length: 7993 dropped: 0 num samples 16 - rank: 4 max len: 529 min len: 474 avg len: 499.5625 num_loss_counted_tokens: 7145
 total length: 7995 dropped: 0 num samples 21 - rank: 6 max len: 406 min len: 334 avg len: 380.7142857142857 num_loss_counted_tokens: 6882
Epoch: 3, Step: 138, Rank: 5, loss = 0.054931640625Epoch: 3, Step: 138, Rank: 0, loss = 0.03271484375Epoch: 3, Step: 138, Rank: 6, loss = 0.0546875Epoch: 3, Step: 138, Rank: 2, loss = 0.034423828125Epoch: 3, Step: 138, Rank: 7, loss = 0.06494140625Epoch: 3, Step: 138, Rank: 3, loss = 0.044921875





Epoch: 3, Step: 138, Rank: 4, loss = 0.051025390625
Epoch: 3, Step: 138, Rank: 1, loss = 0.032470703125


Epoch 3:  45%|████▌     | 18/40 [00:29<00:36,  1.65s/it]{
    "epoch": 3,
    "step": 138,
    "rank": 0,
    "overall_throughput": 94.36530262249215,
    "lr": 1.177690969760269e-05,
    "cuda_mem_allocated": 6.576291561126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 55891,
    "batch_size": 134,
    "total_loss": 0.04626863001198762,
    "samples_seen": 17454,
    "gradnorm": 0.30859375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:24.190879"
}
 total length: 7984 dropped: 0 num samples 13 - rank: 2 max len: 631 min len: 557 avg len: 614.1538461538462 num_loss_counted_tokens: 7295
 total length: 7794 dropped: 0 num samples 24 - rank: 7 max len: 354 min len: 259 avg len: 324.75 num_loss_counted_tokens: 6522
 total length: 7988 dropped: 0 num samples 17 - rank: 4 max len: 521 min len: 277 avg len: 469.88235294117646 num_loss_counted_tokens: 7087
 total length: 7882 dropped: 0 num samples 21 - rank: 6 max len: 392 min len: 354 avg len: 375.3333333333333 num_loss_counted_tokens: 6769 total length: 7856 dropped: 0 num samples 14 - rank: 3 max len: 596 min len: 522 avg len: 561.1428571428571 num_loss_counted_tokens: 7114

 total length: 7806 dropped: 0 num samples 12 - rank: 1 max len: 688 min len: 631 avg len: 650.5 num_loss_counted_tokens: 7170
 total length: 7965 dropped: 0 num samples 10 - rank: 0 max len: 1062 min len: 688 avg len: 796.5 num_loss_counted_tokens: 7435
 total length: 7928 dropped: 0 num samples 19 - rank: 5 max len: 439 min len: 393 avg len: 417.2631578947368 num_loss_counted_tokens: 6921
Epoch: 3, Step: 139, Rank: 5, loss = 0.06396484375Epoch: 3, Step: 139, Rank: 2, loss = 0.02978515625Epoch: 3, Step: 139, Rank: 3, loss = 0.03662109375Epoch: 3, Step: 139, Rank: 7, loss = 0.057373046875Epoch: 3, Step: 139, Rank: 1, loss = 0.04248046875Epoch: 3, Step: 139, Rank: 6, loss = 0.058349609375
Epoch: 3, Step: 139, Rank: 4, loss = 0.04296875





Epoch: 3, Step: 139, Rank: 0, loss = 0.047607421875


Epoch 3:  48%|████▊     | 19/40 [00:31<00:34,  1.65s/it]{
    "epoch": 3,
    "step": 139,
    "rank": 0,
    "overall_throughput": 94.2590035042231,
    "lr": 1.16555387618413e-05,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56722,
    "batch_size": 126,
    "total_loss": 0.0474066499770812,
    "samples_seen": 17580,
    "gradnorm": 0.3125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:25.838811"
}
 total length: 7977 dropped: 0 num samples 13 - rank: 2 max len: 628 min len: 559 avg len: 613.6153846153846 num_loss_counted_tokens: 7288
 total length: 7816 dropped: 0 num samples 20 - rank: 6 max len: 414 min len: 370 avg len: 390.8 num_loss_counted_tokens: 6756
 total length: 7996 dropped: 0 num samples 11 - rank: 0 max len: 968 min len: 583 avg len: 726.9090909090909 num_loss_counted_tokens: 7413
 total length: 7860 dropped: 0 num samples 18 - rank: 5 max len: 456 min len: 418 avg len: 436.6666666666667 num_loss_counted_tokens: 6906
 total length: 7909 dropped: 0 num samples 16 - rank: 4 max len: 516 min len: 460 avg len: 494.3125 num_loss_counted_tokens: 7061
 total length: 7985 dropped: 0 num samples 24 - rank: 7 max len: 364 min len: 276 avg len: 332.7083333333333 num_loss_counted_tokens: 6713
 total length: 7986 dropped: 0 num samples 13 - rank: 1 max len: 653 min len: 284 avg len: 614.3076923076923 num_loss_counted_tokens: 7297
 total length: 7780 dropped: 0 num samples 14 - rank: 3 max len: 588 min len: 522 avg len: 555.7142857142857 num_loss_counted_tokens: 7038
Epoch: 3, Step: 140, Rank: 6, loss = 0.047607421875Epoch: 3, Step: 140, Rank: 2, loss = 0.0439453125Epoch: 3, Step: 140, Rank: 7, loss = 0.0634765625Epoch: 3, Step: 140, Rank: 5, loss = 0.05078125
Epoch: 3, Step: 140, Rank: 1, loss = 0.04052734375Epoch: 3, Step: 140, Rank: 4, loss = 0.046630859375Epoch: 3, Step: 140, Rank: 3, loss = 0.038818359375





Epoch: 3, Step: 140, Rank: 0, loss = 0.029541015625


Epoch 3:  50%|█████     | 20/40 [00:32<00:32,  1.65s/it]{
    "epoch": 3,
    "step": 140,
    "rank": 0,
    "overall_throughput": 94.24174589616311,
    "lr": 1.1533916548786856e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56493,
    "batch_size": 130,
    "total_loss": 0.0451560370311366,
    "samples_seen": 17710,
    "gradnorm": 0.306640625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:27.490861"
}
 total length: 7790 dropped: 0 num samples 8 - rank: 0 max len: 1072 min len: 850 avg len: 973.75 num_loss_counted_tokens: 7366
 total length: 7680 dropped: 0 num samples 23 - rank: 7 max len: 367 min len: 282 avg len: 333.9130434782609 num_loss_counted_tokens: 6461
 total length: 7990 dropped: 0 num samples 15 - rank: 3 max len: 568 min len: 294 avg len: 532.6666666666666 num_loss_counted_tokens: 7195
 total length: 7990 dropped: 0 num samples 16 - rank: 4 max len: 530 min len: 346 avg len: 499.375 num_loss_counted_tokens: 7142
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 658 min len: 468 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7891 dropped: 0 num samples 20 - rank: 6 max len: 416 min len: 370 avg len: 394.55 num_loss_counted_tokens: 6831
 total length: 7996 dropped: 0 num samples 18 - rank: 5 max len: 478 min len: 354 avg len: 444.22222222222223 num_loss_counted_tokens: 7042
 total length: 7877 dropped: 0 num samples 11 - rank: 1 max len: 841 min len: 667 avg len: 716.0909090909091 num_loss_counted_tokens: 7294
Epoch: 3, Step: 141, Rank: 5, loss = 0.0634765625Epoch: 3, Step: 141, Rank: 7, loss = 0.064453125Epoch: 3, Step: 141, Rank: 6, loss = 0.057861328125Epoch: 3, Step: 141, Rank: 1, loss = 0.046630859375Epoch: 3, Step: 141, Rank: 2, loss = 0.0380859375


Epoch: 3, Step: 141, Rank: 4, loss = 0.039794921875
Epoch: 3, Step: 141, Rank: 3, loss = 0.043212890625


Epoch: 3, Step: 141, Rank: 0, loss = 0.0223388671875


Epoch 3:  52%|█████▎    | 21/40 [00:34<00:31,  1.65s/it]{
    "epoch": 3,
    "step": 141,
    "rank": 0,
    "overall_throughput": 93.72168789179018,
    "lr": 1.1412061518230916e-05,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56656,
    "batch_size": 124,
    "total_loss": 0.04702061564529794,
    "samples_seen": 17834,
    "gradnorm": 0.314453125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:29.146233"
}
 total length: 7974 dropped: 0 num samples 20 - rank: 6 max len: 428 min len: 362 avg len: 398.7 num_loss_counted_tokens: 6914
 total length: 7823 dropped: 0 num samples 17 - rank: 5 max len: 478 min len: 434 avg len: 460.1764705882353 num_loss_counted_tokens: 6922
 total length: 7976 dropped: 0 num samples 9 - rank: 0 max len: 1058 min len: 524 avg len: 886.2222222222222 num_loss_counted_tokens: 7499
 total length: 7858 dropped: 0 num samples 14 - rank: 3 max len: 602 min len: 518 avg len: 561.2857142857143 num_loss_counted_tokens: 7116
 total length: 7852 dropped: 0 num samples 11 - rank: 1 max len: 825 min len: 664 avg len: 713.8181818181819 num_loss_counted_tokens: 7269
 total length: 7977 dropped: 0 num samples 26 - rank: 7 max len: 361 min len: 245 avg len: 306.8076923076923 num_loss_counted_tokens: 6599 total length: 7995 dropped: 0 num samples 13 - rank: 2 max len: 657 min len: 439 avg len: 615.0 num_loss_counted_tokens: 7306

 total length: 7949 dropped: 0 num samples 16 - rank: 4 max len: 515 min len: 479 avg len: 496.8125 num_loss_counted_tokens: 7101
Epoch: 3, Step: 142, Rank: 1, loss = 0.03759765625Epoch: 3, Step: 142, Rank: 2, loss = 0.041259765625Epoch: 3, Step: 142, Rank: 5, loss = 0.055419921875Epoch: 3, Step: 142, Rank: 3, loss = 0.03857421875

Epoch: 3, Step: 142, Rank: 6, loss = 0.0673828125
Epoch: 3, Step: 142, Rank: 7, loss = 0.05322265625
Epoch: 3, Step: 142, Rank: 4, loss = 0.046630859375


Epoch: 3, Step: 142, Rank: 0, loss = 0.0186767578125


Epoch 3:  55%|█████▌    | 22/40 [00:36<00:29,  1.65s/it]{
    "epoch": 3,
    "step": 142,
    "rank": 0,
    "overall_throughput": 93.76768491652233,
    "lr": 1.1289992165302036e-05,
    "cuda_mem_allocated": 6.584104061126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56573,
    "batch_size": 124,
    "total_loss": 0.044862390186131196,
    "samples_seen": 17958,
    "gradnorm": 0.318359375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:30.797671"
}
 total length: 7203 dropped: 0 num samples 24 - rank: 7 max len: 337 min len: 250 avg len: 300.125 num_loss_counted_tokens: 5931
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 778 min len: 380 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7997 dropped: 0 num samples 9 - rank: 0 max len: 1072 min len: 391 avg len: 888.5555555555555 num_loss_counted_tokens: 7520
 total length: 7996 dropped: 0 num samples 15 - rank: 3 max len: 566 min len: 248 avg len: 533.0666666666667 num_loss_counted_tokens: 7201
 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 537 min len: 400 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7900 dropped: 0 num samples 21 - rank: 6 max len: 406 min len: 339 avg len: 376.1904761904762 num_loss_counted_tokens: 6787
 total length: 7995 dropped: 0 num samples 13 - rank: 2 max len: 651 min len: 540 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7780 dropped: 0 num samples 18 - rank: 5 max len: 464 min len: 407 avg len: 432.22222222222223 num_loss_counted_tokens: 6826
Epoch: 3, Step: 143, Rank: 2, loss = 0.034912109375Epoch: 3, Step: 143, Rank: 6, loss = 0.047607421875Epoch: 3, Step: 143, Rank: 5, loss = 0.0634765625Epoch: 3, Step: 143, Rank: 1, loss = 0.03662109375Epoch: 3, Step: 143, Rank: 7, loss = 0.05908203125
Epoch: 3, Step: 143, Rank: 3, loss = 0.0439453125




Epoch: 3, Step: 143, Rank: 4, loss = 0.059814453125
Epoch: 3, Step: 143, Rank: 0, loss = 0.055908203125


Epoch 3:  57%|█████▊    | 23/40 [00:38<00:29,  1.74s/it]{
    "epoch": 3,
    "step": 143,
    "rank": 0,
    "overall_throughput": 77.49623677573938,
    "lr": 1.1167727017658562e-05,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56092,
    "batch_size": 133,
    "total_loss": 0.05014975397561149,
    "samples_seen": 18091,
    "gradnorm": 0.3203125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:32.739470"
}
 total length: 7997 dropped: 0 num samples 9 - rank: 0 max len: 1164 min len: 425 avg len: 888.5555555555555 num_loss_counted_tokens: 7520
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 659 min len: 390 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7821 dropped: 0 num samples 24 - rank: 7 max len: 364 min len: 255 avg len: 325.875 num_loss_counted_tokens: 6549
 total length: 7904 dropped: 0 num samples 15 - rank: 4 max len: 572 min len: 497 avg len: 526.9333333333333 num_loss_counted_tokens: 7109
 total length: 7950 dropped: 0 num samples 14 - rank: 3 max len: 615 min len: 202 avg len: 567.8571428571429 num_loss_counted_tokens: 7208 total length: 7883 dropped: 0 num samples 17 - rank: 5 max len: 496 min len: 437 avg len: 463.70588235294116 num_loss_counted_tokens: 6982

 total length: 7978 dropped: 0 num samples 12 - rank: 1 max len: 778 min len: 182 avg len: 664.8333333333334 num_loss_counted_tokens: 7342
 total length: 7812 dropped: 0 num samples 20 - rank: 6 max len: 426 min len: 366 avg len: 390.6 num_loss_counted_tokens: 6752
Epoch: 3, Step: 144, Rank: 6, loss = 0.053955078125Epoch: 3, Step: 144, Rank: 2, loss = 0.040283203125Epoch: 3, Step: 144, Rank: 5, loss = 0.056396484375Epoch: 3, Step: 144, Rank: 3, loss = 0.031982421875
Epoch: 3, Step: 144, Rank: 1, loss = 0.040283203125Epoch: 3, Step: 144, Rank: 7, loss = 0.057861328125

Epoch: 3, Step: 144, Rank: 4, loss = 0.037353515625



Epoch: 3, Step: 144, Rank: 0, loss = 0.0322265625


Epoch 3:  60%|██████    | 24/40 [00:39<00:27,  1.71s/it]{
    "epoch": 3,
    "step": 144,
    "rank": 0,
    "overall_throughput": 93.60153954892388,
    "lr": 1.1045284632676535e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56913,
    "batch_size": 125,
    "total_loss": 0.04380370038479785,
    "samples_seen": 18216,
    "gradnorm": 0.294921875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:34.395870"
}
Epoch: 3, Step: 145, Rank: 5, loss = 0.041259765625Epoch: 3, Step: 145, Rank: 2, loss = 0.03662109375Epoch: 3, Step: 145, Rank: 7, loss = 0.057373046875Epoch: 3, Step: 145, Rank: 3, loss = 0.038818359375
Epoch: 3, Step: 145, Rank: 6, loss = 0.0556640625

Epoch: 3, Step: 145, Rank: 1, loss = 0.0341796875
Epoch: 3, Step: 145, Rank: 4, loss = 0.048828125


Epoch: 3, Step: 145, Rank: 0, loss = 0.0380859375


Epoch 3:  62%|██████▎   | 25/40 [00:41<00:25,  1.69s/it]{
    "epoch": 3,
    "step": 145,
    "rank": 0,
    "overall_throughput": 94.47263972334856,
    "lr": 1.092268359463302e-05,
    "cuda_mem_allocated": 6.588062763214111,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56354,
    "batch_size": 128,
    "total_loss": 0.04383007417397168,
    "samples_seen": 18344,
    "gradnorm": 0.3046875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:36.042414"
}
Epoch: 3, Step: 146, Rank: 5, loss = 0.0634765625Epoch: 3, Step: 146, Rank: 6, loss = 0.050048828125
Epoch: 3, Step: 146, Rank: 7, loss = 0.06005859375Epoch: 3, Step: 146, Rank: 2, loss = 0.0361328125Epoch: 3, Step: 146, Rank: 4, loss = 0.051025390625Epoch: 3, Step: 146, Rank: 3, loss = 0.03955078125
Epoch: 3, Step: 146, Rank: 1, loss = 0.0301513671875




Epoch: 3, Step: 146, Rank: 0, loss = 0.0302734375


Epoch 3:  65%|██████▌   | 26/40 [00:43<00:23,  1.68s/it]{
    "epoch": 3,
    "step": 146,
    "rank": 0,
    "overall_throughput": 94.6747566917377,
    "lr": 1.0799942511885417e-05,
    "cuda_mem_allocated": 6.586390018463135,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56182,
    "batch_size": 129,
    "total_loss": 0.04512121319995728,
    "samples_seen": 18473,
    "gradnorm": 0.30078125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:37.689552"
}
Epoch: 3, Step: 147, Rank: 5, loss = 0.0478515625Epoch: 3, Step: 147, Rank: 6, loss = 0.0458984375Epoch: 3, Step: 147, Rank: 2, loss = 0.039306640625Epoch: 3, Step: 147, Rank: 0, loss = 0.033935546875Epoch: 3, Step: 147, Rank: 3, loss = 0.036865234375

Epoch: 3, Step: 147, Rank: 7, loss = 0.05126953125

Epoch: 3, Step: 147, Rank: 4, loss = 0.03564453125


Epoch: 3, Step: 147, Rank: 1, loss = 0.031005859375


Epoch 3:  68%|██████▊   | 27/40 [00:44<00:21,  1.67s/it]{
    "epoch": 3,
    "step": 147,
    "rank": 0,
    "overall_throughput": 94.71444174463963,
    "lr": 1.0677080014047076e-05,
    "cuda_mem_allocated": 6.574780464172363,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56484,
    "batch_size": 123,
    "total_loss": 0.040223780185539265,
    "samples_seen": 18596,
    "gradnorm": 0.279296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:39.338265"
}
Epoch: 3, Step: 148, Rank: 1, loss = 0.037841796875Epoch: 3, Step: 148, Rank: 6, loss = 0.05419921875Epoch: 3, Step: 148, Rank: 2, loss = 0.043212890625Epoch: 3, Step: 148, Rank: 5, loss = 0.0458984375Epoch: 3, Step: 148, Rank: 3, loss = 0.043701171875
Epoch: 3, Step: 148, Rank: 7, loss = 0.0556640625



Epoch: 3, Step: 148, Rank: 4, loss = 0.049560546875

Epoch: 3, Step: 148, Rank: 0, loss = 0.025634765625


Epoch 3:  70%|███████   | 28/40 [00:46<00:19,  1.66s/it]{
    "epoch": 3,
    "step": 148,
    "rank": 0,
    "overall_throughput": 94.46382968914983,
    "lr": 1.05541147491597e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56601,
    "batch_size": 128,
    "total_loss": 0.0444691789897705,
    "samples_seen": 18724,
    "gradnorm": 0.287109375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:40.984464"
}
Epoch: 3, Step: 149, Rank: 5, loss = 0.0595703125Epoch: 3, Step: 149, Rank: 1, loss = 0.058349609375Epoch: 3, Step: 149, Rank: 2, loss = 0.04296875Epoch: 3, Step: 149, Rank: 4, loss = 0.040283203125

Epoch: 3, Step: 149, Rank: 3, loss = 0.03955078125
Epoch: 3, Step: 149, Rank: 6, loss = 0.05615234375
Epoch: 3, Step: 149, Rank: 7, loss = 0.058349609375


Epoch: 3, Step: 149, Rank: 0, loss = 0.0255126953125


Epoch 3:  72%|███████▎  | 29/40 [00:48<00:18,  1.66s/it]{
    "epoch": 3,
    "step": 149,
    "rank": 0,
    "overall_throughput": 93.35367414168304,
    "lr": 1.0431065380862959e-05,
    "cuda_mem_allocated": 6.588062286376953,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56732,
    "batch_size": 125,
    "total_loss": 0.04757456109426778,
    "samples_seen": 18849,
    "gradnorm": 0.296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:42.640697"
}
Epoch: 3, Step: 150, Rank: 6, loss = 0.0517578125Epoch: 3, Step: 150, Rank: 5, loss = 0.056640625Epoch: 3, Step: 150, Rank: 2, loss = 0.03857421875
Epoch: 3, Step: 150, Rank: 3, loss = 0.032958984375
Epoch: 3, Step: 150, Rank: 0, loss = 0.0185546875
Epoch: 3, Step: 150, Rank: 4, loss = 0.03857421875

Epoch: 3, Step: 150, Rank: 7, loss = 0.07275390625

Epoch: 3, Step: 150, Rank: 1, loss = 0.04052734375


Epoch 3:  75%|███████▌  | 30/40 [00:49<00:16,  1.66s/it]{
    "epoch": 3,
    "step": 150,
    "rank": 0,
    "overall_throughput": 94.5634121750869,
    "lr": 1.0307950585561705e-05,
    "cuda_mem_allocated": 6.566525936126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56862,
    "batch_size": 126,
    "total_loss": 0.043772642538074634,
    "samples_seen": 18975,
    "gradnorm": 0.27734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:44.288242"
}
Epoch: 3, Step: 151, Rank: 1, loss = 0.038818359375Epoch: 3, Step: 151, Rank: 5, loss = 0.046630859375Epoch: 3, Step: 151, Rank: 4, loss = 0.04345703125
Epoch: 3, Step: 151, Rank: 2, loss = 0.044677734375Epoch: 3, Step: 151, Rank: 3, loss = 0.052490234375Epoch: 3, Step: 151, Rank: 7, loss = 0.05419921875
Epoch: 3, Step: 151, Rank: 6, loss = 0.052978515625




Epoch: 3, Step: 151, Rank: 0, loss = 0.02294921875


Epoch 3:  78%|███████▊  | 31/40 [00:51<00:14,  1.66s/it]{
    "epoch": 3,
    "step": 151,
    "rank": 0,
    "overall_throughput": 94.36070838518269,
    "lr": 1.01847890495913e-05,
    "cuda_mem_allocated": 6.588063716888428,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56485,
    "batch_size": 127,
    "total_loss": 0.04450739134283438,
    "samples_seen": 19102,
    "gradnorm": 0.271484375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:45.939742"
}
Epoch: 3, Step: 152, Rank: 5, loss = 0.03955078125Epoch: 3, Step: 152, Rank: 7, loss = 0.059814453125Epoch: 3, Step: 152, Rank: 1, loss = 0.0419921875Epoch: 3, Step: 152, Rank: 2, loss = 0.036376953125Epoch: 3, Step: 152, Rank: 6, loss = 0.0576171875



Epoch: 3, Step: 152, Rank: 3, loss = 0.0400390625Epoch: 3, Step: 152, Rank: 4, loss = 0.038330078125


Epoch: 3, Step: 152, Rank: 0, loss = 0.02099609375


Epoch 3:  80%|████████  | 32/40 [00:53<00:13,  1.65s/it]{
    "epoch": 3,
    "step": 152,
    "rank": 0,
    "overall_throughput": 94.2882880541208,
    "lr": 1.0061599466381388e-05,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56616,
    "batch_size": 127,
    "total_loss": 0.04191394658753709,
    "samples_seen": 19229,
    "gradnorm": 0.2578125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:47.589158"
}
Epoch: 3, Step: 153, Rank: 1, loss = 0.0303955078125Epoch: 3, Step: 153, Rank: 2, loss = 0.03173828125Epoch: 3, Step: 153, Rank: 6, loss = 0.0556640625
Epoch: 3, Step: 153, Rank: 5, loss = 0.04541015625Epoch: 3, Step: 153, Rank: 7, loss = 0.0625
Epoch: 3, Step: 153, Rank: 4, loss = 0.041748046875


Epoch: 3, Step: 153, Rank: 3, loss = 0.046630859375
Epoch: 3, Step: 153, Rank: 0, loss = 0.0283203125



Epoch 3:  82%|████████▎ | 33/40 [00:54<00:11,  1.65s/it]{
    "epoch": 3,
    "step": 153,
    "rank": 0,
    "overall_throughput": 94.56169661576192,
    "lr": 9.938400533618615e-06,
    "cuda_mem_allocated": 6.5880632400512695,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56680,
    "batch_size": 127,
    "total_loss": 0.04281933662667608,
    "samples_seen": 19356,
    "gradnorm": 0.279296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:49.237605"
}
Epoch: 3, Step: 154, Rank: 5, loss = 0.06689453125Epoch: 3, Step: 154, Rank: 3, loss = 0.046875Epoch: 3, Step: 154, Rank: 6, loss = 0.05712890625Epoch: 3, Step: 154, Rank: 1, loss = 0.037841796875Epoch: 3, Step: 154, Rank: 7, loss = 0.05517578125Epoch: 3, Step: 154, Rank: 4, loss = 0.04248046875Epoch: 3, Step: 154, Rank: 2, loss = 0.036376953125






Epoch: 3, Step: 154, Rank: 0, loss = 0.01953125


Epoch 3:  85%|████████▌ | 34/40 [00:56<00:10,  1.74s/it]{
    "epoch": 3,
    "step": 154,
    "rank": 0,
    "overall_throughput": 77.81588190941444,
    "lr": 9.815210950408703e-06,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56208,
    "batch_size": 127,
    "total_loss": 0.04529604326786223,
    "samples_seen": 19483,
    "gradnorm": 0.29296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:51.178180"
}
Epoch: 3, Step: 155, Rank: 6, loss = 0.054931640625Epoch: 3, Step: 155, Rank: 3, loss = 0.044677734375Epoch: 3, Step: 155, Rank: 1, loss = 0.034912109375Epoch: 3, Step: 155, Rank: 5, loss = 0.04931640625Epoch: 3, Step: 155, Rank: 2, loss = 0.034912109375

Epoch: 3, Step: 155, Rank: 7, loss = 0.06298828125



Epoch: 3, Step: 155, Rank: 4, loss = 0.054931640625
Epoch: 3, Step: 155, Rank: 0, loss = 0.033935546875


Epoch 3:  88%|████████▊ | 35/40 [00:58<00:09,  1.87s/it]{
    "epoch": 3,
    "step": 155,
    "rank": 0,
    "overall_throughput": 68.41822635922802,
    "lr": 9.692049414438298e-06,
    "cuda_mem_allocated": 6.584970951080322,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56313,
    "batch_size": 130,
    "total_loss": 0.046312574361159944,
    "samples_seen": 19613,
    "gradnorm": 0.30078125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:53.341082"
}
Epoch: 3, Step: 156, Rank: 0, loss = 0.03955078125Epoch: 3, Step: 156, Rank: 6, loss = 0.054443359375Epoch: 3, Step: 156, Rank: 3, loss = 0.03857421875Epoch: 3, Step: 156, Rank: 2, loss = 0.046630859375


Epoch: 3, Step: 156, Rank: 7, loss = 0.06494140625Epoch: 3, Step: 156, Rank: 4, loss = 0.0390625

Epoch: 3, Step: 156, Rank: 5, loss = 0.04736328125

Epoch: 3, Step: 156, Rank: 1, loss = 0.038818359375


Epoch 3:  90%|█████████ | 36/40 [01:00<00:07,  1.80s/it]{
    "epoch": 3,
    "step": 156,
    "rank": 0,
    "overall_throughput": 93.36205100880476,
    "lr": 9.568934619137048e-06,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56472,
    "batch_size": 129,
    "total_loss": 0.04618217877886386,
    "samples_seen": 19742,
    "gradnorm": 0.302734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:54.989969"
}
Epoch: 3, Step: 157, Rank: 5, loss = 0.05908203125Epoch: 3, Step: 157, Rank: 1, loss = 0.043701171875Epoch: 3, Step: 157, Rank: 0, loss = 0.01611328125

Epoch: 3, Step: 157, Rank: 2, loss = 0.035888671875
Epoch: 3, Step: 157, Rank: 3, loss = 0.037109375Epoch: 3, Step: 157, Rank: 7, loss = 0.056396484375

Epoch: 3, Step: 157, Rank: 4, loss = 0.044189453125

Epoch: 3, Step: 157, Rank: 6, loss = 0.0654296875


Epoch 3:  92%|█████████▎| 37/40 [01:02<00:05,  1.75s/it]{
    "epoch": 3,
    "step": 157,
    "rank": 0,
    "overall_throughput": 93.65772726836599,
    "lr": 9.445885250840301e-06,
    "cuda_mem_allocated": 6.568904876708984,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56638,
    "batch_size": 124,
    "total_loss": 0.04474910837247078,
    "samples_seen": 19866,
    "gradnorm": 0.30078125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:56.632735"
}
Epoch: 3, Step: 158, Rank: 6, loss = 0.05419921875Epoch: 3, Step: 158, Rank: 5, loss = 0.047119140625
Epoch: 3, Step: 158, Rank: 4, loss = 0.046875Epoch: 3, Step: 158, Rank: 3, loss = 0.039794921875Epoch: 3, Step: 158, Rank: 2, loss = 0.03955078125

Epoch: 3, Step: 158, Rank: 7, loss = 0.06591796875

Epoch: 3, Step: 158, Rank: 1, loss = 0.046630859375

Epoch: 3, Step: 158, Rank: 0, loss = 0.0164794921875


Epoch 3:  95%|█████████▌| 38/40 [01:03<00:03,  1.72s/it]{
    "epoch": 3,
    "step": 158,
    "rank": 0,
    "overall_throughput": 94.07729721204899,
    "lr": 9.322919985952926e-06,
    "cuda_mem_allocated": 6.586057186126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56726,
    "batch_size": 126,
    "total_loss": 0.04450340231992384,
    "samples_seen": 19992,
    "gradnorm": 0.298828125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:58.289957"
}
Epoch: 3, Step: 159, Rank: 6, loss = 0.05126953125Epoch: 3, Step: 159, Rank: 5, loss = 0.05029296875Epoch: 3, Step: 159, Rank: 1, loss = 0.0289306640625Epoch: 3, Step: 159, Rank: 3, loss = 0.040771484375
Epoch: 3, Step: 159, Rank: 2, loss = 0.04296875Epoch: 3, Step: 159, Rank: 4, loss = 0.040771484375


Epoch: 3, Step: 159, Rank: 7, loss = 0.06396484375


Epoch: 3, Step: 159, Rank: 0, loss = 0.04345703125


Epoch 3:  98%|█████████▊| 39/40 [01:05<00:01,  1.70s/it]{
    "epoch": 3,
    "step": 159,
    "rank": 0,
    "overall_throughput": 94.21983139194397,
    "lr": 9.200057488114585e-06,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56081,
    "batch_size": 128,
    "total_loss": 0.04534512580018188,
    "samples_seen": 20120,
    "gradnorm": 0.2890625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:37:59.941731"
}
Epoch: 3, Step: 160, Rank: 6, loss = 0.050537109375Epoch: 3, Step: 160, Rank: 7, loss = 0.0625Epoch: 3, Step: 160, Rank: 1, loss = 0.0390625Epoch: 3, Step: 160, Rank: 3, loss = 0.0400390625Epoch: 3, Step: 160, Rank: 5, loss = 0.049560546875



Epoch: 3, Step: 160, Rank: 2, loss = 0.04541015625Epoch: 3, Step: 160, Rank: 4, loss = 0.0341796875


Epoch: 3, Step: 160, Rank: 0, loss = 0.02587890625


Epoch 3: 100%|██████████| 40/40 [01:07<00:00,  1.69s/it]{
    "epoch": 3,
    "step": 160,
    "rank": 0,
    "overall_throughput": 94.35223426913302,
    "lr": 9.07731640536698e-06,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56770,
    "batch_size": 124,
    "total_loss": 0.04342082085608596,
    "samples_seen": 20244,
    "gradnorm": 0.27734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:38:01.592935"
}
Saving model in huggingface format at: samples_20244
[19:38:28] INFO     The model is bigger than the maximum     accelerator.py:2924
                    size per checkpoint (5GB) and is going                      
                    to be split in 4 checkpoint shards. You                     
                    can find where each parameters has been                     
                    saved in the index located at                               
                    /tmp/tmpv92rybegw/model.safetensors.inde                    
                    x.json.                                                     
Model saved in /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/hf_format/samples_20244
[19:38:44] INFO     saving took 42.07989287376404 seconds           utils.py:879
Saving full model state in /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/full_state/epoch_3
           INFO     Saving current state to                  accelerator.py:3030
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_3                                                  
           INFO     Saving FSDP model                        accelerator.py:3040
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
[19:38:52] INFO     Saving model to                             fsdp_utils.py:88
                    /var/home/cloud-user/.local/share/instructl                 
                    ab/phased/phase1/checkpoints/full_state/epo                 
                    ch_3/pytorch_model_fsdp.bin                                 
[19:39:03] INFO     Model saved to                              fsdp_utils.py:90
                    /var/home/cloud-user/.local/share/instructl                 
                    ab/phased/phase1/checkpoints/full_state/epo                 
                    ch_3/pytorch_model_fsdp.bin                                 
           INFO     FSDP Model saved to output dir           accelerator.py:3042
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_3                                                  
           INFO     Saving FSDP Optimizer                    accelerator.py:3059
 total length: 7879 dropped: 0 num samples 18 - rank: 5 max len: 453 min len: 421 avg len: 437.72222222222223 num_loss_counted_tokens: 6925 total length: 7994 dropped: 0 num samples 18 - rank: 5 max len: 481 min len: 295 avg len: 444.1111111111111 num_loss_counted_tokens: 7040
 total length: 7987 dropped: 0 num samples 18 - rank: 5 max len: 481 min len: 267 avg len: 443.72222222222223 num_loss_counted_tokens: 7033
 total length: 7993 dropped: 0 num samples 18 - rank: 5 max len: 483 min len: 264 avg len: 444.05555555555554 num_loss_counted_tokens: 7039
 total length: 7808 dropped: 0 num samples 13 - rank: 2 max len: 629 min len: 565 avg len: 600.6153846153846 num_loss_counted_tokens: 7119
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 634 min len: 476 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7965 dropped: 0 num samples 13 - rank: 2 max len: 671 min len: 256 avg len: 612.6923076923077 num_loss_counted_tokens: 7276
 total length: 7988 dropped: 0 num samples 13 - rank: 2 max len: 646 min len: 436 avg len: 614.4615384615385 num_loss_counted_tokens: 7299
 total length: 7858 dropped: 0 num samples 13 - rank: 2 max len: 619 min len: 578 avg len: 604.4615384615385 num_loss_counted_tokens: 7169
 total length: 7996 dropped: 0 num samples 17 - rank: 5 max len: 507 min len: 309 avg len: 470.3529411764706 num_loss_counted_tokens: 7095
 total length: 7879 dropped: 0 num samples 14 - rank: 3 max len: 610 min len: 521 avg len: 562.7857142857143 num_loss_counted_tokens: 7137
 total length: 7971 dropped: 0 num samples 18 - rank: 5 max len: 474 min len: 302 avg len: 442.8333333333333 num_loss_counted_tokens: 7017
 total length: 7997 dropped: 0 num samples 14 - rank: 3 max len: 612 min len: 335 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7840 dropped: 0 num samples 15 - rank: 3 max len: 549 min len: 501 avg len: 522.6666666666666 num_loss_counted_tokens: 7045
 total length: 7910 dropped: 0 num samples 13 - rank: 2 max len: 626 min len: 590 avg len: 608.4615384615385 num_loss_counted_tokens: 7221

 total length: 7980 dropped: 0 num samples 16 - rank: 3 max len: 555 min len: 191 avg len: 498.75 num_loss_counted_tokens: 7132
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 565 min len: 348 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7997 dropped: 0 num samples 14 - rank: 2 max len: 613 min len: 434 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7787 dropped: 0 num samples 14 - rank: 3 max len: 586 min len: 526 avg len: 556.2142857142857 num_loss_counted_tokens: 7045
 total length: 7837 dropped: 0 num samples 20 - rank: 6 max len: 420 min len: 367 avg len: 391.85 num_loss_counted_tokens: 6777
 total length: 7902 dropped: 0 num samples 18 - rank: 5 max len: 477 min len: 416 avg len: 439.0 num_loss_counted_tokens: 6948
 total length: 7995 dropped: 0 num samples 14 - rank: 3 max len: 608 min len: 456 avg len: 571.0714285714286 num_loss_counted_tokens: 7253 total length: 7893 dropped: 0 num samples 18 - rank: 5 max len: 466 min len: 415 avg len: 438.5 num_loss_counted_tokens: 6939
 total length: 7854 dropped: 0 num samples 15 - rank: 3 max len: 548 min len: 498 avg len: 523.6 num_loss_counted_tokens: 7059
 total length: 7974 dropped: 0 num samples 18 - rank: 5 max len: 480 min len: 244 avg len: 443.0 num_loss_counted_tokens: 7020

 total length: 7885 dropped: 0 num samples 14 - rank: 3 max len: 596 min len: 537 avg len: 563.2142857142857 num_loss_counted_tokens: 7143
[19:39:13] INFO     Saving Optimizer state to                  fsdp_utils.py:192
                    /var/home/cloud-user/.local/share/instruct                  
                    lab/phased/phase1/checkpoints/full_state/e                  
                    poch_3/optimizer.bin                                        
 total length: 7994 dropped: 0 num samples 14 - rank: 2 max len: 627 min len: 317 avg len: 571.0 num_loss_counted_tokens: 7252
 total length: 7911 dropped: 0 num samples 13 - rank: 2 max len: 637 min len: 570 avg len: 608.5384615384615 num_loss_counted_tokens: 7222
 total length: 7764 dropped: 0 num samples 19 - rank: 6 max len: 429 min len: 385 avg len: 408.63157894736844 num_loss_counted_tokens: 6757
 total length: 7808 dropped: 0 num samples 20 - rank: 6 max len: 419 min len: 366 avg len: 390.4 num_loss_counted_tokens: 6748
 total length: 7816 dropped: 0 num samples 18 - rank: 5 max len: 454 min len: 408 avg len: 434.22222222222223 num_loss_counted_tokens: 6862
 total length: 7990 dropped: 0 num samples 13 - rank: 2 max len: 636 min len: 543 avg len: 614.6153846153846 num_loss_counted_tokens: 7301
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 470 min len: 337 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7991 dropped: 0 num samples 13 - rank: 2 max len: 633 min len: 586 avg len: 614.6923076923077 num_loss_counted_tokens: 7302
 total length: 7985 dropped: 0 num samples 15 - rank: 3 max len: 578 min len: 273 avg len: 532.3333333333334 num_loss_counted_tokens: 7190
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 583 min len: 270 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 633 min len: 517 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7886 dropped: 0 num samples 19 - rank: 6 max len: 454 min len: 386 avg len: 415.05263157894734 num_loss_counted_tokens: 6879
 total length: 7837 dropped: 0 num samples 17 - rank: 5 max len: 506 min len: 419 avg len: 461.0 num_loss_counted_tokens: 6936
 total length: 7933 dropped: 0 num samples 14 - rank: 3 max len: 586 min len: 532 avg len: 566.6428571428571 num_loss_counted_tokens: 7191
 total length: 7827 dropped: 0 num samples 13 - rank: 2 max len: 629 min len: 584 avg len: 602.0769230769231 num_loss_counted_tokens: 7138
 total length: 7980 dropped: 0 num samples 13 - rank: 2 max len: 667 min len: 290 avg len: 613.8461538461538 num_loss_counted_tokens: 7291
 total length: 7996 dropped: 0 num samples 14 - rank: 3 max len: 612 min len: 385 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7986 dropped: 0 num samples 14 - rank: 3 max len: 619 min len: 521 avg len: 570.4285714285714 num_loss_counted_tokens: 7244
 total length: 7987 dropped: 0 num samples 19 - rank: 5 max len: 446 min len: 264 avg len: 420.36842105263156 num_loss_counted_tokens: 6980
 total length: 7929 dropped: 0 num samples 18 - rank: 5 max len: 474 min len: 409 avg len: 440.5 num_loss_counted_tokens: 6975
 total length: 7997 dropped: 0 num samples 13 - rank: 1 max len: 658 min len: 300 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7994 dropped: 0 num samples 14 - rank: 3 max len: 617 min len: 289 avg len: 571.0 num_loss_counted_tokens: 7252
 total length: 7927 dropped: 0 num samples 11 - rank: 1 max len: 804 min len: 677 avg len: 720.6363636363636 num_loss_counted_tokens: 7344
 total length: 7961 dropped: 0 num samples 12 - rank: 1 max len: 687 min len: 648 avg len: 663.4166666666666 num_loss_counted_tokens: 7325
 total length: 7756 dropped: 0 num samples 14 - rank: 3 max len: 591 min len: 531 avg len: 554.0 num_loss_counted_tokens: 7014
 total length: 7986 dropped: 0 num samples 12 - rank: 1 max len: 692 min len: 629 avg len: 665.5 num_loss_counted_tokens: 7350
 total length: 7886 dropped: 0 num samples 12 - rank: 1 max len: 689 min len: 637 avg len: 657.1666666666666 num_loss_counted_tokens: 7250
 total length: 7963 dropped: 0 num samples 18 - rank: 5 max len: 467 min len: 407 avg len: 442.3888888888889 num_loss_counted_tokens: 7009
 total length: 7950 dropped: 0 num samples 19 - rank: 5 max len: 457 min len: 202 avg len: 418.42105263157896 num_loss_counted_tokens: 6943
 total length: 7827 dropped: 0 num samples 17 - rank: 5 max len: 476 min len: 440 avg len: 460.4117647058824 num_loss_counted_tokens: 6926
 total length: 7993 dropped: 0 num samples 13 - rank: 2 max len: 664 min len: 359 avg len: 614.8461538461538 num_loss_counted_tokens: 7304
 total length: 7988 dropped: 0 num samples 20 - rank: 6 max len: 417 min len: 370 avg len: 399.4 num_loss_counted_tokens: 6928
 total length: 7982 dropped: 0 num samples 16 - rank: 4 max len: 514 min len: 484 avg len: 498.875 num_loss_counted_tokens: 7134
 total length: 7985 dropped: 0 num samples 20 - rank: 6 max len: 423 min len: 279 avg len: 399.25 num_loss_counted_tokens: 6925
 total length: 7996 dropped: 0 num samples 17 - rank: 4 max len: 495 min len: 364 avg len: 470.3529411764706 num_loss_counted_tokens: 7095 total length: 7993 dropped: 0 num samples 17 - rank: 4 max len: 488 min len: 415 avg len: 470.1764705882353 num_loss_counted_tokens: 7092

 total length: 7515 dropped: 0 num samples 23 - rank: 7 max len: 385 min len: 236 avg len: 326.7391304347826 num_loss_counted_tokens: 6296
 total length: 7993 dropped: 0 num samples 16 - rank: 4 max len: 535 min len: 337 avg len: 499.5625 num_loss_counted_tokens: 7145
 total length: 7992 dropped: 0 num samples 21 - rank: 6 max len: 406 min len: 243 avg len: 380.57142857142856 num_loss_counted_tokens: 6879
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 643 min len: 543 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7778 dropped: 0 num samples 23 - rank: 7 max len: 383 min len: 255 avg len: 338.17391304347825 num_loss_counted_tokens: 6559
 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 532 min len: 400 avg len: 499.75 num_loss_counted_tokens: 7148
 total length: 7581 dropped: 0 num samples 24 - rank: 7 max len: 368 min len: 238 avg len: 315.875 num_loss_counted_tokens: 6309
 total length: 7954 dropped: 0 num samples 15 - rank: 4 max len: 557 min len: 510 avg len: 530.2666666666667 num_loss_counted_tokens: 7159
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 569 min len: 361 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7994 dropped: 0 num samples 21 - rank: 6 max len: 408 min len: 353 avg len: 380.6666666666667 num_loss_counted_tokens: 6881
 total length: 6916 dropped: 0 num samples 22 - rank: 7 max len: 364 min len: 248 avg len: 314.3636363636364 num_loss_counted_tokens: 5750
 total length: 7910 dropped: 0 num samples 12 - rank: 1 max len: 701 min len: 623 avg len: 659.1666666666666 num_loss_counted_tokens: 7274
 total length: 7771 dropped: 0 num samples 12 - rank: 1 max len: 673 min len: 635 avg len: 647.5833333333334 num_loss_counted_tokens: 7135
 total length: 7889 dropped: 0 num samples 12 - rank: 1 max len: 715 min len: 628 avg len: 657.4166666666666 num_loss_counted_tokens: 7253
 total length: 7938 dropped: 0 num samples 20 - rank: 6 max len: 423 min len: 373 avg len: 396.9 num_loss_counted_tokens: 6878
 total length: 7810 dropped: 0 num samples 24 - rank: 7 max len: 366 min len: 266 avg len: 325.4166666666667 num_loss_counted_tokens: 6538
 total length: 7770 dropped: 0 num samples 11 - rank: 1 max len: 745 min len: 674 avg len: 706.3636363636364 num_loss_counted_tokens: 7187
 total length: 7900 dropped: 0 num samples 12 - rank: 1 max len: 702 min len: 631 avg len: 658.3333333333334 num_loss_counted_tokens: 7264
 total length: 7884 dropped: 0 num samples 26 - rank: 7 max len: 366 min len: 174 avg len: 303.2307692307692 num_loss_counted_tokens: 6506
 total length: 7763 dropped: 0 num samples 12 - rank: 1 max len: 678 min len: 629 avg len: 646.9166666666666 num_loss_counted_tokens: 7127
 total length: 7936 dropped: 0 num samples 12 - rank: 1 max len: 698 min len: 645 avg len: 661.3333333333334 num_loss_counted_tokens: 7300
 total length: 7962 dropped: 0 num samples 20 - rank: 6 max len: 428 min len: 369 avg len: 398.1 num_loss_counted_tokens: 6902
 total length: 7993 dropped: 0 num samples 12 - rank: 1 max len: 826 min len: 501 avg len: 666.0833333333334 num_loss_counted_tokens: 7357
 total length: 7686 dropped: 0 num samples 25 - rank: 7 max len: 359 min len: 257 avg len: 307.44 num_loss_counted_tokens: 6361
 total length: 7801 dropped: 0 num samples 16 - rank: 4 max len: 522 min len: 462 avg len: 487.5625 num_loss_counted_tokens: 6953
 total length: 7995 dropped: 0 num samples 16 - rank: 4 max len: 539 min len: 380 avg len: 499.6875 num_loss_counted_tokens: 7147
 total length: 7995 dropped: 0 num samples 13 - rank: 2 max len: 642 min len: 458 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7920 dropped: 0 num samples 15 - rank: 4 max len: 558 min len: 507 avg len: 528.0 num_loss_counted_tokens: 7125
 total length: 7855 dropped: 0 num samples 23 - rank: 7 max len: 390 min len: 259 avg len: 341.5217391304348 num_loss_counted_tokens: 6636
 total length: 7994 dropped: 0 num samples 16 - rank: 4 max len: 530 min len: 450 avg len: 499.625 num_loss_counted_tokens: 7146
 total length: 7941 dropped: 0 num samples 16 - rank: 4 max len: 520 min len: 473 avg len: 496.3125 num_loss_counted_tokens: 7093
 total length: 7670 dropped: 0 num samples 23 - rank: 7 max len: 370 min len: 256 avg len: 333.4782608695652 num_loss_counted_tokens: 6451
 total length: 7840 dropped: 0 num samples 24 - rank: 7 max len: 363 min len: 257 avg len: 326.6666666666667 num_loss_counted_tokens: 6568
 total length: 7995 dropped: 0 num samples 17 - rank: 4 max len: 496 min len: 263 avg len: 470.29411764705884 num_loss_counted_tokens: 7094
 total length: 7994 dropped: 0 num samples 16 - rank: 4 max len: 530 min len: 416 avg len: 499.625 num_loss_counted_tokens: 7146
 total length: 7741 dropped: 0 num samples 24 - rank: 7 max len: 372 min len: 265 avg len: 322.5416666666667 num_loss_counted_tokens: 6469
 total length: 7996 dropped: 0 num samples 21 - rank: 6 max len: 410 min len: 290 avg len: 380.76190476190476 num_loss_counted_tokens: 6883
 total length: 7989 dropped: 0 num samples 12 - rank: 1 max len: 750 min len: 256 avg len: 665.75 num_loss_counted_tokens: 7353
 total length: 7980 dropped: 0 num samples 12 - rank: 1 max len: 688 min len: 640 avg len: 665.0 num_loss_counted_tokens: 7344
 total length: 7863 dropped: 0 num samples 24 - rank: 7 max len: 370 min len: 270 avg len: 327.625 num_loss_counted_tokens: 6591
 total length: 7961 dropped: 0 num samples 16 - rank: 4 max len: 515 min len: 484 avg len: 497.5625 num_loss_counted_tokens: 7113
 total length: 7568 dropped: 0 num samples 24 - rank: 7 max len: 359 min len: 255 avg len: 315.3333333333333 num_loss_counted_tokens: 6296
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 724 min len: 527 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7460 dropped: 0 num samples 23 - rank: 7 max len: 372 min len: 236 avg len: 324.3478260869565 num_loss_counted_tokens: 6241
 total length: 7978 dropped: 0 num samples 20 - rank: 6 max len: 424 min len: 373 avg len: 398.9 num_loss_counted_tokens: 6918
 total length: 7857 dropped: 0 num samples 20 - rank: 6 max len: 415 min len: 368 avg len: 392.85 num_loss_counted_tokens: 6797
 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 548 min len: 475 avg len: 499.75 num_loss_counted_tokens: 7148
 total length: 7930 dropped: 0 num samples 12 - rank: 1 max len: 695 min len: 638 avg len: 660.8333333333334 num_loss_counted_tokens: 7294
 total length: 7990 dropped: 0 num samples 21 - rank: 6 max len: 409 min len: 229 avg len: 380.4761904761905 num_loss_counted_tokens: 6877
 total length: 7882 dropped: 0 num samples 20 - rank: 6 max len: 411 min len: 371 avg len: 394.1 num_loss_counted_tokens: 6822
 total length: 7988 dropped: 0 num samples 25 - rank: 7 max len: 345 min len: 265 avg len: 319.52 num_loss_counted_tokens: 6663
 total length: 7405 dropped: 0 num samples 23 - rank: 7 max len: 366 min len: 253 avg len: 321.95652173913044 num_loss_counted_tokens: 6186
 total length: 7990 dropped: 0 num samples 17 - rank: 4 max len: 510 min len: 449 avg len: 470.0 num_loss_counted_tokens: 7089
 total length: 7836 dropped: 0 num samples 20 - rank: 6 max len: 412 min len: 367 avg len: 391.8 num_loss_counted_tokens: 6776
 total length: 7995 dropped: 0 num samples 20 - rank: 6 max len: 438 min len: 350 avg len: 399.75 num_loss_counted_tokens: 6935
 total length: 7788 dropped: 0 num samples 24 - rank: 7 max len: 363 min len: 172 avg len: 324.5 num_loss_counted_tokens: 6516
 total length: 7863 dropped: 0 num samples 15 - rank: 4 max len: 552 min len: 482 avg len: 524.2 num_loss_counted_tokens: 7068
[19:39:36] INFO     Optimizer state saved in                   fsdp_utils.py:194
                    /var/home/cloud-user/.local/share/instruct                  
                    lab/phased/phase1/checkpoints/full_state/e                  
                    poch_3/optimizer.bin                                        
           INFO     FSDP Optimizer saved to output dir       accelerator.py:3061
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_3                                                  
           INFO     Scheduler state saved in                checkpointing.py:118
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_3/scheduler.bin                                  
           INFO     Sampler state for dataloader 0 saved in checkpointing.py:135
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_3/sampler.bin                                    
           INFO     Random states saved in                  checkpointing.py:160
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_3/random_states_0.pkl                            
Saving training state: {'current_epoch': 3, 'samples_seen': 20244}
Model state saved in: /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/full_state/epoch_3

Epoch 4:   0%|          | 0/40 [00:00<?, ?it/s]
Epoch 3: 100%|██████████| 40/40 [02:41<00:00,  4.04s/it]
 total length: 7996 dropped: 0 num samples 10 - rank: 0 max len: 1140 min len: 337 avg len: 799.6 num_loss_counted_tokens: 7466 total length: 7985 dropped: 0 num samples 10 - rank: 0 max len: 1057 min len: 547 avg len: 798.5 num_loss_counted_tokens: 7455

 total length: 7995 dropped: 0 num samples 11 - rank: 0 max len: 904 min len: 416 avg len: 726.8181818181819 num_loss_counted_tokens: 7412
 total length: 7783 dropped: 0 num samples 9 - rank: 0 max len: 1064 min len: 706 avg len: 864.7777777777778 num_loss_counted_tokens: 7306
 total length: 7866 dropped: 0 num samples 9 - rank: 0 max len: 1079 min len: 707 avg len: 874.0 num_loss_counted_tokens: 7389
 total length: 7772 dropped: 0 num samples 8 - rank: 0 max len: 1076 min len: 845 avg len: 971.5 num_loss_counted_tokens: 7348
 total length: 7898 dropped: 0 num samples 9 - rank: 0 max len: 1104 min len: 725 avg len: 877.5555555555555 num_loss_counted_tokens: 7421
 total length: 7776 dropped: 0 num samples 8 - rank: 0 max len: 1106 min len: 834 avg len: 972.0 num_loss_counted_tokens: 7352
 total length: 7997 dropped: 0 num samples 10 - rank: 0 max len: 1077 min len: 549 avg len: 799.7 num_loss_counted_tokens: 7467
 total length: 7988 dropped: 0 num samples 10 - rank: 0 max len: 1068 min len: 563 avg len: 798.8 num_loss_counted_tokens: 7458
 total length: 7995 dropped: 0 num samples 9 - rank: 0 max len: 1118 min len: 376 avg len: 888.3333333333334 num_loss_counted_tokens: 7518
 total length: 7994 dropped: 0 num samples 10 - rank: 0 max len: 1071 min len: 449 avg len: 799.4 num_loss_counted_tokens: 7464
 total length: 7991 dropped: 0 num samples 9 - rank: 0 max len: 1155 min len: 478 avg len: 887.8888888888889 num_loss_counted_tokens: 7514
 total length: 7989 dropped: 0 num samples 9 - rank: 0 max len: 1084 min len: 487 avg len: 887.6666666666666 num_loss_counted_tokens: 7512
 total length: 7997 dropped: 0 num samples 10 - rank: 0 max len: 1104 min len: 404 avg len: 799.7 num_loss_counted_tokens: 7467
 total length: 7857 dropped: 0 num samples 9 - rank: 0 max len: 1064 min len: 743 avg len: 873.0 num_loss_counted_tokens: 7380
 total length: 7835 dropped: 0 num samples 8 - rank: 0 max len: 1185 min len: 804 avg len: 979.375 num_loss_counted_tokens: 7411
Epoch: 4, Step: 161, Rank: 6, loss = 0.056640625Epoch: 4, Step: 161, Rank: 1, loss = 0.046142578125Epoch: 4, Step: 161, Rank: 5, loss = 0.046142578125Epoch: 4, Step: 161, Rank: 2, loss = 0.037841796875Epoch: 4, Step: 161, Rank: 3, loss = 0.038818359375




Epoch: 4, Step: 161, Rank: 7, loss = 0.0576171875Epoch: 4, Step: 161, Rank: 4, loss = 0.047119140625

Epoch: 4, Step: 161, Rank: 0, loss = 0.0299072265625

Epoch 4:   2%|▎         | 1/40 [00:01<01:04,  1.65s/it]{
    "epoch": 4,
    "step": 161,
    "rank": 0,
    "overall_throughput": 94.47724486682192,
    "lr": 8.954715367323468e-06,
    "cuda_mem_allocated": 6.588063716888428,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56407,
    "batch_size": 135,
    "total_loss": 0.045047600475118334,
    "samples_seen": 20379,
    "gradnorm": 0.287109375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:39:37.942159"
}
 total length: 7771 dropped: 0 num samples 9 - rank: 0 max len: 1309 min len: 698 avg len: 863.4444444444445 num_loss_counted_tokens: 7294
 total length: 7997 dropped: 0 num samples 19 - rank: 5 max len: 466 min len: 283 avg len: 420.89473684210526 num_loss_counted_tokens: 6990
 total length: 7960 dropped: 0 num samples 16 - rank: 4 max len: 532 min len: 468 avg len: 497.5 num_loss_counted_tokens: 7112
 total length: 7944 dropped: 0 num samples 13 - rank: 2 max len: 638 min len: 583 avg len: 611.0769230769231 num_loss_counted_tokens: 7255
 total length: 7941 dropped: 0 num samples 25 - rank: 7 max len: 354 min len: 258 avg len: 317.64 num_loss_counted_tokens: 6616
 total length: 7997 dropped: 0 num samples 21 - rank: 6 max len: 408 min len: 346 avg len: 380.8095238095238 num_loss_counted_tokens: 6884
 total length: 7930 dropped: 0 num samples 12 - rank: 1 max len: 698 min len: 640 avg len: 660.8333333333334 num_loss_counted_tokens: 7294
 total length: 7854 dropped: 0 num samples 14 - rank: 3 max len: 579 min len: 536 avg len: 561.0 num_loss_counted_tokens: 7112
Epoch: 4, Step: 162, Rank: 6, loss = 0.041259765625Epoch: 4, Step: 162, Rank: 7, loss = 0.054443359375Epoch: 4, Step: 162, Rank: 5, loss = 0.05078125Epoch: 4, Step: 162, Rank: 1, loss = 0.0380859375Epoch: 4, Step: 162, Rank: 2, loss = 0.036376953125Epoch: 4, Step: 162, Rank: 3, loss = 0.0361328125Epoch: 4, Step: 162, Rank: 4, loss = 0.04150390625






Epoch: 4, Step: 162, Rank: 0, loss = 0.049072265625

Epoch 4:   5%|▌         | 2/40 [00:03<01:08,  1.82s/it]{
    "epoch": 4,
    "step": 162,
    "rank": 0,
    "overall_throughput": 77.36984646246876,
    "lr": 8.83227298234144e-06,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56371,
    "batch_size": 126,
    "total_loss": 0.0434265845913679,
    "samples_seen": 20505,
    "gradnorm": 0.298828125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:39:39.874583"
}
 total length: 7871 dropped: 0 num samples 25 - rank: 7 max len: 369 min len: 182 avg len: 314.84 num_loss_counted_tokens: 6546
 total length: 7915 dropped: 0 num samples 14 - rank: 3 max len: 595 min len: 540 avg len: 565.3571428571429 num_loss_counted_tokens: 7173
 total length: 7951 dropped: 0 num samples 8 - rank: 0 max len: 1101 min len: 821 avg len: 993.875 num_loss_counted_tokens: 7527
 total length: 7838 dropped: 0 num samples 11 - rank: 1 max len: 882 min len: 637 avg len: 712.5454545454545 num_loss_counted_tokens: 7255
 total length: 7972 dropped: 0 num samples 20 - rank: 6 max len: 431 min len: 370 avg len: 398.6 num_loss_counted_tokens: 6912
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 637 min len: 568 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 537 min len: 384 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7996 dropped: 0 num samples 18 - rank: 5 max len: 468 min len: 393 avg len: 444.22222222222223 num_loss_counted_tokens: 7042
Epoch: 4, Step: 163, Rank: 1, loss = 0.03955078125Epoch: 4, Step: 163, Rank: 7, loss = 0.053466796875Epoch: 4, Step: 163, Rank: 6, loss = 0.048095703125Epoch: 4, Step: 163, Rank: 4, loss = 0.03662109375
Epoch: 4, Step: 163, Rank: 3, loss = 0.034423828125



Epoch: 4, Step: 163, Rank: 2, loss = 0.0341796875Epoch: 4, Step: 163, Rank: 5, loss = 0.052490234375

Epoch: 4, Step: 163, Rank: 0, loss = 0.0263671875

Epoch 4:   8%|▊         | 3/40 [00:05<01:04,  1.73s/it]{
    "epoch": 4,
    "step": 163,
    "rank": 0,
    "overall_throughput": 94.92211014673147,
    "lr": 8.71000783469797e-06,
    "cuda_mem_allocated": 6.5868072509765625,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 55905,
    "batch_size": 125,
    "total_loss": 0.04064037205974421,
    "samples_seen": 20630,
    "gradnorm": 0.265625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:39:41.512074"
}
 total length: 7995 dropped: 0 num samples 13 - rank: 2 max len: 642 min len: 497 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7978 dropped: 0 num samples 10 - rank: 0 max len: 1094 min len: 344 avg len: 797.8 num_loss_counted_tokens: 7448
 total length: 7281 dropped: 0 num samples 22 - rank: 7 max len: 399 min len: 242 avg len: 330.95454545454544 num_loss_counted_tokens: 6115
 total length: 7995 dropped: 0 num samples 14 - rank: 3 max len: 597 min len: 459 avg len: 571.0714285714286 num_loss_counted_tokens: 7253
 total length: 7885 dropped: 0 num samples 15 - rank: 4 max len: 550 min len: 504 avg len: 525.6666666666666 num_loss_counted_tokens: 7090
 total length: 7977 dropped: 0 num samples 19 - rank: 6 max len: 446 min len: 329 avg len: 419.8421052631579 num_loss_counted_tokens: 6970
 total length: 7970 dropped: 0 num samples 12 - rank: 1 max len: 694 min len: 643 avg len: 664.1666666666666 num_loss_counted_tokens: 7334
 total length: 7992 dropped: 0 num samples 17 - rank: 5 max len: 501 min len: 338 avg len: 470.11764705882354 num_loss_counted_tokens: 7091
Epoch: 4, Step: 164, Rank: 6, loss = 0.05126953125Epoch: 4, Step: 164, Rank: 7, loss = 0.0625Epoch: 4, Step: 164, Rank: 5, loss = 0.0419921875Epoch: 4, Step: 164, Rank: 0, loss = 0.015380859375Epoch: 4, Step: 164, Rank: 1, loss = 0.040771484375Epoch: 4, Step: 164, Rank: 2, loss = 0.03466796875





Epoch: 4, Step: 164, Rank: 4, loss = 0.034912109375
Epoch: 4, Step: 164, Rank: 3, loss = 0.03515625

Epoch 4:  10%|█         | 4/40 [00:06<01:01,  1.70s/it]{
    "epoch": 4,
    "step": 164,
    "rank": 0,
    "overall_throughput": 94.27191361281872,
    "lr": 8.58793848176909e-06,
    "cuda_mem_allocated": 6.567252159118652,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56915,
    "batch_size": 120,
    "total_loss": 0.03957656153913731,
    "samples_seen": 20750,
    "gradnorm": 0.255859375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:39:43.153145"
}
 total length: 7513 dropped: 0 num samples 24 - rank: 7 max len: 352 min len: 257 avg len: 313.0416666666667 num_loss_counted_tokens: 6241
 total length: 7927 dropped: 0 num samples 13 - rank: 2 max len: 630 min len: 579 avg len: 609.7692307692307 num_loss_counted_tokens: 7238
 total length: 7803 dropped: 0 num samples 12 - rank: 1 max len: 665 min len: 633 avg len: 650.25 num_loss_counted_tokens: 7167
 total length: 7996 dropped: 0 num samples 9 - rank: 0 max len: 1196 min len: 472 avg len: 888.4444444444445 num_loss_counted_tokens: 7519
 total length: 7880 dropped: 0 num samples 17 - rank: 4 max len: 497 min len: 435 avg len: 463.52941176470586 num_loss_counted_tokens: 6979
 total length: 7780 dropped: 0 num samples 21 - rank: 6 max len: 388 min len: 352 avg len: 370.4761904761905 num_loss_counted_tokens: 6667
 total length: 7982 dropped: 0 num samples 15 - rank: 3 max len: 568 min len: 497 avg len: 532.1333333333333 num_loss_counted_tokens: 7187
 total length: 7992 dropped: 0 num samples 20 - rank: 5 max len: 427 min len: 280 avg len: 399.6 num_loss_counted_tokens: 6932
Epoch: 4, Step: 165, Rank: 0, loss = 0.02294921875Epoch: 4, Step: 165, Rank: 3, loss = 0.038818359375Epoch: 4, Step: 165, Rank: 7, loss = 0.0546875Epoch: 4, Step: 165, Rank: 5, loss = 0.037353515625Epoch: 4, Step: 165, Rank: 1, loss = 0.0308837890625



Epoch: 4, Step: 165, Rank: 4, loss = 0.04638671875

Epoch: 4, Step: 165, Rank: 2, loss = 0.0306396484375Epoch: 4, Step: 165, Rank: 6, loss = 0.048095703125


Epoch 4:  12%|█▎        | 5/40 [00:08<01:02,  1.78s/it]{
    "epoch": 4,
    "step": 165,
    "rank": 0,
    "overall_throughput": 78.73464444593706,
    "lr": 8.466083451213145e-06,
    "cuda_mem_allocated": 6.5763139724731445,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56467,
    "batch_size": 126,
    "total_loss": 0.038748295464607646,
    "samples_seen": 20876,
    "gradnorm": 0.322265625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:39:45.072542"
}
 total length: 7828 dropped: 0 num samples 24 - rank: 7 max len: 381 min len: 268 avg len: 326.1666666666667 num_loss_counted_tokens: 6556
 total length: 7996 dropped: 0 num samples 20 - rank: 6 max len: 429 min len: 305 avg len: 399.8 num_loss_counted_tokens: 6936
 total length: 7988 dropped: 0 num samples 16 - rank: 3 max len: 526 min len: 320 avg len: 499.25 num_loss_counted_tokens: 7140
 total length: 7996 dropped: 0 num samples 13 - rank: 1 max len: 653 min len: 362 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 467 min len: 330 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7928 dropped: 0 num samples 15 - rank: 2 max len: 580 min len: 180 avg len: 528.5333333333333 num_loss_counted_tokens: 7133
 total length: 7982 dropped: 0 num samples 11 - rank: 0 max len: 888 min len: 591 avg len: 725.6363636363636 num_loss_counted_tokens: 7399
 total length: 7996 dropped: 0 num samples 17 - rank: 4 max len: 490 min len: 387 avg len: 470.3529411764706 num_loss_counted_tokens: 7095
Epoch: 4, Step: 166, Rank: 2, loss = 0.0322265625Epoch: 4, Step: 166, Rank: 7, loss = 0.06005859375Epoch: 4, Step: 166, Rank: 6, loss = 0.05078125Epoch: 4, Step: 166, Rank: 5, loss = 0.0419921875

Epoch: 4, Step: 166, Rank: 1, loss = 0.03466796875Epoch: 4, Step: 166, Rank: 4, loss = 0.051513671875Epoch: 4, Step: 166, Rank: 3, loss = 0.0400390625




Epoch: 4, Step: 166, Rank: 0, loss = 0.024658203125

Epoch 4:  15%|█▌        | 6/40 [00:10<00:58,  1.73s/it]{
    "epoch": 4,
    "step": 166,
    "rank": 0,
    "overall_throughput": 95.60099619890215,
    "lr": 8.3444612381587e-06,
    "cuda_mem_allocated": 6.568479061126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 55993,
    "batch_size": 130,
    "total_loss": 0.04196953190577394,
    "samples_seen": 21006,
    "gradnorm": 0.275390625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:39:46.705157"
}
 total length: 7890 dropped: 0 num samples 12 - rank: 1 max len: 698 min len: 632 avg len: 657.5 num_loss_counted_tokens: 7254
 total length: 7907 dropped: 0 num samples 13 - rank: 2 max len: 625 min len: 581 avg len: 608.2307692307693 num_loss_counted_tokens: 7218
 total length: 7880 dropped: 0 num samples 9 - rank: 0 max len: 1247 min len: 699 avg len: 875.5555555555555 num_loss_counted_tokens: 7403
 total length: 7984 dropped: 0 num samples 15 - rank: 3 max len: 573 min len: 257 avg len: 532.2666666666667 num_loss_counted_tokens: 7189
 total length: 7946 dropped: 0 num samples 20 - rank: 6 max len: 443 min len: 363 avg len: 397.3 num_loss_counted_tokens: 6886
 total length: 7988 dropped: 0 num samples 16 - rank: 4 max len: 530 min len: 425 avg len: 499.25 num_loss_counted_tokens: 7140
 total length: 7821 dropped: 0 num samples 17 - rank: 5 max len: 473 min len: 444 avg len: 460.05882352941177 num_loss_counted_tokens: 6920
 total length: 7643 dropped: 0 num samples 23 - rank: 7 max len: 361 min len: 250 avg len: 332.30434782608694 num_loss_counted_tokens: 6424
Epoch: 4, Step: 167, Rank: 6, loss = 0.0517578125Epoch: 4, Step: 167, Rank: 5, loss = 0.04638671875Epoch: 4, Step: 167, Rank: 3, loss = 0.0322265625Epoch: 4, Step: 167, Rank: 2, loss = 0.03125Epoch: 4, Step: 167, Rank: 7, loss = 0.056884765625Epoch: 4, Step: 167, Rank: 1, loss = 0.041259765625
Epoch: 4, Step: 167, Rank: 4, loss = 0.037109375





Epoch: 4, Step: 167, Rank: 0, loss = 0.018798828125

Epoch 4:  18%|█▊        | 7/40 [00:12<00:56,  1.70s/it]{
    "epoch": 4,
    "step": 167,
    "rank": 0,
    "overall_throughput": 94.9639345677368,
    "lr": 8.223090302397313e-06,
    "cuda_mem_allocated": 6.578819751739502,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56636,
    "batch_size": 126,
    "total_loss": 0.03942721943640087,
    "samples_seen": 21132,
    "gradnorm": 0.251953125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:39:48.345432"
}
 total length: 7885 dropped: 0 num samples 15 - rank: 3 max len: 550 min len: 511 avg len: 525.6666666666666 num_loss_counted_tokens: 7090
 total length: 7921 dropped: 0 num samples 19 - rank: 5 max len: 434 min len: 399 avg len: 416.89473684210526 num_loss_counted_tokens: 6914
 total length: 7997 dropped: 0 num samples 21 - rank: 6 max len: 396 min len: 354 avg len: 380.8095238095238 num_loss_counted_tokens: 6884
 total length: 7991 dropped: 0 num samples 10 - rank: 0 max len: 1068 min len: 318 avg len: 799.1 num_loss_counted_tokens: 7461
 total length: 7986 dropped: 0 num samples 17 - rank: 4 max len: 501 min len: 442 avg len: 469.7647058823529 num_loss_counted_tokens: 7085
 total length: 7851 dropped: 0 num samples 26 - rank: 7 max len: 362 min len: 172 avg len: 301.96153846153845 num_loss_counted_tokens: 6473
 total length: 7997 dropped: 0 num samples 13 - rank: 1 max len: 655 min len: 392 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7992 dropped: 0 num samples 14 - rank: 2 max len: 595 min len: 532 avg len: 570.8571428571429 num_loss_counted_tokens: 7250
Epoch: 4, Step: 168, Rank: 0, loss = 0.0155029296875Epoch: 4, Step: 168, Rank: 6, loss = 0.0498046875Epoch: 4, Step: 168, Rank: 7, loss = 0.049072265625Epoch: 4, Step: 168, Rank: 4, loss = 0.0478515625
Epoch: 4, Step: 168, Rank: 5, loss = 0.046875Epoch: 4, Step: 168, Rank: 2, loss = 0.04638671875



Epoch: 4, Step: 168, Rank: 3, loss = 0.035888671875

Epoch: 4, Step: 168, Rank: 1, loss = 0.035400390625

Epoch 4:  20%|██        | 8/40 [00:13<00:53,  1.69s/it]{
    "epoch": 4,
    "step": 168,
    "rank": 0,
    "overall_throughput": 94.43250919839008,
    "lr": 8.101989065581742e-06,
    "cuda_mem_allocated": 6.568479061126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56458,
    "batch_size": 125,
    "total_loss": 0.040835665450423324,
    "samples_seen": 21257,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:39:49.999653"
}
 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 522 min len: 357 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7996 dropped: 0 num samples 21 - rank: 6 max len: 415 min len: 344 avg len: 380.76190476190476 num_loss_counted_tokens: 6883
 total length: 7787 dropped: 0 num samples 14 - rank: 3 max len: 587 min len: 523 avg len: 556.2142857142857 num_loss_counted_tokens: 7045
 total length: 7757 dropped: 0 num samples 17 - rank: 5 max len: 496 min len: 419 avg len: 456.29411764705884 num_loss_counted_tokens: 6856
 total length: 7990 dropped: 0 num samples 13 - rank: 2 max len: 632 min len: 573 avg len: 614.6153846153846 num_loss_counted_tokens: 7301
 total length: 7997 dropped: 0 num samples 10 - rank: 0 max len: 1098 min len: 574 avg len: 799.7 num_loss_counted_tokens: 7467
 total length: 7931 dropped: 0 num samples 25 - rank: 7 max len: 363 min len: 245 avg len: 317.24 num_loss_counted_tokens: 6606
 total length: 7896 dropped: 0 num samples 12 - rank: 1 max len: 684 min len: 633 avg len: 658.0 num_loss_counted_tokens: 7260
Epoch: 4, Step: 169, Rank: 6, loss = 0.0498046875Epoch: 4, Step: 169, Rank: 5, loss = 0.050537109375Epoch: 4, Step: 169, Rank: 3, loss = 0.036865234375
Epoch: 4, Step: 169, Rank: 4, loss = 0.039306640625
Epoch: 4, Step: 169, Rank: 2, loss = 0.03759765625Epoch: 4, Step: 169, Rank: 7, loss = 0.05078125
Epoch: 4, Step: 169, Rank: 1, loss = 0.031005859375



Epoch: 4, Step: 169, Rank: 0, loss = 0.0283203125

Epoch 4:  22%|██▎       | 9/40 [00:15<00:54,  1.76s/it]{
    "epoch": 4,
    "step": 169,
    "rank": 0,
    "overall_throughput": 77.9339704261647,
    "lr": 7.9811759084299e-06,
    "cuda_mem_allocated": 6.587082386016846,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56440,
    "batch_size": 126,
    "total_loss": 0.04055634301913536,
    "samples_seen": 21383,
    "gradnorm": 0.265625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:39:51.938814"
}
 total length: 7984 dropped: 0 num samples 9 - rank: 0 max len: 1482 min len: 703 avg len: 887.1111111111111 num_loss_counted_tokens: 7507
 total length: 7994 dropped: 0 num samples 13 - rank: 2 max len: 640 min len: 583 avg len: 614.9230769230769 num_loss_counted_tokens: 7305
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 581 min len: 298 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7993 dropped: 0 num samples 20 - rank: 6 max len: 424 min len: 281 avg len: 399.65 num_loss_counted_tokens: 6933
 total length: 7858 dropped: 0 num samples 17 - rank: 5 max len: 490 min len: 424 avg len: 462.2352941176471 num_loss_counted_tokens: 6957
 total length: 7865 dropped: 0 num samples 12 - rank: 1 max len: 682 min len: 642 avg len: 655.4166666666666 num_loss_counted_tokens: 7229
 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 523 min len: 344 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7844 dropped: 0 num samples 24 - rank: 7 max len: 386 min len: 253 avg len: 326.8333333333333 num_loss_counted_tokens: 6572
Epoch: 4, Step: 170, Rank: 5, loss = 0.035400390625Epoch: 4, Step: 170, Rank: 1, loss = 0.02685546875Epoch: 4, Step: 170, Rank: 6, loss = 0.05078125Epoch: 4, Step: 170, Rank: 3, loss = 0.04541015625Epoch: 4, Step: 170, Rank: 7, loss = 0.0517578125Epoch: 4, Step: 170, Rank: 4, loss = 0.0361328125
Epoch: 4, Step: 170, Rank: 2, loss = 0.03173828125





Epoch: 4, Step: 170, Rank: 0, loss = 0.030517578125

Epoch 4:  25%|██▌       | 10/40 [00:17<00:54,  1.82s/it]{
    "epoch": 4,
    "step": 170,
    "rank": 0,
    "overall_throughput": 78.21171042355341,
    "lr": 7.860669167935028e-06,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56832,
    "batch_size": 126,
    "total_loss": 0.0385522240990991,
    "samples_seen": 21509,
    "gradnorm": 0.26953125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:39:53.868564"
}
 total length: 7899 dropped: 0 num samples 13 - rank: 3 max len: 635 min len: 576 avg len: 607.6153846153846 num_loss_counted_tokens: 7210
 total length: 7997 dropped: 0 num samples 11 - rank: 1 max len: 850 min len: 344 avg len: 727.0 num_loss_counted_tokens: 7414
 total length: 7988 dropped: 0 num samples 8 - rank: 0 max len: 1110 min len: 700 avg len: 998.5 num_loss_counted_tokens: 7564
 total length: 7980 dropped: 0 num samples 17 - rank: 5 max len: 516 min len: 437 avg len: 469.4117647058824 num_loss_counted_tokens: 7079
 total length: 7971 dropped: 0 num samples 23 - rank: 7 max len: 383 min len: 263 avg len: 346.5652173913044 num_loss_counted_tokens: 6752
 total length: 7839 dropped: 0 num samples 12 - rank: 2 max len: 676 min len: 636 avg len: 653.25 num_loss_counted_tokens: 7203
 total length: 7994 dropped: 0 num samples 20 - rank: 6 max len: 431 min len: 275 avg len: 399.7 num_loss_counted_tokens: 6934
 total length: 7993 dropped: 0 num samples 15 - rank: 4 max len: 572 min len: 366 avg len: 532.8666666666667 num_loss_counted_tokens: 7198
Epoch: 4, Step: 171, Rank: 6, loss = 0.053955078125Epoch: 4, Step: 171, Rank: 7, loss = 0.06396484375Epoch: 4, Step: 171, Rank: 2, loss = 0.032470703125Epoch: 4, Step: 171, Rank: 5, loss = 0.0419921875Epoch: 4, Step: 171, Rank: 3, loss = 0.02978515625Epoch: 4, Step: 171, Rank: 1, loss = 0.0419921875Epoch: 4, Step: 171, Rank: 4, loss = 0.040283203125






Epoch: 4, Step: 171, Rank: 0, loss = 0.021484375

Epoch 4:  28%|██▊       | 11/40 [00:19<00:51,  1.77s/it]{
    "epoch": 4,
    "step": 171,
    "rank": 0,
    "overall_throughput": 94.17071379445045,
    "lr": 7.740487134582527e-06,
    "cuda_mem_allocated": 6.588063716888428,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56828,
    "batch_size": 123,
    "total_loss": 0.040789751530935454,
    "samples_seen": 21632,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:39:55.520203"
}
 total length: 7643 dropped: 0 num samples 23 - rank: 7 max len: 370 min len: 233 avg len: 332.30434782608694 num_loss_counted_tokens: 6424
 total length: 7996 dropped: 0 num samples 14 - rank: 3 max len: 609 min len: 421 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7996 dropped: 0 num samples 10 - rank: 0 max len: 1056 min len: 501 avg len: 799.6 num_loss_counted_tokens: 7466
 total length: 7994 dropped: 0 num samples 13 - rank: 2 max len: 649 min len: 434 avg len: 614.9230769230769 num_loss_counted_tokens: 7305
 total length: 7997 dropped: 0 num samples 20 - rank: 6 max len: 429 min len: 355 avg len: 399.85 num_loss_counted_tokens: 6937
 total length: 7769 dropped: 0 num samples 17 - rank: 5 max len: 477 min len: 442 avg len: 457.0 num_loss_counted_tokens: 6868
 total length: 7775 dropped: 0 num samples 15 - rank: 4 max len: 546 min len: 477 avg len: 518.3333333333334 num_loss_counted_tokens: 6980
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 702 min len: 552 avg len: 666.25 num_loss_counted_tokens: 7359
Epoch: 4, Step: 172, Rank: 5, loss = 0.04736328125Epoch: 4, Step: 172, Rank: 2, loss = 0.037841796875Epoch: 4, Step: 172, Rank: 3, loss = 0.046875Epoch: 4, Step: 172, Rank: 6, loss = 0.055419921875Epoch: 4, Step: 172, Rank: 7, loss = 0.051025390625


Epoch: 4, Step: 172, Rank: 1, loss = 0.0283203125Epoch: 4, Step: 172, Rank: 4, loss = 0.03564453125



Epoch: 4, Step: 172, Rank: 0, loss = 0.019775390625

Epoch 4:  30%|███       | 12/40 [00:20<00:48,  1.73s/it]{
    "epoch": 4,
    "step": 172,
    "rank": 0,
    "overall_throughput": 94.59780311172977,
    "lr": 7.620648049573815e-06,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56665,
    "batch_size": 128,
    "total_loss": 0.04030706785493691,
    "samples_seen": 21760,
    "gradnorm": 0.265625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:39:57.162272"
}
 total length: 7939 dropped: 0 num samples 20 - rank: 6 max len: 418 min len: 376 avg len: 396.95 num_loss_counted_tokens: 6879
 total length: 7985 dropped: 0 num samples 13 - rank: 2 max len: 651 min len: 461 avg len: 614.2307692307693 num_loss_counted_tokens: 7296
 total length: 7946 dropped: 0 num samples 25 - rank: 7 max len: 370 min len: 253 avg len: 317.84 num_loss_counted_tokens: 6621
 total length: 7988 dropped: 0 num samples 14 - rank: 3 max len: 602 min len: 491 avg len: 570.5714285714286 num_loss_counted_tokens: 7246
 total length: 7995 dropped: 0 num samples 10 - rank: 0 max len: 1059 min len: 312 avg len: 799.5 num_loss_counted_tokens: 7465
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 486 min len: 262 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7768 dropped: 0 num samples 15 - rank: 4 max len: 543 min len: 486 avg len: 517.8666666666667 num_loss_counted_tokens: 6973
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 697 min len: 615 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
Epoch: 4, Step: 173, Rank: 5, loss = 0.048828125Epoch: 4, Step: 173, Rank: 3, loss = 0.040771484375Epoch: 4, Step: 173, Rank: 7, loss = 0.050048828125Epoch: 4, Step: 173, Rank: 6, loss = 0.051513671875
Epoch: 4, Step: 173, Rank: 2, loss = 0.032470703125

Epoch: 4, Step: 173, Rank: 1, loss = 0.0341796875


Epoch: 4, Step: 173, Rank: 4, loss = 0.036376953125
Epoch: 4, Step: 173, Rank: 0, loss = 0.0218505859375

Epoch 4:  32%|███▎      | 13/40 [00:22<00:46,  1.71s/it]{
    "epoch": 4,
    "step": 173,
    "rank": 0,
    "overall_throughput": 93.95784741041888,
    "lr": 7.501170102057691e-06,
    "cuda_mem_allocated": 6.588062763214111,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56258,
    "batch_size": 128,
    "total_loss": 0.03949660492729923,
    "samples_seen": 21888,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:39:58.818057"
}
 total length: 7840 dropped: 0 num samples 12 - rank: 1 max len: 669 min len: 642 avg len: 653.3333333333334 num_loss_counted_tokens: 7204
 total length: 7996 dropped: 0 num samples 15 - rank: 3 max len: 591 min len: 353 avg len: 533.0666666666667 num_loss_counted_tokens: 7201
 total length: 7614 dropped: 0 num samples 24 - rank: 7 max len: 361 min len: 245 avg len: 317.25 num_loss_counted_tokens: 6342
 total length: 7923 dropped: 0 num samples 20 - rank: 6 max len: 428 min len: 364 avg len: 396.15 num_loss_counted_tokens: 6863
 total length: 7991 dropped: 0 num samples 10 - rank: 0 max len: 1058 min len: 301 avg len: 799.1 num_loss_counted_tokens: 7461
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 469 min len: 298 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7993 dropped: 0 num samples 13 - rank: 2 max len: 640 min len: 572 avg len: 614.8461538461538 num_loss_counted_tokens: 7304
 total length: 7918 dropped: 0 num samples 16 - rank: 4 max len: 518 min len: 477 avg len: 494.875 num_loss_counted_tokens: 7070
Epoch: 4, Step: 174, Rank: 6, loss = 0.050537109375Epoch: 4, Step: 174, Rank: 1, loss = 0.0400390625Epoch: 4, Step: 174, Rank: 7, loss = 0.05029296875Epoch: 4, Step: 174, Rank: 5, loss = 0.0458984375Epoch: 4, Step: 174, Rank: 3, loss = 0.04150390625

Epoch: 4, Step: 174, Rank: 2, loss = 0.031005859375



Epoch: 4, Step: 174, Rank: 4, loss = 0.038818359375
Epoch: 4, Step: 174, Rank: 0, loss = 0.030029296875

Epoch 4:  35%|███▌      | 14/40 [00:24<00:43,  1.69s/it]{
    "epoch": 4,
    "step": 174,
    "rank": 0,
    "overall_throughput": 93.90484679509828,
    "lr": 7.382071426369597e-06,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56742,
    "batch_size": 131,
    "total_loss": 0.04101018645800289,
    "samples_seen": 22019,
    "gradnorm": 0.31640625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:00.471235"
}
 total length: 7990 dropped: 0 num samples 9 - rank: 0 max len: 1068 min len: 638 avg len: 887.7777777777778 num_loss_counted_tokens: 7513
 total length: 7986 dropped: 0 num samples 15 - rank: 3 max len: 580 min len: 250 avg len: 532.4 num_loss_counted_tokens: 7191
 total length: 7872 dropped: 0 num samples 13 - rank: 2 max len: 636 min len: 580 avg len: 605.5384615384615 num_loss_counted_tokens: 7183
 total length: 7918 dropped: 0 num samples 16 - rank: 4 max len: 516 min len: 476 avg len: 494.875 num_loss_counted_tokens: 7070
 total length: 7996 dropped: 0 num samples 20 - rank: 6 max len: 422 min len: 338 avg len: 399.8 num_loss_counted_tokens: 6936
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 756 min len: 514 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
 total length: 7981 dropped: 0 num samples 25 - rank: 7 max len: 379 min len: 240 avg len: 319.24 num_loss_counted_tokens: 6656
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 473 min len: 367 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
Epoch: 4, Step: 175, Rank: 6, loss = 0.05322265625Epoch: 4, Step: 175, Rank: 1, loss = 0.04345703125Epoch: 4, Step: 175, Rank: 5, loss = 0.044677734375Epoch: 4, Step: 175, Rank: 3, loss = 0.035888671875Epoch: 4, Step: 175, Rank: 7, loss = 0.056640625Epoch: 4, Step: 175, Rank: 2, loss = 0.045166015625

Epoch: 4, Step: 175, Rank: 4, loss = 0.053466796875




Epoch: 4, Step: 175, Rank: 0, loss = 0.028564453125

Epoch 4:  38%|███▊      | 15/40 [00:25<00:41,  1.68s/it]{
    "epoch": 4,
    "step": 175,
    "rank": 0,
    "overall_throughput": 94.23873514900322,
    "lr": 7.263370099279173e-06,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56546,
    "batch_size": 130,
    "total_loss": 0.04516676688006225,
    "samples_seen": 22149,
    "gradnorm": 0.296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:02.121654"
}
 total length: 7997 dropped: 0 num samples 9 - rank: 0 max len: 1391 min len: 482 avg len: 888.5555555555555 num_loss_counted_tokens: 7520
 total length: 7992 dropped: 0 num samples 13 - rank: 2 max len: 650 min len: 418 avg len: 614.7692307692307 num_loss_counted_tokens: 7303
 total length: 7703 dropped: 0 num samples 24 - rank: 7 max len: 369 min len: 248 avg len: 320.9583333333333 num_loss_counted_tokens: 6431
 total length: 7995 dropped: 0 num samples 14 - rank: 3 max len: 607 min len: 442 avg len: 571.0714285714286 num_loss_counted_tokens: 7253
 total length: 7872 dropped: 0 num samples 19 - rank: 6 max len: 457 min len: 370 avg len: 414.3157894736842 num_loss_counted_tokens: 6865
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 740 min len: 443 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
 total length: 7795 dropped: 0 num samples 16 - rank: 5 max len: 514 min len: 460 avg len: 487.1875 num_loss_counted_tokens: 6947
 total length: 7927 dropped: 0 num samples 15 - rank: 4 max len: 553 min len: 515 avg len: 528.4666666666667 num_loss_counted_tokens: 7132
Epoch: 4, Step: 176, Rank: 5, loss = 0.046630859375Epoch: 4, Step: 176, Rank: 1, loss = 0.038330078125Epoch: 4, Step: 176, Rank: 3, loss = 0.037353515625Epoch: 4, Step: 176, Rank: 2, loss = 0.035400390625


Epoch: 4, Step: 176, Rank: 6, loss = 0.060546875Epoch: 4, Step: 176, Rank: 4, loss = 0.041015625
Epoch: 4, Step: 176, Rank: 7, loss = 0.056396484375


Epoch: 4, Step: 176, Rank: 0, loss = 0.020263671875

Epoch 4:  40%|████      | 16/40 [00:27<00:40,  1.67s/it]{
    "epoch": 4,
    "step": 176,
    "rank": 0,
    "overall_throughput": 93.99848060530668,
    "lr": 7.1450841372465806e-06,
    "cuda_mem_allocated": 6.575035572052002,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56526,
    "batch_size": 124,
    "total_loss": 0.04198068145632099,
    "samples_seen": 22273,
    "gradnorm": 0.2890625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:03.773539"
}
 total length: 7991 dropped: 0 num samples 16 - rank: 4 max len: 530 min len: 382 avg len: 499.4375 num_loss_counted_tokens: 7143
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 715 min len: 571 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7981 dropped: 0 num samples 9 - rank: 0 max len: 1074 min len: 654 avg len: 886.7777777777778 num_loss_counted_tokens: 7504
 total length: 7914 dropped: 0 num samples 26 - rank: 7 max len: 350 min len: 164 avg len: 304.38461538461536 num_loss_counted_tokens: 6536
 total length: 7997 dropped: 0 num samples 14 - rank: 3 max len: 626 min len: 526 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7899 dropped: 0 num samples 17 - rank: 5 max len: 484 min len: 440 avg len: 464.6470588235294 num_loss_counted_tokens: 6998
 total length: 7985 dropped: 0 num samples 13 - rank: 2 max len: 641 min len: 383 avg len: 614.2307692307693 num_loss_counted_tokens: 7296
 total length: 7896 dropped: 0 num samples 20 - rank: 6 max len: 434 min len: 356 avg len: 394.8 num_loss_counted_tokens: 6836
Epoch: 4, Step: 177, Rank: 5, loss = 0.05078125Epoch: 4, Step: 177, Rank: 6, loss = 0.048828125Epoch: 4, Step: 177, Rank: 3, loss = 0.030029296875Epoch: 4, Step: 177, Rank: 0, loss = 0.02392578125

Epoch: 4, Step: 177, Rank: 4, loss = 0.034912109375Epoch: 4, Step: 177, Rank: 2, loss = 0.048828125
Epoch: 4, Step: 177, Rank: 7, loss = 0.041748046875



Epoch: 4, Step: 177, Rank: 1, loss = 0.03564453125

Epoch 4:  42%|████▎     | 17/40 [00:29<00:38,  1.66s/it]{
    "epoch": 4,
    "step": 177,
    "rank": 0,
    "overall_throughput": 94.38371723222905,
    "lr": 7.027231493687974e-06,
    "cuda_mem_allocated": 6.573036193847656,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56498,
    "batch_size": 125,
    "total_loss": 0.03927572657439202,
    "samples_seen": 22398,
    "gradnorm": 0.271484375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:05.421615"
}
 total length: 7849 dropped: 0 num samples 14 - rank: 3 max len: 584 min len: 533 avg len: 560.6428571428571 num_loss_counted_tokens: 7107
 total length: 7835 dropped: 0 num samples 24 - rank: 7 max len: 375 min len: 243 avg len: 326.4583333333333 num_loss_counted_tokens: 6563
 total length: 7992 dropped: 0 num samples 9 - rank: 0 max len: 1094 min len: 496 avg len: 888.0 num_loss_counted_tokens: 7515
 total length: 7808 dropped: 0 num samples 17 - rank: 5 max len: 477 min len: 448 avg len: 459.29411764705884 num_loss_counted_tokens: 6907
 total length: 7878 dropped: 0 num samples 19 - rank: 6 max len: 445 min len: 377 avg len: 414.63157894736844 num_loss_counted_tokens: 6871
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 643 min len: 510 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7994 dropped: 0 num samples 16 - rank: 4 max len: 532 min len: 314 avg len: 499.625 num_loss_counted_tokens: 7146
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 736 min len: 481 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
Epoch: 4, Step: 178, Rank: 0, loss = 0.051025390625Epoch: 4, Step: 178, Rank: 2, loss = 0.031494140625Epoch: 4, Step: 178, Rank: 3, loss = 0.033935546875Epoch: 4, Step: 178, Rank: 6, loss = 0.051025390625Epoch: 4, Step: 178, Rank: 4, loss = 0.03369140625Epoch: 4, Step: 178, Rank: 5, loss = 0.047607421875




Epoch: 4, Step: 178, Rank: 1, loss = 0.025146484375

Epoch: 4, Step: 178, Rank: 7, loss = 0.055908203125

Epoch 4:  45%|████▌     | 18/40 [00:30<00:36,  1.66s/it]{
    "epoch": 4,
    "step": 178,
    "rank": 0,
    "overall_throughput": 94.79253928700636,
    "lr": 6.909830056250527e-06,
    "cuda_mem_allocated": 6.567160606384277,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56557,
    "batch_size": 129,
    "total_loss": 0.04123273865304029,
    "samples_seen": 22527,
    "gradnorm": 0.28125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:07.066843"
}
 total length: 7986 dropped: 0 num samples 16 - rank: 3 max len: 545 min len: 243 avg len: 499.125 num_loss_counted_tokens: 7138
 total length: 7979 dropped: 0 num samples 9 - rank: 0 max len: 1104 min len: 679 avg len: 886.5555555555555 num_loss_counted_tokens: 7502
 total length: 7765 dropped: 0 num samples 24 - rank: 7 max len: 364 min len: 271 avg len: 323.5416666666667 num_loss_counted_tokens: 6493
 total length: 7989 dropped: 0 num samples 14 - rank: 2 max len: 619 min len: 306 avg len: 570.6428571428571 num_loss_counted_tokens: 7247
 total length: 7964 dropped: 0 num samples 17 - rank: 4 max len: 481 min len: 451 avg len: 468.47058823529414 num_loss_counted_tokens: 7063
 total length: 7795 dropped: 0 num samples 20 - rank: 6 max len: 418 min len: 365 avg len: 389.75 num_loss_counted_tokens: 6735
 total length: 7857 dropped: 0 num samples 12 - rank: 1 max len: 707 min len: 619 avg len: 654.75 num_loss_counted_tokens: 7221
 total length: 7853 dropped: 0 num samples 18 - rank: 5 max len: 450 min len: 418 avg len: 436.27777777777777 num_loss_counted_tokens: 6899
Epoch: 4, Step: 179, Rank: 6, loss = 0.0517578125Epoch: 4, Step: 179, Rank: 1, loss = 0.02978515625Epoch: 4, Step: 179, Rank: 7, loss = 0.060546875Epoch: 4, Step: 179, Rank: 5, loss = 0.04833984375Epoch: 4, Step: 179, Rank: 2, loss = 0.033935546875Epoch: 4, Step: 179, Rank: 3, loss = 0.031494140625Epoch: 4, Step: 179, Rank: 4, loss = 0.041015625






Epoch: 4, Step: 179, Rank: 0, loss = 0.01519775390625

Epoch 4:  48%|████▊     | 19/40 [00:32<00:34,  1.66s/it]{
    "epoch": 4,
    "step": 179,
    "rank": 0,
    "overall_throughput": 93.98174600201453,
    "lr": 6.7928976440974504e-06,
    "cuda_mem_allocated": 6.584146976470947,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56911,
    "batch_size": 125,
    "total_loss": 0.03900827608019539,
    "samples_seen": 22652,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:08.718996"
}
 total length: 7876 dropped: 0 num samples 24 - rank: 7 max len: 378 min len: 268 avg len: 328.1666666666667 num_loss_counted_tokens: 6604
 total length: 7964 dropped: 0 num samples 8 - rank: 0 max len: 1104 min len: 722 avg len: 995.5 num_loss_counted_tokens: 7540
 total length: 7983 dropped: 0 num samples 14 - rank: 3 max len: 616 min len: 272 avg len: 570.2142857142857 num_loss_counted_tokens: 7241
 total length: 7768 dropped: 0 num samples 19 - rank: 6 max len: 434 min len: 388 avg len: 408.8421052631579 num_loss_counted_tokens: 6761
 total length: 7983 dropped: 0 num samples 17 - rank: 5 max len: 505 min len: 434 avg len: 469.5882352941176 num_loss_counted_tokens: 7082
 total length: 7991 dropped: 0 num samples 13 - rank: 2 max len: 673 min len: 281 avg len: 614.6923076923077 num_loss_counted_tokens: 7302
 total length: 7993 dropped: 0 num samples 15 - rank: 4 max len: 564 min len: 498 avg len: 532.8666666666667 num_loss_counted_tokens: 7198
 total length: 7995 dropped: 0 num samples 11 - rank: 1 max len: 858 min len: 372 avg len: 726.8181818181819 num_loss_counted_tokens: 7412
Epoch: 4, Step: 180, Rank: 6, loss = 0.049072265625Epoch: 4, Step: 180, Rank: 3, loss = 0.031982421875Epoch: 4, Step: 180, Rank: 7, loss = 0.052734375Epoch: 4, Step: 180, Rank: 5, loss = 0.0458984375

Epoch: 4, Step: 180, Rank: 2, loss = 0.0262451171875


Epoch: 4, Step: 180, Rank: 4, loss = 0.0361328125Epoch: 4, Step: 180, Rank: 0, loss = 0.020263671875

Epoch: 4, Step: 180, Rank: 1, loss = 0.030517578125

Epoch 4:  50%|█████     | 20/40 [00:34<00:33,  1.65s/it]{
    "epoch": 4,
    "step": 180,
    "rank": 0,
    "overall_throughput": 94.45149840141546,
    "lr": 6.6764520052034054e-06,
    "cuda_mem_allocated": 6.58611536026001,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56607,
    "batch_size": 122,
    "total_loss": 0.036620912607981344,
    "samples_seen": 22774,
    "gradnorm": 0.26953125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:10.368439"
}
 total length: 7858 dropped: 0 num samples 24 - rank: 7 max len: 366 min len: 252 avg len: 327.4166666666667 num_loss_counted_tokens: 6586
 total length: 7997 dropped: 0 num samples 10 - rank: 0 max len: 1072 min len: 468 avg len: 799.7 num_loss_counted_tokens: 7467
 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 638 min len: 597 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7789 dropped: 0 num samples 20 - rank: 6 max len: 411 min len: 366 avg len: 389.45 num_loss_counted_tokens: 6729
 total length: 7928 dropped: 0 num samples 14 - rank: 3 max len: 596 min len: 534 avg len: 566.2857142857143 num_loss_counted_tokens: 7186
 total length: 7876 dropped: 0 num samples 18 - rank: 5 max len: 463 min len: 411 avg len: 437.55555555555554 num_loss_counted_tokens: 6922
 total length: 7891 dropped: 0 num samples 16 - rank: 4 max len: 526 min len: 464 avg len: 493.1875 num_loss_counted_tokens: 7043
 total length: 7982 dropped: 0 num samples 12 - rank: 1 max len: 689 min len: 639 avg len: 665.1666666666666 num_loss_counted_tokens: 7346
Epoch: 4, Step: 181, Rank: 6, loss = 0.04296875Epoch: 4, Step: 181, Rank: 7, loss = 0.0615234375Epoch: 4, Step: 181, Rank: 1, loss = 0.030517578125Epoch: 4, Step: 181, Rank: 5, loss = 0.04541015625Epoch: 4, Step: 181, Rank: 3, loss = 0.034912109375
Epoch: 4, Step: 181, Rank: 2, loss = 0.037109375




Epoch: 4, Step: 181, Rank: 4, loss = 0.04931640625
Epoch: 4, Step: 181, Rank: 0, loss = 0.0233154296875

Epoch 4:  52%|█████▎    | 21/40 [00:35<00:31,  1.65s/it]{
    "epoch": 4,
    "step": 181,
    "rank": 0,
    "overall_throughput": 94.12950249275231,
    "lr": 6.560510813660719e-06,
    "cuda_mem_allocated": 6.588063716888428,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 55930,
    "batch_size": 131,
    "total_loss": 0.040622206329340244,
    "samples_seen": 22905,
    "gradnorm": 0.310546875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:12.020617"
}
 total length: 7956 dropped: 0 num samples 24 - rank: 7 max len: 381 min len: 260 avg len: 331.5 num_loss_counted_tokens: 6684
 total length: 7757 dropped: 0 num samples 14 - rank: 3 max len: 573 min len: 539 avg len: 554.0714285714286 num_loss_counted_tokens: 7015
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 481 min len: 263 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7740 dropped: 0 num samples 19 - rank: 6 max len: 427 min len: 385 avg len: 407.36842105263156 num_loss_counted_tokens: 6733
 total length: 7997 dropped: 0 num samples 11 - rank: 0 max len: 1076 min len: 411 avg len: 727.0 num_loss_counted_tokens: 7414
 total length: 7859 dropped: 0 num samples 12 - rank: 1 max len: 681 min len: 629 avg len: 654.9166666666666 num_loss_counted_tokens: 7223
 total length: 7858 dropped: 0 num samples 13 - rank: 2 max len: 629 min len: 574 avg len: 604.4615384615385 num_loss_counted_tokens: 7169
 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 534 min len: 423 avg len: 499.75 num_loss_counted_tokens: 7148
Epoch: 4, Step: 182, Rank: 6, loss = 0.051513671875Epoch: 4, Step: 182, Rank: 0, loss = 0.0284423828125Epoch: 4, Step: 182, Rank: 5, loss = 0.056884765625Epoch: 4, Step: 182, Rank: 3, loss = 0.036865234375
Epoch: 4, Step: 182, Rank: 7, loss = 0.06103515625Epoch: 4, Step: 182, Rank: 2, loss = 0.03955078125




Epoch: 4, Step: 182, Rank: 1, loss = 0.031494140625Epoch: 4, Step: 182, Rank: 4, loss = 0.041259765625


Epoch 4:  55%|█████▌    | 22/40 [00:37<00:29,  1.65s/it]{
    "epoch": 4,
    "step": 182,
    "rank": 0,
    "overall_throughput": 94.47312182924233,
    "lr": 6.44509166699682e-06,
    "cuda_mem_allocated": 6.58648157119751,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56609,
    "batch_size": 134,
    "total_loss": 0.04334999735024466,
    "samples_seen": 23039,
    "gradnorm": 0.302734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:13.662232"
}
 total length: 7935 dropped: 0 num samples 10 - rank: 0 max len: 942 min len: 717 avg len: 793.5 num_loss_counted_tokens: 7405
 total length: 7985 dropped: 0 num samples 13 - rank: 2 max len: 650 min len: 540 avg len: 614.2307692307693 num_loss_counted_tokens: 7296
 total length: 7881 dropped: 0 num samples 25 - rank: 7 max len: 356 min len: 262 avg len: 315.24 num_loss_counted_tokens: 6556
 total length: 7744 dropped: 0 num samples 20 - rank: 6 max len: 410 min len: 364 avg len: 387.2 num_loss_counted_tokens: 6684
 total length: 7847 dropped: 0 num samples 14 - rank: 3 max len: 588 min len: 524 avg len: 560.5 num_loss_counted_tokens: 7105
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 712 min len: 578 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
 total length: 7832 dropped: 0 num samples 16 - rank: 4 max len: 520 min len: 465 avg len: 489.5 num_loss_counted_tokens: 6984
 total length: 7812 dropped: 0 num samples 18 - rank: 5 max len: 464 min len: 411 avg len: 434.0 num_loss_counted_tokens: 6858
Epoch: 4, Step: 183, Rank: 6, loss = 0.047119140625Epoch: 4, Step: 183, Rank: 5, loss = 0.045166015625Epoch: 4, Step: 183, Rank: 3, loss = 0.03759765625Epoch: 4, Step: 183, Rank: 1, loss = 0.04736328125

Epoch: 4, Step: 183, Rank: 2, loss = 0.037353515625

Epoch: 4, Step: 183, Rank: 4, loss = 0.03564453125
Epoch: 4, Step: 183, Rank: 7, loss = 0.04931640625

Epoch: 4, Step: 183, Rank: 0, loss = 0.025634765625

Epoch 4:  57%|█████▊    | 23/40 [00:39<00:28,  1.65s/it]{
    "epoch": 4,
    "step": 183,
    "rank": 0,
    "overall_throughput": 94.72517043602934,
    "lr": 6.33021208350328e-06,
    "cuda_mem_allocated": 6.577167510986328,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56434,
    "batch_size": 125,
    "total_loss": 0.04066697381011447,
    "samples_seen": 23164,
    "gradnorm": 0.28515625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:15.304568"
}
 total length: 7837 dropped: 0 num samples 14 - rank: 3 max len: 579 min len: 542 avg len: 559.7857142857143 num_loss_counted_tokens: 7095
 total length: 7996 dropped: 0 num samples 17 - rank: 5 max len: 495 min len: 410 avg len: 470.3529411764706 num_loss_counted_tokens: 7095
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 727 min len: 571 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7891 dropped: 0 num samples 24 - rank: 7 max len: 375 min len: 171 avg len: 328.7916666666667 num_loss_counted_tokens: 6619
 total length: 7989 dropped: 0 num samples 9 - rank: 0 max len: 1096 min len: 689 avg len: 887.6666666666666 num_loss_counted_tokens: 7512
 total length: 7992 dropped: 0 num samples 13 - rank: 2 max len: 627 min len: 586 avg len: 614.7692307692307 num_loss_counted_tokens: 7303
 total length: 7989 dropped: 0 num samples 20 - rank: 6 max len: 443 min len: 262 avg len: 399.45 num_loss_counted_tokens: 6929
 total length: 7956 dropped: 0 num samples 16 - rank: 4 max len: 535 min len: 172 avg len: 497.25 num_loss_counted_tokens: 7108
Epoch: 4, Step: 184, Rank: 6, loss = 0.0615234375Epoch: 4, Step: 184, Rank: 5, loss = 0.0439453125Epoch: 4, Step: 184, Rank: 2, loss = 0.03662109375Epoch: 4, Step: 184, Rank: 3, loss = 0.039306640625Epoch: 4, Step: 184, Rank: 7, loss = 0.052978515625
Epoch: 4, Step: 184, Rank: 1, loss = 0.035400390625




Epoch: 4, Step: 184, Rank: 4, loss = 0.03515625
Epoch: 4, Step: 184, Rank: 0, loss = 0.0203857421875

Epoch 4:  60%|██████    | 24/40 [00:40<00:26,  1.65s/it]{
    "epoch": 4,
    "step": 184,
    "rank": 0,
    "overall_throughput": 94.43121362556708,
    "lr": 6.215889499576898e-06,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56465,
    "batch_size": 135,
    "total_loss": 0.04066235721243248,
    "samples_seen": 23299,
    "gradnorm": 0.287109375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:16.951209"
}
Epoch: 4, Step: 185, Rank: 5, loss = 0.045654296875Epoch: 4, Step: 185, Rank: 6, loss = 0.058349609375Epoch: 4, Step: 185, Rank: 1, loss = 0.03271484375Epoch: 4, Step: 185, Rank: 3, loss = 0.044921875
Epoch: 4, Step: 185, Rank: 7, loss = 0.0625


Epoch: 4, Step: 185, Rank: 2, loss = 0.03857421875

Epoch: 4, Step: 185, Rank: 4, loss = 0.040771484375Epoch: 4, Step: 185, Rank: 0, loss = 0.032470703125


Epoch 4:  62%|██████▎   | 25/40 [00:42<00:24,  1.65s/it]{
    "epoch": 4,
    "step": 185,
    "rank": 0,
    "overall_throughput": 94.71013089899762,
    "lr": 6.102141267073207e-06,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56567,
    "batch_size": 128,
    "total_loss": 0.04446055120476603,
    "samples_seen": 23427,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:18.597055"
}
Epoch: 4, Step: 186, Rank: 7, loss = 0.06103515625Epoch: 4, Step: 186, Rank: 3, loss = 0.041259765625Epoch: 4, Step: 186, Rank: 2, loss = 0.03515625Epoch: 4, Step: 186, Rank: 6, loss = 0.047119140625
Epoch: 4, Step: 186, Rank: 5, loss = 0.04345703125Epoch: 4, Step: 186, Rank: 1, loss = 0.0257568359375

Epoch: 4, Step: 186, Rank: 4, loss = 0.03515625



Epoch: 4, Step: 186, Rank: 0, loss = 0.0255126953125

Epoch 4:  65%|██████▌   | 26/40 [00:43<00:23,  1.65s/it]{
    "epoch": 4,
    "step": 186,
    "rank": 0,
    "overall_throughput": 94.23904944877964,
    "lr": 5.988984650672813e-06,
    "cuda_mem_allocated": 6.58666467666626,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56854,
    "batch_size": 126,
    "total_loss": 0.039276040384141835,
    "samples_seen": 23553,
    "gradnorm": 0.28125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:20.248253"
}
Epoch: 4, Step: 187, Rank: 7, loss = 0.057373046875Epoch: 4, Step: 187, Rank: 6, loss = 0.041748046875Epoch: 4, Step: 187, Rank: 3, loss = 0.037109375Epoch: 4, Step: 187, Rank: 5, loss = 0.04345703125Epoch: 4, Step: 187, Rank: 2, loss = 0.03125Epoch: 4, Step: 187, Rank: 1, loss = 0.034912109375

Epoch: 4, Step: 187, Rank: 4, loss = 0.037109375




Epoch: 4, Step: 187, Rank: 0, loss = 0.0216064453125

Epoch 4:  68%|██████▊   | 27/40 [00:45<00:21,  1.65s/it]{
    "epoch": 4,
    "step": 187,
    "rank": 0,
    "overall_throughput": 93.82020286113823,
    "lr": 5.876436825260967e-06,
    "cuda_mem_allocated": 6.587082386016846,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 57354,
    "batch_size": 119,
    "total_loss": 0.03811416814869059,
    "samples_seen": 23672,
    "gradnorm": 0.287109375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:21.904486"
}
Epoch: 4, Step: 188, Rank: 6, loss = 0.04443359375Epoch: 4, Step: 188, Rank: 3, loss = 0.04345703125Epoch: 4, Step: 188, Rank: 7, loss = 0.05224609375
Epoch: 4, Step: 188, Rank: 1, loss = 0.0537109375Epoch: 4, Step: 188, Rank: 2, loss = 0.039794921875Epoch: 4, Step: 188, Rank: 5, loss = 0.04638671875



Epoch: 4, Step: 188, Rank: 4, loss = 0.0311279296875

Epoch: 4, Step: 188, Rank: 0, loss = 0.0302734375

Epoch 4:  70%|███████   | 28/40 [00:47<00:19,  1.65s/it]{
    "epoch": 4,
    "step": 188,
    "rank": 0,
    "overall_throughput": 94.52165682606932,
    "lr": 5.764514873320761e-06,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56593,
    "batch_size": 124,
    "total_loss": 0.042690792147438726,
    "samples_seen": 23796,
    "gradnorm": 0.310546875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:23.546745"
}
Epoch: 4, Step: 189, Rank: 3, loss = 0.044677734375Epoch: 4, Step: 189, Rank: 7, loss = 0.064453125Epoch: 4, Step: 189, Rank: 6, loss = 0.05615234375Epoch: 4, Step: 189, Rank: 1, loss = 0.03125Epoch: 4, Step: 189, Rank: 2, loss = 0.035888671875Epoch: 4, Step: 189, Rank: 5, loss = 0.041748046875


Epoch: 4, Step: 189, Rank: 4, loss = 0.03857421875



Epoch: 4, Step: 189, Rank: 0, loss = 0.0203857421875

Epoch 4:  72%|███████▎  | 29/40 [00:48<00:18,  1.65s/it]{
    "epoch": 4,
    "step": 189,
    "rank": 0,
    "overall_throughput": 94.44697884353972,
    "lr": 5.653235782340351e-06,
    "cuda_mem_allocated": 6.588063716888428,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56884,
    "batch_size": 127,
    "total_loss": 0.04166373672737501,
    "samples_seen": 23923,
    "gradnorm": 0.283203125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:25.193366"
}
Epoch: 4, Step: 190, Rank: 3, loss = 0.033203125Epoch: 4, Step: 190, Rank: 1, loss = 0.027099609375Epoch: 4, Step: 190, Rank: 7, loss = 0.047607421875Epoch: 4, Step: 190, Rank: 5, loss = 0.044189453125
Epoch: 4, Step: 190, Rank: 6, loss = 0.06396484375Epoch: 4, Step: 190, Rank: 2, loss = 0.0308837890625



Epoch: 4, Step: 190, Rank: 4, loss = 0.03857421875

Epoch: 4, Step: 190, Rank: 0, loss = 0.03515625

Epoch 4:  75%|███████▌  | 30/40 [00:50<00:16,  1.65s/it]{
    "epoch": 4,
    "step": 190,
    "rank": 0,
    "overall_throughput": 94.42737694535069,
    "lr": 5.542616442234618e-06,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56488,
    "batch_size": 128,
    "total_loss": 0.04004390312986829,
    "samples_seen": 24051,
    "gradnorm": 0.28125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:26.840739"
}
Epoch: 4, Step: 191, Rank: 6, loss = 0.0546875Epoch: 4, Step: 191, Rank: 5, loss = 0.046875Epoch: 4, Step: 191, Rank: 7, loss = 0.049072265625Epoch: 4, Step: 191, Rank: 3, loss = 0.031982421875Epoch: 4, Step: 191, Rank: 2, loss = 0.03564453125




Epoch: 4, Step: 191, Rank: 1, loss = 0.041259765625Epoch: 4, Step: 191, Rank: 4, loss = 0.035400390625

Epoch: 4, Step: 191, Rank: 0, loss = 0.02197265625

Epoch 4:  78%|███████▊  | 31/40 [00:52<00:14,  1.65s/it]{
    "epoch": 4,
    "step": 191,
    "rank": 0,
    "overall_throughput": 94.47445180204515,
    "lr": 5.432673642781595e-06,
    "cuda_mem_allocated": 6.588062286376953,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56952,
    "batch_size": 128,
    "total_loss": 0.03962986374490799,
    "samples_seen": 24179,
    "gradnorm": 0.431640625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:28.484853"
}
Epoch: 4, Step: 192, Rank: 6, loss = 0.05810546875Epoch: 4, Step: 192, Rank: 3, loss = 0.0419921875Epoch: 4, Step: 192, Rank: 2, loss = 0.044189453125Epoch: 4, Step: 192, Rank: 5, loss = 0.035400390625Epoch: 4, Step: 192, Rank: 7, loss = 0.055419921875Epoch: 4, Step: 192, Rank: 1, loss = 0.04443359375Epoch: 4, Step: 192, Rank: 4, loss = 0.03564453125






Epoch: 4, Step: 192, Rank: 0, loss = 0.0194091796875

Epoch 4:  80%|████████  | 32/40 [00:54<00:13,  1.74s/it]{
    "epoch": 4,
    "step": 192,
    "rank": 0,
    "overall_throughput": 77.21748175388251,
    "lr": 5.3234240710741335e-06,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56811,
    "batch_size": 122,
    "total_loss": 0.041840488637763816,
    "samples_seen": 24301,
    "gradnorm": 0.302734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:30.433761"
}
Epoch: 4, Step: 193, Rank: 5, loss = 0.044189453125Epoch: 4, Step: 193, Rank: 6, loss = 0.046630859375Epoch: 4, Step: 193, Rank: 2, loss = 0.03173828125Epoch: 4, Step: 193, Rank: 1, loss = 0.03076171875Epoch: 4, Step: 193, Rank: 7, loss = 0.064453125Epoch: 4, Step: 193, Rank: 3, loss = 0.03173828125


Epoch: 4, Step: 193, Rank: 4, loss = 0.041259765625



Epoch: 4, Step: 193, Rank: 0, loss = 0.0186767578125

Epoch 4:  82%|████████▎ | 33/40 [00:56<00:12,  1.80s/it]{
    "epoch": 4,
    "step": 193,
    "rank": 0,
    "overall_throughput": 77.61999937252925,
    "lr": 5.214884308987136e-06,
    "cuda_mem_allocated": 6.586440086364746,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56927,
    "batch_size": 127,
    "total_loss": 0.03864598520912748,
    "samples_seen": 24428,
    "gradnorm": 0.275390625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:32.376492"
}
Epoch: 4, Step: 194, Rank: 6, loss = 0.05126953125Epoch: 4, Step: 194, Rank: 1, loss = 0.03955078125Epoch: 4, Step: 194, Rank: 5, loss = 0.040283203125Epoch: 4, Step: 194, Rank: 7, loss = 0.05078125
Epoch: 4, Step: 194, Rank: 3, loss = 0.03857421875

Epoch: 4, Step: 194, Rank: 2, loss = 0.03515625
Epoch: 4, Step: 194, Rank: 4, loss = 0.04248046875


Epoch: 4, Step: 194, Rank: 0, loss = 0.01904296875

Epoch 4:  85%|████████▌ | 34/40 [00:57<00:10,  1.75s/it]{
    "epoch": 4,
    "step": 194,
    "rank": 0,
    "overall_throughput": 94.38678703260274,
    "lr": 5.107070830660765e-06,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56777,
    "batch_size": 124,
    "total_loss": 0.03962872289835673,
    "samples_seen": 24552,
    "gradnorm": 0.302734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:34.026033"
}
Epoch: 4, Step: 195, Rank: 6, loss = 0.04833984375Epoch: 4, Step: 195, Rank: 3, loss = 0.039306640625Epoch: 4, Step: 195, Rank: 7, loss = 0.05029296875Epoch: 4, Step: 195, Rank: 5, loss = 0.056640625
Epoch: 4, Step: 195, Rank: 1, loss = 0.03662109375



Epoch: 4, Step: 195, Rank: 2, loss = 0.0361328125Epoch: 4, Step: 195, Rank: 4, loss = 0.048828125

Epoch: 4, Step: 195, Rank: 0, loss = 0.0286865234375

Epoch 4:  88%|████████▊ | 35/40 [00:59<00:08,  1.72s/it]{
    "epoch": 4,
    "step": 195,
    "rank": 0,
    "overall_throughput": 93.68333700362504,
    "lr": 5.000000000000003e-06,
    "cuda_mem_allocated": 6.586256504058838,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56298,
    "batch_size": 130,
    "total_loss": 0.04309211694909233,
    "samples_seen": 24682,
    "gradnorm": 0.287109375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:35.676278"
}
Epoch: 4, Step: 196, Rank: 6, loss = 0.058837890625Epoch: 4, Step: 196, Rank: 5, loss = 0.037841796875Epoch: 4, Step: 196, Rank: 3, loss = 0.031494140625Epoch: 4, Step: 196, Rank: 7, loss = 0.05224609375Epoch: 4, Step: 196, Rank: 1, loss = 0.035400390625


Epoch: 4, Step: 196, Rank: 2, loss = 0.0380859375


Epoch: 4, Step: 196, Rank: 4, loss = 0.031982421875
Epoch: 4, Step: 196, Rank: 0, loss = 0.015869140625

Epoch 4:  90%|█████████ | 36/40 [01:01<00:06,  1.70s/it]{
    "epoch": 4,
    "step": 196,
    "rank": 0,
    "overall_throughput": 94.49031461400314,
    "lr": 4.893688068190933e-06,
    "cuda_mem_allocated": 6.584833145141602,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 57140,
    "batch_size": 121,
    "total_loss": 0.03773188659432972,
    "samples_seen": 24803,
    "gradnorm": 0.298828125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:37.323267"
}
Epoch: 4, Step: 197, Rank: 7, loss = 0.056640625Epoch: 4, Step: 197, Rank: 2, loss = 0.029296875Epoch: 4, Step: 197, Rank: 6, loss = 0.056640625Epoch: 4, Step: 197, Rank: 1, loss = 0.030517578125Epoch: 4, Step: 197, Rank: 5, loss = 0.040771484375
Epoch: 4, Step: 197, Rank: 3, loss = 0.031494140625

Epoch: 4, Step: 197, Rank: 4, loss = 0.036376953125



Epoch: 4, Step: 197, Rank: 0, loss = 0.0294189453125

Epoch 4:  92%|█████████▎| 37/40 [01:02<00:05,  1.68s/it]{
    "epoch": 4,
    "step": 197,
    "rank": 0,
    "overall_throughput": 94.38482897470617,
    "lr": 4.788151171234149e-06,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56586,
    "batch_size": 127,
    "total_loss": 0.03887887463330152,
    "samples_seen": 24930,
    "gradnorm": 0.283203125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:38.965727"
}
Epoch: 4, Step: 198, Rank: 6, loss = 0.05029296875Epoch: 4, Step: 198, Rank: 1, loss = 0.033935546875Epoch: 4, Step: 198, Rank: 7, loss = 0.05224609375Epoch: 4, Step: 198, Rank: 3, loss = 0.0308837890625
Epoch: 4, Step: 198, Rank: 5, loss = 0.045166015625


Epoch: 4, Step: 198, Rank: 4, loss = 0.036865234375
Epoch: 4, Step: 198, Rank: 2, loss = 0.03857421875

Epoch: 4, Step: 198, Rank: 0, loss = 0.03466796875

Epoch 4:  95%|█████████▌| 38/40 [01:04<00:03,  1.67s/it]{
    "epoch": 4,
    "step": 198,
    "rank": 0,
    "overall_throughput": 94.14899741405041,
    "lr": 4.683405327495638e-06,
    "cuda_mem_allocated": 6.588010311126709,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56429,
    "batch_size": 127,
    "total_loss": 0.0402984281132042,
    "samples_seen": 25057,
    "gradnorm": 0.294921875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:40.611348"
}
Epoch: 4, Step: 199, Rank: 6, loss = 0.0478515625Epoch: 4, Step: 199, Rank: 3, loss = 0.044189453125Epoch: 4, Step: 199, Rank: 5, loss = 0.051025390625Epoch: 4, Step: 199, Rank: 7, loss = 0.06640625

Epoch: 4, Step: 199, Rank: 0, loss = 0.0201416015625


Epoch: 4, Step: 199, Rank: 2, loss = 0.03466796875
Epoch: 4, Step: 199, Rank: 1, loss = 0.03173828125Epoch: 4, Step: 199, Rank: 4, loss = 0.04541015625


Epoch 4:  98%|█████████▊| 39/40 [01:05<00:01,  1.66s/it]{
    "epoch": 4,
    "step": 199,
    "rank": 0,
    "overall_throughput": 94.28048920334201,
    "lr": 4.579466435275506e-06,
    "cuda_mem_allocated": 6.582216739654541,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56248,
    "batch_size": 128,
    "total_loss": 0.04270374057744275,
    "samples_seen": 25185,
    "gradnorm": 0.33203125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:42.257091"
}
Epoch: 4, Step: 200, Rank: 6, loss = 0.054931640625Epoch: 4, Step: 200, Rank: 1, loss = 0.043212890625Epoch: 4, Step: 200, Rank: 5, loss = 0.036376953125Epoch: 4, Step: 200, Rank: 3, loss = 0.04052734375


Epoch: 4, Step: 200, Rank: 2, loss = 0.03173828125Epoch: 4, Step: 200, Rank: 7, loss = 0.05517578125
Epoch: 4, Step: 200, Rank: 4, loss = 0.037109375


Epoch: 4, Step: 200, Rank: 0, loss = 0.0216064453125

Epoch 4: 100%|██████████| 40/40 [01:07<00:00,  1.66s/it]{
    "epoch": 4,
    "step": 200,
    "rank": 0,
    "overall_throughput": 94.53405637873567,
    "lr": 4.476350270394942e-06,
    "cuda_mem_allocated": 6.587998867034912,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 57022,
    "batch_size": 125,
    "total_loss": 0.0401073269965978,
    "samples_seen": 25310,
    "gradnorm": 0.294921875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:40:43.907013"
}
Saving model in huggingface format at: samples_25310
[19:41:10] INFO     The model is bigger than the maximum     accelerator.py:2924
                    size per checkpoint (5GB) and is going                      
                    to be split in 4 checkpoint shards. You                     
                    can find where each parameters has been                     
                    saved in the index located at                               
                    /tmp/tmpcte3bppuw/model.safetensors.inde                    
                    x.json.                                                     
Model saved in /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/hf_format/samples_25310
[19:41:26] INFO     saving took 42.577348947525024 seconds          utils.py:879
Saving full model state in /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/full_state/epoch_4
           INFO     Saving current state to                  accelerator.py:3030
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_4                                                  
           INFO     Saving FSDP model                        accelerator.py:3040
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
[19:41:35] INFO     Saving model to                             fsdp_utils.py:88
                    /var/home/cloud-user/.local/share/instructl                 
                    ab/phased/phase1/checkpoints/full_state/epo                 
                    ch_4/pytorch_model_fsdp.bin                                 
[19:41:46] INFO     Model saved to                              fsdp_utils.py:90
                    /var/home/cloud-user/.local/share/instructl                 
                    ab/phased/phase1/checkpoints/full_state/epo                 
                    ch_4/pytorch_model_fsdp.bin                                 
           INFO     FSDP Model saved to output dir           accelerator.py:3042
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_4                                                  
           INFO     Saving FSDP Optimizer                    accelerator.py:3059
 total length: 7809 dropped: 0 num samples 17 - rank: 5 max len: 492 min len: 424 avg len: 459.3529411764706 num_loss_counted_tokens: 6908
 total length: 7997 dropped: 0 num samples 17 - rank: 5 max len: 514 min len: 303 avg len: 470.4117647058824 num_loss_counted_tokens: 7096
 total length: 7733 dropped: 0 num samples 16 - rank: 5 max len: 505 min len: 456 avg len: 483.3125 num_loss_counted_tokens: 6885
 total length: 7746 dropped: 0 num samples 16 - rank: 5 max len: 503 min len: 463 avg len: 484.125 num_loss_counted_tokens: 6898
 total length: 7646 dropped: 0 num samples 24 - rank: 7 max len: 361 min len: 266 avg len: 318.5833333333333 num_loss_counted_tokens: 6374
 total length: 7619 dropped: 0 num samples 24 - rank: 7 max len: 372 min len: 242 avg len: 317.4583333333333 num_loss_counted_tokens: 6347 total length: 7948 dropped: 0 num samples 18 - rank: 5 max len: 467 min len: 418 avg len: 441.55555555555554 num_loss_counted_tokens: 6994
 total length: 7805 dropped: 0 num samples 17 - rank: 5 max len: 481 min len: 434 avg len: 459.11764705882354 num_loss_counted_tokens: 6904
 total length: 7994 dropped: 0 num samples 17 - rank: 5 max len: 494 min len: 288 avg len: 470.2352941176471 num_loss_counted_tokens: 7093
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 468 min len: 400 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7995 dropped: 0 num samples 19 - rank: 5 max len: 462 min len: 233 avg len: 420.7894736842105 num_loss_counted_tokens: 6988
 total length: 7995 dropped: 0 num samples 13 - rank: 2 max len: 636 min len: 466 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7757 dropped: 0 num samples 12 - rank: 2 max len: 673 min len: 629 avg len: 646.4166666666666 num_loss_counted_tokens: 7121 total length: 7996 dropped: 0 num samples 13 - rank: 2 max len: 664 min len: 253 avg len: 615.0769230769231 num_loss_counted_tokens: 7307

 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 469 min len: 399 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 652 min len: 398 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7997 dropped: 0 num samples 24 - rank: 7 max len: 388 min len: 280 avg len: 333.2083333333333 num_loss_counted_tokens: 6725
 total length: 7877 dropped: 0 num samples 18 - rank: 5 max len: 464 min len: 415 avg len: 437.6111111111111 num_loss_counted_tokens: 6923
 total length: 7866 dropped: 0 num samples 18 - rank: 5 max len: 456 min len: 417 avg len: 437.0 num_loss_counted_tokens: 6912
 total length: 7826 dropped: 0 num samples 24 - rank: 7 max len: 370 min len: 267 avg len: 326.0833333333333 num_loss_counted_tokens: 6554
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 645 min len: 357 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7844 dropped: 0 num samples 24 - rank: 7 max len: 365 min len: 277 avg len: 326.8333333333333 num_loss_counted_tokens: 6572
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 653 min len: 411 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7678 dropped: 0 num samples 23 - rank: 7 max len: 400 min len: 265 avg len: 333.82608695652175 num_loss_counted_tokens: 6459
 total length: 7898 dropped: 0 num samples 13 - rank: 2 max len: 638 min len: 582 avg len: 607.5384615384615 num_loss_counted_tokens: 7209
 total length: 7806 dropped: 0 num samples 18 - rank: 5 max len: 461 min len: 414 avg len: 433.6666666666667 num_loss_counted_tokens: 6852
 total length: 7383 dropped: 0 num samples 23 - rank: 7 max len: 355 min len: 259 avg len: 321.0 num_loss_counted_tokens: 6164
 total length: 7757 dropped: 0 num samples 19 - rank: 6 max len: 434 min len: 382 avg len: 408.2631578947368 num_loss_counted_tokens: 6750

 total length: 7908 dropped: 0 num samples 13 - rank: 2 max len: 632 min len: 555 avg len: 608.3076923076923 num_loss_counted_tokens: 7219
 total length: 7626 dropped: 0 num samples 23 - rank: 7 max len: 382 min len: 260 avg len: 331.5652173913044 num_loss_counted_tokens: 6407
 total length: 7906 dropped: 0 num samples 20 - rank: 6 max len: 423 min len: 369 avg len: 395.3 num_loss_counted_tokens: 6846
 total length: 7996 dropped: 0 num samples 14 - rank: 2 max len: 621 min len: 299 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7852 dropped: 0 num samples 18 - rank: 5 max len: 463 min len: 418 avg len: 436.22222222222223 num_loss_counted_tokens: 6898
 total length: 7989 dropped: 0 num samples 18 - rank: 5 max len: 470 min len: 329 avg len: 443.8333333333333 num_loss_counted_tokens: 7035
 total length: 7821 dropped: 0 num samples 24 - rank: 7 max len: 371 min len: 261 avg len: 325.875 num_loss_counted_tokens: 6549
 total length: 7732 dropped: 0 num samples 25 - rank: 7 max len: 365 min len: 243 avg len: 309.28 num_loss_counted_tokens: 6407
 total length: 7934 dropped: 0 num samples 19 - rank: 6 max len: 462 min len: 390 avg len: 417.57894736842104 num_loss_counted_tokens: 6927
 total length: 7980 dropped: 0 num samples 26 - rank: 7 max len: 371 min len: 245 avg len: 306.9230769230769 num_loss_counted_tokens: 6602
 total length: 7462 dropped: 0 num samples 22 - rank: 7 max len: 380 min len: 257 avg len: 339.1818181818182 num_loss_counted_tokens: 6296
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 675 min len: 337 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7988 dropped: 0 num samples 13 - rank: 2 max len: 672 min len: 339 avg len: 614.4615384615385 num_loss_counted_tokens: 7299
 total length: 7997 dropped: 0 num samples 14 - rank: 2 max len: 628 min len: 326 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7456 dropped: 0 num samples 23 - rank: 7 max len: 371 min len: 251 avg len: 324.17391304347825 num_loss_counted_tokens: 6237
 total length: 7995 dropped: 0 num samples 13 - rank: 2 max len: 640 min len: 430 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7784 dropped: 0 num samples 17 - rank: 5 max len: 474 min len: 443 avg len: 457.88235294117646 num_loss_counted_tokens: 6883
 total length: 7829 dropped: 0 num samples 24 - rank: 7 max len: 366 min len: 253 avg len: 326.2083333333333 num_loss_counted_tokens: 6557
 total length: 7969 dropped: 0 num samples 17 - rank: 5 max len: 490 min len: 431 avg len: 468.7647058823529 num_loss_counted_tokens: 7068
 total length: 7920 dropped: 0 num samples 13 - rank: 2 max len: 624 min len: 585 avg len: 609.2307692307693 num_loss_counted_tokens: 7231
 total length: 7949 dropped: 0 num samples 24 - rank: 7 max len: 381 min len: 229 avg len: 331.2083333333333 num_loss_counted_tokens: 6677
 total length: 7860 dropped: 0 num samples 15 - rank: 4 max len: 544 min len: 505 avg len: 524.0 num_loss_counted_tokens: 7065
 total length: 7993 dropped: 0 num samples 15 - rank: 4 max len: 568 min len: 408 avg len: 532.8666666666667 num_loss_counted_tokens: 7198
 total length: 7833 dropped: 0 num samples 15 - rank: 4 max len: 548 min len: 507 avg len: 522.2 num_loss_counted_tokens: 7038
 total length: 7879 dropped: 0 num samples 15 - rank: 4 max len: 550 min len: 497 avg len: 525.2666666666667 num_loss_counted_tokens: 7084
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 730 min len: 298 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7993 dropped: 0 num samples 13 - rank: 2 max len: 648 min len: 577 avg len: 614.8461538461538 num_loss_counted_tokens: 7304
 total length: 7993 dropped: 0 num samples 12 - rank: 1 max len: 717 min len: 638 avg len: 666.0833333333334 num_loss_counted_tokens: 7357
 total length: 7989 dropped: 0 num samples 12 - rank: 1 max len: 694 min len: 555 avg len: 665.75 num_loss_counted_tokens: 7353
 total length: 7992 dropped: 0 num samples 16 - rank: 4 max len: 525 min len: 400 avg len: 499.5 num_loss_counted_tokens: 7144
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 731 min len: 473 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
[19:41:56] INFO     Saving Optimizer state to                  fsdp_utils.py:192
                    /var/home/cloud-user/.local/share/instruct                  
                    lab/phased/phase1/checkpoints/full_state/e                  
                    poch_4/optimizer.bin                                        
 total length: 7997 dropped: 0 num samples 11 - rank: 1 max len: 859 min len: 500 avg len: 727.0 num_loss_counted_tokens: 7414
 total length: 7837 dropped: 0 num samples 20 - rank: 6 max len: 412 min len: 365 avg len: 391.85 num_loss_counted_tokens: 6777
 total length: 7991 dropped: 0 num samples 14 - rank: 3 max len: 610 min len: 429 avg len: 570.7857142857143 num_loss_counted_tokens: 7249
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 708 min len: 592 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7804 dropped: 0 num samples 13 - rank: 2 max len: 622 min len: 579 avg len: 600.3076923076923 num_loss_counted_tokens: 7115
 total length: 7806 dropped: 0 num samples 20 - rank: 6 max len: 418 min len: 372 avg len: 390.3 num_loss_counted_tokens: 6746
 total length: 7994 dropped: 0 num samples 14 - rank: 3 max len: 603 min len: 418 avg len: 571.0 num_loss_counted_tokens: 7252
 total length: 7781 dropped: 0 num samples 13 - rank: 3 max len: 631 min len: 545 avg len: 598.5384615384615 num_loss_counted_tokens: 7092 total length: 7994 dropped: 0 num samples 14 - rank: 3 max len: 609 min len: 356 avg len: 571.0 num_loss_counted_tokens: 7252

 total length: 7730 dropped: 0 num samples 13 - rank: 3 max len: 628 min len: 550 avg len: 594.6153846153846 num_loss_counted_tokens: 7041
 total length: 7934 dropped: 0 num samples 14 - rank: 3 max len: 611 min len: 526 avg len: 566.7142857142857 num_loss_counted_tokens: 7192
 total length: 7894 dropped: 0 num samples 20 - rank: 6 max len: 417 min len: 372 avg len: 394.7 num_loss_counted_tokens: 6834
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 571 min len: 461 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7654 dropped: 0 num samples 24 - rank: 7 max len: 367 min len: 261 avg len: 318.9166666666667 num_loss_counted_tokens: 6382
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 699 min len: 635 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7931 dropped: 0 num samples 19 - rank: 6 max len: 454 min len: 383 avg len: 417.42105263157896 num_loss_counted_tokens: 6924
 total length: 7909 dropped: 0 num samples 20 - rank: 6 max len: 433 min len: 367 avg len: 395.45 num_loss_counted_tokens: 6849
 total length: 7759 dropped: 0 num samples 16 - rank: 4 max len: 517 min len: 461 avg len: 484.9375 num_loss_counted_tokens: 6911
 total length: 7991 dropped: 0 num samples 11 - rank: 1 max len: 836 min len: 559 avg len: 726.4545454545455 num_loss_counted_tokens: 7408
 total length: 7985 dropped: 0 num samples 21 - rank: 6 max len: 408 min len: 243 avg len: 380.23809523809524 num_loss_counted_tokens: 6872
 total length: 7809 dropped: 0 num samples 11 - rank: 1 max len: 809 min len: 672 avg len: 709.9090909090909 num_loss_counted_tokens: 7226
 total length: 7780 dropped: 0 num samples 18 - rank: 6 max len: 466 min len: 402 avg len: 432.22222222222223 num_loss_counted_tokens: 6826
 total length: 7819 dropped: 0 num samples 12 - rank: 1 max len: 677 min len: 627 avg len: 651.5833333333334 num_loss_counted_tokens: 7183
 total length: 7994 dropped: 0 num samples 17 - rank: 4 max len: 515 min len: 344 avg len: 470.2352941176471 num_loss_counted_tokens: 7093
 total length: 7946 dropped: 0 num samples 20 - rank: 6 max len: 413 min len: 383 avg len: 397.3 num_loss_counted_tokens: 6886
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 745 min len: 576 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
 total length: 7913 dropped: 0 num samples 14 - rank: 3 max len: 615 min len: 531 avg len: 565.2142857142857 num_loss_counted_tokens: 7171
 total length: 7992 dropped: 0 num samples 13 - rank: 2 max len: 655 min len: 397 avg len: 614.7692307692307 num_loss_counted_tokens: 7303
 total length: 7894 dropped: 0 num samples 16 - rank: 4 max len: 523 min len: 470 avg len: 493.375 num_loss_counted_tokens: 7046
 total length: 7976 dropped: 0 num samples 16 - rank: 4 max len: 532 min len: 330 avg len: 498.5 num_loss_counted_tokens: 7128
 total length: 7882 dropped: 0 num samples 14 - rank: 3 max len: 608 min len: 536 avg len: 563.0 num_loss_counted_tokens: 7140
 total length: 7894 dropped: 0 num samples 16 - rank: 4 max len: 531 min len: 467 avg len: 493.375 num_loss_counted_tokens: 7046
 total length: 7886 dropped: 0 num samples 14 - rank: 3 max len: 608 min len: 523 avg len: 563.2857142857143 num_loss_counted_tokens: 7144
 total length: 7827 dropped: 0 num samples 14 - rank: 3 max len: 580 min len: 526 avg len: 559.0714285714286 num_loss_counted_tokens: 7085 total length: 7993 dropped: 0 num samples 15 - rank: 3 max len: 572 min len: 359 avg len: 532.8666666666667 num_loss_counted_tokens: 7198

 total length: 7864 dropped: 0 num samples 16 - rank: 4 max len: 518 min len: 470 avg len: 491.5 num_loss_counted_tokens: 7016
 total length: 7892 dropped: 0 num samples 12 - rank: 1 max len: 682 min len: 621 avg len: 657.6666666666666 num_loss_counted_tokens: 7256
 total length: 7995 dropped: 0 num samples 15 - rank: 3 max len: 567 min len: 410 avg len: 533.0 num_loss_counted_tokens: 7200
 total length: 7995 dropped: 0 num samples 13 - rank: 1 max len: 665 min len: 267 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7920 dropped: 0 num samples 20 - rank: 6 max len: 415 min len: 367 avg len: 396.0 num_loss_counted_tokens: 6860
 total length: 7962 dropped: 0 num samples 25 - rank: 7 max len: 373 min len: 250 avg len: 318.48 num_loss_counted_tokens: 6637
 total length: 7785 dropped: 0 num samples 12 - rank: 1 max len: 672 min len: 631 avg len: 648.75 num_loss_counted_tokens: 7149
 total length: 7995 dropped: 0 num samples 15 - rank: 3 max len: 559 min len: 357 avg len: 533.0 num_loss_counted_tokens: 7200
 total length: 7841 dropped: 0 num samples 12 - rank: 1 max len: 689 min len: 633 avg len: 653.4166666666666 num_loss_counted_tokens: 7205
 total length: 7996 dropped: 0 num samples 20 - rank: 6 max len: 426 min len: 265 avg len: 399.8 num_loss_counted_tokens: 6936
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 719 min len: 405 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7995 dropped: 0 num samples 20 - rank: 6 max len: 443 min len: 290 avg len: 399.75 num_loss_counted_tokens: 6935
 total length: 7948 dropped: 0 num samples 16 - rank: 4 max len: 518 min len: 474 avg len: 496.75 num_loss_counted_tokens: 7100
 total length: 7828 dropped: 0 num samples 16 - rank: 4 max len: 519 min len: 465 avg len: 489.25 num_loss_counted_tokens: 6980
 total length: 7996 dropped: 0 num samples 15 - rank: 3 max len: 552 min len: 476 avg len: 533.0666666666667 num_loss_counted_tokens: 7201
 total length: 7991 dropped: 0 num samples 15 - rank: 3 max len: 582 min len: 302 avg len: 532.7333333333333 num_loss_counted_tokens: 7196
 total length: 7909 dropped: 0 num samples 15 - rank: 4 max len: 556 min len: 505 avg len: 527.2666666666667 num_loss_counted_tokens: 7114
 total length: 7997 dropped: 0 num samples 20 - rank: 6 max len: 429 min len: 328 avg len: 399.85 num_loss_counted_tokens: 6937
 total length: 7997 dropped: 0 num samples 20 - rank: 6 max len: 417 min len: 363 avg len: 399.85 num_loss_counted_tokens: 6937
 total length: 7926 dropped: 0 num samples 14 - rank: 3 max len: 605 min len: 528 avg len: 566.1428571428571 num_loss_counted_tokens: 7184
 total length: 7991 dropped: 0 num samples 12 - rank: 1 max len: 744 min len: 510 avg len: 665.9166666666666 num_loss_counted_tokens: 7355
 total length: 7986 dropped: 0 num samples 16 - rank: 4 max len: 525 min len: 328 avg len: 499.125 num_loss_counted_tokens: 7138
 total length: 7996 dropped: 0 num samples 17 - rank: 4 max len: 510 min len: 291 avg len: 470.3529411764706 num_loss_counted_tokens: 7095
 total length: 7995 dropped: 0 num samples 20 - rank: 6 max len: 430 min len: 263 avg len: 399.75 num_loss_counted_tokens: 6935
 total length: 7739 dropped: 0 num samples 16 - rank: 4 max len: 506 min len: 465 avg len: 483.6875 num_loss_counted_tokens: 6891
[19:42:18] INFO     Optimizer state saved in                   fsdp_utils.py:194
                    /var/home/cloud-user/.local/share/instruct                  
                    lab/phased/phase1/checkpoints/full_state/e                  
                    poch_4/optimizer.bin                                        
[19:42:19] INFO     FSDP Optimizer saved to output dir       accelerator.py:3061
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_4                                                  
           INFO     Scheduler state saved in                checkpointing.py:118
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_4/scheduler.bin                                  
           INFO     Sampler state for dataloader 0 saved in checkpointing.py:135
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_4/sampler.bin                                    
           INFO     Random states saved in                  checkpointing.py:160
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_4/random_states_0.pkl                            
Saving training state: {'current_epoch': 4, 'samples_seen': 25310}
Model state saved in: /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/full_state/epoch_4


Epoch 5:   0%|          | 0/40 [00:00<?, ?it/s]
Epoch 4: 100%|██████████| 40/40 [02:42<00:00,  4.07s/it]
 total length: 7991 dropped: 0 num samples 9 - rank: 0 max len: 1155 min len: 278 avg len: 887.8888888888889 num_loss_counted_tokens: 7514
 total length: 7992 dropped: 0 num samples 9 - rank: 0 max len: 1074 min len: 596 avg len: 888.0 num_loss_counted_tokens: 7515
 total length: 7990 dropped: 0 num samples 8 - rank: 0 max len: 1101 min len: 685 avg len: 998.75 num_loss_counted_tokens: 7566
 total length: 7841 dropped: 0 num samples 9 - rank: 0 max len: 1060 min len: 719 avg len: 871.2222222222222 num_loss_counted_tokens: 7364
 total length: 7860 dropped: 0 num samples 8 - rank: 0 max len: 1110 min len: 735 avg len: 982.5 num_loss_counted_tokens: 7436
 total length: 7995 dropped: 0 num samples 11 - rank: 0 max len: 846 min len: 534 avg len: 726.8181818181819 num_loss_counted_tokens: 7412
 total length: 7997 dropped: 0 num samples 9 - rank: 0 max len: 1104 min len: 573 avg len: 888.5555555555555 num_loss_counted_tokens: 7520
 total length: 7924 dropped: 0 num samples 9 - rank: 0 max len: 1391 min len: 172 avg len: 880.4444444444445 num_loss_counted_tokens: 7447
 total length: 7982 dropped: 0 num samples 9 - rank: 0 max len: 1076 min len: 551 avg len: 886.8888888888889 num_loss_counted_tokens: 7505
 total length: 7997 dropped: 0 num samples 10 - rank: 0 max len: 1060 min len: 597 avg len: 799.7 num_loss_counted_tokens: 7467 total length: 7985 dropped: 0 num samples 9 - rank: 0 max len: 1118 min len: 172 avg len: 887.2222222222222 num_loss_counted_tokens: 7508

 total length: 7911 dropped: 0 num samples 9 - rank: 0 max len: 1098 min len: 701 avg len: 879.0 num_loss_counted_tokens: 7434
 total length: 7798 dropped: 0 num samples 8 - rank: 0 max len: 1164 min len: 769 avg len: 974.75 num_loss_counted_tokens: 7374
 total length: 7996 dropped: 0 num samples 9 - rank: 0 max len: 1247 min len: 612 avg len: 888.4444444444445 num_loss_counted_tokens: 7519
 total length: 7749 dropped: 0 num samples 8 - rank: 0 max len: 1072 min len: 844 avg len: 968.625 num_loss_counted_tokens: 7325
 total length: 7957 dropped: 0 num samples 10 - rank: 0 max len: 904 min len: 695 avg len: 795.7 num_loss_counted_tokens: 7427
 total length: 7875 dropped: 0 num samples 9 - rank: 0 max len: 1064 min len: 695 avg len: 875.0 num_loss_counted_tokens: 7398
Epoch: 5, Step: 201, Rank: 6, loss = 0.05615234375Epoch: 5, Step: 201, Rank: 3, loss = 0.034423828125

Epoch: 5, Step: 201, Rank: 1, loss = 0.04296875Epoch: 5, Step: 201, Rank: 2, loss = 0.0291748046875Epoch: 5, Step: 201, Rank: 5, loss = 0.04736328125Epoch: 5, Step: 201, Rank: 7, loss = 0.048583984375



Epoch: 5, Step: 201, Rank: 4, loss = 0.042724609375
Epoch: 5, Step: 201, Rank: 0, loss = 0.0174560546875


Epoch 5:   2%|▎         | 1/40 [00:01<01:03,  1.64s/it]{
    "epoch": 5,
    "step": 201,
    "rank": 0,
    "overall_throughput": 94.15659287948881,
    "lr": 4.374072483801769e-06,
    "cuda_mem_allocated": 6.588095188140869,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56643,
    "batch_size": 124,
    "total_loss": 0.039881362215984324,
    "samples_seen": 25434,
    "gradnorm": 0.30078125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:20.729630"
}
 total length: 7995 dropped: 0 num samples 10 - rank: 0 max len: 1196 min len: 448 avg len: 799.5 num_loss_counted_tokens: 7465
 total length: 7997 dropped: 0 num samples 13 - rank: 1 max len: 660 min len: 285 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7841 dropped: 0 num samples 15 - rank: 4 max len: 545 min len: 497 avg len: 522.7333333333333 num_loss_counted_tokens: 7046
 total length: 7776 dropped: 0 num samples 20 - rank: 6 max len: 418 min len: 360 avg len: 388.8 num_loss_counted_tokens: 6716
 total length: 7882 dropped: 0 num samples 13 - rank: 2 max len: 627 min len: 583 avg len: 606.3076923076923 num_loss_counted_tokens: 7193
 total length: 7798 dropped: 0 num samples 17 - rank: 5 max len: 495 min len: 418 avg len: 458.70588235294116 num_loss_counted_tokens: 6897
 total length: 7954 dropped: 0 num samples 25 - rank: 7 max len: 356 min len: 248 avg len: 318.16 num_loss_counted_tokens: 6629
 total length: 7935 dropped: 0 num samples 14 - rank: 3 max len: 582 min len: 547 avg len: 566.7857142857143 num_loss_counted_tokens: 7193
Epoch: 5, Step: 202, Rank: 1, loss = 0.036376953125Epoch: 5, Step: 202, Rank: 5, loss = 0.039794921875
Epoch: 5, Step: 202, Rank: 6, loss = 0.044189453125Epoch: 5, Step: 202, Rank: 3, loss = 0.034423828125Epoch: 5, Step: 202, Rank: 4, loss = 0.031005859375

Epoch: 5, Step: 202, Rank: 2, loss = 0.03515625
Epoch: 5, Step: 202, Rank: 7, loss = 0.052001953125


Epoch: 5, Step: 202, Rank: 0, loss = 0.0181884765625


Epoch 5:   5%|▌         | 2/40 [00:03<01:02,  1.64s/it]{
    "epoch": 5,
    "step": 202,
    "rank": 0,
    "overall_throughput": 94.3480226524847,
    "lr": 4.272648599194948e-06,
    "cuda_mem_allocated": 6.587977886199951,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56821,
    "batch_size": 123,
    "total_loss": 0.036377395681174214,
    "samples_seen": 25557,
    "gradnorm": 0.26953125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:22.372701"
}
 total length: 7892 dropped: 0 num samples 14 - rank: 3 max len: 583 min len: 547 avg len: 563.7142857142857 num_loss_counted_tokens: 7150
 total length: 7881 dropped: 0 num samples 15 - rank: 4 max len: 547 min len: 503 avg len: 525.4 num_loss_counted_tokens: 7086
 total length: 7986 dropped: 0 num samples 13 - rank: 2 max len: 624 min len: 584 avg len: 614.3076923076923 num_loss_counted_tokens: 7297
 total length: 7888 dropped: 0 num samples 19 - rank: 6 max len: 450 min len: 376 avg len: 415.1578947368421 num_loss_counted_tokens: 6881
 total length: 7885 dropped: 0 num samples 10 - rank: 0 max len: 1056 min len: 672 avg len: 788.5 num_loss_counted_tokens: 7355
 total length: 7941 dropped: 0 num samples 13 - rank: 1 max len: 672 min len: 171 avg len: 610.8461538461538 num_loss_counted_tokens: 7252
 total length: 7997 dropped: 0 num samples 17 - rank: 5 max len: 501 min len: 361 avg len: 470.4117647058824 num_loss_counted_tokens: 7096
 total length: 7915 dropped: 0 num samples 25 - rank: 7 max len: 373 min len: 243 avg len: 316.6 num_loss_counted_tokens: 6590
Epoch: 5, Step: 203, Rank: 6, loss = 0.044921875Epoch: 5, Step: 203, Rank: 4, loss = 0.0380859375Epoch: 5, Step: 203, Rank: 7, loss = 0.057861328125Epoch: 5, Step: 203, Rank: 5, loss = 0.037841796875Epoch: 5, Step: 203, Rank: 3, loss = 0.03955078125

Epoch: 5, Step: 203, Rank: 2, loss = 0.03076171875

Epoch: 5, Step: 203, Rank: 1, loss = 0.041259765625


Epoch: 5, Step: 203, Rank: 0, loss = 0.0194091796875


Epoch 5:   8%|▊         | 3/40 [00:04<01:00,  1.64s/it]{
    "epoch": 5,
    "step": 203,
    "rank": 0,
    "overall_throughput": 94.27732697970285,
    "lr": 4.1720940106683915e-06,
    "cuda_mem_allocated": 6.588095188140869,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56730,
    "batch_size": 118,
    "total_loss": 0.03867442270403666,
    "samples_seen": 25675,
    "gradnorm": 0.271484375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:24.018915"
}
 total length: 7984 dropped: 0 num samples 17 - rank: 4 max len: 508 min len: 259 avg len: 469.6470588235294 num_loss_counted_tokens: 7083
 total length: 7994 dropped: 0 num samples 15 - rank: 3 max len: 577 min len: 395 avg len: 532.9333333333333 num_loss_counted_tokens: 7199
 total length: 7992 dropped: 0 num samples 13 - rank: 1 max len: 679 min len: 290 avg len: 614.7692307692307 num_loss_counted_tokens: 7303
 total length: 7863 dropped: 0 num samples 13 - rank: 2 max len: 624 min len: 577 avg len: 604.8461538461538 num_loss_counted_tokens: 7174
 total length: 7931 dropped: 0 num samples 21 - rank: 6 max len: 392 min len: 358 avg len: 377.6666666666667 num_loss_counted_tokens: 6818
 total length: 7980 dropped: 0 num samples 19 - rank: 5 max len: 459 min len: 256 avg len: 420.0 num_loss_counted_tokens: 6973
 total length: 7997 dropped: 0 num samples 10 - rank: 0 max len: 1065 min len: 457 avg len: 799.7 num_loss_counted_tokens: 7467
 total length: 7874 dropped: 0 num samples 24 - rank: 7 max len: 355 min len: 282 avg len: 328.0833333333333 num_loss_counted_tokens: 6602
Epoch: 5, Step: 204, Rank: 2, loss = 0.0302734375Epoch: 5, Step: 204, Rank: 4, loss = 0.026611328125Epoch: 5, Step: 204, Rank: 3, loss = 0.0390625Epoch: 5, Step: 204, Rank: 5, loss = 0.03759765625Epoch: 5, Step: 204, Rank: 6, loss = 0.052001953125Epoch: 5, Step: 204, Rank: 7, loss = 0.050537109375
Epoch: 5, Step: 204, Rank: 1, loss = 0.0458984375





Epoch: 5, Step: 204, Rank: 0, loss = 0.01806640625


Epoch 5:  10%|█         | 4/40 [00:06<00:59,  1.65s/it]{
    "epoch": 5,
    "step": 204,
    "rank": 0,
    "overall_throughput": 94.62591427615625,
    "lr": 4.0724239803744524e-06,
    "cuda_mem_allocated": 6.575246334075928,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56622,
    "batch_size": 120,
    "total_loss": 0.03746776871180813,
    "samples_seen": 25795,
    "gradnorm": 0.271484375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:25.665711"
}
 total length: 7909 dropped: 0 num samples 9 - rank: 0 max len: 1068 min len: 725 avg len: 878.7777777777778 num_loss_counted_tokens: 7432
 total length: 7993 dropped: 0 num samples 15 - rank: 4 max len: 554 min len: 489 avg len: 532.8666666666667 num_loss_counted_tokens: 7198
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 721 min len: 337 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7716 dropped: 0 num samples 23 - rank: 7 max len: 387 min len: 243 avg len: 335.4782608695652 num_loss_counted_tokens: 6497
 total length: 7990 dropped: 0 num samples 14 - rank: 3 max len: 618 min len: 294 avg len: 570.7142857142857 num_loss_counted_tokens: 7248
 total length: 7992 dropped: 0 num samples 13 - rank: 2 max len: 656 min len: 319 avg len: 614.7692307692307 num_loss_counted_tokens: 7303
 total length: 7821 dropped: 0 num samples 17 - rank: 5 max len: 515 min len: 427 avg len: 460.05882352941177 num_loss_counted_tokens: 6920
 total length: 7981 dropped: 0 num samples 20 - rank: 6 max len: 426 min len: 277 avg len: 399.05 num_loss_counted_tokens: 6921
Epoch: 5, Step: 205, Rank: 6, loss = 0.043212890625Epoch: 5, Step: 205, Rank: 5, loss = 0.03955078125Epoch: 5, Step: 205, Rank: 7, loss = 0.058837890625Epoch: 5, Step: 205, Rank: 2, loss = 0.0361328125Epoch: 5, Step: 205, Rank: 4, loss = 0.039794921875Epoch: 5, Step: 205, Rank: 1, loss = 0.033203125




Epoch: 5, Step: 205, Rank: 3, loss = 0.0380859375

Epoch: 5, Step: 205, Rank: 0, loss = 0.03515625


Epoch 5:  12%|█▎        | 5/40 [00:08<00:57,  1.65s/it]{
    "epoch": 5,
    "step": 205,
    "rank": 0,
    "overall_throughput": 94.20791087508432,
    "lr": 3.973653636207437e-06,
    "cuda_mem_allocated": 6.588095188140869,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56734,
    "batch_size": 127,
    "total_loss": 0.04050481192935453,
    "samples_seen": 25922,
    "gradnorm": 0.30859375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:27.311523"
}
 total length: 7995 dropped: 0 num samples 16 - rank: 4 max len: 545 min len: 304 avg len: 499.6875 num_loss_counted_tokens: 7147
 total length: 7996 dropped: 0 num samples 18 - rank: 5 max len: 483 min len: 332 avg len: 444.22222222222223 num_loss_counted_tokens: 7042
 total length: 7990 dropped: 0 num samples 14 - rank: 3 max len: 624 min len: 384 avg len: 570.7142857142857 num_loss_counted_tokens: 7248
 total length: 7993 dropped: 0 num samples 12 - rank: 1 max len: 713 min len: 612 avg len: 666.0833333333334 num_loss_counted_tokens: 7357
 total length: 7988 dropped: 0 num samples 13 - rank: 2 max len: 648 min len: 316 avg len: 614.4615384615385 num_loss_counted_tokens: 7299
 total length: 7769 dropped: 0 num samples 25 - rank: 7 max len: 367 min len: 236 avg len: 310.76 num_loss_counted_tokens: 6444
 total length: 7951 dropped: 0 num samples 20 - rank: 6 max len: 422 min len: 373 avg len: 397.55 num_loss_counted_tokens: 6891
 total length: 7883 dropped: 0 num samples 9 - rank: 0 max len: 1106 min len: 720 avg len: 875.8888888888889 num_loss_counted_tokens: 7406
Epoch: 5, Step: 206, Rank: 7, loss = 0.048828125Epoch: 5, Step: 206, Rank: 5, loss = 0.03955078125Epoch: 5, Step: 206, Rank: 2, loss = 0.038818359375Epoch: 5, Step: 206, Rank: 0, loss = 0.031494140625
Epoch: 5, Step: 206, Rank: 6, loss = 0.043701171875
Epoch: 5, Step: 206, Rank: 4, loss = 0.0341796875


Epoch: 5, Step: 206, Rank: 3, loss = 0.03466796875

Epoch: 5, Step: 206, Rank: 1, loss = 0.0400390625


Epoch 5:  15%|█▌        | 6/40 [00:09<00:56,  1.65s/it]{
    "epoch": 5,
    "step": 206,
    "rank": 0,
    "overall_throughput": 94.67836304955974,
    "lr": 3.875797969507502e-06,
    "cuda_mem_allocated": 6.574270248413086,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56775,
    "batch_size": 121,
    "total_loss": 0.03889035667107001,
    "samples_seen": 26043,
    "gradnorm": 0.3046875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:28.962527"
}
 total length: 7964 dropped: 0 num samples 12 - rank: 1 max len: 691 min len: 637 avg len: 663.6666666666666 num_loss_counted_tokens: 7328
 total length: 7995 dropped: 0 num samples 10 - rank: 0 max len: 1064 min len: 327 avg len: 799.5 num_loss_counted_tokens: 7465
 total length: 7988 dropped: 0 num samples 15 - rank: 3 max len: 574 min len: 312 avg len: 532.5333333333333 num_loss_counted_tokens: 7193
 total length: 7947 dropped: 0 num samples 16 - rank: 4 max len: 523 min len: 468 avg len: 496.6875 num_loss_counted_tokens: 7099
 total length: 7932 dropped: 0 num samples 13 - rank: 2 max len: 633 min len: 582 avg len: 610.1538461538462 num_loss_counted_tokens: 7243
 total length: 7992 dropped: 0 num samples 21 - rank: 6 max len: 415 min len: 164 avg len: 380.57142857142856 num_loss_counted_tokens: 6879
 total length: 7994 dropped: 0 num samples 18 - rank: 5 max len: 467 min len: 325 avg len: 444.1111111111111 num_loss_counted_tokens: 7040
 total length: 7817 dropped: 0 num samples 25 - rank: 7 max len: 369 min len: 172 avg len: 312.68 num_loss_counted_tokens: 6492
Epoch: 5, Step: 207, Rank: 5, loss = 0.049560546875Epoch: 5, Step: 207, Rank: 1, loss = 0.03369140625Epoch: 5, Step: 207, Rank: 2, loss = 0.03271484375Epoch: 5, Step: 207, Rank: 3, loss = 0.0380859375Epoch: 5, Step: 207, Rank: 7, loss = 0.04736328125Epoch: 5, Step: 207, Rank: 6, loss = 0.053955078125





Epoch: 5, Step: 207, Rank: 4, loss = 0.03662109375
Epoch: 5, Step: 207, Rank: 0, loss = 0.0203857421875


Epoch 5:  18%|█▊        | 7/40 [00:11<00:57,  1.75s/it]{
    "epoch": 5,
    "step": 207,
    "rank": 0,
    "overall_throughput": 76.69632666262474,
    "lr": 3.7788718327852625e-06,
    "cuda_mem_allocated": 6.588095188140869,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56409,
    "batch_size": 129,
    "total_loss": 0.03903632399085252,
    "samples_seen": 26172,
    "gradnorm": 0.2890625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:30.921744"
}
 total length: 7996 dropped: 0 num samples 14 - rank: 3 max len: 622 min len: 462 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7762 dropped: 0 num samples 19 - rank: 6 max len: 442 min len: 391 avg len: 408.5263157894737 num_loss_counted_tokens: 6755
 total length: 7995 dropped: 0 num samples 17 - rank: 5 max len: 499 min len: 414 avg len: 470.29411764705884 num_loss_counted_tokens: 7094
 total length: 7989 dropped: 0 num samples 12 - rank: 1 max len: 829 min len: 332 avg len: 665.75 num_loss_counted_tokens: 7353
 total length: 7899 dropped: 0 num samples 8 - rank: 0 max len: 1104 min len: 829 avg len: 987.375 num_loss_counted_tokens: 7475
 total length: 7797 dropped: 0 num samples 15 - rank: 4 max len: 548 min len: 501 avg len: 519.8 num_loss_counted_tokens: 7002
 total length: 7993 dropped: 0 num samples 13 - rank: 2 max len: 664 min len: 271 avg len: 614.8461538461538 num_loss_counted_tokens: 7304
 total length: 7964 dropped: 0 num samples 24 - rank: 7 max len: 391 min len: 241 avg len: 331.8333333333333 num_loss_counted_tokens: 6692
Epoch: 5, Step: 208, Rank: 5, loss = 0.046875Epoch: 5, Step: 208, Rank: 2, loss = 0.034912109375Epoch: 5, Step: 208, Rank: 7, loss = 0.05712890625Epoch: 5, Step: 208, Rank: 1, loss = 0.033203125Epoch: 5, Step: 208, Rank: 6, loss = 0.0537109375Epoch: 5, Step: 208, Rank: 4, loss = 0.03662109375

Epoch: 5, Step: 208, Rank: 3, loss = 0.0400390625




Epoch: 5, Step: 208, Rank: 0, loss = 0.0286865234375


Epoch 5:  20%|██        | 8/40 [00:13<00:54,  1.71s/it]{
    "epoch": 5,
    "step": 208,
    "rank": 0,
    "overall_throughput": 95.27611641388171,
    "lr": 3.6828899374674933e-06,
    "cuda_mem_allocated": 6.581121921539307,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56054,
    "batch_size": 127,
    "total_loss": 0.04133514111392586,
    "samples_seen": 26299,
    "gradnorm": 0.298828125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:32.559693"
}
 total length: 7990 dropped: 0 num samples 14 - rank: 3 max len: 610 min len: 378 avg len: 570.7142857142857 num_loss_counted_tokens: 7248
 total length: 7989 dropped: 0 num samples 10 - rank: 0 max len: 1103 min len: 540 avg len: 798.9 num_loss_counted_tokens: 7459
 total length: 7994 dropped: 0 num samples 12 - rank: 1 max len: 701 min len: 564 avg len: 666.1666666666666 num_loss_counted_tokens: 7358
 total length: 7768 dropped: 0 num samples 20 - rank: 6 max len: 421 min len: 357 avg len: 388.4 num_loss_counted_tokens: 6708
 total length: 7988 dropped: 0 num samples 13 - rank: 2 max len: 647 min len: 456 avg len: 614.4615384615385 num_loss_counted_tokens: 7299
 total length: 7708 dropped: 0 num samples 24 - rank: 7 max len: 357 min len: 253 avg len: 321.1666666666667 num_loss_counted_tokens: 6436
 total length: 7992 dropped: 0 num samples 18 - rank: 5 max len: 476 min len: 273 avg len: 444.0 num_loss_counted_tokens: 7038
 total length: 7763 dropped: 0 num samples 15 - rank: 4 max len: 563 min len: 478 avg len: 517.5333333333333 num_loss_counted_tokens: 6968
Epoch: 5, Step: 209, Rank: 7, loss = 0.0634765625Epoch: 5, Step: 209, Rank: 3, loss = 0.03466796875Epoch: 5, Step: 209, Rank: 1, loss = 0.04150390625Epoch: 5, Step: 209, Rank: 6, loss = 0.052978515625Epoch: 5, Step: 209, Rank: 5, loss = 0.036865234375

Epoch: 5, Step: 209, Rank: 4, loss = 0.031982421875Epoch: 5, Step: 209, Rank: 2, loss = 0.044921875




Epoch: 5, Step: 209, Rank: 0, loss = 0.019775390625


Epoch 5:  22%|██▎       | 9/40 [00:15<00:52,  1.69s/it]{
    "epoch": 5,
    "step": 209,
    "rank": 0,
    "overall_throughput": 94.18158400289384,
    "lr": 3.587866851664219e-06,
    "cuda_mem_allocated": 6.58656644821167,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56566,
    "batch_size": 127,
    "total_loss": 0.04080189513135099,
    "samples_seen": 26426,
    "gradnorm": 0.294921875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:34.206604"
}
 total length: 7994 dropped: 0 num samples 21 - rank: 6 max len: 410 min len: 298 avg len: 380.6666666666667 num_loss_counted_tokens: 6881
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 639 min len: 481 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7924 dropped: 0 num samples 12 - rank: 1 max len: 690 min len: 640 avg len: 660.3333333333334 num_loss_counted_tokens: 7288
 total length: 7926 dropped: 0 num samples 18 - rank: 5 max len: 471 min len: 410 avg len: 440.3333333333333 num_loss_counted_tokens: 6972
 total length: 7995 dropped: 0 num samples 14 - rank: 3 max len: 601 min len: 506 avg len: 571.0714285714286 num_loss_counted_tokens: 7253
 total length: 7717 dropped: 0 num samples 24 - rank: 7 max len: 354 min len: 266 avg len: 321.5416666666667 num_loss_counted_tokens: 6445
 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 533 min len: 426 avg len: 499.75 num_loss_counted_tokens: 7148
 total length: 7997 dropped: 0 num samples 11 - rank: 0 max len: 895 min len: 363 avg len: 727.0 num_loss_counted_tokens: 7414
Epoch: 5, Step: 210, Rank: 6, loss = 0.047119140625Epoch: 5, Step: 210, Rank: 3, loss = 0.034912109375Epoch: 5, Step: 210, Rank: 7, loss = 0.06103515625Epoch: 5, Step: 210, Rank: 5, loss = 0.05615234375Epoch: 5, Step: 210, Rank: 1, loss = 0.032470703125


Epoch: 5, Step: 210, Rank: 4, loss = 0.035400390625
Epoch: 5, Step: 210, Rank: 2, loss = 0.032958984375


Epoch: 5, Step: 210, Rank: 0, loss = 0.0164794921875


Epoch 5:  25%|██▌       | 10/40 [00:16<00:50,  1.68s/it]{
    "epoch": 5,
    "step": 210,
    "rank": 0,
    "overall_throughput": 94.06514903274447,
    "lr": 3.493816997957582e-06,
    "cuda_mem_allocated": 6.586722373962402,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56692,
    "batch_size": 127,
    "total_loss": 0.03958230438157059,
    "samples_seen": 26553,
    "gradnorm": 0.30078125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:35.867900"
}
 total length: 7843 dropped: 0 num samples 12 - rank: 1 max len: 672 min len: 643 avg len: 653.5833333333334 num_loss_counted_tokens: 7207
 total length: 7748 dropped: 0 num samples 24 - rank: 7 max len: 391 min len: 191 avg len: 322.8333333333333 num_loss_counted_tokens: 6476
 total length: 7857 dropped: 0 num samples 19 - rank: 6 max len: 440 min len: 392 avg len: 413.5263157894737 num_loss_counted_tokens: 6850
 total length: 7994 dropped: 0 num samples 16 - rank: 4 max len: 547 min len: 276 avg len: 499.625 num_loss_counted_tokens: 7146
 total length: 7996 dropped: 0 num samples 14 - rank: 3 max len: 605 min len: 471 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7993 dropped: 0 num samples 13 - rank: 2 max len: 633 min len: 580 avg len: 614.8461538461538 num_loss_counted_tokens: 7304
 total length: 7992 dropped: 0 num samples 11 - rank: 0 max len: 973 min len: 384 avg len: 726.5454545454545 num_loss_counted_tokens: 7409
 total length: 7809 dropped: 0 num samples 17 - rank: 5 max len: 491 min len: 440 avg len: 459.3529411764706 num_loss_counted_tokens: 6908
Epoch: 5, Step: 211, Rank: 6, loss = 0.057861328125Epoch: 5, Step: 211, Rank: 1, loss = 0.0302734375Epoch: 5, Step: 211, Rank: 5, loss = 0.049072265625Epoch: 5, Step: 211, Rank: 7, loss = 0.045166015625Epoch: 5, Step: 211, Rank: 3, loss = 0.037109375
Epoch: 5, Step: 211, Rank: 4, loss = 0.0274658203125

Epoch: 5, Step: 211, Rank: 2, loss = 0.032470703125



Epoch: 5, Step: 211, Rank: 0, loss = 0.03271484375


Epoch 5:  28%|██▊       | 11/40 [00:18<00:48,  1.67s/it]{
    "epoch": 5,
    "step": 211,
    "rank": 0,
    "overall_throughput": 94.508145850042,
    "lr": 3.4007546512127764e-06,
    "cuda_mem_allocated": 6.588095188140869,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56442,
    "batch_size": 130,
    "total_loss": 0.038995783281953154,
    "samples_seen": 26683,
    "gradnorm": 0.267578125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:37.516738"
}
 total length: 7996 dropped: 0 num samples 14 - rank: 2 max len: 612 min len: 353 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7960 dropped: 0 num samples 24 - rank: 7 max len: 370 min len: 253 avg len: 331.6666666666667 num_loss_counted_tokens: 6688
 total length: 7976 dropped: 0 num samples 19 - rank: 5 max len: 454 min len: 275 avg len: 419.7894736842105 num_loss_counted_tokens: 6969
 total length: 7996 dropped: 0 num samples 13 - rank: 1 max len: 658 min len: 315 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7986 dropped: 0 num samples 15 - rank: 3 max len: 567 min len: 456 avg len: 532.4 num_loss_counted_tokens: 7191
 total length: 7994 dropped: 0 num samples 11 - rank: 0 max len: 1094 min len: 290 avg len: 726.7272727272727 num_loss_counted_tokens: 7411
 total length: 7995 dropped: 0 num samples 21 - rank: 6 max len: 396 min len: 337 avg len: 380.7142857142857 num_loss_counted_tokens: 6882
 total length: 7821 dropped: 0 num samples 16 - rank: 4 max len: 508 min len: 468 avg len: 488.8125 num_loss_counted_tokens: 6973
Epoch: 5, Step: 212, Rank: 7, loss = 0.049560546875Epoch: 5, Step: 212, Rank: 5, loss = 0.05078125
Epoch: 5, Step: 212, Rank: 6, loss = 0.053466796875Epoch: 5, Step: 212, Rank: 0, loss = 0.01611328125
Epoch: 5, Step: 212, Rank: 2, loss = 0.035400390625Epoch: 5, Step: 212, Rank: 3, loss = 0.039306640625

Epoch: 5, Step: 212, Rank: 4, loss = 0.0380859375


Epoch: 5, Step: 212, Rank: 1, loss = 0.0220947265625


Epoch 5:  30%|███       | 12/40 [00:20<00:46,  1.67s/it]{
    "epoch": 5,
    "step": 212,
    "rank": 0,
    "overall_throughput": 94.60630472669473,
    "lr": 3.308693936411421e-06,
    "cuda_mem_allocated": 6.565056324005127,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56448,
    "batch_size": 123,
    "total_loss": 0.03810586734693878,
    "samples_seen": 26806,
    "gradnorm": 0.3046875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:39.168697"
}
 total length: 7994 dropped: 0 num samples 15 - rank: 3 max len: 565 min len: 457 avg len: 532.9333333333333 num_loss_counted_tokens: 7199
 total length: 7996 dropped: 0 num samples 12 - rank: 1 max len: 767 min len: 574 avg len: 666.3333333333334 num_loss_counted_tokens: 7360
 total length: 7991 dropped: 0 num samples 14 - rank: 2 max len: 630 min len: 240 avg len: 570.7857142857143 num_loss_counted_tokens: 7249
 total length: 7875 dropped: 0 num samples 16 - rank: 4 max len: 515 min len: 467 avg len: 492.1875 num_loss_counted_tokens: 7027
 total length: 7851 dropped: 0 num samples 23 - rank: 7 max len: 379 min len: 265 avg len: 341.3478260869565 num_loss_counted_tokens: 6632
 total length: 7897 dropped: 0 num samples 8 - rank: 0 max len: 1309 min len: 830 avg len: 987.125 num_loss_counted_tokens: 7473
 total length: 7986 dropped: 0 num samples 20 - rank: 6 max len: 421 min len: 379 avg len: 399.3 num_loss_counted_tokens: 6926
 total length: 7827 dropped: 0 num samples 18 - rank: 5 max len: 464 min len: 421 avg len: 434.8333333333333 num_loss_counted_tokens: 6873
Epoch: 5, Step: 213, Rank: 6, loss = 0.0400390625Epoch: 5, Step: 213, Rank: 5, loss = 0.04345703125Epoch: 5, Step: 213, Rank: 1, loss = 0.0308837890625Epoch: 5, Step: 213, Rank: 7, loss = 0.053466796875
Epoch: 5, Step: 213, Rank: 2, loss = 0.03759765625Epoch: 5, Step: 213, Rank: 3, loss = 0.032958984375




Epoch: 5, Step: 213, Rank: 0, loss = 0.033447265625
Epoch: 5, Step: 213, Rank: 4, loss = 0.04296875


Epoch 5:  32%|███▎      | 13/40 [00:21<00:44,  1.66s/it]{
    "epoch": 5,
    "step": 213,
    "rank": 0,
    "overall_throughput": 94.92316747606503,
    "lr": 3.21764882650766e-06,
    "cuda_mem_allocated": 6.57676887512207,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56180,
    "batch_size": 124,
    "total_loss": 0.03939124243503026,
    "samples_seen": 26930,
    "gradnorm": 0.29296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:40.811093"
}
 total length: 7931 dropped: 0 num samples 16 - rank: 3 max len: 544 min len: 180 avg len: 495.6875 num_loss_counted_tokens: 7083
 total length: 7997 dropped: 0 num samples 13 - rank: 1 max len: 646 min len: 476 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7755 dropped: 0 num samples 24 - rank: 7 max len: 366 min len: 249 avg len: 323.125 num_loss_counted_tokens: 6483
 total length: 7995 dropped: 0 num samples 19 - rank: 5 max len: 447 min len: 306 avg len: 420.7894736842105 num_loss_counted_tokens: 6988
 total length: 7995 dropped: 0 num samples 14 - rank: 2 max len: 612 min len: 495 avg len: 571.0714285714286 num_loss_counted_tokens: 7253
 total length: 7989 dropped: 0 num samples 21 - rank: 6 max len: 405 min len: 174 avg len: 380.42857142857144 num_loss_counted_tokens: 6876
 total length: 7995 dropped: 0 num samples 11 - rank: 0 max len: 1048 min len: 435 avg len: 726.8181818181819 num_loss_counted_tokens: 7412
 total length: 7940 dropped: 0 num samples 17 - rank: 4 max len: 485 min len: 449 avg len: 467.05882352941177 num_loss_counted_tokens: 7039
Epoch: 5, Step: 214, Rank: 5, loss = 0.042236328125Epoch: 5, Step: 214, Rank: 3, loss = 0.040771484375Epoch: 5, Step: 214, Rank: 7, loss = 0.052734375Epoch: 5, Step: 214, Rank: 1, loss = 0.035400390625Epoch: 5, Step: 214, Rank: 0, loss = 0.0164794921875Epoch: 5, Step: 214, Rank: 2, loss = 0.031982421875
Epoch: 5, Step: 214, Rank: 4, loss = 0.037109375





Epoch: 5, Step: 214, Rank: 6, loss = 0.05029296875


Epoch 5:  35%|███▌      | 14/40 [00:23<00:43,  1.66s/it]{
    "epoch": 5,
    "step": 214,
    "rank": 0,
    "overall_throughput": 94.37565373491363,
    "lr": 3.1276331403073733e-06,
    "cuda_mem_allocated": 6.580146312713623,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56192,
    "batch_size": 126,
    "total_loss": 0.038323960706150344,
    "samples_seen": 27056,
    "gradnorm": 0.2890625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:42.459112"
}
 total length: 7745 dropped: 0 num samples 25 - rank: 7 max len: 357 min len: 202 avg len: 309.8 num_loss_counted_tokens: 6420
 total length: 7963 dropped: 0 num samples 16 - rank: 4 max len: 516 min len: 475 avg len: 497.6875 num_loss_counted_tokens: 7115
 total length: 7944 dropped: 0 num samples 11 - rank: 0 max len: 879 min len: 656 avg len: 722.1818181818181 num_loss_counted_tokens: 7361
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 567 min len: 481 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7984 dropped: 0 num samples 14 - rank: 2 max len: 617 min len: 243 avg len: 570.2857142857143 num_loss_counted_tokens: 7242
 total length: 7993 dropped: 0 num samples 18 - rank: 5 max len: 466 min len: 415 avg len: 444.05555555555554 num_loss_counted_tokens: 7039
 total length: 7996 dropped: 0 num samples 13 - rank: 1 max len: 654 min len: 342 avg len: 615.0769230769231 num_loss_counted_tokens: 7307
 total length: 7975 dropped: 0 num samples 20 - rank: 6 max len: 426 min len: 364 avg len: 398.75 num_loss_counted_tokens: 6915
Epoch: 5, Step: 215, Rank: 6, loss = 0.048828125Epoch: 5, Step: 215, Rank: 1, loss = 0.0302734375Epoch: 5, Step: 215, Rank: 5, loss = 0.042236328125Epoch: 5, Step: 215, Rank: 2, loss = 0.032958984375Epoch: 5, Step: 215, Rank: 4, loss = 0.03466796875

Epoch: 5, Step: 215, Rank: 3, loss = 0.03564453125
Epoch: 5, Step: 215, Rank: 7, loss = 0.058837890625



Epoch: 5, Step: 215, Rank: 0, loss = 0.028564453125


Epoch 5:  38%|███▊      | 15/40 [00:25<00:41,  1.65s/it]{
    "epoch": 5,
    "step": 215,
    "rank": 0,
    "overall_throughput": 94.17689199417963,
    "lr": 3.0386605403707347e-06,
    "cuda_mem_allocated": 6.584277153015137,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56604,
    "batch_size": 128,
    "total_loss": 0.03899017737262384,
    "samples_seen": 27184,
    "gradnorm": 0.267578125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:44.107107"
}
 total length: 7461 dropped: 0 num samples 23 - rank: 7 max len: 370 min len: 269 avg len: 324.39130434782606 num_loss_counted_tokens: 6242
 total length: 7989 dropped: 0 num samples 21 - rank: 6 max len: 407 min len: 236 avg len: 380.42857142857144 num_loss_counted_tokens: 6876
 total length: 7825 dropped: 0 num samples 10 - rank: 0 max len: 1100 min len: 664 avg len: 782.5 num_loss_counted_tokens: 7295
 total length: 7890 dropped: 0 num samples 13 - rank: 2 max len: 624 min len: 581 avg len: 606.9230769230769 num_loss_counted_tokens: 7201
 total length: 7992 dropped: 0 num samples 19 - rank: 5 max len: 456 min len: 295 avg len: 420.63157894736844 num_loss_counted_tokens: 6985
 total length: 7994 dropped: 0 num samples 13 - rank: 1 max len: 655 min len: 356 avg len: 614.9230769230769 num_loss_counted_tokens: 7305
 total length: 7887 dropped: 0 num samples 16 - rank: 4 max len: 532 min len: 460 avg len: 492.9375 num_loss_counted_tokens: 7039
 total length: 7977 dropped: 0 num samples 15 - rank: 3 max len: 574 min len: 238 avg len: 531.8 num_loss_counted_tokens: 7182
Epoch: 5, Step: 216, Rank: 6, loss = 0.05126953125Epoch: 5, Step: 216, Rank: 4, loss = 0.03759765625Epoch: 5, Step: 216, Rank: 0, loss = 0.0166015625Epoch: 5, Step: 216, Rank: 7, loss = 0.0537109375
Epoch: 5, Step: 216, Rank: 2, loss = 0.0322265625


Epoch: 5, Step: 216, Rank: 3, loss = 0.036865234375
Epoch: 5, Step: 216, Rank: 5, loss = 0.0380859375

Epoch: 5, Step: 216, Rank: 1, loss = 0.03369140625


Epoch 5:  40%|████      | 16/40 [00:26<00:39,  1.65s/it]{
    "epoch": 5,
    "step": 216,
    "rank": 0,
    "overall_throughput": 94.41633369725483,
    "lr": 2.95074453093853e-06,
    "cuda_mem_allocated": 6.570359230041504,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56994,
    "batch_size": 125,
    "total_loss": 0.037512720637260064,
    "samples_seen": 27309,
    "gradnorm": 0.283203125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:45.762662"
}
 total length: 7304 dropped: 0 num samples 22 - rank: 7 max len: 379 min len: 265 avg len: 332.0 num_loss_counted_tokens: 6138
 total length: 7990 dropped: 0 num samples 17 - rank: 5 max len: 498 min len: 443 avg len: 470.0 num_loss_counted_tokens: 7089
 total length: 7994 dropped: 0 num samples 20 - rank: 6 max len: 429 min len: 339 avg len: 399.7 num_loss_counted_tokens: 6934
 total length: 7860 dropped: 0 num samples 14 - rank: 3 max len: 574 min len: 546 avg len: 561.4285714285714 num_loss_counted_tokens: 7118
 total length: 7876 dropped: 0 num samples 12 - rank: 1 max len: 694 min len: 632 avg len: 656.3333333333334 num_loss_counted_tokens: 7240
 total length: 7830 dropped: 0 num samples 13 - rank: 2 max len: 631 min len: 575 avg len: 602.3076923076923 num_loss_counted_tokens: 7141
 total length: 7838 dropped: 0 num samples 15 - rank: 4 max len: 542 min len: 499 avg len: 522.5333333333333 num_loss_counted_tokens: 7043
 total length: 7996 dropped: 0 num samples 9 - rank: 0 max len: 1482 min len: 532 avg len: 888.4444444444445 num_loss_counted_tokens: 7519
Epoch: 5, Step: 217, Rank: 7, loss = 0.057373046875Epoch: 5, Step: 217, Rank: 5, loss = 0.050537109375Epoch: 5, Step: 217, Rank: 6, loss = 0.05322265625Epoch: 5, Step: 217, Rank: 1, loss = 0.038330078125
Epoch: 5, Step: 217, Rank: 2, loss = 0.037109375



Epoch: 5, Step: 217, Rank: 4, loss = 0.039306640625Epoch: 5, Step: 217, Rank: 3, loss = 0.036865234375

Epoch: 5, Step: 217, Rank: 0, loss = 0.0255126953125


Epoch 5:  42%|████▎     | 17/40 [00:28<00:40,  1.74s/it]{
    "epoch": 5,
    "step": 217,
    "rank": 0,
    "overall_throughput": 77.31054068037825,
    "lr": 2.8638984558824777e-06,
    "cuda_mem_allocated": 6.588095188140869,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56890,
    "batch_size": 128,
    "total_loss": 0.04227456494990332,
    "samples_seen": 27437,
    "gradnorm": 0.306640625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:47.715444"
}
 total length: 7980 dropped: 0 num samples 21 - rank: 6 max len: 411 min len: 256 avg len: 380.0 num_loss_counted_tokens: 6867
 total length: 7784 dropped: 0 num samples 16 - rank: 4 max len: 500 min len: 469 avg len: 486.5 num_loss_counted_tokens: 6936
 total length: 7997 dropped: 0 num samples 11 - rank: 1 max len: 842 min len: 518 avg len: 727.0 num_loss_counted_tokens: 7414
 total length: 7993 dropped: 0 num samples 13 - rank: 2 max len: 661 min len: 500 avg len: 614.8461538461538 num_loss_counted_tokens: 7304
 total length: 7983 dropped: 0 num samples 18 - rank: 5 max len: 464 min len: 424 avg len: 443.5 num_loss_counted_tokens: 7029
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 582 min len: 332 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7921 dropped: 0 num samples 8 - rank: 0 max len: 1076 min len: 851 avg len: 990.125 num_loss_counted_tokens: 7497
 total length: 7860 dropped: 0 num samples 25 - rank: 7 max len: 354 min len: 253 avg len: 314.4 num_loss_counted_tokens: 6535
Epoch: 5, Step: 218, Rank: 5, loss = 0.037353515625Epoch: 5, Step: 218, Rank: 6, loss = 0.04638671875Epoch: 5, Step: 218, Rank: 2, loss = 0.03662109375Epoch: 5, Step: 218, Rank: 7, loss = 0.056884765625Epoch: 5, Step: 218, Rank: 3, loss = 0.035400390625

Epoch: 5, Step: 218, Rank: 4, loss = 0.042236328125Epoch: 5, Step: 218, Rank: 1, loss = 0.0341796875




Epoch: 5, Step: 218, Rank: 0, loss = 0.0272216796875


Epoch 5:  45%|████▌     | 18/40 [00:30<00:37,  1.71s/it]{
    "epoch": 5,
    "step": 218,
    "rank": 0,
    "overall_throughput": 94.43644596470324,
    "lr": 2.778135496679908e-06,
    "cuda_mem_allocated": 6.587978839874268,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56447,
    "batch_size": 127,
    "total_loss": 0.03955923255443159,
    "samples_seen": 27564,
    "gradnorm": 0.30859375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:49.363136"
}
 total length: 7983 dropped: 0 num samples 15 - rank: 3 max len: 587 min len: 337 avg len: 532.2 num_loss_counted_tokens: 7188
 total length: 7799 dropped: 0 num samples 20 - rank: 6 max len: 412 min len: 371 avg len: 389.95 num_loss_counted_tokens: 6739
 total length: 7989 dropped: 0 num samples 13 - rank: 2 max len: 645 min len: 595 avg len: 614.5384615384615 num_loss_counted_tokens: 7300
 total length: 7965 dropped: 0 num samples 8 - rank: 0 max len: 1068 min len: 778 avg len: 995.625 num_loss_counted_tokens: 7541
 total length: 6909 dropped: 0 num samples 22 - rank: 7 max len: 369 min len: 245 avg len: 314.04545454545456 num_loss_counted_tokens: 5743
 total length: 7757 dropped: 0 num samples 11 - rank: 1 max len: 871 min len: 649 avg len: 705.1818181818181 num_loss_counted_tokens: 7174
 total length: 7992 dropped: 0 num samples 19 - rank: 5 max len: 450 min len: 270 avg len: 420.63157894736844 num_loss_counted_tokens: 6985
 total length: 7996 dropped: 0 num samples 17 - rank: 4 max len: 517 min len: 267 avg len: 470.3529411764706 num_loss_counted_tokens: 7095
Epoch: 5, Step: 219, Rank: 7, loss = 0.064453125Epoch: 5, Step: 219, Rank: 6, loss = 0.050537109375Epoch: 5, Step: 219, Rank: 0, loss = 0.0235595703125Epoch: 5, Step: 219, Rank: 5, loss = 0.041259765625Epoch: 5, Step: 219, Rank: 1, loss = 0.036376953125Epoch: 5, Step: 219, Rank: 3, loss = 0.041748046875


Epoch: 5, Step: 219, Rank: 4, loss = 0.0299072265625



Epoch: 5, Step: 219, Rank: 2, loss = 0.037353515625


Epoch 5:  48%|████▊     | 19/40 [00:31<00:35,  1.69s/it]{
    "epoch": 5,
    "step": 219,
    "rank": 0,
    "overall_throughput": 94.67378836466814,
    "lr": 2.6934686704130698e-06,
    "cuda_mem_allocated": 6.578329563140869,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56707,
    "batch_size": 126,
    "total_loss": 0.04064753910451972,
    "samples_seen": 27690,
    "gradnorm": 0.296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:51.009455"
}
 total length: 7958 dropped: 0 num samples 14 - rank: 3 max len: 601 min len: 531 avg len: 568.4285714285714 num_loss_counted_tokens: 7216
 total length: 7358 dropped: 0 num samples 22 - rank: 7 max len: 377 min len: 290 avg len: 334.45454545454544 num_loss_counted_tokens: 6192
 total length: 7898 dropped: 0 num samples 9 - rank: 0 max len: 1056 min len: 800 avg len: 877.5555555555555 num_loss_counted_tokens: 7421
 total length: 7722 dropped: 0 num samples 19 - rank: 6 max len: 430 min len: 379 avg len: 406.42105263157896 num_loss_counted_tokens: 6715
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 727 min len: 499 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 659 min len: 388 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7980 dropped: 0 num samples 18 - rank: 5 max len: 473 min len: 248 avg len: 443.3333333333333 num_loss_counted_tokens: 7026
 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 526 min len: 417 avg len: 499.75 num_loss_counted_tokens: 7148
Epoch: 5, Step: 220, Rank: 0, loss = 0.0301513671875Epoch: 5, Step: 220, Rank: 5, loss = 0.04296875
Epoch: 5, Step: 220, Rank: 2, loss = 0.0311279296875Epoch: 5, Step: 220, Rank: 6, loss = 0.051513671875Epoch: 5, Step: 220, Rank: 7, loss = 0.048095703125


Epoch: 5, Step: 220, Rank: 4, loss = 0.0390625Epoch: 5, Step: 220, Rank: 1, loss = 0.033203125


Epoch: 5, Step: 220, Rank: 3, loss = 0.037353515625


Epoch 5:  50%|█████     | 20/40 [00:33<00:33,  1.68s/it]{
    "epoch": 5,
    "step": 220,
    "rank": 0,
    "overall_throughput": 93.70642554761623,
    "lr": 2.6099108277934105e-06,
    "cuda_mem_allocated": 6.587979316711426,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56619,
    "batch_size": 132,
    "total_loss": 0.039138804994789736,
    "samples_seen": 27822,
    "gradnorm": 0.275390625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:52.666235"
}
 total length: 7994 dropped: 0 num samples 14 - rank: 3 max len: 611 min len: 436 avg len: 571.0 num_loss_counted_tokens: 7252
 total length: 7907 dropped: 0 num samples 15 - rank: 4 max len: 548 min len: 507 avg len: 527.1333333333333 num_loss_counted_tokens: 7112
 total length: 7992 dropped: 0 num samples 12 - rank: 1 max len: 697 min len: 597 avg len: 666.0 num_loss_counted_tokens: 7356
 total length: 7822 dropped: 0 num samples 20 - rank: 6 max len: 414 min len: 361 avg len: 391.1 num_loss_counted_tokens: 6762
 total length: 7991 dropped: 0 num samples 13 - rank: 2 max len: 647 min len: 388 avg len: 614.6923076923077 num_loss_counted_tokens: 7302
 total length: 7846 dropped: 0 num samples 26 - rank: 7 max len: 360 min len: 182 avg len: 301.7692307692308 num_loss_counted_tokens: 6468
 total length: 7873 dropped: 0 num samples 8 - rank: 0 max len: 1143 min len: 705 avg len: 984.125 num_loss_counted_tokens: 7449
 total length: 7991 dropped: 0 num samples 17 - rank: 5 max len: 507 min len: 424 avg len: 470.05882352941177 num_loss_counted_tokens: 7090
Epoch: 5, Step: 221, Rank: 7, loss = 0.0517578125Epoch: 5, Step: 221, Rank: 3, loss = 0.033447265625Epoch: 5, Step: 221, Rank: 0, loss = 0.020263671875Epoch: 5, Step: 221, Rank: 5, loss = 0.0478515625Epoch: 5, Step: 221, Rank: 1, loss = 0.038330078125


Epoch: 5, Step: 221, Rank: 6, loss = 0.043701171875
Epoch: 5, Step: 221, Rank: 4, loss = 0.0419921875


Epoch: 5, Step: 221, Rank: 2, loss = 0.039306640625


Epoch 5:  52%|█████▎    | 21/40 [00:35<00:33,  1.76s/it]{
    "epoch": 5,
    "step": 221,
    "rank": 0,
    "overall_throughput": 78.00994437294639,
    "lr": 2.527474651211089e-06,
    "cuda_mem_allocated": 6.580282688140869,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56878,
    "batch_size": 123,
    "total_loss": 0.0395583529659974,
    "samples_seen": 27945,
    "gradnorm": 0.28515625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:54.607480"
}
 total length: 7988 dropped: 0 num samples 16 - rank: 3 max len: 544 min len: 271 avg len: 499.25 num_loss_counted_tokens: 7140
 total length: 7977 dropped: 0 num samples 18 - rank: 5 max len: 461 min len: 428 avg len: 443.1666666666667 num_loss_counted_tokens: 7023
 total length: 7740 dropped: 0 num samples 12 - rank: 1 max len: 710 min len: 601 avg len: 645.0 num_loss_counted_tokens: 7104
 total length: 7995 dropped: 0 num samples 14 - rank: 2 max len: 599 min len: 511 avg len: 571.0714285714286 num_loss_counted_tokens: 7253
 total length: 7971 dropped: 0 num samples 20 - rank: 6 max len: 425 min len: 372 avg len: 398.55 num_loss_counted_tokens: 6911
 total length: 7964 dropped: 0 num samples 9 - rank: 0 max len: 1066 min len: 656 avg len: 884.8888888888889 num_loss_counted_tokens: 7487
 total length: 7925 dropped: 0 num samples 24 - rank: 7 max len: 372 min len: 259 avg len: 330.2083333333333 num_loss_counted_tokens: 6653
 total length: 7987 dropped: 0 num samples 17 - rank: 4 max len: 490 min len: 386 avg len: 469.8235294117647 num_loss_counted_tokens: 7086
Epoch: 5, Step: 222, Rank: 7, loss = 0.060791015625Epoch: 5, Step: 222, Rank: 5, loss = 0.04052734375Epoch: 5, Step: 222, Rank: 2, loss = 0.026123046875
Epoch: 5, Step: 222, Rank: 0, loss = 0.031494140625Epoch: 5, Step: 222, Rank: 6, loss = 0.043701171875

Epoch: 5, Step: 222, Rank: 4, loss = 0.03564453125
Epoch: 5, Step: 222, Rank: 3, loss = 0.033447265625


Epoch: 5, Step: 222, Rank: 1, loss = 0.0400390625


Epoch 5:  55%|█████▌    | 22/40 [00:37<00:31,  1.73s/it]{
    "epoch": 5,
    "step": 222,
    "rank": 0,
    "overall_throughput": 94.22667754600008,
    "lr": 2.4461726528100615e-06,
    "cuda_mem_allocated": 6.578186511993408,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56834,
    "batch_size": 127,
    "total_loss": 0.038955554773551046,
    "samples_seen": 28072,
    "gradnorm": 0.26953125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:56.258187"
}
 total length: 7994 dropped: 0 num samples 10 - rank: 0 max len: 1104 min len: 528 avg len: 799.4 num_loss_counted_tokens: 7464
 total length: 7791 dropped: 0 num samples 15 - rank: 3 max len: 539 min len: 501 avg len: 519.4 num_loss_counted_tokens: 6996
 total length: 7816 dropped: 0 num samples 12 - rank: 1 max len: 681 min len: 622 avg len: 651.3333333333334 num_loss_counted_tokens: 7180
 total length: 7997 dropped: 0 num samples 17 - rank: 4 max len: 495 min len: 449 avg len: 470.4117647058824 num_loss_counted_tokens: 7096
 total length: 7997 dropped: 0 num samples 14 - rank: 2 max len: 619 min len: 388 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7991 dropped: 0 num samples 21 - rank: 6 max len: 412 min len: 261 avg len: 380.5238095238095 num_loss_counted_tokens: 6878
 total length: 7821 dropped: 0 num samples 18 - rank: 5 max len: 449 min len: 416 avg len: 434.5 num_loss_counted_tokens: 6867
 total length: 7487 dropped: 0 num samples 23 - rank: 7 max len: 359 min len: 276 avg len: 325.5217391304348 num_loss_counted_tokens: 6268
Epoch: 5, Step: 223, Rank: 7, loss = 0.0556640625Epoch: 5, Step: 223, Rank: 1, loss = 0.033935546875Epoch: 5, Step: 223, Rank: 5, loss = 0.040283203125Epoch: 5, Step: 223, Rank: 6, loss = 0.0498046875Epoch: 5, Step: 223, Rank: 2, loss = 0.04345703125



Epoch: 5, Step: 223, Rank: 4, loss = 0.047607421875

Epoch: 5, Step: 223, Rank: 3, loss = 0.02978515625
Epoch: 5, Step: 223, Rank: 0, loss = 0.0179443359375


Epoch 5:  57%|█████▊    | 23/40 [00:38<00:28,  1.70s/it]{
    "epoch": 5,
    "step": 223,
    "rank": 0,
    "overall_throughput": 94.35142176263417,
    "lr": 2.3660171725889703e-06,
    "cuda_mem_allocated": 6.588095188140869,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56739,
    "batch_size": 130,
    "total_loss": 0.03979626006803081,
    "samples_seen": 28202,
    "gradnorm": 0.279296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:57.908094"
}
 total length: 7990 dropped: 0 num samples 15 - rank: 3 max len: 579 min len: 262 avg len: 532.6666666666666 num_loss_counted_tokens: 7195
 total length: 7918 dropped: 0 num samples 12 - rank: 1 max len: 684 min len: 637 avg len: 659.8333333333334 num_loss_counted_tokens: 7282
 total length: 7997 dropped: 0 num samples 10 - rank: 0 max len: 1185 min len: 416 avg len: 799.7 num_loss_counted_tokens: 7467
 total length: 7904 dropped: 0 num samples 20 - rank: 6 max len: 427 min len: 367 avg len: 395.2 num_loss_counted_tokens: 6844
 total length: 7967 dropped: 0 num samples 13 - rank: 2 max len: 633 min len: 583 avg len: 612.8461538461538 num_loss_counted_tokens: 7278
 total length: 7750 dropped: 0 num samples 17 - rank: 5 max len: 474 min len: 429 avg len: 455.88235294117646 num_loss_counted_tokens: 6849
 total length: 7946 dropped: 0 num samples 24 - rank: 7 max len: 365 min len: 256 avg len: 331.0833333333333 num_loss_counted_tokens: 6674
 total length: 7950 dropped: 0 num samples 16 - rank: 4 max len: 524 min len: 474 avg len: 496.875 num_loss_counted_tokens: 7102
Epoch: 5, Step: 224, Rank: 0, loss = 0.0196533203125Epoch: 5, Step: 224, Rank: 1, loss = 0.03564453125
Epoch: 5, Step: 224, Rank: 7, loss = 0.049560546875Epoch: 5, Step: 224, Rank: 6, loss = 0.0478515625Epoch: 5, Step: 224, Rank: 5, loss = 0.032958984375



Epoch: 5, Step: 224, Rank: 3, loss = 0.029541015625
Epoch: 5, Step: 224, Rank: 4, loss = 0.030517578125
Epoch: 5, Step: 224, Rank: 2, loss = 0.034912109375


Epoch 5:  60%|██████    | 24/40 [00:40<00:27,  1.69s/it]{
    "epoch": 5,
    "step": 224,
    "rank": 0,
    "overall_throughput": 94.54534363168534,
    "lr": 2.287020376528193e-06,
    "cuda_mem_allocated": 6.578826904296875,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56929,
    "batch_size": 122,
    "total_loss": 0.03502608512357498,
    "samples_seen": 28324,
    "gradnorm": 0.267578125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:42:59.559787"
}
Epoch: 5, Step: 225, Rank: 6, loss = 0.049560546875Epoch: 5, Step: 225, Rank: 5, loss = 0.05615234375Epoch: 5, Step: 225, Rank: 7, loss = 0.05078125

Epoch: 5, Step: 225, Rank: 3, loss = 0.03173828125Epoch: 5, Step: 225, Rank: 2, loss = 0.037353515625
Epoch: 5, Step: 225, Rank: 1, loss = 0.035888671875


Epoch: 5, Step: 225, Rank: 4, loss = 0.03662109375
Epoch: 5, Step: 225, Rank: 0, loss = 0.0286865234375


Epoch 5:  62%|██████▎   | 25/40 [00:42<00:25,  1.68s/it]{
    "epoch": 5,
    "step": 225,
    "rank": 0,
    "overall_throughput": 94.51074125503827,
    "lr": 2.209194254743295e-06,
    "cuda_mem_allocated": 6.588095188140869,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56514,
    "batch_size": 126,
    "total_loss": 0.040839438015359025,
    "samples_seen": 28450,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:43:01.208537"
}
Epoch: 5, Step: 226, Rank: 5, loss = 0.043212890625Epoch: 5, Step: 226, Rank: 7, loss = 0.060302734375Epoch: 5, Step: 226, Rank: 6, loss = 0.04833984375Epoch: 5, Step: 226, Rank: 2, loss = 0.039306640625Epoch: 5, Step: 226, Rank: 1, loss = 0.04345703125
Epoch: 5, Step: 226, Rank: 4, loss = 0.037841796875



Epoch: 5, Step: 226, Rank: 3, loss = 0.03466796875

Epoch: 5, Step: 226, Rank: 0, loss = 0.027099609375


Epoch 5:  65%|██████▌   | 26/40 [00:43<00:23,  1.67s/it]{
    "epoch": 5,
    "step": 226,
    "rank": 0,
    "overall_throughput": 94.7923719169843,
    "lr": 2.132550619665168e-06,
    "cuda_mem_allocated": 6.587979316711426,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56709,
    "batch_size": 129,
    "total_loss": 0.04175704032869562,
    "samples_seen": 28579,
    "gradnorm": 0.287109375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:43:02.854324"
}
Epoch: 5, Step: 227, Rank: 5, loss = 0.043212890625Epoch: 5, Step: 227, Rank: 7, loss = 0.055419921875Epoch: 5, Step: 227, Rank: 2, loss = 0.03125Epoch: 5, Step: 227, Rank: 1, loss = 0.0250244140625Epoch: 5, Step: 227, Rank: 3, loss = 0.034423828125
Epoch: 5, Step: 227, Rank: 4, loss = 0.041259765625Epoch: 5, Step: 227, Rank: 6, loss = 0.0498046875





Epoch: 5, Step: 227, Rank: 0, loss = 0.027099609375


Epoch 5:  68%|██████▊   | 27/40 [00:45<00:22,  1.75s/it]{
    "epoch": 5,
    "step": 227,
    "rank": 0,
    "overall_throughput": 77.83149503425844,
    "lr": 2.05710110424714e-06,
    "cuda_mem_allocated": 6.588095188140869,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56554,
    "batch_size": 126,
    "total_loss": 0.03847650033596209,
    "samples_seen": 28705,
    "gradnorm": 0.275390625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:43:04.792352"
}
Epoch: 5, Step: 228, Rank: 5, loss = 0.05419921875Epoch: 5, Step: 228, Rank: 1, loss = 0.03759765625Epoch: 5, Step: 228, Rank: 6, loss = 0.044921875Epoch: 5, Step: 228, Rank: 7, loss = 0.043212890625Epoch: 5, Step: 228, Rank: 3, loss = 0.042724609375


Epoch: 5, Step: 228, Rank: 2, loss = 0.0341796875

Epoch: 5, Step: 228, Rank: 4, loss = 0.03955078125

Epoch: 5, Step: 228, Rank: 0, loss = 0.025146484375


Epoch 5:  70%|███████   | 28/40 [00:47<00:21,  1.81s/it]{
    "epoch": 5,
    "step": 228,
    "rank": 0,
    "overall_throughput": 77.50166257218987,
    "lr": 1.982857160199334e-06,
    "cuda_mem_allocated": 6.587978363037109,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56675,
    "batch_size": 133,
    "total_loss": 0.040176444640494045,
    "samples_seen": 28838,
    "gradnorm": 0.27734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:43:06.734778"
}
Epoch: 5, Step: 229, Rank: 6, loss = 0.046142578125Epoch: 5, Step: 229, Rank: 0, loss = 0.0235595703125Epoch: 5, Step: 229, Rank: 7, loss = 0.047119140625Epoch: 5, Step: 229, Rank: 5, loss = 0.04150390625
Epoch: 5, Step: 229, Rank: 3, loss = 0.033203125

Epoch: 5, Step: 229, Rank: 2, loss = 0.03369140625Epoch: 5, Step: 229, Rank: 4, loss = 0.0400390625



Epoch: 5, Step: 229, Rank: 1, loss = 0.0272216796875


Epoch 5:  72%|███████▎  | 29/40 [00:49<00:19,  1.76s/it]{
    "epoch": 5,
    "step": 229,
    "rank": 0,
    "overall_throughput": 95.23253003029372,
    "lr": 1.9098300562505266e-06,
    "cuda_mem_allocated": 6.57878303527832,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56739,
    "batch_size": 126,
    "total_loss": 0.03657096529723823,
    "samples_seen": 28964,
    "gradnorm": 0.28125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:43:08.376249"
}
Epoch: 5, Step: 230, Rank: 6, loss = 0.0478515625Epoch: 5, Step: 230, Rank: 1, loss = 0.0281982421875Epoch: 5, Step: 230, Rank: 7, loss = 0.04931640625Epoch: 5, Step: 230, Rank: 3, loss = 0.0380859375

Epoch: 5, Step: 230, Rank: 5, loss = 0.045166015625
Epoch: 5, Step: 230, Rank: 2, loss = 0.02392578125


Epoch: 5, Step: 230, Rank: 4, loss = 0.039306640625
Epoch: 5, Step: 230, Rank: 0, loss = 0.031005859375


Epoch 5:  75%|███████▌  | 30/40 [00:51<00:18,  1.81s/it]{
    "epoch": 5,
    "step": 230,
    "rank": 0,
    "overall_throughput": 77.60919391351361,
    "lr": 1.8380308764377841e-06,
    "cuda_mem_allocated": 6.587978839874268,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56442,
    "batch_size": 135,
    "total_loss": 0.037861875908011766,
    "samples_seen": 29099,
    "gradnorm": 0.2734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:43:10.316460"
}
Epoch: 5, Step: 231, Rank: 1, loss = 0.034423828125Epoch: 5, Step: 231, Rank: 3, loss = 0.033203125Epoch: 5, Step: 231, Rank: 7, loss = 0.0498046875
Epoch: 5, Step: 231, Rank: 5, loss = 0.041748046875
Epoch: 5, Step: 231, Rank: 4, loss = 0.03271484375Epoch: 5, Step: 231, Rank: 6, loss = 0.0458984375Epoch: 5, Step: 231, Rank: 2, loss = 0.03759765625




Epoch: 5, Step: 231, Rank: 0, loss = 0.033935546875


Epoch 5:  78%|███████▊  | 31/40 [00:52<00:15,  1.76s/it]{
    "epoch": 5,
    "step": 231,
    "rank": 0,
    "overall_throughput": 94.79086561338168,
    "lr": 1.767470518424129e-06,
    "cuda_mem_allocated": 6.583086967468262,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56601,
    "batch_size": 132,
    "total_loss": 0.038638893305771985,
    "samples_seen": 29231,
    "gradnorm": 0.294921875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:43:11.960459"
}
Epoch: 5, Step: 232, Rank: 6, loss = 0.049072265625Epoch: 5, Step: 232, Rank: 7, loss = 0.052001953125
Epoch: 5, Step: 232, Rank: 2, loss = 0.032470703125Epoch: 5, Step: 232, Rank: 3, loss = 0.033935546875Epoch: 5, Step: 232, Rank: 4, loss = 0.041015625Epoch: 5, Step: 232, Rank: 5, loss = 0.0458984375Epoch: 5, Step: 232, Rank: 0, loss = 0.026611328125





Epoch: 5, Step: 232, Rank: 1, loss = 0.02685546875


Epoch 5:  80%|████████  | 32/40 [00:54<00:13,  1.73s/it]{
    "epoch": 5,
    "step": 232,
    "rank": 0,
    "overall_throughput": 94.87416907708533,
    "lr": 1.6981596918444953e-06,
    "cuda_mem_allocated": 6.57231330871582,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56125,
    "batch_size": 130,
    "total_loss": 0.03844988864142539,
    "samples_seen": 29361,
    "gradnorm": 0.298828125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:43:13.604311"
}
Epoch: 5, Step: 233, Rank: 6, loss = 0.05224609375Epoch: 5, Step: 233, Rank: 7, loss = 0.04638671875Epoch: 5, Step: 233, Rank: 5, loss = 0.037109375Epoch: 5, Step: 233, Rank: 2, loss = 0.037353515625Epoch: 5, Step: 233, Rank: 1, loss = 0.031494140625Epoch: 5, Step: 233, Rank: 4, loss = 0.046630859375


Epoch: 5, Step: 233, Rank: 3, loss = 0.032470703125



Epoch: 5, Step: 233, Rank: 0, loss = 0.033203125


Epoch 5:  82%|████████▎ | 33/40 [00:56<00:11,  1.70s/it]{
    "epoch": 5,
    "step": 233,
    "rank": 0,
    "overall_throughput": 93.76562144965068,
    "lr": 1.6301089166802232e-06,
    "cuda_mem_allocated": 6.588095188140869,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56222,
    "batch_size": 122,
    "total_loss": 0.03961082850129843,
    "samples_seen": 29483,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:43:15.254783"
}
Epoch: 5, Step: 234, Rank: 0, loss = 0.01397705078125Epoch: 5, Step: 234, Rank: 7, loss = 0.0517578125Epoch: 5, Step: 234, Rank: 6, loss = 0.049072265625Epoch: 5, Step: 234, Rank: 5, loss = 0.056396484375Epoch: 5, Step: 234, Rank: 3, loss = 0.03857421875



Epoch: 5, Step: 234, Rank: 2, loss = 0.033447265625
Epoch: 5, Step: 234, Rank: 4, loss = 0.040771484375

Epoch: 5, Step: 234, Rank: 1, loss = 0.028564453125


Epoch 5:  85%|████████▌ | 34/40 [00:57<00:10,  1.68s/it]{
    "epoch": 5,
    "step": 234,
    "rank": 0,
    "overall_throughput": 94.93756963983485,
    "lr": 1.5633285216623384e-06,
    "cuda_mem_allocated": 6.580846786499023,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56784,
    "batch_size": 127,
    "total_loss": 0.03907790927021696,
    "samples_seen": 29610,
    "gradnorm": 0.279296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:43:16.895888"
}
Epoch: 5, Step: 235, Rank: 7, loss = 0.048583984375Epoch: 5, Step: 235, Rank: 6, loss = 0.0419921875Epoch: 5, Step: 235, Rank: 1, loss = 0.024658203125

Epoch: 5, Step: 235, Rank: 4, loss = 0.049560546875Epoch: 5, Step: 235, Rank: 2, loss = 0.027099609375Epoch: 5, Step: 235, Rank: 5, loss = 0.050537109375Epoch: 5, Step: 235, Rank: 3, loss = 0.042236328125




Epoch: 5, Step: 235, Rank: 0, loss = 0.01220703125


Epoch 5:  88%|████████▊ | 35/40 [00:59<00:08,  1.68s/it]{
    "epoch": 5,
    "step": 235,
    "rank": 0,
    "overall_throughput": 93.83527268322771,
    "lr": 1.4978286427038602e-06,
    "cuda_mem_allocated": 6.585009574890137,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 55765,
    "batch_size": 125,
    "total_loss": 0.03708419259392092,
    "samples_seen": 29735,
    "gradnorm": 0.275390625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:43:18.551069"
}
Epoch: 5, Step: 236, Rank: 3, loss = 0.028076171875Epoch: 5, Step: 236, Rank: 5, loss = 0.042236328125Epoch: 5, Step: 236, Rank: 6, loss = 0.04443359375Epoch: 5, Step: 236, Rank: 0, loss = 0.0172119140625

Epoch: 5, Step: 236, Rank: 7, loss = 0.05078125
Epoch: 5, Step: 236, Rank: 4, loss = 0.041748046875Epoch: 5, Step: 236, Rank: 1, loss = 0.02685546875



Epoch: 5, Step: 236, Rank: 2, loss = 0.031982421875


Epoch 5:  90%|█████████ | 36/40 [01:01<00:06,  1.67s/it]{
    "epoch": 5,
    "step": 236,
    "rank": 0,
    "overall_throughput": 95.06142710355387,
    "lr": 1.4336192213613742e-06,
    "cuda_mem_allocated": 6.578734874725342,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56387,
    "batch_size": 123,
    "total_loss": 0.03541596467270825,
    "samples_seen": 29858,
    "gradnorm": 0.26953125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:43:20.193615"
}
Epoch: 5, Step: 237, Rank: 6, loss = 0.052978515625Epoch: 5, Step: 237, Rank: 5, loss = 0.041015625Epoch: 5, Step: 237, Rank: 7, loss = 0.0625Epoch: 5, Step: 237, Rank: 1, loss = 0.037841796875Epoch: 5, Step: 237, Rank: 2, loss = 0.035400390625




Epoch: 5, Step: 237, Rank: 4, loss = 0.04345703125Epoch: 5, Step: 237, Rank: 3, loss = 0.036376953125

Epoch: 5, Step: 237, Rank: 0, loss = 0.013671875


Epoch 5:  92%|█████████▎| 37/40 [01:02<00:04,  1.66s/it]{
    "epoch": 5,
    "step": 237,
    "rank": 0,
    "overall_throughput": 94.20643961959523,
    "lr": 1.3707100033261035e-06,
    "cuda_mem_allocated": 6.57658576965332,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56791,
    "batch_size": 125,
    "total_loss": 0.04039372435773274,
    "samples_seen": 29983,
    "gradnorm": 0.30078125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:43:21.839439"
}
Epoch: 5, Step: 238, Rank: 1, loss = 0.037109375Epoch: 5, Step: 238, Rank: 6, loss = 0.048828125Epoch: 5, Step: 238, Rank: 3, loss = 0.03125Epoch: 5, Step: 238, Rank: 5, loss = 0.047119140625Epoch: 5, Step: 238, Rank: 7, loss = 0.0517578125
Epoch: 5, Step: 238, Rank: 2, loss = 0.04541015625




Epoch: 5, Step: 238, Rank: 4, loss = 0.0390625
Epoch: 5, Step: 238, Rank: 0, loss = 0.0218505859375


Epoch 5:  95%|█████████▌| 38/40 [01:04<00:03,  1.66s/it]{
    "epoch": 5,
    "step": 238,
    "rank": 0,
    "overall_throughput": 94.75974605341958,
    "lr": 1.3091105369447166e-06,
    "cuda_mem_allocated": 6.584794044494629,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56657,
    "batch_size": 130,
    "total_loss": 0.04029510916568121,
    "samples_seen": 30113,
    "gradnorm": 0.2734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:43:23.484669"
}
Epoch: 5, Step: 239, Rank: 5, loss = 0.05419921875Epoch: 5, Step: 239, Rank: 6, loss = 0.047607421875Epoch: 5, Step: 239, Rank: 2, loss = 0.032470703125Epoch: 5, Step: 239, Rank: 3, loss = 0.0341796875
Epoch: 5, Step: 239, Rank: 1, loss = 0.0267333984375
Epoch: 5, Step: 239, Rank: 7, loss = 0.047607421875

Epoch: 5, Step: 239, Rank: 4, loss = 0.0419921875


Epoch: 5, Step: 239, Rank: 0, loss = 0.0250244140625


Epoch 5:  98%|█████████▊| 39/40 [01:06<00:01,  1.65s/it]{
    "epoch": 5,
    "step": 239,
    "rank": 0,
    "overall_throughput": 94.71278753654023,
    "lr": 1.2488301717700735e-06,
    "cuda_mem_allocated": 6.588095188140869,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56004,
    "batch_size": 130,
    "total_loss": 0.038747232340547104,
    "samples_seen": 30243,
    "gradnorm": 0.294921875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:43:25.128892"
}
Epoch: 5, Step: 240, Rank: 7, loss = 0.048095703125Epoch: 5, Step: 240, Rank: 1, loss = 0.052490234375Epoch: 5, Step: 240, Rank: 6, loss = 0.04541015625Epoch: 5, Step: 240, Rank: 5, loss = 0.04052734375Epoch: 5, Step: 240, Rank: 3, loss = 0.03564453125

Epoch: 5, Step: 240, Rank: 2, loss = 0.0390625



Epoch: 5, Step: 240, Rank: 0, loss = 0.03955078125
Epoch: 5, Step: 240, Rank: 4, loss = 0.035888671875


Epoch 5: 100%|██████████| 40/40 [01:07<00:00,  1.74s/it]{
    "epoch": 5,
    "step": 240,
    "rank": 0,
    "overall_throughput": 78.11415207142485,
    "lr": 1.1898780571421554e-06,
    "cuda_mem_allocated": 6.587979316711426,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56691,
    "batch_size": 127,
    "total_loss": 0.04205253038401157,
    "samples_seen": 30370,
    "gradnorm": 0.3046875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:43:27.060796"
}
Saving model in huggingface format at: samples_30370
[19:43:53] INFO     The model is bigger than the maximum     accelerator.py:2924
                    size per checkpoint (5GB) and is going                      
                    to be split in 4 checkpoint shards. You                     
                    can find where each parameters has been                     
                    saved in the index located at                               
                    /tmp/tmpsexwu5eaw/model.safetensors.inde                    
                    x.json.                                                     
Model saved in /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/hf_format/samples_30370
[19:44:09] INFO     saving took 41.67494249343872 seconds           utils.py:879
Saving full model state in /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/full_state/epoch_5
           INFO     Saving current state to                  accelerator.py:3030
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_5                                                  
           INFO     Saving FSDP model                        accelerator.py:3040
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
[19:44:17] INFO     Saving model to                             fsdp_utils.py:88
                    /var/home/cloud-user/.local/share/instructl                 
                    ab/phased/phase1/checkpoints/full_state/epo                 
                    ch_5/pytorch_model_fsdp.bin                                 
[19:44:28] INFO     Model saved to                              fsdp_utils.py:90
                    /var/home/cloud-user/.local/share/instructl                 
                    ab/phased/phase1/checkpoints/full_state/epo                 
                    ch_5/pytorch_model_fsdp.bin                                 
           INFO     FSDP Model saved to output dir           accelerator.py:3042
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_5                                                  
           INFO     Saving FSDP Optimizer                    accelerator.py:3059
 total length: 7679 dropped: 0 num samples 24 - rank: 7 max len: 367 min len: 269 avg len: 319.9583333333333 num_loss_counted_tokens: 6407
 total length: 7996 dropped: 0 num samples 14 - rank: 2 max len: 623 min len: 310 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7978 dropped: 0 num samples 25 - rank: 7 max len: 381 min len: 236 avg len: 319.12 num_loss_counted_tokens: 6653 total length: 7995 dropped: 0 num samples 14 - rank: 2 max len: 615 min len: 301 avg len: 571.0714285714286 num_loss_counted_tokens: 7253
 total length: 7988 dropped: 0 num samples 13 - rank: 2 max len: 650 min len: 578 avg len: 614.4615384615385 num_loss_counted_tokens: 7299

 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 646 min len: 501 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7744 dropped: 0 num samples 23 - rank: 7 max len: 365 min len: 257 avg len: 336.69565217391306 num_loss_counted_tokens: 6525
 total length: 7992 dropped: 0 num samples 13 - rank: 2 max len: 665 min len: 280 avg len: 614.7692307692307 num_loss_counted_tokens: 7303
[19:44:38] INFO     Saving Optimizer state to                  fsdp_utils.py:192
                    /var/home/cloud-user/.local/share/instruct                  
                    lab/phased/phase1/checkpoints/full_state/e                  
                    poch_5/optimizer.bin                                        
 total length: 7849 dropped: 0 num samples 13 - rank: 2 max len: 626 min len: 574 avg len: 603.7692307692307 num_loss_counted_tokens: 7160 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 642 min len: 539 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7977 dropped: 0 num samples 13 - rank: 2 max len: 631 min len: 596 avg len: 613.6153846153846 num_loss_counted_tokens: 7288
 total length: 7996 dropped: 0 num samples 14 - rank: 2 max len: 625 min len: 439 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7803 dropped: 0 num samples 25 - rank: 7 max len: 352 min len: 241 avg len: 312.12 num_loss_counted_tokens: 6478
 total length: 7645 dropped: 0 num samples 24 - rank: 7 max len: 368 min len: 248 avg len: 318.5416666666667 num_loss_counted_tokens: 6373
 total length: 7797 dropped: 0 num samples 25 - rank: 7 max len: 371 min len: 259 avg len: 311.88 num_loss_counted_tokens: 6472
 total length: 7996 dropped: 0 num samples 15 - rank: 3 max len: 569 min len: 348 avg len: 533.0666666666667 num_loss_counted_tokens: 7201
 total length: 7829 dropped: 0 num samples 13 - rank: 2 max len: 618 min len: 581 avg len: 602.2307692307693 num_loss_counted_tokens: 7140
 total length: 7774 dropped: 0 num samples 18 - rank: 5 max len: 460 min len: 412 avg len: 431.8888888888889 num_loss_counted_tokens: 6820
 total length: 7817 dropped: 0 num samples 17 - rank: 5 max len: 487 min len: 440 avg len: 459.8235294117647 num_loss_counted_tokens: 6916
 total length: 7908 dropped: 0 num samples 13 - rank: 2 max len: 629 min len: 583 avg len: 608.3076923076923 num_loss_counted_tokens: 7219
 total length: 7994 dropped: 0 num samples 18 - rank: 5 max len: 468 min len: 344 avg len: 444.1111111111111 num_loss_counted_tokens: 7040
 total length: 7849 dropped: 0 num samples 25 - rank: 7 max len: 364 min len: 245 avg len: 313.96 num_loss_counted_tokens: 6524
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 479 min len: 257 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7994 dropped: 0 num samples 13 - rank: 1 max len: 667 min len: 323 avg len: 614.9230769230769 num_loss_counted_tokens: 7305

 total length: 7882 dropped: 0 num samples 17 - rank: 5 max len: 484 min len: 437 avg len: 463.6470588235294 num_loss_counted_tokens: 6981
 total length: 7935 dropped: 0 num samples 11 - rank: 1 max len: 825 min len: 665 avg len: 721.3636363636364 num_loss_counted_tokens: 7352
 total length: 7767 dropped: 0 num samples 12 - rank: 1 max len: 673 min len: 630 avg len: 647.25 num_loss_counted_tokens: 7131
 total length: 7915 dropped: 0 num samples 12 - rank: 1 max len: 707 min len: 630 avg len: 659.5833333333334 num_loss_counted_tokens: 7279
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 468 min len: 412 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7919 dropped: 0 num samples 14 - rank: 3 max len: 604 min len: 530 avg len: 565.6428571428571 num_loss_counted_tokens: 7177
 total length: 7992 dropped: 0 num samples 12 - rank: 1 max len: 703 min len: 611 avg len: 666.0 num_loss_counted_tokens: 7356
 total length: 7708 dropped: 0 num samples 24 - rank: 7 max len: 366 min len: 253 avg len: 321.1666666666667 num_loss_counted_tokens: 6436
 total length: 7996 dropped: 0 num samples 15 - rank: 3 max len: 568 min len: 357 avg len: 533.0666666666667 num_loss_counted_tokens: 7201
 total length: 7910 dropped: 0 num samples 13 - rank: 2 max len: 631 min len: 586 avg len: 608.4615384615385 num_loss_counted_tokens: 7221
 total length: 7945 dropped: 0 num samples 10 - rank: 1 max len: 895 min len: 665 avg len: 794.5 num_loss_counted_tokens: 7415
 total length: 7753 dropped: 0 num samples 13 - rank: 3 max len: 619 min len: 545 avg len: 596.3846153846154 num_loss_counted_tokens: 7064
 total length: 7994 dropped: 0 num samples 12 - rank: 1 max len: 780 min len: 337 avg len: 666.1666666666666 num_loss_counted_tokens: 7358
 total length: 7296 dropped: 0 num samples 22 - rank: 7 max len: 377 min len: 257 avg len: 331.6363636363636 num_loss_counted_tokens: 6130
 total length: 7986 dropped: 0 num samples 15 - rank: 3 max len: 582 min len: 274 avg len: 532.4 num_loss_counted_tokens: 7191
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 571 min len: 481 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7997 dropped: 0 num samples 18 - rank: 5 max len: 460 min len: 422 avg len: 444.27777777777777 num_loss_counted_tokens: 7043
 total length: 7598 dropped: 0 num samples 23 - rank: 7 max len: 373 min len: 243 avg len: 330.3478260869565 num_loss_counted_tokens: 6379
 total length: 7532 dropped: 0 num samples 24 - rank: 7 max len: 364 min len: 243 avg len: 313.8333333333333 num_loss_counted_tokens: 6260
 total length: 7435 dropped: 0 num samples 23 - rank: 7 max len: 360 min len: 281 avg len: 323.2608695652174 num_loss_counted_tokens: 6216
 total length: 7953 dropped: 0 num samples 18 - rank: 5 max len: 480 min len: 236 avg len: 441.8333333333333 num_loss_counted_tokens: 6999
 total length: 7764 dropped: 0 num samples 12 - rank: 1 max len: 678 min len: 632 avg len: 647.0 num_loss_counted_tokens: 7128
 total length: 7661 dropped: 0 num samples 23 - rank: 7 max len: 372 min len: 261 avg len: 333.0869565217391 num_loss_counted_tokens: 6442
 total length: 7791 dropped: 0 num samples 12 - rank: 2 max len: 675 min len: 628 avg len: 649.25 num_loss_counted_tokens: 7155
 total length: 7996 dropped: 0 num samples 14 - rank: 2 max len: 615 min len: 366 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7935 dropped: 0 num samples 13 - rank: 2 max len: 647 min len: 555 avg len: 610.3846153846154 num_loss_counted_tokens: 7246
 total length: 7994 dropped: 0 num samples 12 - rank: 1 max len: 783 min len: 359 avg len: 666.1666666666666 num_loss_counted_tokens: 7358
 total length: 7914 dropped: 0 num samples 25 - rank: 7 max len: 365 min len: 243 avg len: 316.56 num_loss_counted_tokens: 6589
 total length: 7924 dropped: 0 num samples 18 - rank: 5 max len: 477 min len: 410 avg len: 440.22222222222223 num_loss_counted_tokens: 6970
 total length: 7910 dropped: 0 num samples 11 - rank: 1 max len: 826 min len: 679 avg len: 719.0909090909091 num_loss_counted_tokens: 7327
 total length: 7813 dropped: 0 num samples 14 - rank: 3 max len: 573 min len: 544 avg len: 558.0714285714286 num_loss_counted_tokens: 7071
 total length: 7904 dropped: 0 num samples 12 - rank: 1 max len: 686 min len: 636 avg len: 658.6666666666666 num_loss_counted_tokens: 7268
 total length: 7997 dropped: 0 num samples 13 - rank: 1 max len: 654 min len: 300 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7798 dropped: 0 num samples 13 - rank: 3 max len: 623 min len: 582 avg len: 599.8461538461538 num_loss_counted_tokens: 7109
 total length: 7995 dropped: 0 num samples 13 - rank: 1 max len: 657 min len: 338 avg len: 615.0 num_loss_counted_tokens: 7306
 total length: 7769 dropped: 0 num samples 17 - rank: 5 max len: 486 min len: 437 avg len: 457.0 num_loss_counted_tokens: 6868
 total length: 7930 dropped: 0 num samples 15 - rank: 3 max len: 551 min len: 509 avg len: 528.6666666666666 num_loss_counted_tokens: 7135
 total length: 7962 dropped: 0 num samples 18 - rank: 5 max len: 466 min len: 420 avg len: 442.3333333333333 num_loss_counted_tokens: 7008
 total length: 7759 dropped: 0 num samples 24 - rank: 7 max len: 357 min len: 287 avg len: 323.2916666666667 num_loss_counted_tokens: 6487
 total length: 7990 dropped: 0 num samples 13 - rank: 1 max len: 665 min len: 267 avg len: 614.6153846153846 num_loss_counted_tokens: 7301
 total length: 7962 dropped: 0 num samples 15 - rank: 3 max len: 548 min len: 514 avg len: 530.8 num_loss_counted_tokens: 7167
 total length: 7748 dropped: 0 num samples 18 - rank: 5 max len: 450 min len: 412 avg len: 430.44444444444446 num_loss_counted_tokens: 6794
 total length: 7834 dropped: 0 num samples 14 - rank: 3 max len: 574 min len: 532 avg len: 559.5714285714286 num_loss_counted_tokens: 7092
 total length: 7696 dropped: 0 num samples 24 - rank: 7 max len: 361 min len: 256 avg len: 320.6666666666667 num_loss_counted_tokens: 6424
 total length: 7779 dropped: 0 num samples 14 - rank: 3 max len: 583 min len: 528 avg len: 555.6428571428571 num_loss_counted_tokens: 7037
 total length: 7878 dropped: 0 num samples 18 - rank: 5 max len: 468 min len: 401 avg len: 437.6666666666667 num_loss_counted_tokens: 6924
 total length: 7958 dropped: 0 num samples 14 - rank: 3 max len: 590 min len: 543 avg len: 568.4285714285714 num_loss_counted_tokens: 7216
 total length: 7786 dropped: 0 num samples 17 - rank: 5 max len: 479 min len: 442 avg len: 458.0 num_loss_counted_tokens: 6885
 total length: 7779 dropped: 0 num samples 20 - rank: 6 max len: 408 min len: 370 avg len: 388.95 num_loss_counted_tokens: 6719
 total length: 7997 dropped: 0 num samples 14 - rank: 2 max len: 617 min len: 410 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7997 dropped: 0 num samples 20 - rank: 6 max len: 425 min len: 334 avg len: 399.85 num_loss_counted_tokens: 6937
 total length: 7845 dropped: 0 num samples 12 - rank: 1 max len: 672 min len: 639 avg len: 653.75 num_loss_counted_tokens: 7209
 total length: 7994 dropped: 0 num samples 13 - rank: 2 max len: 637 min len: 532 avg len: 614.9230769230769 num_loss_counted_tokens: 7305
 total length: 7745 dropped: 0 num samples 19 - rank: 6 max len: 435 min len: 372 avg len: 407.63157894736844 num_loss_counted_tokens: 6738
 total length: 7996 dropped: 0 num samples 20 - rank: 6 max len: 423 min len: 348 avg len: 399.8 num_loss_counted_tokens: 6936
 total length: 7831 dropped: 0 num samples 25 - rank: 7 max len: 364 min len: 253 avg len: 313.24 num_loss_counted_tokens: 6506
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 555 min len: 437 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7993 dropped: 0 num samples 13 - rank: 1 max len: 652 min len: 345 avg len: 614.8461538461538 num_loss_counted_tokens: 7304
 total length: 7977 dropped: 0 num samples 20 - rank: 6 max len: 436 min len: 366 avg len: 398.85 num_loss_counted_tokens: 6917
 total length: 7815 dropped: 0 num samples 16 - rank: 4 max len: 508 min len: 469 avg len: 488.4375 num_loss_counted_tokens: 6967
 total length: 7768 dropped: 0 num samples 16 - rank: 4 max len: 516 min len: 470 avg len: 485.5 num_loss_counted_tokens: 6920
 total length: 7976 dropped: 0 num samples 18 - rank: 5 max len: 474 min len: 411 avg len: 443.1111111111111 num_loss_counted_tokens: 7022
 total length: 7992 dropped: 0 num samples 14 - rank: 3 max len: 586 min len: 543 avg len: 570.8571428571429 num_loss_counted_tokens: 7250
 total length: 7947 dropped: 0 num samples 19 - rank: 5 max len: 453 min len: 406 avg len: 418.2631578947368 num_loss_counted_tokens: 6940
 total length: 7816 dropped: 0 num samples 19 - rank: 6 max len: 437 min len: 382 avg len: 411.36842105263156 num_loss_counted_tokens: 6809
 total length: 7994 dropped: 0 num samples 16 - rank: 4 max len: 544 min len: 291 avg len: 499.625 num_loss_counted_tokens: 7146 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 517 min len: 454 avg len: 499.8125 num_loss_counted_tokens: 7149

 total length: 7923 dropped: 0 num samples 15 - rank: 3 max len: 559 min len: 505 avg len: 528.2 num_loss_counted_tokens: 7128
 total length: 7991 dropped: 0 num samples 20 - rank: 6 max len: 431 min len: 243 avg len: 399.55 num_loss_counted_tokens: 6931
 total length: 7775 dropped: 0 num samples 16 - rank: 4 max len: 508 min len: 468 avg len: 485.9375 num_loss_counted_tokens: 6927
 total length: 7982 dropped: 0 num samples 17 - rank: 4 max len: 499 min len: 305 avg len: 469.52941176470586 num_loss_counted_tokens: 7081
 total length: 7982 dropped: 0 num samples 20 - rank: 6 max len: 433 min len: 371 avg len: 399.1 num_loss_counted_tokens: 6922
 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 525 min len: 425 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7928 dropped: 0 num samples 20 - rank: 6 max len: 427 min len: 370 avg len: 396.4 num_loss_counted_tokens: 6868
 total length: 7771 dropped: 0 num samples 12 - rank: 1 max len: 667 min len: 638 avg len: 647.5833333333334 num_loss_counted_tokens: 7135
 total length: 7797 dropped: 0 num samples 20 - rank: 6 max len: 423 min len: 367 avg len: 389.85 num_loss_counted_tokens: 6737
 total length: 7879 dropped: 0 num samples 21 - rank: 6 max len: 393 min len: 356 avg len: 375.1904761904762 num_loss_counted_tokens: 6766
 total length: 7990 dropped: 0 num samples 15 - rank: 4 max len: 577 min len: 425 avg len: 532.6666666666666 num_loss_counted_tokens: 7195
 total length: 7791 dropped: 0 num samples 16 - rank: 4 max len: 519 min len: 468 avg len: 486.9375 num_loss_counted_tokens: 6943
 total length: 7763 dropped: 0 num samples 20 - rank: 6 max len: 405 min len: 369 avg len: 388.15 num_loss_counted_tokens: 6703
 total length: 7826 dropped: 0 num samples 20 - rank: 6 max len: 407 min len: 373 avg len: 391.3 num_loss_counted_tokens: 6766
 total length: 7812 dropped: 0 num samples 20 - rank: 6 max len: 411 min len: 373 avg len: 390.6 num_loss_counted_tokens: 6752
 total length: 7994 dropped: 0 num samples 19 - rank: 5 max len: 453 min len: 354 avg len: 420.7368421052632 num_loss_counted_tokens: 6987
 total length: 7992 dropped: 0 num samples 15 - rank: 3 max len: 584 min len: 447 avg len: 532.8 num_loss_counted_tokens: 7197
 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 529 min len: 404 avg len: 499.75 num_loss_counted_tokens: 7148
 total length: 7801 dropped: 0 num samples 20 - rank: 6 max len: 414 min len: 368 avg len: 390.05 num_loss_counted_tokens: 6741
 total length: 7800 dropped: 0 num samples 15 - rank: 4 max len: 540 min len: 485 avg len: 520.0 num_loss_counted_tokens: 7005
 total length: 7986 dropped: 0 num samples 17 - rank: 4 max len: 505 min len: 340 avg len: 469.7647058823529 num_loss_counted_tokens: 7085
 total length: 7997 dropped: 0 num samples 16 - rank: 4 max len: 542 min len: 390 avg len: 499.8125 num_loss_counted_tokens: 7149
 total length: 7726 dropped: 0 num samples 20 - rank: 6 max len: 407 min len: 365 avg len: 386.3 num_loss_counted_tokens: 6666
 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 526 min len: 427 avg len: 499.75 num_loss_counted_tokens: 7148
 total length: 7994 dropped: 0 num samples 17 - rank: 4 max len: 512 min len: 363 avg len: 470.2352941176471 num_loss_counted_tokens: 7093
 total length: 7984 dropped: 0 num samples 17 - rank: 4 max len: 501 min len: 261 avg len: 469.6470588235294 num_loss_counted_tokens: 7083
 total length: 7996 dropped: 0 num samples 21 - rank: 6 max len: 400 min len: 324 avg len: 380.76190476190476 num_loss_counted_tokens: 6883
 total length: 7995 dropped: 0 num samples 16 - rank: 4 max len: 541 min len: 366 avg len: 499.6875 num_loss_counted_tokens: 7147
[19:45:01] INFO     Optimizer state saved in                   fsdp_utils.py:194
                    /var/home/cloud-user/.local/share/instruct                  
                    lab/phased/phase1/checkpoints/full_state/e                  
                    poch_5/optimizer.bin                                        
           INFO     FSDP Optimizer saved to output dir       accelerator.py:3061
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_5                                                  
           INFO     Scheduler state saved in                checkpointing.py:118
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_5/scheduler.bin                                  
           INFO     Sampler state for dataloader 0 saved in checkpointing.py:135
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_5/sampler.bin                                    
           INFO     Random states saved in                  checkpointing.py:160
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_5/random_states_0.pkl                            
Saving training state: {'current_epoch': 5, 'samples_seen': 30370}
Model state saved in: /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/full_state/epoch_5

Epoch 6:   0%|          | 0/40 [00:00<?, ?it/s]
Epoch 5: 100%|██████████| 40/40 [02:42<00:00,  4.06s/it]
 total length: 7935 dropped: 0 num samples 10 - rank: 0 max len: 1104 min len: 670 avg len: 793.5 num_loss_counted_tokens: 7405
 total length: 7983 dropped: 0 num samples 7 - rank: 0 max len: 1391 min len: 950 avg len: 1140.4285714285713 num_loss_counted_tokens: 7612
 total length: 7994 dropped: 0 num samples 10 - rank: 0 max len: 1062 min len: 454 avg len: 799.4 num_loss_counted_tokens: 7464
 total length: 7878 dropped: 0 num samples 8 - rank: 0 max len: 1101 min len: 825 avg len: 984.75 num_loss_counted_tokens: 7454
 total length: 7996 dropped: 0 num samples 9 - rank: 0 max len: 1076 min len: 282 avg len: 888.4444444444445 num_loss_counted_tokens: 7519
 total length: 7989 dropped: 0 num samples 9 - rank: 0 max len: 1087 min len: 688 avg len: 887.6666666666666 num_loss_counted_tokens: 7512
 total length: 7997 dropped: 0 num samples 9 - rank: 0 max len: 1196 min len: 464 avg len: 888.5555555555555 num_loss_counted_tokens: 7520
 total length: 7900 dropped: 0 num samples 10 - rank: 0 max len: 1060 min len: 661 avg len: 790.0 num_loss_counted_tokens: 7370
 total length: 7837 dropped: 0 num samples 8 - rank: 0 max len: 1155 min len: 838 avg len: 979.625 num_loss_counted_tokens: 7413
 total length: 7993 dropped: 0 num samples 10 - rank: 0 max len: 1098 min len: 551 avg len: 799.3 num_loss_counted_tokens: 7463
 total length: 7996 dropped: 0 num samples 11 - rank: 0 max len: 1057 min len: 542 avg len: 726.9090909090909 num_loss_counted_tokens: 7413
 total length: 7976 dropped: 0 num samples 8 - rank: 0 max len: 1164 min len: 781 avg len: 997.0 num_loss_counted_tokens: 7552
 total length: 7984 dropped: 0 num samples 10 - rank: 0 max len: 972 min len: 672 avg len: 798.4 num_loss_counted_tokens: 7454
 total length: 7993 dropped: 0 num samples 10 - rank: 0 max len: 1185 min len: 301 avg len: 799.3 num_loss_counted_tokens: 7463
 total length: 7897 dropped: 0 num samples 10 - rank: 0 max len: 1084 min len: 689 avg len: 789.7 num_loss_counted_tokens: 7367
 total length: 7989 dropped: 0 num samples 10 - rank: 0 max len: 1098 min len: 430 avg len: 798.9 num_loss_counted_tokens: 7459
 total length: 7997 dropped: 0 num samples 11 - rank: 0 max len: 1084 min len: 403 avg len: 727.0 num_loss_counted_tokens: 7414
Epoch: 6, Step: 241, Rank: 1, loss = 0.0296630859375Epoch: 6, Step: 241, Rank: 5, loss = 0.044189453125Epoch: 6, Step: 241, Rank: 2, loss = 0.0303955078125Epoch: 6, Step: 241, Rank: 3, loss = 0.038330078125
Epoch: 6, Step: 241, Rank: 6, loss = 0.045654296875
Epoch: 6, Step: 241, Rank: 4, loss = 0.042236328125
Epoch: 6, Step: 241, Rank: 7, loss = 0.059326171875



Epoch: 6, Step: 241, Rank: 0, loss = 0.0361328125

Epoch 6:   2%|▎         | 1/40 [00:01<01:04,  1.65s/it]{
    "epoch": 6,
    "step": 241,
    "rank": 0,
    "overall_throughput": 94.30991964338413,
    "lr": 1.132263140799381e-06,
    "cuda_mem_allocated": 6.588095188140869,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56078,
    "batch_size": 130,
    "total_loss": 0.04072898462855309,
    "samples_seen": 30500,
    "gradnorm": 0.30078125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:03.035185"
}
 total length: 7735 dropped: 0 num samples 24 - rank: 7 max len: 357 min len: 285 avg len: 322.2916666666667 num_loss_counted_tokens: 6463
 total length: 7909 dropped: 0 num samples 12 - rank: 1 max len: 695 min len: 644 avg len: 659.0833333333334 num_loss_counted_tokens: 7273
 total length: 7970 dropped: 0 num samples 15 - rank: 3 max len: 588 min len: 248 avg len: 531.3333333333334 num_loss_counted_tokens: 7175
 total length: 7859 dropped: 0 num samples 16 - rank: 4 max len: 520 min len: 467 avg len: 491.1875 num_loss_counted_tokens: 7011
 total length: 7786 dropped: 0 num samples 20 - rank: 6 max len: 414 min len: 357 avg len: 389.3 num_loss_counted_tokens: 6726
 total length: 7920 dropped: 0 num samples 9 - rank: 0 max len: 1482 min len: 703 avg len: 880.0 num_loss_counted_tokens: 7443
 total length: 7952 dropped: 0 num samples 18 - rank: 5 max len: 461 min len: 416 avg len: 441.77777777777777 num_loss_counted_tokens: 6998
 total length: 7962 dropped: 0 num samples 13 - rank: 2 max len: 637 min len: 590 avg len: 612.4615384615385 num_loss_counted_tokens: 7273
Epoch: 6, Step: 242, Rank: 5, loss = 0.04052734375Epoch: 6, Step: 242, Rank: 3, loss = 0.033935546875Epoch: 6, Step: 242, Rank: 1, loss = 0.0361328125Epoch: 6, Step: 242, Rank: 6, loss = 0.044677734375

Epoch: 6, Step: 242, Rank: 4, loss = 0.0380859375Epoch: 6, Step: 242, Rank: 2, loss = 0.032958984375Epoch: 6, Step: 242, Rank: 0, loss = 0.03076171875




Epoch: 6, Step: 242, Rank: 7, loss = 0.05224609375

Epoch 6:   5%|▌         | 2/40 [00:03<01:02,  1.64s/it]{
    "epoch": 6,
    "step": 242,
    "rank": 0,
    "overall_throughput": 95.18324556462643,
    "lr": 1.0759941675205221e-06,
    "cuda_mem_allocated": 6.582131862640381,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56721,
    "batch_size": 131,
    "total_loss": 0.03866292907388798,
    "samples_seen": 30631,
    "gradnorm": 0.283203125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:04.671039"
}
 total length: 7986 dropped: 0 num samples 9 - rank: 0 max len: 1104 min len: 720 avg len: 887.3333333333334 num_loss_counted_tokens: 7509
 total length: 7323 dropped: 0 num samples 23 - rank: 7 max len: 357 min len: 266 avg len: 318.39130434782606 num_loss_counted_tokens: 6104
 total length: 7993 dropped: 0 num samples 15 - rank: 3 max len: 558 min len: 394 avg len: 532.8666666666667 num_loss_counted_tokens: 7198
 total length: 7994 dropped: 0 num samples 16 - rank: 4 max len: 526 min len: 344 avg len: 499.625 num_loss_counted_tokens: 7146
 total length: 7973 dropped: 0 num samples 14 - rank: 2 max len: 629 min len: 171 avg len: 569.5 num_loss_counted_tokens: 7231
 total length: 7990 dropped: 0 num samples 18 - rank: 5 max len: 470 min len: 307 avg len: 443.8888888888889 num_loss_counted_tokens: 7036
 total length: 7819 dropped: 0 num samples 20 - rank: 6 max len: 426 min len: 364 avg len: 390.95 num_loss_counted_tokens: 6759
 total length: 7889 dropped: 0 num samples 12 - rank: 1 max len: 706 min len: 633 avg len: 657.4166666666666 num_loss_counted_tokens: 7253
Epoch: 6, Step: 243, Rank: 6, loss = 0.041748046875Epoch: 6, Step: 243, Rank: 5, loss = 0.038330078125Epoch: 6, Step: 243, Rank: 3, loss = 0.04052734375Epoch: 6, Step: 243, Rank: 1, loss = 0.0250244140625
Epoch: 6, Step: 243, Rank: 4, loss = 0.043701171875

Epoch: 6, Step: 243, Rank: 2, loss = 0.033203125Epoch: 6, Step: 243, Rank: 7, loss = 0.0537109375



Epoch: 6, Step: 243, Rank: 0, loss = 0.0233154296875

Epoch 6:   8%|▊         | 3/40 [00:04<01:00,  1.65s/it]{
    "epoch": 6,
    "step": 243,
    "rank": 0,
    "overall_throughput": 93.87734292965987,
    "lr": 1.0210796777974196e-06,
    "cuda_mem_allocated": 6.586592674255371,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 57019,
    "batch_size": 120,
    "total_loss": 0.037443659131166804,
    "samples_seen": 30751,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:06.327687"
}
 total length: 7848 dropped: 0 num samples 9 - rank: 0 max len: 1076 min len: 705 avg len: 872.0 num_loss_counted_tokens: 7371
 total length: 7980 dropped: 0 num samples 13 - rank: 2 max len: 647 min len: 422 avg len: 613.8461538461538 num_loss_counted_tokens: 7291
 total length: 7997 dropped: 0 num samples 14 - rank: 3 max len: 612 min len: 352 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7997 dropped: 0 num samples 20 - rank: 6 max len: 453 min len: 319 avg len: 399.85 num_loss_counted_tokens: 6937
 total length: 7997 dropped: 0 num samples 15 - rank: 4 max len: 556 min len: 472 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7850 dropped: 0 num samples 24 - rank: 7 max len: 372 min len: 273 avg len: 327.0833333333333 num_loss_counted_tokens: 6578
 total length: 7987 dropped: 0 num samples 12 - rank: 1 max len: 700 min len: 586 avg len: 665.5833333333334 num_loss_counted_tokens: 7351
 total length: 7798 dropped: 0 num samples 16 - rank: 5 max len: 520 min len: 456 avg len: 487.375 num_loss_counted_tokens: 6950
Epoch: 6, Step: 244, Rank: 6, loss = 0.05029296875Epoch: 6, Step: 244, Rank: 5, loss = 0.048828125Epoch: 6, Step: 244, Rank: 3, loss = 0.04052734375Epoch: 6, Step: 244, Rank: 2, loss = 0.036865234375


Epoch: 6, Step: 244, Rank: 1, loss = 0.03759765625
Epoch: 6, Step: 244, Rank: 4, loss = 0.039306640625

Epoch: 6, Step: 244, Rank: 7, loss = 0.05224609375
Epoch: 6, Step: 244, Rank: 0, loss = 0.0133056640625

Epoch 6:  10%|█         | 4/40 [00:06<00:59,  1.64s/it]{
    "epoch": 6,
    "step": 244,
    "rank": 0,
    "overall_throughput": 94.70588727148476,
    "lr": 9.675280065387117e-07,
    "cuda_mem_allocated": 6.577012538909912,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56570,
    "batch_size": 124,
    "total_loss": 0.039879794944316774,
    "samples_seen": 30875,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:07.967602"
}
 total length: 7785 dropped: 0 num samples 25 - rank: 7 max len: 359 min len: 253 avg len: 311.4 num_loss_counted_tokens: 6460
 total length: 7995 dropped: 0 num samples 9 - rank: 0 max len: 1103 min len: 639 avg len: 888.3333333333334 num_loss_counted_tokens: 7518
 total length: 7996 dropped: 0 num samples 18 - rank: 5 max len: 475 min len: 414 avg len: 444.22222222222223 num_loss_counted_tokens: 7042
 total length: 7796 dropped: 0 num samples 20 - rank: 6 max len: 412 min len: 362 avg len: 389.8 num_loss_counted_tokens: 6736
 total length: 7995 dropped: 0 num samples 16 - rank: 4 max len: 537 min len: 415 avg len: 499.6875 num_loss_counted_tokens: 7147
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 652 min len: 369 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 722 min len: 531 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7997 dropped: 0 num samples 14 - rank: 3 max len: 617 min len: 462 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
Epoch: 6, Step: 245, Rank: 1, loss = 0.03076171875Epoch: 6, Step: 245, Rank: 5, loss = 0.05029296875Epoch: 6, Step: 245, Rank: 6, loss = 0.04296875Epoch: 6, Step: 245, Rank: 2, loss = 0.03076171875
Epoch: 6, Step: 245, Rank: 3, loss = 0.033203125
Epoch: 6, Step: 245, Rank: 7, loss = 0.055908203125
Epoch: 6, Step: 245, Rank: 4, loss = 0.0390625



Epoch: 6, Step: 245, Rank: 0, loss = 0.0172119140625

Epoch 6:  12%|█▎        | 5/40 [00:08<00:57,  1.65s/it]{
    "epoch": 6,
    "step": 245,
    "rank": 0,
    "overall_throughput": 94.24660981720538,
    "lr": 9.153472818047627e-07,
    "cuda_mem_allocated": 6.588009834289551,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56545,
    "batch_size": 121,
    "total_loss": 0.03753647537359625,
    "samples_seen": 30996,
    "gradnorm": 0.275390625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:09.613470"
}
 total length: 7749 dropped: 0 num samples 23 - rank: 7 max len: 370 min len: 270 avg len: 336.9130434782609 num_loss_counted_tokens: 6530
 total length: 7884 dropped: 0 num samples 16 - rank: 4 max len: 515 min len: 462 avg len: 492.75 num_loss_counted_tokens: 7036
 total length: 7994 dropped: 0 num samples 18 - rank: 5 max len: 461 min len: 306 avg len: 444.1111111111111 num_loss_counted_tokens: 7040
 total length: 7852 dropped: 0 num samples 13 - rank: 2 max len: 623 min len: 580 avg len: 604.0 num_loss_counted_tokens: 7163
 total length: 7994 dropped: 0 num samples 20 - rank: 6 max len: 430 min len: 301 avg len: 399.7 num_loss_counted_tokens: 6934
 total length: 7993 dropped: 0 num samples 15 - rank: 3 max len: 575 min len: 322 avg len: 532.8666666666667 num_loss_counted_tokens: 7198
 total length: 7997 dropped: 0 num samples 11 - rank: 0 max len: 850 min len: 481 avg len: 727.0 num_loss_counted_tokens: 7414
 total length: 7874 dropped: 0 num samples 12 - rank: 1 max len: 683 min len: 630 avg len: 656.1666666666666 num_loss_counted_tokens: 7238
Epoch: 6, Step: 246, Rank: 5, loss = 0.05126953125Epoch: 6, Step: 246, Rank: 2, loss = 0.0322265625
Epoch: 6, Step: 246, Rank: 6, loss = 0.042236328125Epoch: 6, Step: 246, Rank: 1, loss = 0.0281982421875Epoch: 6, Step: 246, Rank: 7, loss = 0.0556640625Epoch: 6, Step: 246, Rank: 3, loss = 0.0306396484375


Epoch: 6, Step: 246, Rank: 4, loss = 0.03759765625


Epoch: 6, Step: 246, Rank: 0, loss = 0.0184326171875

Epoch 6:  15%|█▌        | 6/40 [00:09<00:55,  1.64s/it]{
    "epoch": 6,
    "step": 246,
    "rank": 0,
    "overall_throughput": 94.74564859372121,
    "lr": 8.645454235739903e-07,
    "cuda_mem_allocated": 6.588062763214111,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56425,
    "batch_size": 128,
    "total_loss": 0.03704031900753212,
    "samples_seen": 31124,
    "gradnorm": 0.275390625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:11.256100"
}
 total length: 7994 dropped: 0 num samples 14 - rank: 3 max len: 602 min len: 537 avg len: 571.0 num_loss_counted_tokens: 7252
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 648 min len: 456 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7988 dropped: 0 num samples 24 - rank: 7 max len: 383 min len: 288 avg len: 332.8333333333333 num_loss_counted_tokens: 6716
 total length: 7965 dropped: 0 num samples 9 - rank: 0 max len: 1059 min len: 723 avg len: 885.0 num_loss_counted_tokens: 7488
 total length: 7969 dropped: 0 num samples 16 - rank: 4 max len: 542 min len: 253 avg len: 498.0625 num_loss_counted_tokens: 7121
 total length: 7924 dropped: 0 num samples 20 - rank: 6 max len: 442 min len: 164 avg len: 396.2 num_loss_counted_tokens: 6864
 total length: 7988 dropped: 0 num samples 12 - rank: 1 max len: 721 min len: 360 avg len: 665.6666666666666 num_loss_counted_tokens: 7352
 total length: 7994 dropped: 0 num samples 17 - rank: 5 max len: 493 min len: 437 avg len: 470.2352941176471 num_loss_counted_tokens: 7093
Epoch: 6, Step: 247, Rank: 1, loss = 0.0279541015625Epoch: 6, Step: 247, Rank: 6, loss = 0.045166015625Epoch: 6, Step: 247, Rank: 2, loss = 0.033935546875Epoch: 6, Step: 247, Rank: 3, loss = 0.04150390625Epoch: 6, Step: 247, Rank: 4, loss = 0.0380859375Epoch: 6, Step: 247, Rank: 7, loss = 0.04833984375

Epoch: 6, Step: 247, Rank: 5, loss = 0.042724609375




Epoch: 6, Step: 247, Rank: 0, loss = 0.01806640625

Epoch 6:  18%|█▊        | 7/40 [00:11<00:54,  1.65s/it]{
    "epoch": 6,
    "step": 247,
    "rank": 0,
    "overall_throughput": 94.47747762967468,
    "lr": 8.151301425407699e-07,
    "cuda_mem_allocated": 6.588097095489502,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56634,
    "batch_size": 127,
    "total_loss": 0.03697425574743087,
    "samples_seen": 31251,
    "gradnorm": 0.318359375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:12.904536"
}
 total length: 7984 dropped: 0 num samples 10 - rank: 0 max len: 958 min len: 630 avg len: 798.4 num_loss_counted_tokens: 7454
 total length: 7920 dropped: 0 num samples 25 - rank: 7 max len: 365 min len: 255 avg len: 316.8 num_loss_counted_tokens: 6595
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 712 min len: 347 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7994 dropped: 0 num samples 13 - rank: 2 max len: 661 min len: 408 avg len: 614.9230769230769 num_loss_counted_tokens: 7305
 total length: 7835 dropped: 0 num samples 20 - rank: 6 max len: 406 min len: 368 avg len: 391.75 num_loss_counted_tokens: 6775
 total length: 7996 dropped: 0 num samples 14 - rank: 3 max len: 602 min len: 487 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7992 dropped: 0 num samples 16 - rank: 4 max len: 542 min len: 318 avg len: 499.5 num_loss_counted_tokens: 7144
 total length: 7967 dropped: 0 num samples 18 - rank: 5 max len: 477 min len: 407 avg len: 442.6111111111111 num_loss_counted_tokens: 7013
Epoch: 6, Step: 248, Rank: 1, loss = 0.04150390625Epoch: 6, Step: 248, Rank: 2, loss = 0.032470703125Epoch: 6, Step: 248, Rank: 3, loss = 0.0296630859375Epoch: 6, Step: 248, Rank: 6, loss = 0.0517578125Epoch: 6, Step: 248, Rank: 5, loss = 0.05224609375
Epoch: 6, Step: 248, Rank: 7, loss = 0.05224609375



Epoch: 6, Step: 248, Rank: 4, loss = 0.04052734375

Epoch: 6, Step: 248, Rank: 0, loss = 0.02978515625

Epoch 6:  20%|██        | 8/40 [00:13<00:52,  1.64s/it]{
    "epoch": 6,
    "step": 248,
    "rank": 0,
    "overall_throughput": 95.16242601646653,
    "lr": 7.671089389451059e-07,
    "cuda_mem_allocated": 6.578916549682617,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56044,
    "batch_size": 132,
    "total_loss": 0.04125330097780316,
    "samples_seen": 31383,
    "gradnorm": 0.287109375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:14.540713"
}
 total length: 7870 dropped: 0 num samples 19 - rank: 6 max len: 445 min len: 381 avg len: 414.2105263157895 num_loss_counted_tokens: 6863
 total length: 7993 dropped: 0 num samples 13 - rank: 2 max len: 657 min len: 306 avg len: 614.8461538461538 num_loss_counted_tokens: 7304
 total length: 7590 dropped: 0 num samples 24 - rank: 7 max len: 381 min len: 182 avg len: 316.25 num_loss_counted_tokens: 6318
 total length: 7996 dropped: 0 num samples 17 - rank: 5 max len: 502 min len: 346 avg len: 470.3529411764706 num_loss_counted_tokens: 7095
 total length: 7997 dropped: 0 num samples 14 - rank: 3 max len: 604 min len: 426 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7995 dropped: 0 num samples 9 - rank: 0 max len: 1100 min len: 608 avg len: 888.3333333333334 num_loss_counted_tokens: 7518
 total length: 7995 dropped: 0 num samples 15 - rank: 4 max len: 561 min len: 451 avg len: 533.0 num_loss_counted_tokens: 7200
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 733 min len: 407 avg len: 666.25 num_loss_counted_tokens: 7359
Epoch: 6, Step: 249, Rank: 2, loss = 0.0306396484375Epoch: 6, Step: 249, Rank: 1, loss = 0.0400390625
Epoch: 6, Step: 249, Rank: 7, loss = 0.050537109375Epoch: 6, Step: 249, Rank: 6, loss = 0.051513671875

Epoch: 6, Step: 249, Rank: 3, loss = 0.03759765625Epoch: 6, Step: 249, Rank: 4, loss = 0.0400390625

Epoch: 6, Step: 249, Rank: 5, loss = 0.038818359375

Epoch: 6, Step: 249, Rank: 0, loss = 0.0157470703125

Epoch 6:  22%|██▎       | 9/40 [00:14<00:50,  1.64s/it]{
    "epoch": 6,
    "step": 249,
    "rank": 0,
    "overall_throughput": 95.30957264238049,
    "lr": 7.204891014342552e-07,
    "cuda_mem_allocated": 6.573290824890137,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56331,
    "batch_size": 123,
    "total_loss": 0.038105128614794696,
    "samples_seen": 31506,
    "gradnorm": 0.283203125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:16.177777"
}
 total length: 7992 dropped: 0 num samples 14 - rank: 2 max len: 623 min len: 265 avg len: 570.8571428571429 num_loss_counted_tokens: 7250
 total length: 7788 dropped: 0 num samples 20 - rank: 6 max len: 408 min len: 371 avg len: 389.4 num_loss_counted_tokens: 6728
 total length: 7994 dropped: 0 num samples 19 - rank: 5 max len: 451 min len: 319 avg len: 420.7368421052632 num_loss_counted_tokens: 6987
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 565 min len: 496 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7703 dropped: 0 num samples 24 - rank: 7 max len: 371 min len: 248 avg len: 320.9583333333333 num_loss_counted_tokens: 6431
 total length: 7997 dropped: 0 num samples 9 - rank: 0 max len: 1065 min len: 537 avg len: 888.5555555555555 num_loss_counted_tokens: 7520
 total length: 7886 dropped: 0 num samples 12 - rank: 1 max len: 724 min len: 623 avg len: 657.1666666666666 num_loss_counted_tokens: 7250
 total length: 7997 dropped: 0 num samples 17 - rank: 4 max len: 506 min len: 361 avg len: 470.4117647058824 num_loss_counted_tokens: 7096
Epoch: 6, Step: 250, Rank: 6, loss = 0.046875Epoch: 6, Step: 250, Rank: 7, loss = 0.050537109375Epoch: 6, Step: 250, Rank: 1, loss = 0.03369140625Epoch: 6, Step: 250, Rank: 3, loss = 0.032958984375Epoch: 6, Step: 250, Rank: 2, loss = 0.033447265625Epoch: 6, Step: 250, Rank: 4, loss = 0.04443359375



Epoch: 6, Step: 250, Rank: 5, loss = 0.04443359375


Epoch: 6, Step: 250, Rank: 0, loss = 0.0306396484375

Epoch 6:  25%|██▌       | 10/40 [00:16<00:49,  1.64s/it]{
    "epoch": 6,
    "step": 250,
    "rank": 0,
    "overall_throughput": 94.46565805168497,
    "lr": 6.752777059564431e-07,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56412,
    "batch_size": 129,
    "total_loss": 0.039636956675884565,
    "samples_seen": 31635,
    "gradnorm": 0.296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:17.827472"
}
 total length: 7847 dropped: 0 num samples 13 - rank: 2 max len: 624 min len: 578 avg len: 603.6153846153846 num_loss_counted_tokens: 7158
 total length: 7992 dropped: 0 num samples 15 - rank: 3 max len: 567 min len: 407 avg len: 532.8 num_loss_counted_tokens: 7197
 total length: 7974 dropped: 0 num samples 21 - rank: 6 max len: 400 min len: 361 avg len: 379.7142857142857 num_loss_counted_tokens: 6861
 total length: 7758 dropped: 0 num samples 24 - rank: 7 max len: 360 min len: 191 avg len: 323.25 num_loss_counted_tokens: 6486
 total length: 7983 dropped: 0 num samples 9 - rank: 0 max len: 1068 min len: 519 avg len: 887.0 num_loss_counted_tokens: 7506
 total length: 7882 dropped: 0 num samples 16 - rank: 4 max len: 506 min len: 477 avg len: 492.625 num_loss_counted_tokens: 7034
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 732 min len: 611 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7876 dropped: 0 num samples 18 - rank: 5 max len: 474 min len: 403 avg len: 437.55555555555554 num_loss_counted_tokens: 6922
Epoch: 6, Step: 251, Rank: 2, loss = 0.0400390625Epoch: 6, Step: 251, Rank: 3, loss = 0.034423828125Epoch: 6, Step: 251, Rank: 5, loss = 0.05126953125Epoch: 6, Step: 251, Rank: 7, loss = 0.05615234375Epoch: 6, Step: 251, Rank: 6, loss = 0.0478515625Epoch: 6, Step: 251, Rank: 1, loss = 0.044677734375

Epoch: 6, Step: 251, Rank: 4, loss = 0.037353515625




Epoch: 6, Step: 251, Rank: 0, loss = 0.0299072265625

Epoch 6:  28%|██▊       | 11/40 [00:18<00:47,  1.65s/it]{
    "epoch": 6,
    "step": 251,
    "rank": 0,
    "overall_throughput": 94.68086762636543,
    "lr": 6.314816146868951e-07,
    "cuda_mem_allocated": 6.588009834289551,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56253,
    "batch_size": 133,
    "total_loss": 0.042664391232467604,
    "samples_seen": 31768,
    "gradnorm": 0.298828125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:19.478553"
}
 total length: 7996 dropped: 0 num samples 14 - rank: 3 max len: 618 min len: 372 avg len: 571.1428571428571 num_loss_counted_tokens: 7254
 total length: 7745 dropped: 0 num samples 11 - rank: 1 max len: 846 min len: 649 avg len: 704.0909090909091 num_loss_counted_tokens: 7162
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 648 min len: 385 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7995 dropped: 0 num samples 16 - rank: 4 max len: 549 min len: 301 avg len: 499.6875 num_loss_counted_tokens: 7147
 total length: 7899 dropped: 0 num samples 18 - rank: 5 max len: 463 min len: 415 avg len: 438.8333333333333 num_loss_counted_tokens: 6945
 total length: 7619 dropped: 0 num samples 25 - rank: 7 max len: 344 min len: 263 avg len: 304.76 num_loss_counted_tokens: 6294
 total length: 7985 dropped: 0 num samples 21 - rank: 6 max len: 412 min len: 346 avg len: 380.23809523809524 num_loss_counted_tokens: 6872
 total length: 7968 dropped: 0 num samples 8 - rank: 0 max len: 1140 min len: 732 avg len: 996.0 num_loss_counted_tokens: 7544
Epoch: 6, Step: 252, Rank: 3, loss = 0.033935546875Epoch: 6, Step: 252, Rank: 2, loss = 0.03466796875Epoch: 6, Step: 252, Rank: 6, loss = 0.048828125Epoch: 6, Step: 252, Rank: 1, loss = 0.0390625
Epoch: 6, Step: 252, Rank: 7, loss = 0.05029296875



Epoch: 6, Step: 252, Rank: 4, loss = 0.03125
Epoch: 6, Step: 252, Rank: 5, loss = 0.036376953125
Epoch: 6, Step: 252, Rank: 0, loss = 0.02783203125

Epoch 6:  30%|███       | 12/40 [00:19<00:45,  1.64s/it]{
    "epoch": 6,
    "step": 252,
    "rank": 0,
    "overall_throughput": 95.65354330147501,
    "lr": 5.891074749862857e-07,
    "cuda_mem_allocated": 6.578728675842285,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56225,
    "batch_size": 126,
    "total_loss": 0.037812361049355266,
    "samples_seen": 31894,
    "gradnorm": 0.271484375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:21.109033"
}
 total length: 7931 dropped: 0 num samples 12 - rank: 1 max len: 689 min len: 631 avg len: 660.9166666666666 num_loss_counted_tokens: 7295
 total length: 7845 dropped: 0 num samples 13 - rank: 2 max len: 629 min len: 569 avg len: 603.4615384615385 num_loss_counted_tokens: 7156
 total length: 7994 dropped: 0 num samples 17 - rank: 4 max len: 512 min len: 278 avg len: 470.2352941176471 num_loss_counted_tokens: 7093
 total length: 7992 dropped: 0 num samples 10 - rank: 0 max len: 1003 min len: 587 avg len: 799.2 num_loss_counted_tokens: 7462
 total length: 7870 dropped: 0 num samples 25 - rank: 7 max len: 357 min len: 245 avg len: 314.8 num_loss_counted_tokens: 6545
 total length: 7974 dropped: 0 num samples 15 - rank: 3 max len: 569 min len: 514 avg len: 531.6 num_loss_counted_tokens: 7179
 total length: 7997 dropped: 0 num samples 21 - rank: 6 max len: 408 min len: 350 avg len: 380.8095238095238 num_loss_counted_tokens: 6884
 total length: 7945 dropped: 0 num samples 19 - rank: 5 max len: 451 min len: 172 avg len: 418.1578947368421 num_loss_counted_tokens: 6938
Epoch: 6, Step: 253, Rank: 6, loss = 0.049072265625Epoch: 6, Step: 253, Rank: 2, loss = 0.033447265625Epoch: 6, Step: 253, Rank: 7, loss = 0.048828125Epoch: 6, Step: 253, Rank: 1, loss = 0.038330078125Epoch: 6, Step: 253, Rank: 5, loss = 0.051025390625Epoch: 6, Step: 253, Rank: 3, loss = 0.033447265625




Epoch: 6, Step: 253, Rank: 4, loss = 0.031982421875

Epoch: 6, Step: 253, Rank: 0, loss = 0.0223388671875

Epoch 6:  32%|███▎      | 13/40 [00:21<00:44,  1.64s/it]{
    "epoch": 6,
    "step": 253,
    "rank": 0,
    "overall_throughput": 94.53468892634012,
    "lr": 5.481617183918053e-07,
    "cuda_mem_allocated": 6.586664199829102,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56436,
    "batch_size": 128,
    "total_loss": 0.03853923027854561,
    "samples_seen": 32022,
    "gradnorm": 0.287109375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:22.756703"
}
 total length: 7767 dropped: 0 num samples 13 - rank: 2 max len: 628 min len: 558 avg len: 597.4615384615385 num_loss_counted_tokens: 7078
 total length: 7882 dropped: 0 num samples 12 - rank: 1 max len: 702 min len: 630 avg len: 656.8333333333334 num_loss_counted_tokens: 7246
 total length: 7864 dropped: 0 num samples 16 - rank: 4 max len: 514 min len: 468 avg len: 491.5 num_loss_counted_tokens: 7016
 total length: 7797 dropped: 0 num samples 19 - rank: 6 max len: 437 min len: 382 avg len: 410.36842105263156 num_loss_counted_tokens: 6790
 total length: 7993 dropped: 0 num samples 18 - rank: 5 max len: 467 min len: 303 avg len: 444.05555555555554 num_loss_counted_tokens: 7039
 total length: 7891 dropped: 0 num samples 24 - rank: 7 max len: 375 min len: 244 avg len: 328.7916666666667 num_loss_counted_tokens: 6619
 total length: 7997 dropped: 0 num samples 9 - rank: 0 max len: 1071 min len: 533 avg len: 888.5555555555555 num_loss_counted_tokens: 7520
 total length: 7984 dropped: 0 num samples 15 - rank: 3 max len: 553 min len: 515 avg len: 532.2666666666667 num_loss_counted_tokens: 7189
Epoch: 6, Step: 254, Rank: 5, loss = 0.04296875Epoch: 6, Step: 254, Rank: 6, loss = 0.05029296875Epoch: 6, Step: 254, Rank: 2, loss = 0.04296875
Epoch: 6, Step: 254, Rank: 7, loss = 0.05078125
Epoch: 6, Step: 254, Rank: 1, loss = 0.0291748046875Epoch: 6, Step: 254, Rank: 3, loss = 0.031494140625
Epoch: 6, Step: 254, Rank: 4, loss = 0.036376953125



Epoch: 6, Step: 254, Rank: 0, loss = 0.02490234375

Epoch 6:  35%|███▌      | 14/40 [00:23<00:42,  1.65s/it]{
    "epoch": 6,
    "step": 254,
    "rank": 0,
    "overall_throughput": 94.1913329804549,
    "lr": 5.086505596409885e-07,
    "cuda_mem_allocated": 6.586106300354004,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56190,
    "batch_size": 120,
    "total_loss": 0.03861897134721481,
    "samples_seen": 32142,
    "gradnorm": 0.28515625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:24.410277"
}
 total length: 7992 dropped: 0 num samples 13 - rank: 2 max len: 653 min len: 373 avg len: 614.7692307692307 num_loss_counted_tokens: 7303
 total length: 7986 dropped: 0 num samples 15 - rank: 4 max len: 553 min len: 511 avg len: 532.4 num_loss_counted_tokens: 7191
 total length: 7929 dropped: 0 num samples 19 - rank: 6 max len: 451 min len: 396 avg len: 417.3157894736842 num_loss_counted_tokens: 6922
 total length: 7996 dropped: 0 num samples 9 - rank: 0 max len: 1118 min len: 456 avg len: 888.4444444444445 num_loss_counted_tokens: 7519
 total length: 7997 dropped: 0 num samples 14 - rank: 3 max len: 600 min len: 513 avg len: 571.2142857142857 num_loss_counted_tokens: 7255
 total length: 7991 dropped: 0 num samples 17 - rank: 5 max len: 511 min len: 233 avg len: 470.05882352941177 num_loss_counted_tokens: 7090
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 699 min len: 595 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7854 dropped: 0 num samples 23 - rank: 7 max len: 396 min len: 249 avg len: 341.4782608695652 num_loss_counted_tokens: 6635
Epoch: 6, Step: 255, Rank: 2, loss = 0.0274658203125Epoch: 6, Step: 255, Rank: 6, loss = 0.048095703125Epoch: 6, Step: 255, Rank: 5, loss = 0.04833984375
Epoch: 6, Step: 255, Rank: 7, loss = 0.049560546875
Epoch: 6, Step: 255, Rank: 3, loss = 0.03564453125Epoch: 6, Step: 255, Rank: 1, loss = 0.0361328125


Epoch: 6, Step: 255, Rank: 4, loss = 0.03857421875

Epoch: 6, Step: 255, Rank: 0, loss = 0.02587890625

Epoch 6:  38%|███▊      | 15/40 [00:24<00:41,  1.65s/it]{
    "epoch": 6,
    "step": 255,
    "rank": 0,
    "overall_throughput": 94.27068865580695,
    "lr": 4.7057999572843516e-07,
    "cuda_mem_allocated": 6.588009834289551,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56502,
    "batch_size": 131,
    "total_loss": 0.03874199143393154,
    "samples_seen": 32273,
    "gradnorm": 0.283203125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:26.057086"
}
 total length: 7995 dropped: 0 num samples 12 - rank: 1 max len: 717 min len: 527 avg len: 666.25 num_loss_counted_tokens: 7359
 total length: 7361 dropped: 0 num samples 23 - rank: 7 max len: 359 min len: 250 avg len: 320.04347826086956 num_loss_counted_tokens: 6142
 total length: 7984 dropped: 0 num samples 13 - rank: 2 max len: 640 min len: 591 avg len: 614.1538461538462 num_loss_counted_tokens: 7295
 total length: 7960 dropped: 0 num samples 20 - rank: 6 max len: 440 min len: 361 avg len: 398.0 num_loss_counted_tokens: 6900
 total length: 7915 dropped: 0 num samples 16 - rank: 4 max len: 516 min len: 477 avg len: 494.6875 num_loss_counted_tokens: 7067
 total length: 7988 dropped: 0 num samples 15 - rank: 3 max len: 589 min len: 298 avg len: 532.5333333333333 num_loss_counted_tokens: 7193
 total length: 7824 dropped: 0 num samples 9 - rank: 0 max len: 1065 min len: 736 avg len: 869.3333333333334 num_loss_counted_tokens: 7347
 total length: 7877 dropped: 0 num samples 17 - rank: 5 max len: 476 min len: 443 avg len: 463.3529411764706 num_loss_counted_tokens: 6976
Epoch: 6, Step: 256, Rank: 6, loss = 0.049072265625Epoch: 6, Step: 256, Rank: 7, loss = 0.051513671875Epoch: 6, Step: 256, Rank: 5, loss = 0.04541015625Epoch: 6, Step: 256, Rank: 2, loss = 0.0279541015625

Epoch: 6, Step: 256, Rank: 1, loss = 0.03564453125


Epoch: 6, Step: 256, Rank: 3, loss = 0.027587890625Epoch: 6, Step: 256, Rank: 4, loss = 0.0361328125

Epoch: 6, Step: 256, Rank: 0, loss = 0.02294921875

Epoch 6:  40%|████      | 16/40 [00:26<00:39,  1.65s/it]{
    "epoch": 6,
    "step": 256,
    "rank": 0,
    "overall_throughput": 93.87770407004366,
    "lr": 4.3395580499559276e-07,
    "cuda_mem_allocated": 6.588062763214111,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56473,
    "batch_size": 131,
    "total_loss": 0.03700883608095904,
    "samples_seen": 32404,
    "gradnorm": 0.267578125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:27.719898"
}
 total length: 7864 dropped: 0 num samples 25 - rank: 7 max len: 357 min len: 257 avg len: 314.56 num_loss_counted_tokens: 6539
 total length: 7992 dropped: 0 num samples 11 - rank: 1 max len: 826 min len: 657 avg len: 726.5454545454545 num_loss_counted_tokens: 7409
 total length: 7983 dropped: 0 num samples 8 - rank: 0 max len: 1247 min len: 831 avg len: 997.875 num_loss_counted_tokens: 7559
 total length: 7982 dropped: 0 num samples 14 - rank: 3 max len: 626 min len: 172 avg len: 570.1428571428571 num_loss_counted_tokens: 7240
 total length: 7989 dropped: 0 num samples 13 - rank: 2 max len: 657 min len: 299 avg len: 614.5384615384615 num_loss_counted_tokens: 7300
 total length: 7996 dropped: 0 num samples 15 - rank: 4 max len: 565 min len: 475 avg len: 533.0666666666667 num_loss_counted_tokens: 7201
 total length: 7816 dropped: 0 num samples 20 - rank: 6 max len: 408 min len: 367 avg len: 390.8 num_loss_counted_tokens: 6756
 total length: 7874 dropped: 0 num samples 17 - rank: 5 max len: 508 min len: 412 avg len: 463.1764705882353 num_loss_counted_tokens: 6973
Epoch: 6, Step: 257, Rank: 5, loss = 0.04833984375Epoch: 6, Step: 257, Rank: 1, loss = 0.0322265625Epoch: 6, Step: 257, Rank: 6, loss = 0.047119140625Epoch: 6, Step: 257, Rank: 3, loss = 0.03369140625Epoch: 6, Step: 257, Rank: 2, loss = 0.03515625

Epoch: 6, Step: 257, Rank: 7, loss = 0.0458984375



Epoch: 6, Step: 257, Rank: 4, loss = 0.039306640625
Epoch: 6, Step: 257, Rank: 0, loss = 0.0279541015625

Epoch 6:  42%|████▎     | 17/40 [00:27<00:37,  1.65s/it]{
    "epoch": 6,
    "step": 257,
    "rank": 0,
    "overall_throughput": 94.44712838081513,
    "lr": 3.9878354625371927e-07,
    "cuda_mem_allocated": 6.588097095489502,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56408,
    "batch_size": 127,
    "total_loss": 0.03871791235285775,
    "samples_seen": 32531,
    "gradnorm": 0.28515625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:29.367999"
}
 total length: 7898 dropped: 0 num samples 25 - rank: 7 max len: 375 min len: 240 avg len: 315.92 num_loss_counted_tokens: 6573
 total length: 7833 dropped: 0 num samples 19 - rank: 6 max len: 440 min len: 387 avg len: 412.2631578947368 num_loss_counted_tokens: 6826
 total length: 7991 dropped: 0 num samples 16 - rank: 4 max len: 530 min len: 378 avg len: 499.4375 num_loss_counted_tokens: 7143
 total length: 7914 dropped: 0 num samples 14 - rank: 3 max len: 603 min len: 537 avg len: 565.2857142857143 num_loss_counted_tokens: 7172
 total length: 7995 dropped: 0 num samples 10 - rank: 0 max len: 1064 min len: 675 avg len: 799.5 num_loss_counted_tokens: 7465
 total length: 7996 dropped: 0 num samples 18 - rank: 5 max len: 482 min len: 174 avg len: 444.22222222222223 num_loss_counted_tokens: 7042
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 640 min len: 489 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7935 dropped: 0 num samples 12 - rank: 1 max len: 691 min len: 640 avg len: 661.25 num_loss_counted_tokens: 7299
Epoch: 6, Step: 258, Rank: 6, loss = 0.043212890625Epoch: 6, Step: 258, Rank: 5, loss = 0.03857421875Epoch: 6, Step: 258, Rank: 2, loss = 0.039794921875Epoch: 6, Step: 258, Rank: 7, loss = 0.056884765625
Epoch: 6, Step: 258, Rank: 1, loss = 0.0390625

Epoch: 6, Step: 258, Rank: 3, loss = 0.040771484375


Epoch: 6, Step: 258, Rank: 0, loss = 0.0245361328125
Epoch: 6, Step: 258, Rank: 4, loss = 0.0380859375

Epoch 6:  45%|████▌     | 18/40 [00:29<00:38,  1.74s/it]{
    "epoch": 6,
    "step": 258,
    "rank": 0,
    "overall_throughput": 77.37700540385664,
    "lr": 3.650685579401692e-07,
    "cuda_mem_allocated": 6.580752849578857,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56362,
    "batch_size": 127,
    "total_loss": 0.04011568077782903,
    "samples_seen": 32658,
    "gradnorm": 0.302734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:31.315070"
}
 total length: 7890 dropped: 0 num samples 14 - rank: 3 max len: 586 min len: 532 avg len: 563.5714285714286 num_loss_counted_tokens: 7148
 total length: 7831 dropped: 0 num samples 16 - rank: 4 max len: 530 min len: 454 avg len: 489.4375 num_loss_counted_tokens: 6983
 total length: 7923 dropped: 0 num samples 10 - rank: 0 max len: 1064 min len: 172 avg len: 792.3 num_loss_counted_tokens: 7393
 total length: 7785 dropped: 0 num samples 18 - rank: 5 max len: 450 min len: 419 avg len: 432.5 num_loss_counted_tokens: 6831
 total length: 7970 dropped: 0 num samples 13 - rank: 2 max len: 635 min len: 590 avg len: 613.0769230769231 num_loss_counted_tokens: 7281
 total length: 7907 dropped: 0 num samples 20 - rank: 6 max len: 418 min len: 370 avg len: 395.35 num_loss_counted_tokens: 6847
 total length: 7815 dropped: 0 num samples 24 - rank: 7 max len: 365 min len: 255 avg len: 325.625 num_loss_counted_tokens: 6543
 total length: 7935 dropped: 0 num samples 12 - rank: 1 max len: 695 min len: 639 avg len: 661.25 num_loss_counted_tokens: 7299
Epoch: 6, Step: 259, Rank: 6, loss = 0.045166015625Epoch: 6, Step: 259, Rank: 7, loss = 0.044921875Epoch: 6, Step: 259, Rank: 3, loss = 0.031982421875Epoch: 6, Step: 259, Rank: 5, loss = 0.053955078125
Epoch: 6, Step: 259, Rank: 1, loss = 0.0302734375
Epoch: 6, Step: 259, Rank: 2, loss = 0.0361328125



Epoch: 6, Step: 259, Rank: 4, loss = 0.041748046875
Epoch: 6, Step: 259, Rank: 0, loss = 0.02587890625

Epoch 6:  48%|████▊     | 19/40 [00:31<00:35,  1.71s/it]{
    "epoch": 6,
    "step": 259,
    "rank": 0,
    "overall_throughput": 94.43001774374432,
    "lr": 3.328159573081258e-07,
    "cuda_mem_allocated": 6.586934566497803,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56236,
    "batch_size": 127,
    "total_loss": 0.038800768191194254,
    "samples_seen": 32785,
    "gradnorm": 0.2890625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:32.964552"
}
 total length: 7992 dropped: 0 num samples 9 - rank: 0 max len: 1094 min len: 570 avg len: 888.0 num_loss_counted_tokens: 7515
 total length: 7994 dropped: 0 num samples 16 - rank: 4 max len: 519 min len: 424 avg len: 499.625 num_loss_counted_tokens: 7146
 total length: 7989 dropped: 0 num samples 13 - rank: 2 max len: 648 min len: 523 avg len: 614.5384615384615 num_loss_counted_tokens: 7300
 total length: 7969 dropped: 0 num samples 14 - rank: 3 max len: 601 min len: 522 avg len: 569.2142857142857 num_loss_counted_tokens: 7227
 total length: 7546 dropped: 0 num samples 23 - rank: 7 max len: 376 min len: 256 avg len: 328.0869565217391 num_loss_counted_tokens: 6327
 total length: 7989 dropped: 0 num samples 18 - rank: 5 max len: 483 min len: 242 avg len: 443.8333333333333 num_loss_counted_tokens: 7035
 total length: 7996 dropped: 0 num samples 20 - rank: 6 max len: 423 min len: 337 avg len: 399.8 num_loss_counted_tokens: 6936
 total length: 7992 dropped: 0 num samples 12 - rank: 1 max len: 744 min len: 557 avg len: 666.0 num_loss_counted_tokens: 7356
Epoch: 6, Step: 260, Rank: 0, loss = 0.0186767578125Epoch: 6, Step: 260, Rank: 5, loss = 0.04248046875Epoch: 6, Step: 260, Rank: 1, loss = 0.027099609375

Epoch: 6, Step: 260, Rank: 7, loss = 0.0517578125Epoch: 6, Step: 260, Rank: 6, loss = 0.044677734375Epoch: 6, Step: 260, Rank: 3, loss = 0.031494140625



Epoch: 6, Step: 260, Rank: 4, loss = 0.03857421875
Epoch: 6, Step: 260, Rank: 2, loss = 0.0341796875

Epoch 6:  50%|█████     | 20/40 [00:33<00:33,  1.69s/it]{
    "epoch": 6,
    "step": 260,
    "rank": 0,
    "overall_throughput": 94.23521182657184,
    "lr": 3.020306396499062e-07,
    "cuda_mem_allocated": 6.574269771575928,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56935,
    "batch_size": 123,
    "total_loss": 0.036111355054008955,
    "samples_seen": 32908,
    "gradnorm": 0.259765625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:34.611358"
}
 total length: 7996 dropped: 0 num samples 15 - rank: 3 max len: 595 min len: 264 avg len: 533.0666666666667 num_loss_counted_tokens: 7201
 total length: 7997 dropped: 0 num samples 11 - rank: 0 max len: 879 min len: 486 avg len: 727.0 num_loss_counted_tokens: 7414
 total length: 7883 dropped: 0 num samples 19 - rank: 6 max len: 439 min len: 395 avg len: 414.89473684210526 num_loss_counted_tokens: 6876
 total length: 7994 dropped: 0 num samples 13 - rank: 2 max len: 633 min len: 549 avg len: 614.9230769230769 num_loss_counted_tokens: 7305
 total length: 7996 dropped: 0 num samples 16 - rank: 4 max len: 523 min len: 432 avg len: 499.75 num_loss_counted_tokens: 7148
 total length: 7788 dropped: 0 num samples 23 - rank: 7 max len: 393 min len: 229 avg len: 338.60869565217394 num_loss_counted_tokens: 6569
 total length: 7808 dropped: 0 num samples 17 - rank: 5 max len: 478 min len: 440 avg len: 459.29411764705884 num_loss_counted_tokens: 6907
 total length: 7992 dropped: 0 num samples 12 - rank: 1 max len: 688 min len: 643 avg len: 666.0 num_loss_counted_tokens: 7356
Epoch: 6, Step: 261, Rank: 5, loss = 0.048095703125Epoch: 6, Step: 261, Rank: 1, loss = 0.0301513671875Epoch: 6, Step: 261, Rank: 2, loss = 0.02978515625Epoch: 6, Step: 261, Rank: 7, loss = 0.047119140625Epoch: 6, Step: 261, Rank: 6, loss = 0.05029296875


Epoch: 6, Step: 261, Rank: 3, loss = 0.034912109375

Epoch: 6, Step: 261, Rank: 4, loss = 0.042236328125

Epoch: 6, Step: 261, Rank: 0, loss = 0.0186767578125

Epoch 6:  52%|█████▎    | 21/40 [00:34<00:31,  1.68s/it]{
    "epoch": 6,
    "step": 261,
    "rank": 0,
    "overall_throughput": 94.01474372545857,
    "lr": 2.727172775539522e-07,
    "cuda_mem_allocated": 6.588097095489502,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56825,
    "batch_size": 127,
    "total_loss": 0.03767707875054994,
    "samples_seen": 33035,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:36.267293"
}
 total length: 7910 dropped: 0 num samples 12 - rank: 1 max len: 687 min len: 637 avg len: 659.1666666666666 num_loss_counted_tokens: 7274
 total length: 7977 dropped: 0 num samples 25 - rank: 7 max len: 366 min len: 252 avg len: 319.08 num_loss_counted_tokens: 6652
 total length: 7934 dropped: 0 num samples 13 - rank: 2 max len: 636 min len: 570 avg len: 610.3076923076923 num_loss_counted_tokens: 7245
 total length: 7838 dropped: 0 num samples 16 - rank: 4 max len: 515 min len: 468 avg len: 489.875 num_loss_counted_tokens: 6990
 total length: 7996 dropped: 0 num samples 18 - rank: 5 max len: 467 min len: 417 avg len: 444.22222222222223 num_loss_counted_tokens: 7042
 total length: 7997 dropped: 0 num samples 15 - rank: 3 max len: 570 min len: 346 avg len: 533.1333333333333 num_loss_counted_tokens: 7202
 total length: 7990 dropped: 0 num samples 9 - rank: 0 max len: 1104 min len: 326 avg len: 887.7777777777778 num_loss_counted_tokens: 7513
 total length: 7904 dropped: 0 num samples 20 - rank: 6 max len: 424 min len: 366 avg len: 395.2 num_loss_counted_tokens: 6844
Epoch: 6, Step: 262, Rank: 2, loss = 0.03857421875Epoch: 6, Step: 262, Rank: 3, loss = 0.04296875Epoch: 6, Step: 262, Rank: 6, loss = 0.052001953125Epoch: 6, Step: 262, Rank: 5, loss = 0.046142578125Epoch: 6, Step: 262, Rank: 7, loss = 0.046875Epoch: 6, Step: 262, Rank: 1, loss = 0.034912109375





Epoch: 6, Step: 262, Rank: 0, loss = 0.0260009765625
Epoch: 6, Step: 262, Rank: 4, loss = 0.038818359375

Epoch 6:  55%|█████▌    | 22/40 [00:36<00:30,  1.67s/it]{
    "epoch": 6,
    "step": 262,
    "rank": 0,
    "overall_throughput": 94.7237331183158,
    "lr": 2.44880320195634e-07,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56553,
    "batch_size": 128,
    "total_loss": 0.040793591851891145,
    "samples_seen": 33163,
    "gradnorm": 0.291015625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:37.910760"
}
 total length: 7724 dropped: 0 num samples 23 - rank: 7 max len: 378 min len: 250 avg len: 335.82608695652175 num_loss_counted_tokens: 6505
 total length: 7957 dropped: 0 num samples 10 - rank: 0 max len: 1110 min len: 180 avg len: 795.7 num_loss_counted_tokens: 7427
 total length: 7993 dropped: 0 num samples 14 - rank: 3 max len: 626 min len: 326 avg len: 570.9285714285714 num_loss_counted_tokens: 7251
 total length: 7896 dropped: 0 num samples 19 - rank: 6 max len: 454 min len: 383 avg len: 415.57894736842104 num_loss_counted_tokens: 6889
 total length: 7824 dropped: 0 num samples 15 - rank: 4 max len: 546 min len: 497 avg len: 521.6 num_loss_counted_tokens: 7029
 total length: 7997 dropped: 0 num samples 13 - rank: 2 max len: 651 min len: 316 avg len: 615.1538461538462 num_loss_counted_tokens: 7308
 total length: 7997 dropped: 0 num samples 12 - rank: 1 max len: 690 min len: 579 avg len: 666.4166666666666 num_loss_counted_tokens: 7361
 total length: 7997 dropped: 0 num samples 17 - rank: 5 max len: 494 min len: 406 avg len: 470.4117647058824 num_loss_counted_tokens: 7096
Epoch: 6, Step: 263, Rank: 2, loss = 0.033203125Epoch: 6, Step: 263, Rank: 3, loss = 0.035400390625Epoch: 6, Step: 263, Rank: 5, loss = 0.03515625Epoch: 6, Step: 263, Rank: 6, loss = 0.04833984375Epoch: 6, Step: 263, Rank: 7, loss = 0.054931640625Epoch: 6, Step: 263, Rank: 1, loss = 0.04345703125





Epoch: 6, Step: 263, Rank: 0, loss = 0.038818359375
Epoch: 6, Step: 263, Rank: 4, loss = 0.033935546875

Epoch 6:  57%|█████▊    | 23/40 [00:38<00:28,  1.66s/it]{
    "epoch": 6,
    "step": 263,
    "rank": 0,
    "overall_throughput": 94.10358229978549,
    "lr": 2.1852399266194312e-07,
    "cuda_mem_allocated": 6.584917068481445,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 57194,
    "batch_size": 125,
    "total_loss": 0.040388851977480156,
    "samples_seen": 33288,
    "gradnorm": 0.2890625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:39.558220"
}
 total length: 7996 dropped: 0 num samples 10 - rank: 0 max len: 1072 min len: 515 avg len: 799.6 num_loss_counted_tokens: 7466
 total length: 7996 dropped: 0 num samples 15 - rank: 3 max len: 572 min len: 434 avg len: 533.0666666666667 num_loss_counted_tokens: 7201
 total length: 7936 dropped: 0 num samples 25 - rank: 7 max len: 371 min len: 259 avg len: 317.44 num_loss_counted_tokens: 6611
 total length: 7880 dropped: 0 num samples 20 - rank: 6 max len: 411 min len: 376 avg len: 394.0 num_loss_counted_tokens: 6820
 total length: 7785 dropped: 0 num samples 12 - rank: 1 max len: 675 min len: 626 avg len: 648.75 num_loss_counted_tokens: 7149
 total length: 7842 dropped: 0 num samples 13 - rank: 2 max len: 624 min len: 581 avg len: 603.2307692307693 num_loss_counted_tokens: 7153
 total length: 7845 dropped: 0 num samples 18 - rank: 5 max len: 461 min len: 416 avg len: 435.8333333333333 num_loss_counted_tokens: 6891
 total length: 7834 dropped: 0 num samples 16 - rank: 4 max len: 511 min len: 469 avg len: 489.625 num_loss_counted_tokens: 6986
Epoch: 6, Step: 264, Rank: 6, loss = 0.0517578125Epoch: 6, Step: 264, Rank: 3, loss = 0.03271484375Epoch: 6, Step: 264, Rank: 5, loss = 0.04443359375Epoch: 6, Step: 264, Rank: 7, loss = 0.057373046875
Epoch: 6, Step: 264, Rank: 2, loss = 0.026611328125
Epoch: 6, Step: 264, Rank: 1, loss = 0.03564453125



Epoch: 6, Step: 264, Rank: 4, loss = 0.035888671875
Epoch: 6, Step: 264, Rank: 0, loss = 0.02392578125

Epoch 6:  60%|██████    | 24/40 [00:39<00:26,  1.66s/it]{
    "epoch": 6,
    "step": 264,
    "rank": 0,
    "overall_throughput": 94.06641809873567,
    "lr": 1.9365229531022267e-07,
    "cuda_mem_allocated": 6.5867228507995605,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56901,
    "batch_size": 128,
    "total_loss": 0.03855819757122019,
    "samples_seen": 33416,
    "gradnorm": 0.326171875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:41.211427"
}
Epoch: 6, Step: 265, Rank: 1, loss = 0.034912109375Epoch: 6, Step: 265, Rank: 6, loss = 0.046630859375Epoch: 6, Step: 265, Rank: 2, loss = 0.040771484375Epoch: 6, Step: 265, Rank: 7, loss = 0.052490234375
Epoch: 6, Step: 265, Rank: 3, loss = 0.033203125

Epoch: 6, Step: 265, Rank: 4, loss = 0.03662109375Epoch: 6, Step: 265, Rank: 5, loss = 0.036865234375



Epoch: 6, Step: 265, Rank: 0, loss = 0.01513671875

Epoch 6:  62%|██████▎   | 25/40 [00:41<00:24,  1.66s/it]{
    "epoch": 6,
    "step": 265,
    "rank": 0,
    "overall_throughput": 94.20017490002203,
    "lr": 1.7026900316098217e-07,
    "cuda_mem_allocated": 6.588098526000977,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56912,
    "batch_size": 123,
    "total_loss": 0.037083567613157155,
    "samples_seen": 33539,
    "gradnorm": 0.27734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:42.861274"
}
Epoch: 6, Step: 266, Rank: 6, loss = 0.0498046875Epoch: 6, Step: 266, Rank: 1, loss = 0.0478515625Epoch: 6, Step: 266, Rank: 2, loss = 0.031982421875Epoch: 6, Step: 266, Rank: 3, loss = 0.03759765625



Epoch: 6, Step: 266, Rank: 7, loss = 0.047119140625Epoch: 6, Step: 266, Rank: 4, loss = 0.045654296875Epoch: 6, Step: 266, Rank: 5, loss = 0.052978515625


Epoch: 6, Step: 266, Rank: 0, loss = 0.0162353515625

Epoch 6:  65%|██████▌   | 26/40 [00:43<00:23,  1.65s/it]{
    "epoch": 6,
    "step": 266,
    "rank": 0,
    "overall_throughput": 94.59610297208903,
    "lr": 1.4837766532493469e-07,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56464,
    "batch_size": 130,
    "total_loss": 0.04117667894587702,
    "samples_seen": 33669,
    "gradnorm": 0.306640625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:44.503275"
}
Epoch: 6, Step: 267, Rank: 1, loss = 0.041015625Epoch: 6, Step: 267, Rank: 5, loss = 0.046142578125Epoch: 6, Step: 267, Rank: 6, loss = 0.046142578125Epoch: 6, Step: 267, Rank: 7, loss = 0.05078125Epoch: 6, Step: 267, Rank: 3, loss = 0.033447265625


Epoch: 6, Step: 267, Rank: 2, loss = 0.033447265625
Epoch: 6, Step: 267, Rank: 4, loss = 0.03662109375


Epoch: 6, Step: 267, Rank: 0, loss = 0.0234375

Epoch 6:  68%|██████▊   | 27/40 [00:44<00:21,  1.65s/it]{
    "epoch": 6,
    "step": 267,
    "rank": 0,
    "overall_throughput": 93.55748241628731,
    "lr": 1.2798160446431006e-07,
    "cuda_mem_allocated": 6.586569786071777,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56523,
    "batch_size": 128,
    "total_loss": 0.038886824832369125,
    "samples_seen": 33797,
    "gradnorm": 0.2734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:46.155335"
}
Epoch: 6, Step: 268, Rank: 6, loss = 0.0458984375Epoch: 6, Step: 268, Rank: 1, loss = 0.04541015625Epoch: 6, Step: 268, Rank: 3, loss = 0.041259765625


Epoch: 6, Step: 268, Rank: 7, loss = 0.05859375Epoch: 6, Step: 268, Rank: 2, loss = 0.033935546875Epoch: 6, Step: 268, Rank: 5, loss = 0.043212890625

Epoch: 6, Step: 268, Rank: 4, loss = 0.041015625

Epoch: 6, Step: 268, Rank: 0, loss = 0.01348876953125

Epoch 6:  70%|███████   | 28/40 [00:46<00:19,  1.65s/it]{
    "epoch": 6,
    "step": 268,
    "rank": 0,
    "overall_throughput": 94.40912791767092,
    "lr": 1.0908391628854042e-07,
    "cuda_mem_allocated": 6.58610725402832,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56526,
    "batch_size": 126,
    "total_loss": 0.040344266355305525,
    "samples_seen": 33923,
    "gradnorm": 0.2890625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:47.805029"
}
Epoch: 6, Step: 269, Rank: 6, loss = 0.05029296875Epoch: 6, Step: 269, Rank: 1, loss = 0.03564453125Epoch: 6, Step: 269, Rank: 2, loss = 0.03369140625Epoch: 6, Step: 269, Rank: 7, loss = 0.05810546875Epoch: 6, Step: 269, Rank: 5, loss = 0.042236328125Epoch: 6, Step: 269, Rank: 3, loss = 0.032958984375
Epoch: 6, Step: 269, Rank: 4, loss = 0.050048828125





Epoch: 6, Step: 269, Rank: 0, loss = 0.0252685546875

Epoch 6:  72%|███████▎  | 29/40 [00:48<00:18,  1.65s/it]{
    "epoch": 6,
    "step": 269,
    "rank": 0,
    "overall_throughput": 94.72690864595289,
    "lr": 9.168746908439718e-08,
    "cuda_mem_allocated": 6.588102340698242,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56552,
    "batch_size": 132,
    "total_loss": 0.041024190125901826,
    "samples_seen": 34055,
    "gradnorm": 0.283203125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:49.446180"
}
Epoch: 6, Step: 270, Rank: 6, loss = 0.052734375Epoch: 6, Step: 270, Rank: 3, loss = 0.03369140625Epoch: 6, Step: 270, Rank: 5, loss = 0.042724609375


Epoch: 6, Step: 270, Rank: 2, loss = 0.02587890625Epoch: 6, Step: 270, Rank: 1, loss = 0.032470703125Epoch: 6, Step: 270, Rank: 4, loss = 0.0400390625
Epoch: 6, Step: 270, Rank: 7, loss = 0.05810546875


Epoch: 6, Step: 270, Rank: 0, loss = 0.018310546875

Epoch 6:  75%|███████▌  | 30/40 [00:49<00:16,  1.65s/it]{
    "epoch": 6,
    "step": 270,
    "rank": 0,
    "overall_throughput": 94.74675215808053,
    "lr": 7.579490328064265e-08,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56497,
    "batch_size": 126,
    "total_loss": 0.03798431775138503,
    "samples_seen": 34181,
    "gradnorm": 0.279296875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:51.092527"
}
Epoch: 6, Step: 271, Rank: 5, loss = 0.052734375Epoch: 6, Step: 271, Rank: 6, loss = 0.050537109375Epoch: 6, Step: 271, Rank: 3, loss = 0.0439453125Epoch: 6, Step: 271, Rank: 1, loss = 0.03369140625Epoch: 6, Step: 271, Rank: 7, loss = 0.04248046875
Epoch: 6, Step: 271, Rank: 2, loss = 0.031494140625




Epoch: 6, Step: 271, Rank: 4, loss = 0.036865234375
Epoch: 6, Step: 271, Rank: 0, loss = 0.0213623046875

Epoch 6:  78%|███████▊  | 31/40 [00:51<00:14,  1.65s/it]{
    "epoch": 6,
    "step": 271,
    "rank": 0,
    "overall_throughput": 93.94119302155323,
    "lr": 6.140863104726391e-08,
    "cuda_mem_allocated": 6.588009834289551,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 57276,
    "batch_size": 122,
    "total_loss": 0.03916125427753335,
    "samples_seen": 34303,
    "gradnorm": 0.3046875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:52.741265"
}
Epoch: 6, Step: 272, Rank: 6, loss = 0.04296875Epoch: 6, Step: 272, Rank: 1, loss = 0.029296875Epoch: 6, Step: 272, Rank: 0, loss = 0.018310546875Epoch: 6, Step: 272, Rank: 7, loss = 0.059814453125



Epoch: 6, Step: 272, Rank: 5, loss = 0.044921875Epoch: 6, Step: 272, Rank: 3, loss = 0.05322265625

Epoch: 6, Step: 272, Rank: 4, loss = 0.033203125
Epoch: 6, Step: 272, Rank: 2, loss = 0.027099609375

Epoch 6:  80%|████████  | 32/40 [00:53<00:13,  1.74s/it]{
    "epoch": 6,
    "step": 272,
    "rank": 0,
    "overall_throughput": 77.3885717885282,
    "lr": 4.85308359293557e-08,
    "cuda_mem_allocated": 6.5723981857299805,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56279,
    "batch_size": 125,
    "total_loss": 0.038593436272854886,
    "samples_seen": 34428,
    "gradnorm": 0.294921875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:54.686412"
}
Epoch: 6, Step: 273, Rank: 1, loss = 0.03173828125Epoch: 6, Step: 273, Rank: 6, loss = 0.04345703125Epoch: 6, Step: 273, Rank: 2, loss = 0.030517578125Epoch: 6, Step: 273, Rank: 7, loss = 0.05078125

Epoch: 6, Step: 273, Rank: 3, loss = 0.033203125
Epoch: 6, Step: 273, Rank: 4, loss = 0.037841796875Epoch: 6, Step: 273, Rank: 5, loss = 0.045654296875



Epoch: 6, Step: 273, Rank: 0, loss = 0.018310546875

Epoch 6:  82%|████████▎ | 33/40 [00:54<00:11,  1.71s/it]{
    "epoch": 6,
    "step": 273,
    "rank": 0,
    "overall_throughput": 94.16833523849348,
    "lr": 3.716347251570551e-08,
    "cuda_mem_allocated": 6.586572647094727,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56977,
    "batch_size": 123,
    "total_loss": 0.03645330571985187,
    "samples_seen": 34551,
    "gradnorm": 0.263671875,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:56.335587"
}
Epoch: 6, Step: 274, Rank: 6, loss = 0.050048828125Epoch: 6, Step: 274, Rank: 1, loss = 0.0281982421875Epoch: 6, Step: 274, Rank: 2, loss = 0.034423828125Epoch: 6, Step: 274, Rank: 3, loss = 0.02783203125
Epoch: 6, Step: 274, Rank: 7, loss = 0.055908203125Epoch: 6, Step: 274, Rank: 4, loss = 0.037353515625
Epoch: 6, Step: 274, Rank: 5, loss = 0.038330078125




Epoch: 6, Step: 274, Rank: 0, loss = 0.0281982421875

Epoch 6:  85%|████████▌ | 34/40 [00:56<00:10,  1.69s/it]{
    "epoch": 6,
    "step": 274,
    "rank": 0,
    "overall_throughput": 94.74544794841785,
    "lr": 2.7308266142119788e-08,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56828,
    "batch_size": 127,
    "total_loss": 0.0375695079890195,
    "samples_seen": 34678,
    "gradnorm": 0.27734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:57.978950"
}
Epoch: 6, Step: 275, Rank: 0, loss = 0.034912109375Epoch: 6, Step: 275, Rank: 6, loss = 0.04541015625Epoch: 6, Step: 275, Rank: 5, loss = 0.045166015625
Epoch: 6, Step: 275, Rank: 1, loss = 0.041748046875Epoch: 6, Step: 275, Rank: 7, loss = 0.062255859375Epoch: 6, Step: 275, Rank: 3, loss = 0.033203125



Epoch: 6, Step: 275, Rank: 4, loss = 0.039794921875

Epoch: 6, Step: 275, Rank: 2, loss = 0.03369140625

Epoch 6:  88%|████████▊ | 35/40 [00:58<00:08,  1.67s/it]{
    "epoch": 6,
    "step": 275,
    "rank": 0,
    "overall_throughput": 95.82617058303808,
    "lr": 1.896671262955896e-08,
    "cuda_mem_allocated": 6.58107852935791,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56325,
    "batch_size": 127,
    "total_loss": 0.042023968042609855,
    "samples_seen": 34805,
    "gradnorm": 0.302734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:45:59.607257"
}
Epoch: 6, Step: 276, Rank: 0, loss = 0.0208740234375Epoch: 6, Step: 276, Rank: 6, loss = 0.04833984375Epoch: 6, Step: 276, Rank: 7, loss = 0.051513671875
Epoch: 6, Step: 276, Rank: 5, loss = 0.04931640625
Epoch: 6, Step: 276, Rank: 3, loss = 0.029296875Epoch: 6, Step: 276, Rank: 2, loss = 0.042236328125

Epoch: 6, Step: 276, Rank: 4, loss = 0.03515625


Epoch: 6, Step: 276, Rank: 1, loss = 0.0247802734375

Epoch 6:  90%|█████████ | 36/40 [00:59<00:06,  1.66s/it]{
    "epoch": 6,
    "step": 276,
    "rank": 0,
    "overall_throughput": 94.18596253128884,
    "lr": 1.2140078057101269e-08,
    "cuda_mem_allocated": 6.5880632400512695,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56842,
    "batch_size": 125,
    "total_loss": 0.03766581049224165,
    "samples_seen": 34930,
    "gradnorm": 0.271484375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:46:01.255949"
}
Epoch: 6, Step: 277, Rank: 0, loss = 0.0291748046875Epoch: 6, Step: 277, Rank: 5, loss = 0.0439453125Epoch: 6, Step: 277, Rank: 3, loss = 0.038818359375Epoch: 6, Step: 277, Rank: 6, loss = 0.04345703125
Epoch: 6, Step: 277, Rank: 7, loss = 0.0517578125



Epoch: 6, Step: 277, Rank: 2, loss = 0.035888671875Epoch: 6, Step: 277, Rank: 4, loss = 0.033935546875

Epoch: 6, Step: 277, Rank: 1, loss = 0.03271484375

Epoch 6:  92%|█████████▎| 37/40 [01:01<00:04,  1.66s/it]{
    "epoch": 6,
    "step": 277,
    "rank": 0,
    "overall_throughput": 94.554118918858,
    "lr": 6.82939856977094e-09,
    "cuda_mem_allocated": 6.588096618652344,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56776,
    "batch_size": 126,
    "total_loss": 0.038731154008736085,
    "samples_seen": 35056,
    "gradnorm": 0.28125,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:46:02.904327"
}
Epoch: 6, Step: 278, Rank: 7, loss = 0.046875Epoch: 6, Step: 278, Rank: 1, loss = 0.027587890625Epoch: 6, Step: 278, Rank: 2, loss = 0.031982421875Epoch: 6, Step: 278, Rank: 6, loss = 0.05078125Epoch: 6, Step: 278, Rank: 5, loss = 0.042236328125
Epoch: 6, Step: 278, Rank: 4, loss = 0.03515625




Epoch: 6, Step: 278, Rank: 3, loss = 0.04052734375
Epoch: 6, Step: 278, Rank: 0, loss = 0.0224609375

Epoch 6:  95%|█████████▌| 38/40 [01:03<00:03,  1.74s/it]{
    "epoch": 6,
    "step": 278,
    "rank": 0,
    "overall_throughput": 78.01878684894416,
    "lr": 3.0354802212839705e-09,
    "cuda_mem_allocated": 6.587975978851318,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56762,
    "batch_size": 128,
    "total_loss": 0.03719037384165463,
    "samples_seen": 35184,
    "gradnorm": 0.302734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:46:04.835064"
}
Epoch: 6, Step: 279, Rank: 6, loss = 0.057861328125Epoch: 6, Step: 279, Rank: 2, loss = 0.028076171875Epoch: 6, Step: 279, Rank: 5, loss = 0.0380859375
Epoch: 6, Step: 279, Rank: 7, loss = 0.05224609375
Epoch: 6, Step: 279, Rank: 4, loss = 0.04248046875Epoch: 6, Step: 279, Rank: 3, loss = 0.035888671875Epoch: 6, Step: 279, Rank: 1, loss = 0.03564453125




Epoch: 6, Step: 279, Rank: 0, loss = 0.030517578125

Epoch 6:  98%|█████████▊| 39/40 [01:05<00:01,  1.71s/it]{
    "epoch": 6,
    "step": 279,
    "rank": 0,
    "overall_throughput": 94.51142340436657,
    "lr": 7.588988516937789e-10,
    "cuda_mem_allocated": 6.584278583526611,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56866,
    "batch_size": 123,
    "total_loss": 0.040111841873878944,
    "samples_seen": 35307,
    "gradnorm": 0.302734375,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:46:06.481737"
}
Epoch: 6, Step: 280, Rank: 2, loss = 0.033447265625Epoch: 6, Step: 280, Rank: 6, loss = 0.046875Epoch: 6, Step: 280, Rank: 4, loss = 0.039306640625Epoch: 6, Step: 280, Rank: 7, loss = 0.059814453125Epoch: 6, Step: 280, Rank: 5, loss = 0.048583984375Epoch: 6, Step: 280, Rank: 3, loss = 0.03076171875




Epoch: 6, Step: 280, Rank: 1, loss = 0.0303955078125

Epoch: 6, Step: 280, Rank: 0, loss = 0.0208740234375

Epoch 6: 100%|██████████| 40/40 [01:06<00:00,  1.69s/it]{
    "epoch": 6,
    "step": 280,
    "rank": 0,
    "overall_throughput": 93.96710605538706,
    "lr": 0.0,
    "cuda_mem_allocated": 6.588064193725586,
    "cuda_malloc_retries": 0,
    "num_loss_counted_tokens": 56277,
    "batch_size": 129,
    "total_loss": 0.03875473106242337,
    "samples_seen": 35436,
    "gradnorm": 0.2890625,
    "total_samples": 5090,
    "timestamp": "2025-03-10T19:46:08.125447"
}
Saving model in huggingface format at: samples_35436
[19:46:34] INFO     The model is bigger than the maximum     accelerator.py:2924
                    size per checkpoint (5GB) and is going                      
                    to be split in 4 checkpoint shards. You                     
                    can find where each parameters has been                     
                    saved in the index located at                               
                    /tmp/tmplc3hp5fuw/model.safetensors.inde                    
                    x.json.                                                     
Model saved in /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/hf_format/samples_35436
[19:46:50] INFO     saving took 42.045623540878296 seconds          utils.py:879
Saving full model state in /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/full_state/epoch_6
           INFO     Saving current state to                  accelerator.py:3030
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_6                                                  
           INFO     Saving FSDP model                        accelerator.py:3040
/opt/app-root/lib64/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
[19:46:58] INFO     Saving model to                             fsdp_utils.py:88
                    /var/home/cloud-user/.local/share/instructl                 
                    ab/phased/phase1/checkpoints/full_state/epo                 
                    ch_6/pytorch_model_fsdp.bin                                 
[19:47:10] INFO     Model saved to                              fsdp_utils.py:90
                    /var/home/cloud-user/.local/share/instructl                 
                    ab/phased/phase1/checkpoints/full_state/epo                 
                    ch_6/pytorch_model_fsdp.bin                                 
           INFO     FSDP Model saved to output dir           accelerator.py:3042
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_6                                                  
           INFO     Saving FSDP Optimizer                    accelerator.py:3059
[19:47:19] INFO     Saving Optimizer state to                  fsdp_utils.py:192
                    /var/home/cloud-user/.local/share/instruct                  
                    lab/phased/phase1/checkpoints/full_state/e                  
                    poch_6/optimizer.bin                                        
[19:47:42] INFO     Optimizer state saved in                   fsdp_utils.py:194
                    /var/home/cloud-user/.local/share/instruct                  
                    lab/phased/phase1/checkpoints/full_state/e                  
                    poch_6/optimizer.bin                                        
           INFO     FSDP Optimizer saved to output dir       accelerator.py:3061
                    /var/home/cloud-user/.local/share/instru                    
                    ctlab/phased/phase1/checkpoints/full_sta                    
                    te/epoch_6                                                  
           INFO     Scheduler state saved in                checkpointing.py:118
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_6/scheduler.bin                                  
           INFO     Sampler state for dataloader 0 saved in checkpointing.py:135
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_6/sampler.bin                                    
           INFO     Random states saved in                  checkpointing.py:160
                    /var/home/cloud-user/.local/share/instr                     
                    uctlab/phased/phase1/checkpoints/full_s                     
                    tate/epoch_6/random_states_0.pkl                            
Saving training state: {'current_epoch': 6, 'samples_seen': 35436}
Model state saved in: /var/home/cloud-user/.local/share/instructlab/phased/phase1/checkpoints/full_state/epoch_6

Epoch 6: 100%|██████████| 40/40 [02:41<00:00,  4.03s/it]
Training subprocess has not exited yet. Sending SIGTERM.
Waiting for process to exit, 60s...
--- Logging error ---
Traceback (most recent call last):
  File "/opt/app-root/lib64/python3.11/site-packages/instructlab/model/accelerated_train.py", line 261, in _run_phase
    _training_phase(
  File "/opt/app-root/lib64/python3.11/site-packages/instructlab/model/accelerated_train.py", line 563, in _training_phase
    run_training(train_args=train_args, torch_args=torch_args)
  File "/opt/app-root/lib64/python3.11/site-packages/instructlab/training/__init__.py", line 36, in run_training
    return run_training(torch_args=torch_args, train_args=train_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/instructlab/training/main_ds.py", line 837, in run_training
    raise RuntimeError(
RuntimeError: Suffered a failure during distributed training. Please see the training logs for more context.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.11/logging/__init__.py", line 1110, in emit
    msg = self.format(record)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/logging/__init__.py", line 953, in format
    return fmt.format(record)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/instructlab/log.py", line 19, in format
    return super().format(record)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/logging/__init__.py", line 687, in format
    record.message = record.getMessage()
                     ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/logging/__init__.py", line 377, in getMessage
    msg = msg % self.args
          ~~~~^~~~~~~~~~~
TypeError: not all arguments converted during string formatting
Call stack:
  File "/opt/app-root/bin/ilab", line 8, in <module>
    sys.exit(ilab())
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/app-root/lib64/python3.11/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/opt/app-root/lib64/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/app-root/lib64/python3.11/site-packages/instructlab/clickext.py", line 356, in wrapper
    return f(*args, **kwargs)
  File "/opt/app-root/lib64/python3.11/site-packages/instructlab/cli/model/train.py", line 469, in train
    accelerated_train.accelerated_train(
  File "/opt/app-root/lib64/python3.11/site-packages/instructlab/model/accelerated_train.py", line 202, in accelerated_train
    _run_phased_training(
  File "/opt/app-root/lib64/python3.11/site-packages/instructlab/model/accelerated_train.py", line 342, in _run_phased_training
    _run_phase(
  File "/opt/app-root/lib64/python3.11/site-packages/instructlab/model/accelerated_train.py", line 276, in _run_phase
    logger.error("Failed during training loop: ", e)
Message: 'Failed during training loop: '
Arguments: (RuntimeError('Suffered a failure during distributed training. Please see the training logs for more context.'),)
Accelerated Training failed with 1