chore🍟:
This commit is contained in:
@ -1,136 +0,0 @@
|
||||
WARNING:torch.distributed.run:
|
||||
*****************************************
|
||||
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
|
||||
*****************************************
|
||||
Training model with params:
|
||||
base_model: /home/tushilong/hf/models/Llama-2-7b-hf
|
||||
output_dir: ../ckpts/stylish
|
||||
micro_batch_size: 2
|
||||
gradient_accumulation_steps: 1
|
||||
train_batch_size: 2
|
||||
gradient_checkpointing: True
|
||||
num_epochs: 1
|
||||
learning_rate: 2e-05
|
||||
weight_decay: 0.0001
|
||||
warmup_ratio: 0.06
|
||||
deepspeed_config: None
|
||||
fsdp: shard_grad_op auto_wrap offload
|
||||
fsdp_config: ./configs/fsdp/llama2_fsdp_config.json
|
||||
smart_embedding: False
|
||||
wandb_project:
|
||||
wandb_run_name:
|
||||
wandb_watch:
|
||||
wandb_log_model:
|
||||
resume_from_checkpoint: False
|
||||
|
||||
|
||||
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
|
||||
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
|
||||
Loading checkpoint shards: 50%|█████ | 1/2 [00:03<00:03, 3.52s/it]
|
||||
Loading checkpoint shards: 50%|█████ | 1/2 [00:03<00:03, 3.51s/it]
|
||||
Loading checkpoint shards: 100%|██████████| 2/2 [00:04<00:00, 2.20s/it]
|
||||
Loading checkpoint shards: 100%|██████████| 2/2 [00:04<00:00, 2.40s/it]
|
||||
|
||||
Loading checkpoint shards: 100%|██████████| 2/2 [00:04<00:00, 2.20s/it]
|
||||
Loading checkpoint shards: 100%|██████████| 2/2 [00:04<00:00, 2.39s/it]
|
||||
|
||||
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
|
||||
Loading checkpoint shards: 50%|█████ | 1/2 [00:03<00:03, 3.45s/it]
|
||||
Loading checkpoint shards: 100%|██████████| 2/2 [00:04<00:00, 2.17s/it]
|
||||
Loading checkpoint shards: 100%|██████████| 2/2 [00:04<00:00, 2.36s/it]
|
||||
Detected kernel version 4.15.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
|
||||
StateDictType.FULL_STATE_DICT FullStateDictConfig(offload_to_cpu=False, rank0_only=False)
|
||||
You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
|
||||
StateDictType.FULL_STATE_DICT FullStateDictConfig(offload_to_cpu=False, rank0_only=False)
|
||||
|
||||
0%| | 0/167 [00:00<?, ?it/s]You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
|
||||
StateDictType.FULL_STATE_DICT FullStateDictConfig(offload_to_cpu=False, rank0_only=False)
|
||||
You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
|
||||
|
||||
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
|
||||
|
||||
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s][A
|
||||
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
|
||||
Loading checkpoint shards: 50%|█████ | 1/2 [00:00<00:00, 4.90it/s]
|
||||
Loading checkpoint shards: 50%|█████ | 1/2 [00:00<00:00, 4.20it/s]
|
||||
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 6.97it/s]
|
||||
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 6.55it/s]
|
||||
|
||||
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 6.33it/s]
|
||||
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00, 5.88it/s]
|
||||
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...
|
||||
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...
|
||||
|
||||
|
||||
Loading checkpoint shards: 50%|█████ | 1/2 [00:14<00:14, 14.71s/it][A
|
||||
|
||||
Loading checkpoint shards: 100%|██████████| 2/2 [00:19<00:00, 8.65s/it][A
|
||||
Loading checkpoint shards: 100%|██████████| 2/2 [00:19<00:00, 9.55s/it]
|
||||
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...
|
||||
Traceback (most recent call last):
|
||||
File "/home/tushilong/code/realign/train/train.py", line 167, in <module>
|
||||
fire.Fire(train)
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/fire/core.py", line 141, in Fire
|
||||
component_trace = _Fire(component, args, parsed_flag_args, context, name)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/fire/core.py", line 475, in _Fire
|
||||
component, remaining_args = _CallAndUpdateTrace(
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
|
||||
component = fn(*varargs, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/code/realign/train/train.py", line 160, in train
|
||||
trainer.train(resume_from_checkpoint=resume_from_checkpoint)
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/transformers/trainer.py", line 1555, in train
|
||||
return inner_training_loop(
|
||||
^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/transformers/trainer.py", line 1837, in _inner_training_loop
|
||||
tr_loss_step = self.training_step(model, inputs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/transformers/trainer.py", line 2682, in training_step
|
||||
loss = self.compute_loss(model, inputs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/transformers/trainer.py", line 2707, in compute_loss
|
||||
outputs = model(**inputs)
|
||||
^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
|
||||
return forward_call(*args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/accelerate/utils/operations.py", line 659, in forward
|
||||
return model_forward(*args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/accelerate/utils/operations.py", line 647, in __call__
|
||||
return convert_to_fp32(self.model_forward(*args, **kwargs))
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
|
||||
return func(*args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/accelerate/utils/operations.py", line 659, in forward
|
||||
return model_forward(*args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/accelerate/utils/operations.py", line 647, in __call__
|
||||
return convert_to_fp32(self.model_forward(*args, **kwargs))
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
|
||||
return func(*args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 748, in forward
|
||||
output = self._fsdp_wrapped_module(*args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
|
||||
return forward_call(*args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/code/realign/llama/rellama.py", line 129, in forward
|
||||
assert torch.isnan(target_logits).sum() == 0, f"target_logits has nan: {torch.isnan(target_logits).sum()}"
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
AssertionError: target_logits has nan: 10752000
|
||||
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 20205 closing signal SIGTERM
|
||||
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 20206 closing signal SIGTERM
|
||||
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 2 (pid: 20207) of binary: /home/tushilong/anaconda3/envs/realign/bin/python
|
||||
Traceback (most recent call last):
|
||||
File "/home/tushilong/anaconda3/envs/realign/bin/torchrun", line 33, in <module>
|
||||
sys.exit(load_entry_point('torch==2.0.1', 'console_scripts', 'torchrun')())
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/home/tushilong/anaconda3/envs/realign/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
|
||||
return f(*args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
Reference in New Issue
Block a user