Windows 11: FileExistsError: [WinError 183] Cannot create a file when that file already exists

[VyBui]

Can someone help on this issue?

(mlenv) C:\Users\Admin\nanoGPT>python train.py config/train_shakespeare_char.py
Overriding config with config/train_shakespeare_char.py:

train a miniature character-level shakespeare model

good for debugging and playing on macbooks and such

out_dir = 'out-shakespeare-char'
eval_interval = 250 # keep frequent because we'll overfit
eval_iters = 200
log_interval = 10 # don't print too too often

we expect to overfit on this small dataset, so only save when val improves

always_save_checkpoint = False

wandb_log = False # override via command line if you like
wandb_project = 'shakespeare-char'
wandb_run_name = 'mini-gpt'

dataset = 'shakespeare_char'
gradient_accumulation_steps = 1
batch_size = 64
block_size = 256 # context of up to 256 previous characters

baby GPT model :)

n_layer = 6
n_head = 6
n_embd = 384
dropout = 0.2

learning_rate = 1e-3 # with baby networks can afford to go a bit higher
max_iters = 5000
lr_decay_iters = 5000 # make equal to max_iters usually
min_lr = 1e-4 # learning_rate / 10 usually
beta2 = 0.99 # make a bit bigger because number of tokens per iter is small

warmup_iters = 100 # not super necessary potentially

on macbook also add

device = 'cpu' # run on cpu only

compile = False # do not torch compile the model

tokens per iteration will be: 16,384
found vocab_size = 65 (inside data\shakespeare_char\meta.pkl)
Initializing a new model from scratch
number of parameters: 10.65M
C:\Users\Admin\nanoGPT\train.py:196: FutureWarning: torch.cuda.amp.GradScaler(args...) is deprecated. Please use torch.amp.GradScaler('cuda', args...) instead.
scaler = torch.cuda.amp.GradScaler(enabled=(dtype == 'float16'))
num decayed parameter tensors: 26, with 10,740,096 parameters
num non-decayed parameter tensors: 13, with 4,992 parameters
using fused AdamW: True
compiling the model... (takes a ~minute)
C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\utils.py:1903: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:555.)
return node.target(*args, **kwargs)
Traceback (most recent call last):
File "C:\Users\Admin\nanoGPT\train.py", line 264, in
losses = estimate_loss()
^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch\utils_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\nanoGPT\train.py", line 224, in estimate_loss
logits, loss = model(X, Y)
^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\eval_frame.py", line 433, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch\nn\modules\module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch\nn\modules\module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\convert_frame.py", line 1116, in call
return self._torchdynamo_orig_callable(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\convert_frame.py", line 948, in call
result = self._inner_convert(
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\convert_frame.py", line 472, in call
return _compile(
^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_utils_internal.py", line 84, in wrapper_function
return StrobelightCompileTimeProfiler.profile_compile_time(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_strobelight\compile_time_profiler.py", line 129, in profile_compile_time
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\convert_frame.py", line 817, in _compile
guarded_code = compile_inner(code, one_graph, hooks, transform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\utils.py", line 231, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\convert_frame.py", line 636, in compile_inner
out_code = transform_code_object(code, transform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\bytecode_transformation.py", line 1185, in transform_code_object
transformations(instructions, code_options)
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\convert_frame.py", line 178, in fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\convert_frame.py", line 582, in transform
tracer.run()
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\symbolic_convert.py", line 2451, in run
super().run()
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\symbolic_convert.py", line 893, in run
while self.step():
^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\symbolic_convert.py", line 805, in step
self.dispatch_table[inst.opcode](self, inst)
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\symbolic_convert.py", line 2642, in RETURN_VALUE
self.return(inst)
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\symbolic_convert.py", line 2627, in return
self.output.compile_subgraph(
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\output_graph.py", line 1123, in compile_subgraph
self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\output_graph.py", line 1318, in compile_and_call_fx_graph
compiled_fn = self.call_user_compiler(gm)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\utils.py", line 231, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\output_graph.py", line 1409, in call_user_compiler
raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\output_graph.py", line 1390, in call_user_compiler
compiled_fn = compiler_fn(gm, self.example_inputs())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\repro\after_dynamo.py", line 129, in call
compiled_gm = compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_init.py", line 1951, in call
return compile_fx(model, inputs, config_patches=self.config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_inductor\compile_fx.py", line 1505, in compile_fx
return aot_autograd(
^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\backends\common.py", line 69, in call
cg = aot_module_simplified(gm, example_inputs, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_functorch\aot_autograd.py", line 954, in aot_module_simplified
compiled_fn, _ = create_aot_dispatcher_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\utils.py", line 231, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_functorch\aot_autograd.py", line 687, in create_aot_dispatcher_function
compiled_fn, fw_metadata = compiler_fn(
^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_functorch_aot_autograd\jit_compile_runtime_wrappers.py", line 168, in aot_dispatch_base
compiled_fw = compiler(fw_module, updated_flat_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_dynamo\utils.py", line 231, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_inductor\compile_fx.py", line 1352, in fw_compiler_base
_recursive_joint_graph_passes(model)
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_inductor\compile_fx.py", line 256, in _recursive_joint_graph_passes
joint_graph_passes(gm)
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_inductor\fx_passes\joint_graph.py", line 326, in joint_graph_passes
count += patterns.apply(graph.graph) # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_inductor\pattern_matcher.py", line 1698, in apply
if is_match(m) and entry.extra_check(m):
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_inductor\pattern_matcher.py", line 1314, in check_fn
if is_match(specific_pattern_match) and extra_check(specific_pattern_match):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_inductor\fx_passes\pad_mm.py", line 502, in should_pad_mm
return should_pad_common(mat1, mat2) and should_pad_bench(
^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_inductor\fx_passes\pad_mm.py", line 492, in should_pad_bench
set_cached_should_pad(key, should_pad)
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_inductor\fx_passes\pad_mm.py", line 237, in set_cached_should_pad
return get_pad_cache().set_value(key, value=value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_inductor\codecache.py", line 230, in set_value
self.update_local_cache(cache)
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_inductor\codecache.py", line 201, in update_local_cache
write_atomic(
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\site-packages\torch_inductor\codecache.py", line 404, in write_atomic
tmp_path.rename(path)
File "C:\Users\Admin\anaconda3\envs\mlenv\Lib\pathlib.py", line 1175, in rename
os.rename(self, target)
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\Users\Admin\AppData\Local\Temp\torchinductor_Admin\cache\.13268.32284.tmp' -> 'C:\Users\Admin\AppData\Local\Temp\torchinductor_Admin\cache\e2263ba6e33073368c52f4aa78f67071b5f51eedea3cc454388bc54a5d4af969'

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information

You can suppress this exception and fall back to eager by setting:
import torch._dynamo
torch._dynamo.config.suppress_errors = True

[fhyfhytt]

I meet same problem,please help.

[mkowalczyk88]

I found setting all of this a little bit tricky on windows, but it is possible. I used conda + python 3.10 + https://github.com/woct0rdho/triton-windows with a 'standalone' cuda installation. For your specific problem I don't remember if I had to apply this fix: pytorch/pytorch@d5035f0 or just change the pytorch version. Hope that's helpful.