runtime error

Exit code: 1. Reason: s] model_jay1.safetensors: 75%|███████▍ | 1.01G/1.35G [00:02<00:00, 500MB/s] model_jay1.safetensors: 100%|██████████| 1.35G/1.35G [00:02<00:00, 601MB/s] vocab : /usr/local/lib/python3.10/site-packages/f5_tts/infer/examples/vocab.txt token : custom model : /home/user/.cache/huggingface/hub/models--kevinwang676--F5-TTS/snapshots/c37c3bc0a95bbc1533ce720b1e377b45d46b5804/model_jay1.safetensors E2TTS_Base/model_1200000.safetensors: 0%| | 0.00/1.33G [00:00<?, ?B/s] E2TTS_Base/model_1200000.safetensors: 30%|███ | 403M/1.33G [00:01<00:02, 320MB/s] E2TTS_Base/model_1200000.safetensors: 100%|██████████| 1.33G/1.33G [00:01<00:00, 677MB/s] vocab : /usr/local/lib/python3.10/site-packages/f5_tts/infer/examples/vocab.txt token : custom model : /home/user/.cache/huggingface/hub/models--SWivid--E2-TTS/snapshots/851141880b5ca38050025e98dfdee27dc553f86e/E2TTS_Base/model_1200000.safetensors SPACES_ZERO_GPU_DEBUG self.arg_queue._writer.fileno()=32 SPACES_ZERO_GPU_DEBUG self.res_queue._writer.fileno()=34 Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/spaces/zero/wrappers.py", line 148, in worker_init torch.init(nvidia_uuid) File "/usr/local/lib/python3.10/site-packages/spaces/zero/torch/patching.py", line 417, in init torch.Tensor([0]).cuda() torch.AcceleratorError: CUDA error: uncorrectable ECC error encountered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. Traceback (most recent call last): File "/home/user/app/app.py", line 591, in <module> load_chat_model(chat_model_name_list[0]) File "/usr/local/lib/python3.10/site-packages/spaces/zero/wrappers.py", line 238, in gradio_handler raise error("ZeroGPU worker error", res.error_cls) gradio.exceptions.Error: 'AcceleratorError'

Container logs: