Task 5460338

Name	RF_SAVE_ALL_OUT_NOJRAN_IGNORE_THE_REST_validation_env_g_pred_38_16903_4_0
Workunit	4849244
Created	13 Jun 2024, 22:50:07 UTC
Sent	14 Jun 2024, 0:51:48 UTC
Report deadline	15 Jun 2024, 0:51:48 UTC
Received	14 Jun 2024, 2:20:21 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	12 (0x0000000C) Unknown error code
Computer ID	43305
Run time	1 hours 15 min 17 sec
CPU time	50 sec
Validate state	Invalid
Credit	0.00
Device peak FLOPS	3.55 GFLOPS
Application version	Generalized biomolecular modeling and design with RoseTTAFold All-Atom v0.02 (nvidia_alpha) windows_x86_64
Peak working set size	4,323.07 MB
Peak swap size	10,325.16 MB
Peak disk usage	2.11 MB

Stderr output

<core_client_version>8.0.2</core_client_version> <![CDATA[ <message> The access code is invalid. (0xc) - exit code 12 (0xc)</message> <stderr_txt> Traceback (most recent call last): File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\predict.py", line 708, in <module> pred.predict(out_name+f'_{n}', File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\predict.py", line 551, in predict logit_s, logit_aa_s, logit_pae, logit_pde, p_bind, pred_crds, alpha, pred_allatom, pred_lddt_binned, msa_prev, pair_prev, state_prev = self.model( File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\RoseTTAFoldModel.py", line 358, in forward msa, pair, xyz, alpha_s, xyz_allatom, state, symmsub = self.simulator( File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\Track_module.py", line 1082, in forward dchiraldxyz, = calc_chiral_grads(xyz.detach(),chirals) File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\autograd\grad_mode.py", line 28, in decorate_context return func(*args, **kwargs) File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\loss.py", line 1364, in calc_chiral_grads if l.item() == 0.0: RuntimeError: CUDA error: the launch timed out and was terminated CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. </stderr_txt> ]]>