Task 5469297

Name	RF_SAVE_ALL_OUT_NOJRAN_IGNORE_THE_REST_validation_env_j_pred_97_16906_1_0
Workunit	4854429
Created	21 Jun 2024, 10:44:51 UTC
Sent	21 Jun 2024, 16:51:03 UTC
Report deadline	23 Jun 2024, 16:51:03 UTC
Received	21 Jun 2024, 16:53:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	12 (0x0000000C) Unknown error code
Computer ID	50054
Run time	24 sec
CPU time
Validate state	Invalid
Credit	0.00
Device peak FLOPS	30,739.86 GFLOPS
Application version	Generalized biomolecular modeling and design with RoseTTAFold All-Atom v0.03 (nvidia_alpha) windows_x86_64
Peak working set size	1,161.65 MB
Peak swap size	5,126.96 MB
Peak disk usage	2.45 MB

Stderr output

<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
The access code is invalid.
 (0xc) - exit code 12 (0xc)</message>
<stderr_txt>
C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\util.py:450: UserWarning: Using torch.cross without specifying the dim arg is deprecated.
Please either pass the dim explicitly or simply use torch.linalg.cross.
The default value of dim will change to agree with that of linalg.cross in a future release. (Triggered internally at C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\Cross.cpp:66.)
  Z = torch.cross(Xn,Yn)
Traceback (most recent call last):
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\predict.py", line 742, in <module>
    pred.predict(out_name+f'_{n}', 
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\predict.py", line 550, in predict
    logit_s, logit_aa_s, logit_pae, logit_pde, p_bind, pred_crds, alpha, pred_allatom, pred_lddt_binned,                msa_prev, pair_prev, state_prev = self.model(
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev1\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev1\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\RoseTTAFoldModel.py", line 358, in forward
    msa, pair, xyz, alpha_s, xyz_allatom, state, symmsub = self.simulator(
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev1\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev1\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\Track_module.py", line 1084, in forward
    msa_full, pair, xyz, state, alpha, symmsub = self.extra_block[i_m](msa_full, pair,
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev1\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev1\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\Track_module.py", line 929, in forward
    xyz, state, alpha = self.str2str(
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev1\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev1\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev1\lib\site-packages\torch\amp\autocast_mode.py", line 16, in decorate_autocast
    return func(*args, **kwargs)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\Track_module.py", line 476, in forward
    neighbor = get_seqsep_protein_sm(idx, bond_feats, dist_matrix, rotation_mask)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\util_module.py", line 106, in get_seqsep_protein_sm
    res_dist, atom_dist = get_res_atom_dist(idx, bond_feats, dist_matrix, sm_mask)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\util_module.py", line 147, in get_res_atom_dist
    res_dist_inter[sm_mask,:] = res_dist_prot[closest_prot_res,:]
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


</stderr_txt>
]]>