Task 5469723

Name	RF_SAVE_ALL_OUT_NOJRAN_IGNORE_THE_REST_validation_env_j_pred_148_16906_6_0
Workunit	4854855
Created	21 Jun 2024, 10:46:57 UTC
Sent	21 Jun 2024, 18:02:20 UTC
Report deadline	23 Jun 2024, 18:02:20 UTC
Received	21 Jun 2024, 18:04:59 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	12 (0x0000000C) Unknown error code
Computer ID	50054
Run time	23 sec
CPU time
Validate state	Invalid
Credit	0.00
Device peak FLOPS	30,739.86 GFLOPS
Application version	Generalized biomolecular modeling and design with RoseTTAFold All-Atom v0.03 (nvidia_alpha) windows_x86_64
Peak working set size	1,213.80 MB
Peak swap size	5,561.10 MB
Peak disk usage	2.45 MB

Stderr output

<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
The access code is invalid.
 (0xc) - exit code 12 (0xc)</message>
<stderr_txt>
C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\util.py:450: UserWarning: Using torch.cross without specifying the dim arg is deprecated.
Please either pass the dim explicitly or simply use torch.linalg.cross.
The default value of dim will change to agree with that of linalg.cross in a future release. (Triggered internally at C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\Cross.cpp:66.)
  Z = torch.cross(Xn,Yn)
Traceback (most recent call last):
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\predict.py", line 742, in <module>
    pred.predict(out_name+f'_{n}', 
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\predict.py", line 550, in predict
    logit_s, logit_aa_s, logit_pae, logit_pde, p_bind, pred_crds, alpha, pred_allatom, pred_lddt_binned,                msa_prev, pair_prev, state_prev = self.model(
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev1\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev1\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\RoseTTAFoldModel.py", line 358, in forward
    msa, pair, xyz, alpha_s, xyz_allatom, state, symmsub = self.simulator(
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev1\lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev1\lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\Track_module.py", line 1135, in forward
    dljdxyz, dljdalpha = calc_lj_grads(
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev1\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\loss.py", line 1316, in calc_lj_grads
    Elj = calc_lj(
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\loss.py", line 1143, in calc_lj
    ljval = lj(
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev1\lib\site-packages\torch\autograd\function.py", line 598, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\loss.py", line 1098, in forward
    ljval_batch,dljEdd_i = LJLoss.ljVdV(deltas,ljrs,ljss,lj_lin,eps)
  File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv1\rf2aa\loss.py", line 996, in ljVdV
    -12 * sd12[linpart]/deff[linpart] + 12 * sd6[linpart]/deff[linpart]
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


</stderr_txt>
]]>