Task 5461656

Name	RF_SAVE_ALL_OUT_NOJRAN_IGNORE_THE_REST_validation_env_g_pred_44_16903_3_1
Workunit	4849225
Created	14 Jun 2024, 1:31:47 UTC
Sent	14 Jun 2024, 4:12:37 UTC
Report deadline	15 Jun 2024, 4:12:37 UTC
Received	14 Jun 2024, 10:27:13 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	40032
Run time	6 hours 3 min 28 sec
CPU time	1 min 37 sec
Validate state	Valid
Credit	200.71
Device peak FLOPS	5.06 GFLOPS
Application version	Generalized biomolecular modeling and design with RoseTTAFold All-Atom v0.02 (nvidia_alpha) windows_x86_64
Peak working set size	4,916.46 MB
Peak swap size	11,475.56 MB
Peak disk usage	6.38 MB

Stderr output

<core_client_version>7.24.1</core_client_version>
<![CDATA[
<stderr_txt>
Traceback (most recent call last):
  File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\predict.py", line 708, in <module>
    pred.predict(out_name+f'_{n}', 
  File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\predict.py", line 551, in predict
    logit_s, logit_aa_s, logit_pae, logit_pde, p_bind, pred_crds, alpha, pred_allatom, pred_lddt_binned,                msa_prev, pair_prev, state_prev = self.model(
  File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\RoseTTAFoldModel.py", line 358, in forward
    msa, pair, xyz, alpha_s, xyz_allatom, state, symmsub = self.simulator(
  File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\Track_module.py", line 1106, in forward
    msa, pair, xyz, state, alpha, symmsub = self.main_block[i_m](msa, pair,
  File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\Track_module.py", line 929, in forward
    xyz, state, alpha = self.str2str(
  File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\cuda\amp\autocast_mode.py", line 141, in decorate_autocast
    return func(*args, **kwargs)
  File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\Track_module.py", line 488, in forward
    G, edge_feats = make_full_graph(xyz[:,:,1,:], edge, idx)
  File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\util_module.py", line 201, in make_full_graph
    G = dgl.graph((src, tgt), num_nodes=B*L).to(device)
  File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\dgl\convert.py", line 181, in graph
    (sparse_fmt, arrays), urange, vrange = utils.graphdata2tensors(data, idtype)
  File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\dgl\utils\data.py", line 177, in graphdata2tensors
    num_src, num_dst = infer_num_nodes(data, bipartite=bipartite)
  File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\dgl\utils\data.py", line 300, in infer_num_nodes
    nsrc = F.as_scalar(F.max(u, dim=0)) + 1 if len(u) > 0 else 0
  File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\dgl\backend\pytorch\tensor.py", line 47, in as_scalar
    return data.item()
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
06:26:50 (55156): called boinc_finish(0)

</stderr_txt>
]]>