Task 5464408

Name	RF_SAVE_ALL_OUT_NOJRAN_IGNORE_THE_REST_validation_env_g_pred_311_16903_4_1
Workunit	4850884
Created	14 Jun 2024, 12:19:28 UTC
Sent	14 Jun 2024, 15:05:48 UTC
Report deadline	15 Jun 2024, 15:05:48 UTC
Received	14 Jun 2024, 17:04:59 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	12 (0x0000000C) Unknown error code
Computer ID	38418
Run time	1 min 7 sec
CPU time	4 sec
Validate state	Invalid
Credit	0.00
Device peak FLOPS	5.71 GFLOPS
Application version	Generalized biomolecular modeling and design with RoseTTAFold All-Atom v0.02 (nvidia_alpha) windows_x86_64
Peak working set size	2,579.30 MB
Peak swap size	8,806.89 MB
Peak disk usage	2.11 MB

Stderr output

<core_client_version>7.24.1</core_client_version>
<![CDATA[
<message>
 - exit code 12 (0xc)</message>
<stderr_txt>
Traceback (most recent call last):
  File "E:\Boinc\projects\ralph.bakerlab.org\cv2\rf2aa\predict.py", line 708, in <module>
    pred.predict(out_name+f'_{n}', 
  File "E:\Boinc\projects\ralph.bakerlab.org\cv2\rf2aa\predict.py", line 551, in predict
    logit_s, logit_aa_s, logit_pae, logit_pde, p_bind, pred_crds, alpha, pred_allatom, pred_lddt_binned,                msa_prev, pair_prev, state_prev = self.model(
  File "E:\Boinc\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "E:\Boinc\projects\ralph.bakerlab.org\cv2\rf2aa\RoseTTAFoldModel.py", line 358, in forward
    msa, pair, xyz, alpha_s, xyz_allatom, state, symmsub = self.simulator(
  File "E:\Boinc\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "E:\Boinc\projects\ralph.bakerlab.org\cv2\rf2aa\Track_module.py", line 1084, in forward
    msa_full, pair, xyz, state, alpha, symmsub = self.extra_block[i_m](msa_full, pair,
  File "E:\Boinc\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "E:\Boinc\projects\ralph.bakerlab.org\cv2\rf2aa\Track_module.py", line 929, in forward
    xyz, state, alpha = self.str2str(
  File "E:\Boinc\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "E:\Boinc\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\cuda\amp\autocast_mode.py", line 141, in decorate_autocast
    return func(*args, **kwargs)
  File "E:\Boinc\projects\ralph.bakerlab.org\cv2\rf2aa\Track_module.py", line 503, in forward
    shift = self.se3(G, node.reshape(B*L, -1, 1), l1_feats, edge_feats)
  File "E:\Boinc\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "E:\Boinc\projects\ralph.bakerlab.org\cv2\rf2aa\SE3_network.py", line 96, in forward
    return self.se3(G, node_features, edge_features)
  File "E:\Boinc\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "E:\Boinc\projects\ralph.bakerlab.org\cv2\rf2aa/SE3Transformer\se3_transformer\model\transformer.py", line 185, in forward
    node_feats = self.graph_modules(node_feats, edge_feats, graph=graph, basis=basis)
  File "E:\Boinc\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "E:\Boinc\projects\ralph.bakerlab.org\cv2\rf2aa/SE3Transformer\se3_transformer\model\transformer.py", line 47, in forward
    input = module(input, *args, **kwargs)
  File "E:\Boinc\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "E:\Boinc\projects\ralph.bakerlab.org\cv2\rf2aa/SE3Transformer\se3_transformer\model\layers\attention.py", line 162, in forward
    fused_key_value = self.to_key_value(node_features, edge_features, graph, basis)
  File "E:\Boinc\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "E:\Boinc\projects\ralph.bakerlab.org\cv2\rf2aa/SE3Transformer\se3_transformer\model\layers\convolution.py", line 319, in forward
    src, dst = graph.edges()
  File "E:\Boinc\projects\ralph.bakerlab.org\ev0\lib\site-packages\dgl\view.py", line 166, in __call__
    return self._graph.all_edges(*args, **kwargs)
  File "E:\Boinc\projects\ralph.bakerlab.org\ev0\lib\site-packages\dgl\heterograph.py", line 3417, in all_edges
    src, dst, eid = self._graph.edges(self.get_etype_id(etype), order)
  File "E:\Boinc\projects\ralph.bakerlab.org\ev0\lib\site-packages\dgl\heterograph_index.py", line 609, in edges
    edge_array = _CAPI_DGLHeteroEdges(self, int(etype), order)
  File "E:\Boinc\projects\ralph.bakerlab.org\ev0\lib\site-packages\dgl\_ffi\_ctypes\function.py", line 188, in __call__
    check_call(_LIB.DGLFuncCall(
  File "E:\Boinc\projects\ralph.bakerlab.org\ev0\lib\site-packages\dgl\_ffi\base.py", line 65, in check_call
    raise DGLError(py_str(_LIB.DGLGetLastError()))
dgl._ffi.base.DGLError: [19:01:08] C:/Users/Administrator/dgl-0.5/src/array/cuda/array_op_impl.cu:259: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: CUDA kernel launch error: out of memory

</stderr_txt>
]]>