Task 5457962
| Name | RF_SAVE_ALL_OUT_NOJRAN_IGNORE_THE_REST_validation_env_f_pred_110_16902_1_1 |
| Workunit | 4847216 |
| Created | 13 Jun 2024, 2:45:27 UTC |
| Sent | 13 Jun 2024, 5:30:41 UTC |
| Report deadline | 14 Jun 2024, 5:30:41 UTC |
| Received | 14 Jun 2024, 8:35:28 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Aborted by user |
| Exit status | 203 (0x000000CB) EXIT_ABORTED_VIA_GUI |
| Computer ID | 48876 |
| Run time | 20 min 29 sec |
| CPU time | |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 4.34 GFLOPS |
| Application version | Generalized biomolecular modeling and design with RoseTTAFold All-Atom v0.02 (nvidia_alpha) windows_x86_64 |
| Peak working set size | 5,950.00 MB |
| Peak swap size | 12,761.62 MB |
| Peak disk usage | 2.09 MB |
Stderr output
<core_client_version>8.0.2</core_client_version>
<![CDATA[
<message>
aborted by user</message>
<stderr_txt>
Traceback (most recent call last):
File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\predict.py", line 708, in <module>
pred.predict(out_name+f'_{n}',
File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\predict.py", line 551, in predict
logit_s, logit_aa_s, logit_pae, logit_pde, p_bind, pred_crds, alpha, pred_allatom, pred_lddt_binned, msa_prev, pair_prev, state_prev = self.model(
File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\RoseTTAFoldModel.py", line 358, in forward
msa, pair, xyz, alpha_s, xyz_allatom, state, symmsub = self.simulator(
File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\Track_module.py", line 1106, in forward
msa, pair, xyz, state, alpha, symmsub = self.main_block[i_m](msa, pair,
File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\Track_module.py", line 929, in forward
xyz, state, alpha = self.str2str(
File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\cuda\amp\autocast_mode.py", line 141, in decorate_autocast
return func(*args, **kwargs)
File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\Track_module.py", line 476, in forward
neighbor = get_seqsep_protein_sm(idx, bond_feats, dist_matrix, rotation_mask)
File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\util_module.py", line 106, in get_seqsep_protein_sm
res_dist, atom_dist = get_res_atom_dist(idx, bond_feats, dist_matrix, sm_mask)
File "C:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\util_module.py", line 140, in get_res_atom_dist
i_s, j_s = torch.where(bond_feats==6)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Unhandled Exception Detected...
- Unhandled Exception Record -
Reason: Breakpoint Encountered (0x80000003) at address 0x00007FFDA3308072
Engaging BOINC Windows Runtime Debugger...
********************
BOINC Windows Runtime Debugger Version 7.9.0
Dump Timestamp : 06/14/24 08:55:29
Install Directory : C:\Program Files\BOINC\
Data Directory : C:\ProgramData\BOINC
Project Symstore :
LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library : dbghelp.dll
LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library : version.dll
Debugger Engine : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\slots\10;C:\ProgramData\BOINC\projects\ralph.bakerlab.org
ModLoad: 0000000040000000 000000000013f000 C:\ProgramData\BOINC\projects\ralph.bakerlab.org\w_0.02_windows_x86_64.exe (-nosymbols- Symbols Loaded)
Linked PDB Filename : C:\Users\User\source\repos\ConsoleApplication1\x64\Release\ConsoleApplication1.pdb
ModLoad: 00000000a5cb0000 0000000000217000 C:\Windows\SYSTEM32\ntdll.dll (6.2.22621.3733) (-exported- Symbols Loaded)
Linked PDB Filename : ntdll.pdb
File Version : 10.0.22621.3374 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3374
ModLoad: 00000000a42e0000 00000000000c4000 C:\Windows\System32\KERNEL32.DLL (6.2.22621.3672) (-exported- Symbols Loaded)
Linked PDB Filename : kernel32.pdb
File Version : 10.0.22621.3374 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3374
ModLoad: 00000000a31f0000 00000000003ac000 C:\Windows\System32\KERNELBASE.dll (6.2.22621.3733) (-exported- Symbols Loaded)
Linked PDB Filename : kernelbase.pdb
File Version : 10.0.22621.3374 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3374
ModLoad: 00000000a5ac0000 00000000001ae000 C:\Windows\System32\USER32.dll (6.2.22621.3672) (-exported- Symbols Loaded)
Linked PDB Filename : user32.pdb
File Version : 10.0.22621.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.1
ModLoad: 00000000a3100000 0000000000026000 C:\Windows\System32\win32u.dll (6.2.22621.3733) (-exported- Symbols Loaded)
Linked PDB Filename : win32u.pdb
File Version : 10.0.22621.3733 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3733
ModLoad: 00000000a3ab0000 0000000000029000 C:\Windows\System32\GDI32.dll (6.2.22621.3672) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32.pdb
File Version : 10.0.22621.3672 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3672
ModLoad: 00000000a38d0000 0000000000119000 C:\Windows\System32\gdi32full.dll (6.2.22621.3672) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32full.pdb
File Version : 10.0.22621.3672 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3672
ModLoad: 00000000a3830000 000000000009a000 C:\Windows\System32\msvcp_win.dll (6.2.22621.3374) (-exported- Symbols Loaded)
Linked PDB Filename : msvcp_win.pdb
File Version : 10.0.22621.3374 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3374
ModLoad: 00000000a2fe0000 0000000000111000 C:\Windows\System32\ucrtbase.dll (6.2.22621.3593) (-exported- Symbols Loaded)
Linked PDB Filename : ucrtbase.pdb
File Version : 10.0.22621.3593 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3593
ModLoad: 00000000a4490000 00000000000b2000 C:\Windows\System32\ADVAPI32.dll (6.2.22621.3733) (-exported- Symbols Loaded)
Linked PDB Filename : advapi32.pdb
File Version : 10.0.22621.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.1
ModLoad: 00000000a3ae0000 00000000000a7000 C:\Windows\System32\msvcrt.dll (7.0.22621.2506) (-exported- Symbols Loaded)
Linked PDB Filename : msvcrt.pdb
File Version : 7.0.22621.2506 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 7.0.22621.2506
ModLoad: 00000000a4dc0000 00000000000aa000 C:\Windows\System32\sechost.dll (6.2.22621.3733) (-exported- Symbols Loaded)
Linked PDB Filename : sechost.pdb
File Version : 10.0.22621.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.1
ModLoad: 00000000a35a0000 0000000000028000 C:\Windows\System32\bcrypt.dll (6.2.22621.2506) (-exported- Symbols Loaded)
Linked PDB Filename : bcrypt.pdb
File Version : 10.0.22621.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.1
ModLoad: 00000000a59a0000 0000000000115000 C:\Windows\System32\RPCRT4.dll (6.2.22621.3672) (-exported- Symbols Loaded)
Linked PDB Filename : rpcrt4.pdb
File Version : 10.0.22621.2506 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.2506
ModLoad: 00000000a5710000 0000000000031000 C:\Windows\System32\IMM32.DLL (6.2.22621.3374) (-exported- Symbols Loaded)
Linked PDB Filename : imm32.pdb
File Version : 10.0.22621.3374 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3374
ModLoad: 00000000a1fd0000 0000000000018000 C:\Windows\SYSTEM32\kernel.appcore.dll (6.2.22621.2715) (-exported- Symbols Loaded)
Linked PDB Filename : Kernel.Appcore.pdb
File Version : 10.0.22621.2715 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.2715
ModLoad: 00000000a06e0000 0000000000232000 C:\Windows\SYSTEM32\dbghelp.dll (6.2.22621.3593) (-exported- Symbols Loaded)
Linked PDB Filename : dbghelp.pdb
File Version : 10.0.22621.3593 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3593
ModLoad: 00000000a3c20000 0000000000388000 C:\Windows\System32\combase.dll (6.2.22621.3672) (-exported- Symbols Loaded)
Linked PDB Filename : combase.pdb
File Version : 10.0.22621.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.1
ModLoad: 00000000a43b0000 00000000000d7000 C:\Windows\System32\OLEAUT32.dll (6.2.22621.3672) (-exported- Symbols Loaded)
Linked PDB Filename : oleaut32.pdb
File Version : 10.0.22621.3672 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3672
ModLoad: 000000009a430000 000000000000a000 C:\Windows\SYSTEM32\version.dll (6.2.22621.1) (-exported- Symbols Loaded)
Linked PDB Filename : version.pdb
File Version : 10.0.22621.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.1
ModLoad: 00000000a3640000 000000000007b000 C:\Windows\System32\bcryptPrimitives.dll (6.2.22621.3672) (-exported- Symbols Loaded)
Linked PDB Filename : bcryptprimitives.pdb
File Version : 10.0.22621.3672 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3672
*** Dump of the Process Statistics: ***
- I/O Operations Counters -
Read: 10, Write: 185, Other 143
- I/O Transfers Counters -
Read: 24054, Write: 194, Other 8532
- Paged Pool Usage -
QuotaPagedPoolUsage: 93168, QuotaPeakPagedPoolUsage: 93368
QuotaNonPagedPoolUsage: 7336, QuotaPeakNonPagedPoolUsage: 15336
- Virtual Memory Usage -
VirtualSize: 2920448, PeakVirtualSize: 86171648
- Pagefile Usage -
PagefileUsage: 2920448, PeakPagefileUsage: 2924544
- Working Set Size -
WorkingSetSize: 6512640, PeakWorkingSetSize: 6512640, PageFaultCount: 12477
*** Dump of thread ID 10272 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 110781248.000000, User Time: 122031248.000000, Wait Time: 5545971.000000
- Unhandled Exception Record -
Reason: Breakpoint Encountered (0x80000003) at address 0x00007FFDA3308072
- Registers -
rax=0000000000000000 rbx=0000000000000001 rcx=0000000040099c58 rdx=00000000023eeb80 rsi=0000000000000000 rdi=0000000000000000
r8=00000000023eeb80 r9=0000000040099c48 r10=0000000000000fff r11=0000000000000ff0 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=00000000a3308072 rsp=00000000023eeb58 rbp=0000000000000000
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
023eeb50 40041010 00000001 023eeb80 023eeb80 40099c48 KERNELBASE!DebugBreak+0x0
023eef90 40041b74 f2e00000 023ef2f3 00000000 000001b0 w_0.02_windows_x86_64!+0x0
023efef0 40041f14 00000000 00000000 fff0bdc0 00000000 w_0.02_windows_x86_64!+0x0
023eff20 a42f257d 00000000 00000000 00000000 00000000 w_0.02_windows_x86_64!+0x0
023eff50 a5d0af28 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
023effd0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 400 (state: Waiting): ***
- Information -
Status: Wait Reason: Suspended, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 67295.000000
- Registers -
rax=0000000000000004 rbx=0000000000000000 rcx=000000000000014c rdx=0000000000000000 rsi=0000000000000000 rdi=000000000000014c
r8=0000000000000004 r9=00000000001392a8 r10=0000000000000080 r11=00000000a4362210 r12=0000000000000000 r13=0000000000000000
r14=000000000000014c r15=0000000000000000 rip=00000000a5d50024 rsp=000000000013b518 rbp=000000000013b679
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
0013b510 a322952e 00000000 a3219100 00000000 00000000 ntdll!NtWaitForSingleObject+0x0
0013b5b0 4006a010 0000014c ffffffff 00000000 0000014c KERNELBASE!WaitForSingleObjectEx+0x0
0013b6d0 40069ddd 00000000 00000000 ffffffff 00000000 w_0.02_windows_x86_64!+0x0
0013b730 4004be8b 00000000 00000000 0013b7a0 005abf60 w_0.02_windows_x86_64!+0x0
0013b7a0 40003e04 005b75a0 00000031 005ac000 0013b8b0 w_0.02_windows_x86_64!+0x0
0014fee0 400447b0 005abce0 00000000 005b8aa0 00000000 w_0.02_windows_x86_64!+0x0
0014ff20 a42f257d 00000000 00000000 00000000 00000000 w_0.02_windows_x86_64!+0x0
0014ff50 a5d0af28 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
0014ffd0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Debug Message Dump ****
*** Foreground Window Data ***
Window Name :
Window Class :
Window Process ID: 0
Window Thread ID : 0
Exiting...
</stderr_txt>
]]>
©2024 University of Washington
http://www.bakerlab.org