Task 5465240
| Name | RF_SAVE_ALL_OUT_NOJRAN_IGNORE_THE_REST_validation_env_g_pred_277_16903_1_1 |
| Workunit | 4850587 |
| Created | 15 Jun 2024, 9:30:13 UTC |
| Sent | 15 Jun 2024, 9:30:26 UTC |
| Report deadline | 16 Jun 2024, 9:30:26 UTC |
| Received | 15 Jun 2024, 14:25:12 UTC |
| Server state | Over |
| Outcome | Computation error |
| Client state | Aborted by user |
| Exit status | 203 (0x000000CB) EXIT_ABORTED_VIA_GUI |
| Computer ID | 49964 |
| Run time | 2 hours 14 min 16 sec |
| CPU time | 1 sec |
| Validate state | Invalid |
| Credit | 0.00 |
| Device peak FLOPS | 1.00 GFLOPS |
| Application version | Generalized biomolecular modeling and design with RoseTTAFold All-Atom v0.02 (nvidia_alpha) windows_x86_64 |
| Peak working set size | 3,926.48 MB |
| Peak swap size | 9,892.30 MB |
| Peak disk usage | 2.10 MB |
Stderr output
<core_client_version>7.24.1</core_client_version>
<![CDATA[
<message>
aborted by user</message>
<stderr_txt>
Unhandled Exception Detected...
- Unhandled Exception Record -
Reason: Breakpoint Encountered (0x80000003) at address 0x00007FFD757E8072
Engaging BOINC Windows Runtime Debugger...
********************
BOINC Windows Runtime Debugger Version 7.9.0
Dump Timestamp : 06/15/24 10:24:00
Install Directory : D:\Program Files\BOINC\
Data Directory : D:\ProgramData\BOINC
Project Symstore : https://boinc.bakerlab.org/rosetta/symstore
LoadLibraryA( D:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library : dbghelp.dll
LoadLibraryA( D:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library : version.dll
Debugger Engine : 4.0.5.0
Symbol Search Path: D:\ProgramData\BOINC\slots\7;D:\ProgramData\BOINC\projects\ralph.bakerlab.org;srv*D:\ProgramData\BOINC\projects\ralph.bakerlab.org\symbols*http://msdl.microsoft.com/download/symbols;srv*D:\ProgramData\BOINC\projects\ralph.bakerlab.org\symbols*https://boinc.bakerlab.org/rosetta/symstore
ModLoad: 0000000040000000 000000000013f000 D:\ProgramData\BOINC\projects\ralph.bakerlab.org\w_0.02_windows_x86_64.exe (-nosymbols- Symbols Loaded)
Linked PDB Filename : C:\Users\User\source\repos\ConsoleApplication1\x64\Release\ConsoleApplication1.pdb
ModLoad: 0000000078210000 0000000000217000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.22621.3733) (-exported- Symbols Loaded)
Linked PDB Filename : ntdll.pdb
File Version : 10.0.22621.3374 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3374
ModLoad: 0000000076350000 00000000000c4000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.22621.3672) (-exported- Symbols Loaded)
Linked PDB Filename : kernel32.pdb
File Version : 10.0.22621.3374 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3374
ModLoad: 00000000756d0000 00000000003ac000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.22621.3733) (-exported- Symbols Loaded)
Linked PDB Filename : kernelbase.pdb
File Version : 10.0.22621.3374 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3374
ModLoad: 0000000076080000 00000000001ae000 C:\WINDOWS\System32\USER32.dll (6.2.22621.3672) (-exported- Symbols Loaded)
Linked PDB Filename : user32.pdb
File Version : 10.0.22621.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.1
ModLoad: 0000000075b00000 0000000000026000 C:\WINDOWS\System32\win32u.dll (6.2.22621.3733) (-exported- Symbols Loaded)
Linked PDB Filename : win32u.pdb
File Version : 10.0.22621.3733 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3733
ModLoad: 0000000076c80000 0000000000029000 C:\WINDOWS\System32\GDI32.dll (6.2.22621.3672) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32.pdb
File Version : 10.0.22621.3672 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3672
ModLoad: 0000000075d60000 0000000000119000 C:\WINDOWS\System32\gdi32full.dll (6.2.22621.3672) (-exported- Symbols Loaded)
Linked PDB Filename : gdi32full.pdb
File Version : 10.0.22621.3672 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3672
ModLoad: 0000000075eb0000 000000000009a000 C:\WINDOWS\System32\msvcp_win.dll (6.2.22621.3374) (-exported- Symbols Loaded)
Linked PDB Filename : msvcp_win.pdb
File Version : 10.0.22621.3374 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3374
ModLoad: 0000000075540000 0000000000111000 C:\WINDOWS\System32\ucrtbase.dll (6.2.22621.3593) (-exported- Symbols Loaded)
Linked PDB Filename : ucrtbase.pdb
File Version : 10.0.22621.3593 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3593
ModLoad: 0000000077230000 00000000000b2000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.22621.3733) (-exported- Symbols Loaded)
Linked PDB Filename : advapi32.pdb
File Version : 10.0.22621.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.1
ModLoad: 00000000774c0000 00000000000a7000 C:\WINDOWS\System32\msvcrt.dll (7.0.22621.2506) (-exported- Symbols Loaded)
Linked PDB Filename : msvcrt.pdb
File Version : 7.0.22621.2506 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 7.0.22621.2506
ModLoad: 00000000777e0000 00000000000aa000 C:\WINDOWS\System32\sechost.dll (6.2.22621.3733) (-exported- Symbols Loaded)
Linked PDB Filename : sechost.pdb
File Version : 10.0.22621.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.1
ModLoad: 0000000075e80000 0000000000028000 C:\WINDOWS\System32\bcrypt.dll (6.2.22621.2506) (-exported- Symbols Loaded)
Linked PDB Filename : bcrypt.pdb
File Version : 10.0.22621.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.1
ModLoad: 0000000076230000 0000000000115000 C:\WINDOWS\System32\RPCRT4.dll (6.2.22621.3672) (-exported- Symbols Loaded)
Linked PDB Filename : rpcrt4.pdb
File Version : 10.0.22621.2506 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.2506
ModLoad: 0000000076040000 0000000000031000 C:\WINDOWS\System32\IMM32.DLL (6.2.22621.3374) (-exported- Symbols Loaded)
Linked PDB Filename : imm32.pdb
File Version : 10.0.22621.3374 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3374
ModLoad: 0000000074620000 0000000000018000 C:\WINDOWS\SYSTEM32\kernel.appcore.dll (6.2.22621.2715) (-exported- Symbols Loaded)
Linked PDB Filename : Kernel.Appcore.pdb
File Version : 10.0.22621.2715 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.2715
ModLoad: 0000000072d60000 0000000000232000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.22621.3593) (-exported- Symbols Loaded)
Linked PDB Filename : dbghelp.pdb
File Version : 10.0.22621.3593 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3593
ModLoad: 0000000077a20000 0000000000388000 C:\WINDOWS\System32\combase.dll (6.2.22621.3672) (-exported- Symbols Loaded)
Linked PDB Filename : combase.pdb
File Version : 10.0.22621.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.1
ModLoad: 0000000077db0000 00000000000d7000 C:\WINDOWS\System32\OLEAUT32.dll (6.2.22621.3672) (-exported- Symbols Loaded)
Linked PDB Filename : oleaut32.pdb
File Version : 10.0.22621.3672 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3672
ModLoad: 000000006d820000 000000000000a000 C:\WINDOWS\SYSTEM32\version.dll (6.2.22621.1) (-exported- Symbols Loaded)
Linked PDB Filename : version.pdb
File Version : 10.0.22621.1 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.1
ModLoad: 0000000075a80000 000000000007b000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.22621.3672) (-exported- Symbols Loaded)
Linked PDB Filename : bcryptprimitives.pdb
File Version : 10.0.22621.3672 (WinBuild.160101.0800)
Company Name : Microsoft Corporation
Product Name : Microsoft® Windows® Operating System
Product Version : 10.0.22621.3672
*** Dump of the Process Statistics: ***
- I/O Operations Counters -
Read: 6, Write: 185, Other 75
- I/O Transfers Counters -
Read: 24152, Write: 194, Other 8670
- Paged Pool Usage -
QuotaPagedPoolUsage: 93168, QuotaPeakPagedPoolUsage: 93440
QuotaNonPagedPoolUsage: 7200, QuotaPeakNonPagedPoolUsage: 14088
- Virtual Memory Usage -
VirtualSize: 2408448, PeakVirtualSize: 85626880
- Pagefile Usage -
PagefileUsage: 2408448, PeakPagefileUsage: 2408448
- Working Set Size -
WorkingSetSize: 6508544, PeakWorkingSetSize: 6508544, PageFaultCount: 12803
*** Dump of thread ID 13684 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 8281250.000000, User Time: 5625000.000000, Wait Time: 19674458.000000
- Unhandled Exception Record -
Reason: Breakpoint Encountered (0x80000003) at address 0x00007FFD757E8072
- Registers -
rax=0000000000000000 rbx=0000000000000001 rcx=0000000040099c58 rdx=000000000250eb80 rsi=0000000000000000 rdi=0000000000000000
r8=000000000250eb80 r9=0000000040099c48 r10=0000000000000fff r11=0000000000000ff0 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000 rip=00000000757e8072 rsp=000000000250eb58 rbp=0000000000000000
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
0250eb50 40041010 00000001 0250eb80 0250eb80 40099c48 KERNELBASE!DebugBreak+0x0
0250ef90 40041b74 da200000 0250f2f3 00000000 00000170 w_0.02_windows_x86_64!+0x0
0250fef0 40041f14 00000000 00000000 fff0bdc0 00000000 w_0.02_windows_x86_64!+0x0
0250ff20 7636257d 00000000 00000000 00000000 00000000 w_0.02_windows_x86_64!+0x0
0250ff50 7826af28 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
0250ffd0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Dump of thread ID 11064 (state: Waiting): ***
- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 19104584.000000
- Registers -
rax=0000000000000004 rbx=0000000000000000 rcx=000000000000014c rdx=0000000000000000 rsi=0000000000000000 rdi=000000000000014c
r8=0000000000000004 r9=00000000001392a8 r10=0000000000000080 r11=00000000763d2210 r12=0000000000000000 r13=0000000000000000
r14=000000000000014c r15=0000000000000000 rip=00000000782b0024 rsp=000000000013b518 rbp=000000000013b679
cs=0033 ss=002b ds=0000 es=0000 fs=0000 gs=0000 efl=00000246
- Callstack -
ChildEBP RetAddr Args to Child
0013b510 7570952e 00000000 756f9100 00000000 00000000 ntdll!NtWaitForSingleObject+0x0
0013b5b0 4006a010 0000014c ffffffff 00000000 0000014c KERNELBASE!WaitForSingleObjectEx+0x0
0013b6d0 40069ddd 00000000 00000000 ffffffff 00000000 w_0.02_windows_x86_64!+0x0
0013b730 4004be8b 00000000 00000000 0013b7a0 0052ad90 w_0.02_windows_x86_64!+0x0
0013b7a0 40003e04 0053c960 00000031 0052ade0 0013b8b0 w_0.02_windows_x86_64!+0x0
0014fee0 400447b0 0052af20 00000000 00537ad0 00000000 w_0.02_windows_x86_64!+0x0
0014ff20 7636257d 00000000 00000000 00000000 00000000 w_0.02_windows_x86_64!+0x0
0014ff50 7826af28 00000000 00000000 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0
0014ffd0 00000000 00000000 00000000 00000000 00000000 ntdll!RtlUserThreadStart+0x0
*** Debug Message Dump ****
*** Foreground Window Data ***
Window Name :
Window Class :
Window Process ID: 0
Window Thread ID : 0
Exiting...
Traceback (most recent call last):
File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\predict.py", line 708, in <module>
pred.predict(out_name+f'_{n}',
File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\predict.py", line 551, in predict
logit_s, logit_aa_s, logit_pae, logit_pde, p_bind, pred_crds, alpha, pred_allatom, pred_lddt_binned, msa_prev, pair_prev, state_prev = self.model(
File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\RoseTTAFoldModel.py", line 346, in forward
msa_recycle, pair_recycle, state_recycle = self.recycle(msa_prev, pair_prev, xyz, state_prev, sctors, mask_recycle)
File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\ev0\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\Embeddings.py", line 357, in forward
dist = rbf(torch.cdist(Ca_or_P, Ca_or_P))
File "D:\ProgramData\BOINC\projects\ralph.bakerlab.org\cv2\rf2aa\util_module.py", line 88, in rbf
D_mu = torch.linspace(D_min, D_max, D_count).to(D.device)
RuntimeError: CUDA error: the launch timed out and was terminated
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
</stderr_txt>
]]>
©2024 University of Washington
http://www.bakerlab.org