RoseTTAFold All-Atom
Message boards : RALPH@home bug list : RoseTTAFold All-Atom
Previous · 1 . . . 7 · 8 · 9 · 10 · 11 · Next
| Author | Message |
|---|---|
|
Henk Haneveld Send message Joined: 13 Apr 21 Posts: 8 Credit: 88 RAC: 0 |
It looks like they changed the definition for BOINC from a CPU app to a GPU app. But because I have a older Nvida card with limited memory a get: 18/06/2024 14:25:32 | ralph@home | Requesting new tasks for NVIDIA GPU 18/06/2024 14:25:33 | ralph@home | Scheduler request completed: got 0 new tasks 18/06/2024 14:25:33 | ralph@home | A minimum of 6144 MB (preferably 6144 MB) of video RAM is needed to process tasks using your computer's NVIDIA GPU 18/06/2024 14:25:33 | ralph@home | Project requested delay of 31 seconds |
 [VENETO] bobovizSend message Joined: 9 Apr 08 Posts: 913 Credit: 1,892,541 RAC: 294 |
18/06/2024 14:25:32 | ralph@home | Requesting new tasks for NVIDIA GPU Not bad, not bad... :-( |
|
Henk Haneveld Send message Joined: 13 Apr 21 Posts: 8 Credit: 88 RAC: 0 |
18/06/2024 14:25:32 | ralph@home | Requesting new tasks for NVIDIA GPU It is bad because the next call was for CPU work and I got this. 18/06/2024 14:44:09 | ralph@home | Requesting new tasks for CPU 18/06/2024 14:44:10 | ralph@home | Scheduler request completed: got 0 new tasks 18/06/2024 14:44:10 | ralph@home | A minimum of 6144 MB (preferably 6144 MB) of video RAM is needed to process tasks using your computer's NVIDIA GPU 18/06/2024 14:44:10 | ralph@home | No tasks sent 18/06/2024 14:44:10 | ralph@home | Project requested delay of 31 seconds A clear sign that mixing CPU and GPU setting won't work. |
|
Grant (SSSF) Send message Joined: 13 Jun 24 Posts: 126 Credit: 193,939 RAC: 2,635 
|
It looks like they changed the definition for BOINC from a CPU app to a GPU app.I have that much VRAM, and it worked with the previous batch of work, but i am unable to get any work due to the same message on my system. For either CPU or GPU. Grant Darwin NT |
kotenok2000 Send message Joined: 26 Feb 21 Posts: 22 Credit: 1,893 RAC: 0 |
You probably have old boinc client that doesn't report more than 4 gb of vram. Install it using this. https://boinc.berkeley.edu/linux_install.php |
|
Grant (SSSF) Send message Joined: 13 Jun 24 Posts: 126 Credit: 193,939 RAC: 2,635
|
You probably have old boinc client that doesn't report more than 4 gb of vram.I'm running Windows, and i've got the current Manager v8.0.2 Grant Darwin NT |
|
Grant (SSSF) Send message Joined: 13 Jun 24 Posts: 126 Credit: 193,939 RAC: 2,635
|
As it worked previously (i'd hope the new application doesn't require that much more VRAM than before), i'd hope they'd lower the requirement significantly. Ideally to less than 4GB, so 4GB and more would be OKIt looks like they changed the definition for BOINC from a CPU app to a GPU app.I have that much VRAM, and it worked with the previous batch of work, but i am unable to get any work due to the same message on my system. Looking at many of the Tasks that were processed by the new v0.02 (nvidia_alpha) application, the Peak working set size used is often well under 3GB. Even Just 1MB (2MB to be sure) less and i'd be able to run a task. Reported specs by BOINC, and what the minimum is set to- GeForce RTX 2060 (6143MB)
minimum of 6144MB Just one lousy MB...I would also hope that they make it so that if you have a video card & the correct driver, but not enough video RAM, you can still download Tasks to run on the CPU (this is one of the problems with having a single application that can run on both the CPU & the GPU- the project has to come up with some way of allowing it to still be able run on one compute resource when the other isn't suitable. Easily done with 2 different applications (even if they are the same application with different names they still count as different applications). Not so easily done with a single one). Grant Darwin NT |
|
Grant (SSSF) Send message Joined: 13 Jun 24 Posts: 126 Credit: 193,939 RAC: 2,635
|
Looking at some of the Tasks that were able to run on a GPU on the new Application, there appears to be an issue with the expected runtime. eg RF_SAVE_ALL_OUT_NOJRAN_IGNORE_THE_REST_validation_env_i_pred_74_16905_2_0 <core_client_version>8.0.0</core_client_version> <
|
GPU tasks are still failing immediately on my RTX 3060Ti. It's not a VRAM problem as the 3060Ti has 8GB. It looks like it's actually a programming problem and there needs to be quotation marks around the file path, or whatever method is appropriate in Python to allow spaces in file paths, since is it failing when it tries to access the BOINC directory inside Program Files. <core_client_version>8.0.2</core_client_version> <![CDATA[ <message> The access code is invalid. (0xc) - exit code 12 (0xc)</message> <stderr_txt> 'C:Program' is not recognized as an internal or external command, operable program or batch file. </stderr_txt> ]]> |
|
Grant (SSSF) Send message Joined: 13 Jun 24 Posts: 126 Credit: 193,939 RAC: 2,635
|
GPU tasks are still failing immediately on my RTX 3060Ti. It's not a VRAM problem as the 3060Ti has 8GB.Don't know what's going on there as i had that error issue with one of my systems early on, then the next time it tried for work it was able to get it and process it OK (that was with 6GB of VRAM). When i could no longer get work on that system, i attached to Ralph with my other system, and it downloaded the files & started processing with no problems (other than some initial weirdness with estimated time & progress indication). As it's unable to do any Ralph work at present, I'd suggest doing a Reset Project on that system- let it download everything from scratch & see if the error still occurs. Grant Darwin NT |
|
Grant (SSSF) Send message Joined: 13 Jun 24 Posts: 126 Credit: 193,939 RAC: 2,635
|
GPU tasks are still failing immediately on my RTX 3060Ti. It's not a VRAM problem as the 3060Ti has 8GB.Looks like you were able to get this system to process some work in the end- 54 Valid Tasks. Grant Darwin NT |
|
Grant (SSSF) Send message Joined: 13 Jun 24 Posts: 126 Credit: 193,939 RAC: 2,635
|
Just noticed this on the Computing, Applications page 0.03 (nvidia_alpha) 21 Jun 2024, 8:41:54 UTCAnother new version, so with a bit of luck there might be some more work over the weekend. (It would be nice to know what to expect with the new version...). Grant Darwin NT |
|
Grant (SSSF) Send message Joined: 13 Jun 24 Posts: 126 Credit: 193,939 RAC: 2,635
|
And new work is flowing. On my system that was able to process work the last time around, The Initial Remaining (estimated) time is very close to the past actual Runtimes, within about 30 sec. This new application isn't working the GPU as hard as the last one- GPU load was around 90% most of the time with the occasional large dips to 50%. This one, is around 85% and it's large dips are as low as 25%. And during those dips CPU usage goes from a bit over 1 thread to a bit under 2 threads for a few seconds (or less). GPU power draw is much choppier, showing the lower sustained load on the GPU. Memory controller load is very close to the previous version levels. My other system that couldn't get work due to only 6GB of VRAM, has now downloaded the new application and Tasks, and is crunching. And all Tasks are now erroring out on 6GB VRAM system. First Task ran for 1:41secs (suspect mostly CPU) then errored out, then the rest for 3 sec, then errored out. Reset Project, but still broken. Doesn't appear to be memory related as such, but a configuration issue. RF_SAVE_ALL_OUT_NOJRAN_IGNORE_THE_REST_validation_env_j_pred_46_16906_2_0 <core_client_version>8.0.2</core_client_version>
<![CDATA[
<message>
The access code is invalid.
(0xc) - exit code 12 (0xc)</message>
<stderr_txt>
C:ProgramDataBOINCprojectsralph.bakerlab.orgcv2rf2aautil.py:450: UserWarning: Using torch.cross without specifying the dim arg is deprecated.
Please either pass the dim explicitly or simply use torch.linalg.cross.
The default value of dim will change to agree with that of linalg.cross in a future release. (Triggered internally at C:cbpytorch_1000000000000workatensrcATennativeCross.cpp:66.)
Z = torch.cross(Xn,Yn)
usage: predict.py [-h] [-checkpoint CHECKPOINT] [-pdb PDB] [-silent SILENT] [-tmpl_chain TMPL_CHAIN]
[-tmpl_conf TMPL_CONF] [-movescale MOVESCALE] [-n_pred N_PRED] [-n_cycle N_CYCLE] [-no_extra_l1]
[-no_atom_frames] [-outcsv OUTCSV]
predict.py: error: unrecognized arguments: -z .venv_j_pred_alpha_46.zip
</stderr_txt>
]]>Interestingly the Tasks of this batch that are running OK & Validating on my other system with a RTX 2060 Super (8GB VRAM) give a similar error message, but don't error out. RF_SAVE_ALL_OUT_NOJRAN_IGNORE_THE_REST_validation_env_j_pred_8_16906_5_0 <core_client_version>8.0.2</core_client_version> <![CDATA[ <stderr_txt> C:ProgramDataBOINCprojectsralph.bakerlab.orgcv1rf2aautil.py:450: UserWarning: Using torch.cross without specifying the dim arg is deprecated. Please either pass the dim explicitly or simply use torch.linalg.cross. The default value of dim will change to agree with that of linalg.cross in a future release. (Triggered internally at C:cbpytorch_1000000000000workatensrcATennativeCross.cpp:66.) Z = torch.cross(Xn,Yn) 20:49:39 (15324): called boinc_finish(0) </stderr_txt> ]]> My other system that is completing Tasks without issue, the most VRAM i've seen in use has been 4790MB. And system RAM is less than 1GB (much, much less than before). Grant Darwin NT |
|
Grant (SSSF) Send message Joined: 13 Jun 24 Posts: 126 Credit: 193,939 RAC: 2,635
|
My other system that couldn't get work due to only 6GB of VRAM, has now downloaded the new application and Tasks, and is crunching.OK, this is odd. After resetting the project, my RTX 2060 (6GB VRAM) is now actually processing work, but everything else errored out on the GTX 1070 (8GB VRAM). (before they errored out on both cards). Damn. And i can't get any more work to see what's going on till tomorrow. (i've hit my daily quota with all the other errors). The Task it was processing finished & Validated, with the same stderr_txt as those on the other system. Monitoring VRAM on the RTX 2060 (6GB VRAM), and it's got up to 6121MB Maximum in use (so far). The other video card has more RAM, but is using less of it... (5314MB max so far) Note to developers- it would be worth including in the stderr_txt the video card details of what card completed the Task eg Device 0 RTX 2060 6GB, Device 1 GTX 1070 8GB, so that for systems with multiple GPUs, if something goes wrong, we know what it went wrong on. Also for when there is a CPU application again- include CPU in the stderr_txt to help avoid confusion with GPU processed Tasks. Grant Darwin NT |
|
[VENETO] boboviz Send message Joined: 9 Apr 08 Posts: 913 Credit: 1,892,541 RAC: 294 |
It mentions that there are CPU tasks available, until it gets the update to not request them, then it doesn't mention the availability of CPU work after that. Ok. I disabled the gpu option in my profile ('cause i have not Nvidia) And the result is: Tasks for NVIDIA GPU are available, but your preferences are set to not accept them Waiting for cpu wus, if there ever will be |
zombie67 [MM] Send message Joined: 8 Aug 06 Posts: 75 Credit: 2,396,363 RAC: 6,299
|
|
|
Grant (SSSF) Send message Joined: 13 Jun 24 Posts: 126 Credit: 193,939 RAC: 2,635
|
Finally able to get more work on system that errored Tasks. However, while the BOINC Manager indicates they are running on different GPUs, they are in fact running on the same GPU (which explains why so many errored out even though many showed as being on the GTX 1070 8GB VRAM GPU). It doesn't explain why that one Task was able to run OK after all the others did error out on the GPU that is now running Tasks OK.... And the Remaining (estimated) time is 3hr 20min, not the actual 26min or so processing time. Re-instated application limit of 1 Task. Exited BOINC and restarted, Task restarted from beginning (still no checkpointing), Looking at progress indicator- appears to be based on time done and initial Remaining (estimated) time, not actual processing done. This time around Monitoring VRAM on the RTX 2060 (6GB VRAM), and it's got up to 6097MB Maximum in use (so far). Grant Darwin NT |
Message boards :
RALPH@home bug list :
RoseTTAFold All-Atom
©2024 University of Washington
http://www.bakerlab.org