Both copies of this test_cc 1.50 workunit failed quickly:

https://ralph.bakerlab.org/workunit.php?wuid=1105459

The mammoths have not been having a happy time. I have had a run of 23 failures (1st time) including:


unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0081E942 read attempt to address 0x00000000

and

ERROR: unknown model name: 1B0NA_10
ERROR:: Exit from: d:boinc_buildminirosetta_windowsminisrcprotocols/abinitio/PairingStatistics.hh line: 170
called boinc_finish

> Step 900,000!

Step 900000 ? WHat does that mean ? "Step" ??

That was the step shown in the graphic. (to the right of the model number). I've just never seen such a large step number.

---

One more with v1.50: test_cc_1_8_nocst4_hb_t367__IGNORE_THE_REST_1UFBA_5_6830_3

<core_client_version>6.2.18</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)
</message>
<stderr_txt>
Watchdog active.
# cpu_run_time_pref: 14400

ERROR: target_strands.size()
ERROR:: Exit from: src/protocols/abinitio/TemplateJumpSetup.cc line: 94
called boinc_finish



And one with v.1.51: test_cc2_1_8_mammoth_mix_cen_cst_hb_t311__IGNORE_THE_REST_1PERL_6_6852_1

<core_client_version>6.2.18</core_client_version>
<![CDATA[
<message>
process exited with code 1 (0x1, -255)
</message>
<stderr_txt>
BOINC:: Initializing ... ok.
[2009- 1-22 11:56:36:] :: BOINC :: boinc_init()
BOINC:: Setting up shared resources ... ok.
BOINC:: Setting up semaphores ... ok.
BOINC:: Updating status ... ok.
BOINC:: Registering timer callback... ok.
BOINC:: Worker initialized successfully.
Initializing options.... ok
Initializing random generators... ok
Initialization complete.
Watchdog active.

ERROR: unknown model name: 1B0NA_10
ERROR:: Exit from: src/protocols/abinitio/PairingStatistics.hh line: 170
called boinc_finish

Snags

I tried loading up the docking task I got. It is 1.51. Displayed graphic just as the task was starting. Waited, waited... finally realized it was using more CPU then the thread working on the protein! Double checked Ralph settings for % of CPU for the graphic, set to default which is 10%.

Here's my screenshot showing the graphic monopolizing one core, while the two running tasks are competing for the other. Net result, nothing shown in the graphic after several minutes, and graphic thread consuming much more then 10% of CPU.



[edit]
I gave up on it, captured the screen, uploaded the screenshot, reported it here... then when I opened the graphic a second time it was better behaved. Not overusing CPU... but was essentially unusable. Go to resize or rotate the images and it wouldn't respond for about 30 seconds each time.

---

[quote]I tried loading up the docking task I got. It is 1.51. Displayed graphic just as the task was starting. Waited, waited... finally realized it was using more CPU then the thread working on the protein! Double checked Ralph settings for % of CPU for the graphic, set to default which is 10%.

Here's my screenshot showing the graphic monopolizing one core, while the two running tasks are competing for the other. Net result, nothing shown in the graphic after several minutes, and graphic thread consuming much more then 10% of CPU.

[quote]

Feet1st , this is awesome - debugging on an unprecedented level :) Nice to get an idea of what all this looks like from your point of view.

These docking tasks are new and not mine - lemme track down the person submiting these and make sure the graphics app can deal with it.

Mike

application version 1.51

https://ralph.bakerlab.org/result.php?resultid=1258676

<core_client_version>6.4.5</core_client_version>
<![CDATA[
<message>
Incorrect function. (0x1) - exit code 1 (0x1)
</message>
<stderr_txt>
BOINC:: Initializing ... ok.
[2009- 1-22 19: 2:51:] :: BOINC :: boinc_init()
BOINC:: Setting up shared resources ... ok.
BOINC:: Setting up semaphores ... ok.
BOINC:: Updating status ... ok.
BOINC:: Registering timer callback... ok.
BOINC:: Worker initialized successfully.
Initializing options.... ok
Initializing random generators... ok
Initialization complete.
Watchdog active.

ERROR: unknown model name: 1DK8A_1
ERROR:: Exit from: d:boinc_buildminirosetta_windowsminisrcprotocols/abinitio/PairingStatistics.hh line: 170
called boinc_finish

</stderr_txt>
]]>

---

Otherwise this release has added debug information to let me figure out where all this stuff is failing. Believe me guys, we're now in the land whre i cannot reproduce these errors here what so ever. Not on the linux boxes, Mac boxes or windows boxes we have. nowhere. Why these remaining segfaults occur is a total mystery to me, so please bear with me. THis is going ot be incredibly difficult to track down.

Thanks for all your help! Every post is super useful to us!!

Questions I ask myself,

Are they happening in the same module?
Are they happening because of the same type of activity?
What is common to all the events?
Could it be something external?

I know BOINC is supposed to be keeping things isolated and that application A is not supposed to affect application B ... but I have seen enough odd things that I am not convinced that this is completely so ...

As to the tasks that fail ... you say they don't fail for you ...

Why not grab the set and issue them to us with high replication counts ... then if they fail for everyone, that tells us one thing ... if they only fail for some and not others ... that tells us something else ...

To my mind, the cases that fail are the ones that you should be saving and using as your issue tasks for each round of testing. Not to keep making new tasks in all cases, but to use those tasks that have proven their ability to cause a problem.

Just some things to consider ... oh, and you are out of work again ...

---

DEP - execution protection on new Intel and AMD cores.

How can I turn graphics off?

---

v1.51

ERROR: unknown model name: 2FRHA_10
ERROR:: Exit from: d:boinc_buildminirosetta_windowsminisrcprotocols/abinitio/PairingStatistics.hh line: 170
called boinc_finish

on task
test_cc2_1_8_mammoth_mix_cen_cst_hb_t327__IGNORE_THE_REST_2F2EA_7_6860_1_1

---

Feet1st , this is awesome - debugging on an unprecedented level :) Nice to get an idea of what all this looks like from your point of view.

These docking tasks are new and not mine - lemme track down the person submiting these and make sure the graphics app can deal with it.

Mike

I've seen a number of other reports of the screen saver/graphic just displaying as a black window. And I had always assumed people just weren't waiting long enough for the display to refresh. It does often take a long time, and I always assumed this was due to the allowed % of CPU time for the graphic. But, on the other hand, this wasn't an issue before about 2 or 3 months ago.

I meant to point out that the screenshot shows the graphic thread has used 4:02 of CPU time, but the corresponding Ralph thread is shown with only 2:11 of CPU received so far. So the % CPU shown in the screenshot is only the last interval, but you can see from the totals that the number is roughly what it's been during the entire 4 minutes the task has been running.

I was wondering if perhaps the graphic could always display at least it's grid lines, and text immediately, and perhaps a "protein information being retrieved... please wait" message in the frames. That way at least it would never just be "blank".

---

Feet1st , this is awesome - debugging on an unprecedented level :) Nice to get an idea of what all this looks like from your point of view.

These docking tasks are new and not mine - lemme track down the person submiting these and make sure the graphics app can deal with it.

Mike

I've seen a number of other reports of the screen saver/graphic just displaying as a black window. And I had always assumed people just weren't waiting long enough for the display to refresh. It does often take a long time, and I always assumed this was due to the allowed % of CPU time for the graphic. But, on the other hand, this wasn't an issue before about 2 or 3 months ago.

I meant to point out that the screenshot shows the graphic thread has used 4:02 of CPU time, but the corresponding Ralph thread is shown with only 2:11 of CPU received so far. So the % CPU shown in the screenshot is only the last interval, but you can see from the totals that the number is roughly what it's been during the entire 4 minutes the task has been running.

I was wondering if perhaps the graphic could always display at least it's grid lines, and text immediately, and perhaps a "protein information being retrieved... please wait" message in the frames. That way at least it would never just be "blank".

Cool idea...I would love to see that as well.

test_cc_1_8_nocst4_hb_t327__IGNORE_THE_REST_2FSWA_6_6888_1
andtest_cc_1_8_nocst4_hb_t327__IGNORE_THE_REST_2F2EA_10_6888_1_1

both ended with:

ERROR: unknown model name: 2FRHA_10
ERROR:: Exit from: src/protocols/abinitio/PairingStatistics.hh line: 170

This time they ran a few minutes instead of a few seconds, claimed to be done instead of declaring a compute error and received a validate error instead of a client error.


Snags

Why not grab the set and issue them to us with high replication counts ... then if they fail for everyone, that tells us one thing ... if they only fail for some and not others ... that tells us something else ...

we already submit identical jobs to many many machines - and they only fail *sometimes*. Even when the same random number seed is used.

To my mind, the cases that fail are the ones that you should be saving and using as your issue tasks for each round of testing. Not to keep making new tasks in all cases, but to use those tasks that have proven their ability to cause a problem.

again .. we're already doing that. But the errors are sporadic in nature so submitting the same task might not throw an error "this time".

I have noticed though that some of the problems seem to be highly machine specific. Like one user will always produce this one kind of an error and nithing else. weired. something strange about their setup ? No idea.

Thank you all for reporting the problems! The problem divides it self into three parts, firstly cornering the error, turning it into an erro message, reproducing it here and then finding the root of the problem.

At any stage there are three types of problem:

a) Ones that fail with an error message like this:

ERROR: unknown model name: 1DK8A_1
ERROR:: Exit from: d:boinc_buildminirosetta_windowsminisrcprotocols/abinitio/PairingStatistics.hh line: 170
called boinc_finish

These ones we've basically identified. Why they occur is merely a matter of tracking down the route. These ones i have already passed on to the relevant programmers who are fixing htem as we speak.

b) Segfaults with traces:

- Callstack -
ChildEBP RetAddr Args to Child
0012d270 00426312 08000000 077bf590 00000010 08000000 minirosetta_1.50_windows_intelx!UnwindUpVec+0x0 (F:SPvctoolscrt_bldSELF_X86crtsrcIntelMEMCPY.ASM:312)
0012d28c 0046d11a 08000000 00000010 077bf590 00000010 minirosetta_1.50_windows_intelx!memmove_s+0xc (f:spvctoolscrt_bldself_x86crtsrcmemmove_s.c:58)
0012d2a8 0046f352 077bf590 077bf5a0 08000000 021a3d3c minirosetta_1.50_windows_intelx!stdext::unchecked_copy+0x2a (c:program files (x86)microsoft visual studio 8vcincludexutility:3409)
0012d2c8 005b1d2e 021a3d40 3d0497dd 0012d500 02006520 minirosetta_1.50_windows_intelx!std::vector >::operator=+0x0 (c:program files (x86)microsoft visual studio 8vcincludevector:565)
0012d3cc 005b1ed1 3d0496dd 02029f88 00000000 02006520 minirosetta_1.50_windows_intelx!core::scoring::Ramachandran::init_rama_sampling_table+0x2e (d:boinc_buildminirosetta_windowsminisrccorescoringramachandran.cc:200)
0012d3ec 005398fb 3d0496fd 02090718 02006520 0012d420 minirosetta_1.50_windows_intelx!core::scoring::Ramachandran::Ramachandran+0x0 (d:boinc_buildminirosetta_windowsminisrccorescoringramachandran.cc:72)
0012d408 005eccf9 3d049111 02029f88 3fc99999 02090718 minirosetta_1.50_windows_intelx!core::scoring::ScoringManager::get_Ramachandran+0x21 (d:boinc_buildminirosetta_windowsminisrccorescoringscoringmanager.cc:339)
0012d428 0053b7e1 3d049131 02012f48 00000001 7c00003d minirosetta_1.50_windows_intelx!core::scoring::methods::RamachandranEnergy::RamachandranEnergy+0x59 (d:boinc_buildminirosetta_windowsminisrccorescoringmethodsramachandranenergy.cc:42)
0012d4b4 004d8a28 0012d4e4 0012d500 02090718 3d0491c9 minirosetta_1.50_windows_intelx!core::scoring::ScoringManager::energy_method+0x2b (d:boinc_buildminirosetta_windowsminisrccorescoringscoringmanager.cc:566)
0012d4dc 004d9135 0012d500 0012d544 3d0491f5 0012da20 minirosetta_1.50_windows_intelx!core::scoring::ScoreFunction::set_weight+0x0 (d:boinc_buildminirosetta_windowsminisrccorescoringscorefunction.cc:1377)
0012d708 004ceacb 0012d738 3d04921d 020c68e0 01321ef0 minirosetta_1.50_windows_intelx!core::scoring::ScoreFunction::apply_patch_from_file+0x0 (d:boinc_buildminirosetta_windowsminisrccorescoringscorefunction.cc:235)
0012d9e0 004cf1b1 0012da20 0000001f 6e617473 64726164 minirosetta_1.50_windows_intelx!core::scoring::ScoreFunctionFactory::create_score_function+0x4e (d:boinc_buildminirosetta_windowsminisrccorescoringscorefunctionfactory.cc:72)
0012da90 005755b2 0012dac4 3d049fa5 00000000 0131af60 minirosetta_1.50_windows_intelx!core::scoring::getScoreFunction+0x31 (d:boinc_buildminirosetta_windowsminisrccorescoringscorefunctionfactory.cc:168)
0012daec 0051b6a9 0131af60 0012dc20 3d049e05 01304498 minirosetta_1.50_windows_intelx!core::pack::pack_missing_sidechains+0xa (d:boinc_buildminirosetta_windowsminisrccorepackpack_missing_sidechains.cc:83)
0012ddf8 0051bc8a 0131af60 01304498 3d049b09 00000010 minirosetta_1.50_windows_intelx!core::io::pdb::FileData::build_pose_as_is+0xe (d:boinc_buildminirosetta_windowsminisrccoreiopdbfile_data.cc:749)
0012de74 004dd0f2 0131af60 01304498 3d049b8d 0012ee68 minirosetta_1.50_windows_intelx!core::io::pdb::FileData::build_pose+0x0 (d:boinc_buildminirosetta_windowsminisrccoreiopdbfile_data.cc:116)
0012e01c 004dd1ae 0131af60 01304498 0012e070 00000000 minirosetta_1.50_windows_intelx!core::io::pdb::pose_from_pdb+0x0 (d:boinc_buildminirosetta_windowsminisrccoreiopdbpose_io.cc:194)
0012e034 00798370 0131af60 0012e070 00000000 3d04a549 minirosetta_1.50_windows_intelx!core::io::pdb::pose_from_pdb+0x0 (d:boinc_buildminirosetta_windowsminisrccoreiopdbpose_io.cc:210)
0012e0f8 007a7282 3d04a401 0000000f 00000007 00000000 minirosetta_1.50_windows_intelx!protocols::abinitio::AbrelaxApplication::setup+0x5d (d:boinc_buildminirosetta_windowsminisrcprotocolsabinitioabrelaxapplication.cc:496)
0012edb4 0041a0d8 3d04a8c5 00000a28 00000002 0012ffc0 minirosetta_1.50_windows_intelx!protocols::abinitio::AbrelaxApplication::run+0x0 (d:boinc_buildminirosetta_windowsminisrcprotocolsabinitioabrelaxapplication.cc:2176)
0012ef14 0041a465 0000001f 0012ef2c 00152310 0012ef2c minirosetta_1.50_windows_intelx!main+0x0 (d:boinc_buildminirosetta_windowsminisrcappspublicboincminirosetta.cc:117)
0012ff28 0042d207 00400000 00000000 00152352 0000000a minirosetta_1.50_windows_intelx!WinMain+0x0 (d:boinc_buildminirosetta_windowsminisrcappspublicboincminirosetta.cc:159)
0012ffc0 7c817067 0032005c 0064005c 7ffd8000 c0000005 minirosetta_1.50_windows_intelx!__tmainCRTStartup+0x1c (f:spvctoolscrt_bldself_x86crtsrccrt0.c:324)
0012fff0 00000000 0042d270 00000000 78746341 00000020 kernel32!_BaseProcessStart@4+0x0 (f:spvctoolscrt_bldself_x86crtsrccrt0.c:324)

Segfault - bad. But at least i get a trace - i can thus look into the code and understand (or guess) at why the program could fail at that point. Then i can turn them into a) type errors.

6.2.19

There are no child processes to wait for. (0x80) - exit code 128 (0x80)


BOINC:: Initializing ... ok.
[2009- 1-22 4:45: 1:] :: BOINC :: boinc_init()
BOINC:: Setting up shared resources ... ok.
BOINC:: Setting up semaphores ... ok.
BOINC:: Updating status ... ok.
BOINC:: Registering timer callback... ok.
BOINC:: Worker initialized successfully.


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x7C81BD02 read attempt to address 0x00000002

Engaging BOINC Windows Runtime Debugger...



Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x7C81BD02 read attempt to address 0x00000002

Engaging BOINC Windows Runtime Debugger...



]]>

These are random segfault that occured somewhere. I can do virtually nothing from here. If i rerun the *exact same commandline* from here all runs fine. So these ones i can only tentatively bracket with stderr statemenets.




Right now i am mainly trying to turn b) and c) into a). Once an error is in the form of a) its usually trivial to solve!

Anyway,.. just incase you guys were interested in what i'm trying to do.


oh i guess there are also errors that are *behavioral* errors. Those are the ones i cannot see from here but you guys have to tell me about. Stuff like the graphics or overrunning models or other strange behaviour.

:-)

Hey, Mike, thanks for all the info you've been giving us lately. It's interesting and certainly provides a bit of extra motivation to keep crunching and reporting what we see at our end.

Snags

Hi,
I'm running RALPH with Ubuntu Jaunty up to date.
My BOINC version is 6.2.18.
When I click to "show graphic" (or suchs, I'm French), I see a small window, like the screenshot in XP in this Post, but all my memory (1.5 Gio) and all my swap (2.0 Gio) are full…
If I do not it (click on "sow graphics"), there isn't any visible problem.

Rosetta mini 1.52
http://francois.linkmauve.fr/boinc.png
The screenshot show when the window has been killed.
If there are questions, ask me…

Hi,
I'm running RALPH with Ubuntu Jaunty up to date.
My BOINC version is 6.2.18.
When I click to "show graphic" (or suchs, I'm French), I see a small window, like the screenshot in XP in this Post, but all my memory (1.5 Gio) and all my swap (2.0 Gio) are full…
If I do not it (click on "sow graphics"), there isn't any visible problem.

Rosetta mini 1.52
http://francois.linkmauve.fr/boinc.png
The screenshot show when the window has been killed.
If there are questions, ask me…

I'm showing the same or very similar behavior on the same type of workunit:lr6_D_score12_rlbn_1gu3_IGNORE_THE_REST_NATIVE_6909_1 running Leopard on an Intel Mac. No cpu use but tons of memory and I don't even get the box so nothing to show in a screen capture. Activity Manager says the process has hung and I killed it without trouble from there.

Snags

Mtyka,

can you disable graphics for some WUs and see if the error rate goes down? Might help turning c)'s into b)'s.

---

FYI, in French "Gio" is "GB" (gigabytes)

---