RALPH@home

minirosetta v1.48-1.51 bug thread

  UW Seal
 
[ Home ] [ Join ] [ About ] [ Participants ] [ Community ] [ Statistics ]
  [ login/out ]


Advanced search

Message boards : RALPH@home bug list : minirosetta v1.48-1.51 bug thread

AuthorMessage
mtyka
Forum moderator
Project developer
Project scientist

Joined: Mar 19 08
Posts: 79
ID: 4144
Credit: 0
RAC: 0
Message 4449 - Posted 19 Jan 2009 1:38:00 UTC

    Last modified: 22 Jan 2009 22:16:01 UTC

    This (long anticipated, yes i know ) new release has a whole lot of new features and bugfixes revolving mainly around making the app more stable, obey user runtimes more precisely as well as give back more information to us when things go wrong. We are (as always but with this one particularily) keen to hear how this app runs out there on your computers. Any feedback in invaluable to us to make progress in getting minirosetta as stable as rosetta++ and other projects. There is more new science on its way too but the priority right now is code stabilty.


    1.48 Release CHANGELOG

    Faster loop closing in FoldCST/Abinitio (affects cc_* cc2_* cs_* WUs), should help with overrunning WUs.

    Bug fix concerning intermittent crashes in _rlbd_ jobs.

    Bug fix for a potential instability in handling text files (affects all types of WUs).

    Bug fix in checkpointing machinery, states were not being correctly restored, probably contributing to long runtimes. (affects cc_* cc2_* cs_* WUs)

    Increased the density of checkpoints to lose less time on restarts and address the weired "backjumping" of the time reported in this thread.

    Added checkpointing to Loopclosing part of FoldCST. (affects cc_* cc2_* cs_* WUs)

    Added checkpointing to Looprelax.

    The Watchdog has been checked and improved, now returning information on the aborted jobs to help us figure out how the remaining long running models come about. The watchdog will now abort if the runtime exceeds your preferred runtime + 4 hours. In other words the WUs should not overrun for more than around 4 hours. If they do please let us know !!


    Thank you all for helping us fix all these problems. Especially all of you who temporarily switched over from Rosetta@HOME, we really appreciate your efford.

    What's next ?

    We will be submiting a whole variety of different WUs on RALPH, see if we have improved the stability or have inadertedly created new problems. Then we will either relewase another app here (1.49) to address outstanding issues or move this version (1.48) directly to BOINC. Fingers crossed.


    EDIT:

    1.49 CHANGELOG

    Something screwed up during the 1.48 release of the database. Supplying the database post-facto seemed to only help those that hadnt already grabbed the new app, so this release is merely a copy of the previous one with proper database to make sure all the clients are downloading the databse correctly.
    THe apps are identical to 1.48 though!



    EDIT:

    1.50 CHANGELOG

    Minor update - essentially identical to 1.49. Added another error-reporting mechanism and repaired the symbol store mechanism to help us figure out remaining problems.

    Pepo
    Avatar

    Joined: Sep 8 06
    Posts: 104
    ID: 1812
    Credit: 36,890
    RAC: 0
    Message 4450 - Posted 19 Jan 2009 8:17:50 UTC - in response to Message 4449.

      As a first preliminary report:

      This (long anticipated, yes i know ) new release ...
      ... is mistakenly dated to December 12, 2008 (probably a copy of the 1.47 release).

      Peter

      mtyka
      Forum moderator
      Project developer
      Project scientist

      Joined: Mar 19 08
      Posts: 79
      ID: 4144
      Credit: 0
      RAC: 0
      Message 4451 - Posted 19 Jan 2009 17:46:15 UTC - in response to Message 4450.

        whoops yes - thanks for noticing that ;)

        mtyka
        Forum moderator
        Project developer
        Project scientist

        Joined: Mar 19 08
        Posts: 79
        ID: 4144
        Credit: 0
        RAC: 0
        Message 4452 - Posted 19 Jan 2009 18:22:53 UTC - in response to Message 4451.

          Update: something went wrong with the database during the update - this has probably nothing todo with the new application itself but wit hthe fact that our update machine went down last week and so this update was done from a new machine that evidently failed to update the databse correctly. We're fixing that right now - the project will be down for a few hours before we get this sorted out. SOrry for the delay.

          Mike

          Profile sslickerson

          Joined: Feb 15 06
          Posts: 17
          ID: 37
          Credit: 4,006
          RAC: 0
          Message 4453 - Posted 19 Jan 2009 19:27:56 UTC - in response to Message 4452.

            Would this be why the last 26 WU or so have failed on my PC with the following message, or is this something different?

            <core_client_version>6.5.0</core_client_version>
            <![CDATA[
            <message>
            Incorrect function. (0x1) - exit code 1 (0x1)
            </message>
            <stderr_txt>

            ERROR: in::file::zip minirosetta_database.zip does not exist!
            ERROR:: Exit from: ..\..\src\apps\public\boinc\minirosetta.cc line: 83
            called boinc_finish

            </stderr_txt>
            ]]>

            mtyka
            Forum moderator
            Project developer
            Project scientist

            Joined: Mar 19 08
            Posts: 79
            ID: 4144
            Credit: 0
            RAC: 0
            Message 4454 - Posted 19 Jan 2009 20:22:07 UTC - in response to Message 4453.

              Yes! Ignore that database error message - for some reason the databse did not get uploaded to the server when i did the update on sunday. Something to do with the move to a new update machine i suspect..

              Profile Paul D. Buck

              Joined: Jan 14 09
              Posts: 62
              ID: 5139
              Credit: 33,293
              RAC: 0
              Message 4455 - Posted 20 Jan 2009 7:51:17 UTC

                I don't know if you like success stories, but I have run 4 tasks now I think on OS-X Intel and they all have completed successfully.
                ____________

                I _ quit

                Joined: Jan 13 09
                Posts: 44
                ID: 5136
                Credit: 88,562
                RAC: 0
                Message 4456 - Posted 20 Jan 2009 15:34:38 UTC

                  8 tasks on win xp home sp3 and no errors so far
                  had a few 1 hour runs before i updated the prefs. to 4hrs

                  mtyka
                  Forum moderator
                  Project developer
                  Project scientist

                  Joined: Mar 19 08
                  Posts: 79
                  ID: 4144
                  Credit: 0
                  RAC: 0
                  Message 4457 - Posted 20 Jan 2009 17:11:56 UTC

                    Awesome guys! Keep me posted on what you see out there. The error rate so far is looking fabulous.

                    I'll probably update the app once more today to fix an issue with the symbol store such that we get code traces in cases where it still fails.

                    Mike :)

                    Profile Paul D. Buck

                    Joined: Jan 14 09
                    Posts: 62
                    ID: 5139
                    Credit: 33,293
                    RAC: 0
                    Message 4458 - Posted 20 Jan 2009 19:55:04 UTC - in response to Message 4457.

                      Awesome guys! Keep me posted on what you see out there. The error rate so far is looking fabulous.

                      I'll probably update the app once more today to fix an issue with the symbol store such that we get code traces in cases where it still fails.

                      Mike :)


                      I only got one 1.5 task so that will be all I have to report ... so, my latest bug report is that version 1.5 is repelling the creation of new tasks ...
                      ____________

                      Profile Ian_D

                      Joined: Feb 16 06
                      Posts: 16
                      ID: 321
                      Credit: 39,518
                      RAC: 0
                      Message 4459 - Posted 20 Jan 2009 20:52:47 UTC

                        http://ralph.bakerlab.org/result.php?resultid=1250838

                        <core_client_version>6.4.5</core_client_version>
                        <![CDATA[
                        <message>
                        Incorrect function. (0x1) - exit code 1 (0x1)
                        </message>
                        <stderr_txt>
                        Watchdog active.

                        ERROR: target_strands.size()
                        ERROR:: Exit from: ..\..\src\protocols\abinitio\TemplateJumpSetup.cc line: 94
                        called boinc_finish

                        </stderr_txt>
                        ]]>

                        ____________

                        Profile feet1st

                        Joined: Mar 7 06
                        Posts: 312
                        ID: 1028
                        Credit: 110,522
                        RAC: 0
                        Message 4460 - Posted 20 Jan 2009 21:42:33 UTC

                          Last modified: 20 Jan 2009 21:44:44 UTC

                          More checkpointing is great! But... this is a bit extreme. My write to disk at MOST every... setting is at 1800 seconds. My harddrive will never be able to spin down and go in to power saver mode all night long if the checkpoints continue at this pace.


                          1/20/2009 3:31:58 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0 checkpointed
                          1/20/2009 3:32:28 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t342__IGNORE_THE_REST_2G0QA_13_6824_1_0 checkpointed
                          1/20/2009 3:32:39 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0 checkpointed
                          1/20/2009 3:33:14 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t342__IGNORE_THE_REST_2G0QA_13_6824_1_0 checkpointed
                          1/20/2009 3:33:22 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0 checkpointed
                          1/20/2009 3:34:02 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t342__IGNORE_THE_REST_2G0QA_13_6824_1_0 checkpointed
                          1/20/2009 3:34:04 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0 checkpointed
                          1/20/2009 3:34:42 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0 checkpointed
                          1/20/2009 3:34:44 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t342__IGNORE_THE_REST_2G0QA_13_6824_1_0 checkpointed
                          1/20/2009 3:35:22 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0 checkpointed
                          1/20/2009 3:35:28 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t342__IGNORE_THE_REST_2G0QA_13_6824_1_0 checkpointed
                          1/20/2009 3:36:04 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0 checkpointed
                          1/20/2009 3:36:17 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t342__IGNORE_THE_REST_2G0QA_13_6824_1_0 checkpointed
                          1/20/2009 3:36:44 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0 checkpointed
                          1/20/2009 3:36:58 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t342__IGNORE_THE_REST_2G0QA_13_6824_1_0 checkpointed
                          1/20/2009 3:37:24 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0 checkpointed
                          1/20/2009 3:37:39 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t342__IGNORE_THE_REST_2G0QA_13_6824_1_0 checkpointed
                          1/20/2009 3:38:04 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0 checkpointed
                          1/20/2009 3:38:21 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t342__IGNORE_THE_REST_2G0QA_13_6824_1_0 checkpointed
                          1/20/2009 3:38:43 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0 checkpointed
                          1/20/2009 3:39:02 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t342__IGNORE_THE_REST_2G0QA_13_6824_1_0 checkpointed
                          1/20/2009 3:39:22 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0 checkpointed
                          1/20/2009 3:39:44 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t342__IGNORE_THE_REST_2G0QA_13_6824_1_0 checkpointed

                          ____________

                          mtyka
                          Forum moderator
                          Project developer
                          Project scientist

                          Joined: Mar 19 08
                          Posts: 79
                          ID: 4144
                          Credit: 0
                          RAC: 0
                          Message 4461 - Posted 20 Jan 2009 22:00:26 UTC - in response to Message 4459.

                            http://ralph.bakerlab.org/result.php?resultid=1250838

                            <core_client_version>6.4.5</core_client_version>
                            <![CDATA[
                            <message>
                            Incorrect function. (0x1) - exit code 1 (0x1)
                            </message>
                            <stderr_txt>
                            Watchdog active.

                            ERROR: target_strands.size()
                            ERROR:: Exit from: ..\..\src\protocols\abinitio\TemplateJumpSetup.cc line: 94
                            called boinc_finish

                            </stderr_txt>
                            ]]>


                            Awesome !! Our new debug tools are working. This rare error (i've never seen it in 1000ds of runs) would have gone unnoticed before and led to a segfault. Now it gets caught at least and we can find its cause.

                            Thanks!

                            mtyka
                            Forum moderator
                            Project developer
                            Project scientist

                            Joined: Mar 19 08
                            Posts: 79
                            ID: 4144
                            Credit: 0
                            RAC: 0
                            Message 4462 - Posted 20 Jan 2009 22:01:10 UTC - in response to Message 4460.

                              More checkpointing is great! But... this is a bit extreme. My write to disk at MOST every... setting is at 1800 seconds. My harddrive will never be able to spin down and go in to power saver mode all night long if the checkpoints continue at this pace.


                              1/20/2009 3:31:58 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0 checkpointed
                              1/20/2009 3:32:28 PM|ralph@home|[checkpoint_debug] result


                              Hmm ok, i'll look into this.

                              Profile Paul D. Buck

                              Joined: Jan 14 09
                              Posts: 62
                              ID: 5139
                              Credit: 33,293
                              RAC: 0
                              Message 4463 - Posted 20 Jan 2009 23:11:44 UTC

                                Need more work ...

                                I have run with 1.50 and both were success ... of course the Mac Application has been stable for me, even the awful 1.47 which really farbled up my XP machines ... well, I be doing my part ... :)
                                ____________

                                mtyka
                                Forum moderator
                                Project developer
                                Project scientist

                                Joined: Mar 19 08
                                Posts: 79
                                ID: 4144
                                Credit: 0
                                RAC: 0
                                Message 4464 - Posted 21 Jan 2009 0:56:23 UTC - in response to Message 4463.

                                  as you wish ...

                                  Profile Paul D. Buck

                                  Joined: Jan 14 09
                                  Posts: 62
                                  ID: 5139
                                  Credit: 33,293
                                  RAC: 0
                                  Message 4465 - Posted 21 Jan 2009 5:31:15 UTC - in response to Message 4464.

                                    as you wish ...


                                    Well, it is the only defect I have found so far on OS-X ...I can't get work ... :)

                                    of course, 1.47 works well on OS-X no hung tasks, no long running tasks ... no illegal functions ... so ... well, I will, be looking to add a windows machine the next drop ...

                                    Anyway, got three more tasks ... thanks ...
                                    ____________

                                    HA-SOFT, s.r.o.

                                    Joined: Jan 19 09
                                    Posts: 6
                                    ID: 5157
                                    Credit: 19,644
                                    RAC: 0
                                    Message 4466 - Posted 21 Jan 2009 9:10:21 UTC

                                      I still have problems on my new W2008 X64 server.
                                      Every taks of 1.5 minirosetta hangs at startup with 3MB memory and stdout:

                                      [2009- 1-21 9:52:36:] :: BOINC :: boinc_init()
                                      Created shared memory segment

                                      These tasks hangs and I have to kill them from taskbar. After killing stderr is:


                                      Unhandled Exception Detected...

                                      - Unhandled Exception Record -
                                      Reason: Access Violation (0xc0000005) at address 0x778806CF read attempt to address 0x00000004

                                      Engaging BOINC Windows Runtime Debugger...
                                      ____________

                                      Profile Paul D. Buck

                                      Joined: Jan 14 09
                                      Posts: 62
                                      ID: 5139
                                      Credit: 33,293
                                      RAC: 0
                                      Message 4467 - Posted 21 Jan 2009 9:35:58 UTC

                                        First ever error Task on OS-X ... I got this error:


                                        <core_client_version>5.10.45</core_client_version>
                                        <![CDATA[
                                        <message>
                                        process exited with code 1 (0x1, -255)
                                        </message>
                                        <stderr_txt>
                                        Watchdog active.

                                        ERROR: target_strands.size()
                                        ERROR:: Exit from: src/protocols/abinitio/TemplateJumpSetup.cc line: 94
                                        called boinc_finish

                                        </stderr_txt>


                                        Which seems to be the same error reported below ...
                                        ____________

                                        Profile sslickerson

                                        Joined: Feb 15 06
                                        Posts: 17
                                        ID: 37
                                        Credit: 4,006
                                        RAC: 0
                                        Message 4468 - Posted 21 Jan 2009 13:54:50 UTC

                                          My Windows Vista 64 laptop has received about 8 WU and all have completed successfully without error so this looks good, hopefully I will be able to attach to Rosetta soon!
                                          ____________


                                          I _ quit

                                          Joined: Jan 13 09
                                          Posts: 44
                                          ID: 5136
                                          Credit: 88,562
                                          RAC: 0
                                          Message 4469 - Posted 21 Jan 2009 14:35:22 UTC

                                            every task given to me so far has completed ok.
                                            I see there is nothing more for jobs in queue, so i take it this test is coming to an end?

                                            Profile feet1st

                                            Joined: Mar 7 06
                                            Posts: 312
                                            ID: 1028
                                            Credit: 110,522
                                            RAC: 0
                                            Message 4470 - Posted 21 Jan 2009 15:55:30 UTC - in response to Message 4469.

                                              so i take it this test is coming to an end?


                                              Don't rush it! There were still 1000 failures and 3000 successes. I'm not sure what time the last of the DB packaging problems were cleared through. But, you've barely given any time for a 24hr runtime to complete.

                                              Greg. Typically, they release a few tasks, if no clear problems like the DB packaging, and successes start coming in, then they release a few thousand tasks over the course of a day or so. And then, when they've made the final adjustments, explained most of the reported errors and confirmed the results, then they do a final push of 10,000+ tasks (again over the course of a day or two) to really seek out those rare and intermittant problems. THEN send it over to Rosetta.
                                              ____________

                                              I _ quit

                                              Joined: Jan 13 09
                                              Posts: 44
                                              ID: 5136
                                              Credit: 88,562
                                              RAC: 0
                                              Message 4471 - Posted 21 Jan 2009 16:19:36 UTC - in response to Message 4470.

                                                so i take it this test is coming to an end?


                                                Don't rush it! There were still 1000 failures and 3000 successes. I'm not sure what time the last of the DB packaging problems were cleared through. But, you've barely given any time for a 24hr runtime to complete.

                                                Greg. Typically, they release a few tasks, if no clear problems like the DB packaging, and successes start coming in, then they release a few thousand tasks over the course of a day or so. And then, when they've made the final adjustments, explained most of the reported errors and confirmed the results, then they do a final push of 10,000+ tasks (again over the course of a day or two) to really seek out those rare and intermittant problems. THEN send it over to Rosetta.



                                                ok...thanks for the explanation, just not sure what to expect here since this is the first time for me on ralph.

                                                Snagletooth

                                                Joined: May 4 07
                                                Posts: 65
                                                ID: 3020
                                                Credit: 112,890
                                                RAC: 0
                                                Message 4472 - Posted 21 Jan 2009 17:33:55 UTC

                                                  test_cc_1_8_nocst4_hb_t367__IGNORE_THE_REST_1WOLA_4_6830_2_0 ended with same rare error as Ian D and Paul D Buck.

                                                  <core_client_version>6.2.18</core_client_version>
                                                  <![CDATA[
                                                  <message>
                                                  process exited with code 1 (0x1, -255)
                                                  </message>
                                                  <stderr_txt>
                                                  Watchdog active.
                                                  # cpu_run_time_pref: 14400

                                                  ERROR: target_strands.size()
                                                  ERROR:: Exit from: src/protocols/abinitio/TemplateJumpSetup.cc line: 94
                                                  called boinc_finish

                                                  Snags

                                                  mtyka
                                                  Forum moderator
                                                  Project developer
                                                  Project scientist

                                                  Joined: Mar 19 08
                                                  Posts: 79
                                                  ID: 4144
                                                  Credit: 0
                                                  RAC: 0
                                                  Message 4473 - Posted 21 Jan 2009 17:58:14 UTC - in response to Message 4466.

                                                    I still have problems on my new W2008 X64 server.
                                                    Every taks of 1.5 minirosetta hangs at startup with 3MB memory and stdout:

                                                    [2009- 1-21 9:52:36:] :: BOINC :: boinc_init()
                                                    Created shared memory segment

                                                    These tasks hangs and I have to kill them from taskbar. After killing stderr is:


                                                    Unhandled Exception Detected...

                                                    - Unhandled Exception Record -
                                                    Reason: Access Violation (0xc0000005) at address 0x778806CF read attempt to address 0x00000004

                                                    Engaging BOINC Windows Runtime Debugger...



                                                    How do you know it *hangs* ?? Rosetta will not print anything to stdout - that's normal. Are the graphics moving ? What happens if you just let it run for a few hours ?

                                                    Mike

                                                    Profile feet1st

                                                    Joined: Mar 7 06
                                                    Posts: 312
                                                    ID: 1028
                                                    Credit: 110,522
                                                    RAC: 0
                                                    Message 4474 - Posted 21 Jan 2009 22:14:45 UTC - in response to Message 4462.

                                                      More checkpointing is great! But... this is a bit extreme. My write to disk at MOST every... setting is at 1800 seconds. My harddrive will never be able to spin down and go in to power saver mode all night long if the checkpoints continue at this pace.


                                                      1/20/2009 3:31:58 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0 checkpointed
                                                      1/20/2009 3:32:28 PM|ralph@home|[checkpoint_debug] result


                                                      Hmm ok, i'll look into this.


                                                      Ya, this afternoon I've been running two tasks for about 4.25hrs and I've got 450 checkpoint taken messages in my messages tab. Sometimes showing two checkpoints on same task within the same second.

                                                      Task names:
                                                      test_cc_1_8_nocst4_hb_t342__IGNORE_THE_REST_2G0QA_13_6824_1_0
                                                      test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0
                                                      (both are running v1.50)
                                                      ____________

                                                      I _ quit

                                                      Joined: Jan 13 09
                                                      Posts: 44
                                                      ID: 5136
                                                      Credit: 88,562
                                                      RAC: 0
                                                      Message 4475 - Posted 22 Jan 2009 1:00:19 UTC - in response to Message 4474.

                                                        Last modified: 22 Jan 2009 1:02:22 UTC

                                                        More checkpointing is great! But... this is a bit extreme. My write to disk at MOST every... setting is at 1800 seconds. My harddrive will never be able to spin down and go in to power saver mode all night long if the checkpoints continue at this pace.


                                                        1/20/2009 3:31:58 PM|ralph@home|[checkpoint_debug] result test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0 checkpointed
                                                        1/20/2009 3:32:28 PM|ralph@home|[checkpoint_debug] result


                                                        Hmm ok, i'll look into this.


                                                        Ya, this afternoon I've been running two tasks for about 4.25hrs and I've got 450 checkpoint taken messages in my messages tab. Sometimes showing two checkpoints on same task within the same second.

                                                        Task names:
                                                        test_cc_1_8_nocst4_hb_t342__IGNORE_THE_REST_2G0QA_13_6824_1_0
                                                        test_cc_1_8_nocst4_hb_t332__IGNORE_THE_REST_1X7OA_6_6823_1_0
                                                        (both are running v1.50)



                                                        interesting how you guys are getting checkpoint messages, i don't see that in my boinc manager. is that due to me using version 6 and your using version 5?

                                                        Profile feet1st

                                                        Joined: Mar 7 06
                                                        Posts: 312
                                                        ID: 1028
                                                        Credit: 110,522
                                                        RAC: 0
                                                        Message 4476 - Posted 22 Jan 2009 5:09:57 UTC

                                                          Greg, this is one of the BOINC debug messages. You get them be setting up a cc_config.xml file. In this case you need "checkpoint_debug" set to 1. You also need at least the first three set to 1.

                                                          Otherwise, the checkpoints are pretty transparent. But, in my case, my BOINC data directory is over on my second drive and it cannot spin down when idle now, because it is never idle. Normally, the drive is set to spin down when not in use, and then every 15-30 minutes BOINC kicks in and wants to write something and it spins up to do so, then goes back to sleep. The timer that makes the drive sleep is longer then the time between checkpoints with this new app.

                                                          It really should be honoring the BOINC setting for "write at most". I'm not clear why, but many projects do not honor that setting. They take then checkpoints as they are able, regardless. ...which was fine, until they started checkpointing every 2 minutes :)
                                                          ____________

                                                          Evan

                                                          Joined: Dec 23 07
                                                          Posts: 75
                                                          ID: 3893
                                                          Credit: 69,584
                                                          RAC: 0
                                                          Message 4477 - Posted 22 Jan 2009 9:28:05 UTC

                                                            I don't see that in my boinc manager


                                                            You can find it in your slots directory. (v6.2.19) You will find the boinc_checkpoint_count file. Open it and you will see how many checkpoints you have. Also there is a list of the checkpoints.

                                                            My work unit has been running for for about 34 minutes and I have accumulated 43 checkpoints.

                                                            Profile Ian_D

                                                            Joined: Feb 16 06
                                                            Posts: 16
                                                            ID: 321
                                                            Credit: 39,518
                                                            RAC: 0
                                                            Message 4478 - Posted 22 Jan 2009 9:30:50 UTC

                                                              Last modified: 22 Jan 2009 9:32:15 UTC

                                                              Version 1.51

                                                              http://ralph.bakerlab.org/result.php?resultid=1258153


                                                              <core_client_version>6.4.5</core_client_version>
                                                              <![CDATA[
                                                              <message>
                                                              - exit code -1073741819 (0xc0000005)
                                                              </message>
                                                              <stderr_txt>
                                                              BOINC:: Initializing ... ok.
                                                              [2009- 1-22 9:15:49:] :: BOINC :: boinc_init()
                                                              BOINC:: Setting up shared resources ... ok.
                                                              BOINC:: Setting up semaphores ... ok.
                                                              BOINC:: Updating status ... ok.
                                                              BOINC:: Registering timer callback... ok.
                                                              BOINC:: Worker initialized successfully.
                                                              Initializing options.... ok
                                                              Initializing random generators... ok
                                                              Initialization complete.
                                                              Watchdog active.


                                                              Unhandled Exception Detected...

                                                              - Unhandled Exception Record -
                                                              Reason: Access Violation (0xc0000005) at address 0x0081E942 read attempt to address 0x00000000

                                                              Engaging BOINC Windows Runtime Debugger...
                                                              ____________

                                                              I _ quit

                                                              Joined: Jan 13 09
                                                              Posts: 44
                                                              ID: 5136
                                                              Credit: 88,562
                                                              RAC: 0
                                                              Message 4479 - Posted 22 Jan 2009 9:37:42 UTC - in response to Message 4477.

                                                                Last modified: 22 Jan 2009 9:40:37 UTC

                                                                I don't see that in my boinc manager


                                                                You can find it in your slots directory. (v6.2.19) You will find the boinc_checkpoint_count file. Open it and you will see how many checkpoints you have. Also there is a list of the checkpoints.

                                                                My work unit has been running for for about 34 minutes and I have accumulated 43 checkpoints.



                                                                thanks for the info evan.
                                                                found the file, in 3 hrs+ run time one task accumulated 561 so far and the other 305. that's a pretty healthy number for 3 hrs. feet1st looks easy enough. i may try that later. thanks

                                                                mtyka - now into the 1.50 tasks and so far no errors on any of the tasks sent to my system.

                                                                HA-SOFT, s.r.o.

                                                                Joined: Jan 19 09
                                                                Posts: 6
                                                                ID: 5157
                                                                Credit: 19,644
                                                                RAC: 0
                                                                Message 4480 - Posted 22 Jan 2009 9:54:07 UTC - in response to Message 4473.

                                                                  Last modified: 22 Jan 2009 10:24:16 UTC

                                                                  Tasks can not be suspended, boinc can do nothing with process. After few days I have about 10-15 death rosetta tasks with 3M RAM allocated.

                                                                  If I don't kill the app, it runs till pc restart (on server usually about 30 days till MS security update and restart).

                                                                  Can be problem with DEP turned on?

                                                                  Ver 1.51:

                                                                  BOINC:: Initializing ... ok.
                                                                  [2009- 1-22 11: 0:26:] :: BOINC :: boinc_init()
                                                                  BOINC:: Setting up shared resources ... ok.
                                                                  BOINC:: Setting up semaphores ... ok.


                                                                  Unhandled Exception Detected...

                                                                  - Unhandled Exception Record -

                                                                  In Graphics application I have got:

                                                                  Starting graphics application...
                                                                  Setting window title to 'minirosetta version 1.51 [workunit: lr6_D_score12_rlbd_2ccv_IGNORE_THE_REST_DECOY_6840_2]'.
                                                                  OpenSemaphore failure
                                                                  Successfully loaded '../../projects/ralph.bakerlab.org/Helvetica.txf'...
                                                                  Close event (shmem not updated) detected, shutting down.
                                                                  Shutting down graphics application...

                                                                  but I'm running boinc like service app (and connecting over rdp) and this may be the problem for graphics.
                                                                  ____________

                                                                  Evan

                                                                  Joined: Dec 23 07
                                                                  Posts: 75
                                                                  ID: 3893
                                                                  Credit: 69,584
                                                                  RAC: 0
                                                                  Message 4481 - Posted 22 Jan 2009 10:47:12 UTC

                                                                    This one 1256978 has failed for the second time.

                                                                    Maybe the same problem as reported by Zdenek Vasku

                                                                    [url]<core_client_version>6.2.19</core_client_version>
                                                                    <![CDATA[
                                                                    <message>
                                                                    Incorrect function. (0x1) - exit code 1 (0x1)
                                                                    </message>
                                                                    <stderr_txt>
                                                                    BOINC:: Initializing ... ok.
                                                                    [2009- 1-22 10:30:57:] :: BOINC :: boinc_init()
                                                                    BOINC:: Setting up shared resources ... ok.
                                                                    BOINC:: Setting up semaphores ... ok.
                                                                    BOINC:: Updating status ... ok.
                                                                    BOINC:: Registering timer callback... ok.
                                                                    BOINC:: Worker initialized successfully.
                                                                    Initializing options.... ok
                                                                    Initializing random generators... ok
                                                                    Initialization complete.
                                                                    Watchdog active.

                                                                    ERROR: unknown model name: 1DK8A_1
                                                                    ERROR:: Exit from: d:\boinc_build\minirosetta_windows\mini\src\protocols/abinitio/PairingStatistics.hh line: 170
                                                                    called boinc_finish
                                                                    [/quote]

                                                                    Profile feet1st

                                                                    Joined: Mar 7 06
                                                                    Posts: 312
                                                                    ID: 1028
                                                                    Credit: 110,522
                                                                    RAC: 0
                                                                    Message 4482 - Posted 22 Jan 2009 14:14:35 UTC

                                                                      This task is running v1.51. It seems a bit on the large side in all dimensions here. It's been running for 6 hours, but is only on model 2. Step 900,000! It's peak memory usage so far was 430MB.

                                                                      Probably what you were expecting for such a large protein, but definitely needs a high memory flag.

                                                                      I believe the memory usage of the tasks is reported back with scheduler requests. Does the project process this data and query for anonomolies in memory usage? Or do you need us to report such things?
                                                                      ____________

                                                                      Profile feet1st

                                                                      Joined: Mar 7 06
                                                                      Posts: 312
                                                                      ID: 1028
                                                                      Credit: 110,522
                                                                      RAC: 0
                                                                      Message 4483 - Posted 22 Jan 2009 14:22:56 UTC

                                                                        This task completed my 24hr runtime preference on v1.48. But it reported 99 starting structures (not the usual "1") and 98 decoys. So, what happened to the last one?
                                                                        ____________

                                                                        Evan

                                                                        Joined: Dec 23 07
                                                                        Posts: 75
                                                                        ID: 3893
                                                                        Credit: 69,584
                                                                        RAC: 0
                                                                        Message 4484 - Posted 22 Jan 2009 17:36:44 UTC

                                                                          Add these to the mammoth second time failures:

                                                                          1257164
                                                                          1257066
                                                                          1257064

                                                                          mtyka
                                                                          Forum moderator
                                                                          Project developer
                                                                          Project scientist

                                                                          Joined: Mar 19 08
                                                                          Posts: 79
                                                                          ID: 4144
                                                                          Credit: 0
                                                                          RAC: 0
                                                                          Message 4485 - Posted 22 Jan 2009 20:41:50 UTC

                                                                            Morning.

                                                                            Things are moving forward - 1.51 is out.

                                                                            This should fix the checkpoint issue - they should not dump as frequently
                                                                            as before and try to honor the user's setting. Now - i have to say this is not *always* possible. To answer your question Feet1st, due to the structure of most simulation software (not just ours) you cant just checkpoint at any arbitrary point. sometime you *need* to checkpoint frequently or not at all, sometimes
                                                                            you can't checkpoint (or at least it would take huge amounts of data which is not ok). So what happens is that we checkpoint, but we "hold on" to the data in memory (i.e buffer it) until it is official time to checkpoint, and then we dump all the gathered checkpoints. Now, of course there is a limit to how much we can hold on to in orde rnot to overflow the memory, so occasionally we *have* to dump, even if it's not time. also at the end of a decoy, all is dumped and deleted and dealt with, the user setting cannot have any control over that. HOpe that explains it.

                                                                            However there was glitch in the buffering mechanism so it was dumping always. 1.51 should fix that - could you confirm that that is so !?

                                                                            Otherwise this release has added debug information to let me figure out where all this stuff is failing. Believe me guys, we're now in the land whre i cannot reproduce these errors here what so ever. Not on the linux boxes, Mac boxes or windows boxes we have. nowhere. Why these remaining segfaults occur is a total mystery to me, so please bear with me. THis is going ot be incredibly difficult to track down.

                                                                            Thanks for all your help! Every post is super useful to us!!

                                                                            Mike

                                                                            mtyka
                                                                            Forum moderator
                                                                            Project developer
                                                                            Project scientist

                                                                            Joined: Mar 19 08
                                                                            Posts: 79
                                                                            ID: 4144
                                                                            Credit: 0
                                                                            RAC: 0
                                                                            Message 4486 - Posted 22 Jan 2009 20:46:38 UTC - in response to Message 4482.

                                                                              This task is running v1.51. It seems a bit on the large side in all dimensions here. It's been running for 6 hours, but is only on model 2.


                                                                              absolutely reasonable isnt it ? 3-4 hours per model is what we expect these days with the mammoth tasks.


                                                                              > Step 900,000!

                                                                              Step 900000 ? WHat does that mean ? "Step" ??


                                                                              >It's peak memory usage so far was 430MB.

                                                                              Hmm yeah, i guess we need to flag these. They're big, i know.


                                                                              I believe the memory usage of the tasks is reported back with scheduler requests. Does the project process this data and query for anonomolies in memory usage? Or do you need us to report such things?


                                                                              Not sure where i can get this info. If you see anomalies let us know.

                                                                              mtyka
                                                                              Forum moderator
                                                                              Project developer
                                                                              Project scientist

                                                                              Joined: Mar 19 08
                                                                              Posts: 79
                                                                              ID: 4144
                                                                              Credit: 0
                                                                              RAC: 0
                                                                              Message 4487 - Posted 22 Jan 2009 20:49:59 UTC - in response to Message 4480.

                                                                                Tasks can not be suspended, boinc can do nothing with process. After few days I have about 10-15 death rosetta tasks with 3M RAM allocated.

                                                                                If I don't kill the app, it runs till pc restart (on server usually about 30 days till MS security update and restart).

                                                                                Can be problem with DEP turned on?

                                                                                Ver 1.51:

                                                                                BOINC:: Initializing ... ok.
                                                                                [2009- 1-22 11: 0:26:] :: BOINC :: boinc_init()
                                                                                BOINC:: Setting up shared resources ... ok.
                                                                                BOINC:: Setting up semaphores ... ok.


                                                                                Unhandled Exception Detected...

                                                                                - Unhandled Exception Record -

                                                                                In Graphics application I have got:

                                                                                Starting graphics application...
                                                                                Setting window title to 'minirosetta version 1.51 [workunit: lr6_D_score12_rlbd_2ccv_IGNORE_THE_REST_DECOY_6840_2]'.
                                                                                OpenSemaphore failure
                                                                                Successfully loaded '../../projects/ralph.bakerlab.org/Helvetica.txf'...
                                                                                Close event (shmem not updated) detected, shutting down.
                                                                                Shutting down graphics application...

                                                                                but I'm running boinc like service app (and connecting over rdp) and this may be the problem for graphics.



                                                                                Is there a way to turn off the graphics ? What's DEP ?

                                                                                THis is interesting, i will have to look at the code to see if there's a way for it to hang somewhere. I wonder what happens to mini if the graphics app fails ..

                                                                                I will have to try that locally. If you can decribe your set-up in more detail that would be excellent. Is this taks 1.51 or 1.50 or 1.48 ?

                                                                                Mike

                                                                                Profile robertmiles

                                                                                Joined: Jan 13 09
                                                                                Posts: 80
                                                                                ID: 5137
                                                                                Credit: 246,177
                                                                                RAC: 24
                                                                                Message 4488 - Posted 22 Jan 2009 21:46:42 UTC

                                                                                  Should the name of this thread be editted to include 1.51, or should a new thread be started for 1.51?

                                                                                  Profile robertmiles

                                                                                  Joined: Jan 13 09
                                                                                  Posts: 80
                                                                                  ID: 5137
                                                                                  Credit: 246,177
                                                                                  RAC: 24
                                                                                  Message 4489 - Posted 22 Jan 2009 21:53:35 UTC

                                                                                    Both copies of this test_cc 1.50 workunit failed quickly:

                                                                                    http://ralph.bakerlab.org/workunit.php?wuid=1105459

                                                                                    Evan

                                                                                    Joined: Dec 23 07
                                                                                    Posts: 75
                                                                                    ID: 3893
                                                                                    Credit: 69,584
                                                                                    RAC: 0
                                                                                    Message 4490 - Posted 22 Jan 2009 22:10:46 UTC

                                                                                      The mammoths have not been having a happy time. I have had a run of 23 failures (1st time) including:


                                                                                      unhandled Exception Detected...

                                                                                      - Unhandled Exception Record -
                                                                                      Reason: Access Violation (0xc0000005) at address 0x0081E942 read attempt to address 0x00000000

                                                                                      and

                                                                                      ERROR: unknown model name: 1B0NA_10
                                                                                      ERROR:: Exit from: d:\boinc_build\minirosetta_windows\mini\src\protocols/abinitio/PairingStatistics.hh line: 170
                                                                                      called boinc_finish




                                                                                      Profile feet1st

                                                                                      Joined: Mar 7 06
                                                                                      Posts: 312
                                                                                      ID: 1028
                                                                                      Credit: 110,522
                                                                                      RAC: 0
                                                                                      Message 4491 - Posted 22 Jan 2009 22:11:54 UTC - in response to Message 4486.


                                                                                        > Step 900,000!

                                                                                        Step 900000 ? WHat does that mean ? "Step" ??


                                                                                        That was the step shown in the graphic. (to the right of the model number). I've just never seen such a large step number.
                                                                                        ____________

                                                                                        Snagletooth

                                                                                        Joined: May 4 07
                                                                                        Posts: 65
                                                                                        ID: 3020
                                                                                        Credit: 112,890
                                                                                        RAC: 0
                                                                                        Message 4492 - Posted 22 Jan 2009 23:02:02 UTC

                                                                                          One more with v1.50: test_cc_1_8_nocst4_hb_t367__IGNORE_THE_REST_1UFBA_5_6830_3

                                                                                          <core_client_version>6.2.18</core_client_version>
                                                                                          <![CDATA[
                                                                                          <message>
                                                                                          process exited with code 1 (0x1, -255)
                                                                                          </message>
                                                                                          <stderr_txt>
                                                                                          Watchdog active.
                                                                                          # cpu_run_time_pref: 14400

                                                                                          ERROR: target_strands.size()
                                                                                          ERROR:: Exit from: src/protocols/abinitio/TemplateJumpSetup.cc line: 94
                                                                                          called boinc_finish



                                                                                          And one with v.1.51: test_cc2_1_8_mammoth_mix_cen_cst_hb_t311__IGNORE_THE_REST_1PERL_6_6852_1

                                                                                          <core_client_version>6.2.18</core_client_version>
                                                                                          <![CDATA[
                                                                                          <message>
                                                                                          process exited with code 1 (0x1, -255)
                                                                                          </message>
                                                                                          <stderr_txt>
                                                                                          BOINC:: Initializing ... ok.
                                                                                          [2009- 1-22 11:56:36:] :: BOINC :: boinc_init()
                                                                                          BOINC:: Setting up shared resources ... ok.
                                                                                          BOINC:: Setting up semaphores ... ok.
                                                                                          BOINC:: Updating status ... ok.
                                                                                          BOINC:: Registering timer callback... ok.
                                                                                          BOINC:: Worker initialized successfully.
                                                                                          Initializing options.... ok
                                                                                          Initializing random generators... ok
                                                                                          Initialization complete.
                                                                                          Watchdog active.

                                                                                          ERROR: unknown model name: 1B0NA_10
                                                                                          ERROR:: Exit from: src/protocols/abinitio/PairingStatistics.hh line: 170
                                                                                          called boinc_finish

                                                                                          Snags

                                                                                          Profile feet1st

                                                                                          Joined: Mar 7 06
                                                                                          Posts: 312
                                                                                          ID: 1028
                                                                                          Credit: 110,522
                                                                                          RAC: 0
                                                                                          Message 4493 - Posted 23 Jan 2009 1:04:16 UTC

                                                                                            Last modified: 23 Jan 2009 1:13:19 UTC

                                                                                            I tried loading up the docking task I got. It is 1.51. Displayed graphic just as the task was starting. Waited, waited... finally realized it was using more CPU then the thread working on the protein! Double checked Ralph settings for % of CPU for the graphic, set to default which is 10%.

                                                                                            Here's my screenshot showing the graphic monopolizing one core, while the two running tasks are competing for the other. Net result, nothing shown in the graphic after several minutes, and graphic thread consuming much more then 10% of CPU.



                                                                                            [edit]
                                                                                            I gave up on it, captured the screen, uploaded the screenshot, reported it here... then when I opened the graphic a second time it was better behaved. Not overusing CPU... but was essentially unusable. Go to resize or rotate the images and it wouldn't respond for about 30 seconds each time.
                                                                                            ____________

                                                                                            mtyka
                                                                                            Forum moderator
                                                                                            Project developer
                                                                                            Project scientist

                                                                                            Joined: Mar 19 08
                                                                                            Posts: 79
                                                                                            ID: 4144
                                                                                            Credit: 0
                                                                                            RAC: 0
                                                                                            Message 4494 - Posted 23 Jan 2009 1:14:21 UTC - in response to Message 4493.

                                                                                              [quote]I tried loading up the docking task I got. It is 1.51. Displayed graphic just as the task was starting. Waited, waited... finally realized it was using more CPU then the thread working on the protein! Double checked Ralph settings for % of CPU for the graphic, set to default which is 10%.

                                                                                              Here's my screenshot showing the graphic monopolizing one core, while the two running tasks are competing for the other. Net result, nothing shown in the graphic after several minutes, and graphic thread consuming much more then 10% of CPU.

                                                                                              [quote]

                                                                                              Feet1st , this is awesome - debugging on an unprecedented level :) Nice to get an idea of what all this looks like from your point of view.

                                                                                              These docking tasks are new and not mine - lemme track down the person submiting these and make sure the graphics app can deal with it.

                                                                                              Mike

                                                                                              Profile Ian_D

                                                                                              Joined: Feb 16 06
                                                                                              Posts: 16
                                                                                              ID: 321
                                                                                              Credit: 39,518
                                                                                              RAC: 0
                                                                                              Message 4495 - Posted 23 Jan 2009 8:31:33 UTC

                                                                                                Last modified: 23 Jan 2009 8:32:55 UTC

                                                                                                application version 1.51

                                                                                                http://ralph.bakerlab.org/result.php?resultid=1258676

                                                                                                <core_client_version>6.4.5</core_client_version>
                                                                                                <![CDATA[
                                                                                                <message>
                                                                                                Incorrect function. (0x1) - exit code 1 (0x1)
                                                                                                </message>
                                                                                                <stderr_txt>
                                                                                                BOINC:: Initializing ... ok.
                                                                                                [2009- 1-22 19: 2:51:] :: BOINC :: boinc_init()
                                                                                                BOINC:: Setting up shared resources ... ok.
                                                                                                BOINC:: Setting up semaphores ... ok.
                                                                                                BOINC:: Updating status ... ok.
                                                                                                BOINC:: Registering timer callback... ok.
                                                                                                BOINC:: Worker initialized successfully.
                                                                                                Initializing options.... ok
                                                                                                Initializing random generators... ok
                                                                                                Initialization complete.
                                                                                                Watchdog active.

                                                                                                ERROR: unknown model name: 1DK8A_1
                                                                                                ERROR:: Exit from: d:\boinc_build\minirosetta_windows\mini\src\protocols/abinitio/PairingStatistics.hh line: 170
                                                                                                called boinc_finish

                                                                                                </stderr_txt>
                                                                                                ]]>
                                                                                                ____________

                                                                                                Profile Paul D. Buck

                                                                                                Joined: Jan 14 09
                                                                                                Posts: 62
                                                                                                ID: 5139
                                                                                                Credit: 33,293
                                                                                                RAC: 0
                                                                                                Message 4496 - Posted 23 Jan 2009 11:25:18 UTC - in response to Message 4485.

                                                                                                  Otherwise this release has added debug information to let me figure out where all this stuff is failing. Believe me guys, we're now in the land whre i cannot reproduce these errors here what so ever. Not on the linux boxes, Mac boxes or windows boxes we have. nowhere. Why these remaining segfaults occur is a total mystery to me, so please bear with me. THis is going ot be incredibly difficult to track down.

                                                                                                  Thanks for all your help! Every post is super useful to us!!


                                                                                                  Questions I ask myself,

                                                                                                  Are they happening in the same module?
                                                                                                  Are they happening because of the same type of activity?
                                                                                                  What is common to all the events?
                                                                                                  Could it be something external?

                                                                                                  I know BOINC is supposed to be keeping things isolated and that application A is not supposed to affect application B ... but I have seen enough odd things that I am not convinced that this is completely so ...

                                                                                                  As to the tasks that fail ... you say they don't fail for you ...

                                                                                                  Why not grab the set and issue them to us with high replication counts ... then if they fail for everyone, that tells us one thing ... if they only fail for some and not others ... that tells us something else ...

                                                                                                  To my mind, the cases that fail are the ones that you should be saving and using as your issue tasks for each round of testing. Not to keep making new tasks in all cases, but to use those tasks that have proven their ability to cause a problem.

                                                                                                  Just some things to consider ... oh, and you are out of work again ...
                                                                                                  ____________

                                                                                                  HA-SOFT, s.r.o.

                                                                                                  Joined: Jan 19 09
                                                                                                  Posts: 6
                                                                                                  ID: 5157
                                                                                                  Credit: 19,644
                                                                                                  RAC: 0
                                                                                                  Message 4497 - Posted 23 Jan 2009 12:56:15 UTC - in response to Message 4487.

                                                                                                    DEP - execution protection on new Intel and AMD cores.

                                                                                                    How can I turn graphics off?


                                                                                                    ____________

                                                                                                    Profile feet1st

                                                                                                    Joined: Mar 7 06
                                                                                                    Posts: 312
                                                                                                    ID: 1028
                                                                                                    Credit: 110,522
                                                                                                    RAC: 0
                                                                                                    Message 4498 - Posted 23 Jan 2009 15:06:37 UTC

                                                                                                      Last modified: 23 Jan 2009 15:18:05 UTC

                                                                                                      v1.51

                                                                                                      ERROR: unknown model name: 2FRHA_10
                                                                                                      ERROR:: Exit from: d:\boinc_build\minirosetta_windows\mini\src\protocols/abinitio/PairingStatistics.hh line: 170
                                                                                                      called boinc_finish

                                                                                                      on task
                                                                                                      test_cc2_1_8_mammoth_mix_cen_cst_hb_t327__IGNORE_THE_REST_2F2EA_7_6860_1_1
                                                                                                      ____________

                                                                                                      Profile feet1st

                                                                                                      Joined: Mar 7 06
                                                                                                      Posts: 312
                                                                                                      ID: 1028
                                                                                                      Credit: 110,522
                                                                                                      RAC: 0
                                                                                                      Message 4499 - Posted 23 Jan 2009 15:16:37 UTC - in response to Message 4494.



                                                                                                        Feet1st , this is awesome - debugging on an unprecedented level :) Nice to get an idea of what all this looks like from your point of view.

                                                                                                        These docking tasks are new and not mine - lemme track down the person submiting these and make sure the graphics app can deal with it.

                                                                                                        Mike


                                                                                                        I've seen a number of other reports of the screen saver/graphic just displaying as a black window. And I had always assumed people just weren't waiting long enough for the display to refresh. It does often take a long time, and I always assumed this was due to the allowed % of CPU time for the graphic. But, on the other hand, this wasn't an issue before about 2 or 3 months ago.

                                                                                                        I meant to point out that the screenshot shows the graphic thread has used 4:02 of CPU time, but the corresponding Ralph thread is shown with only 2:11 of CPU received so far. So the % CPU shown in the screenshot is only the last interval, but you can see from the totals that the number is roughly what it's been during the entire 4 minutes the task has been running.

                                                                                                        I was wondering if perhaps the graphic could always display at least it's grid lines, and text immediately, and perhaps a "protein information being retrieved... please wait" message in the frames. That way at least it would never just be "blank".
                                                                                                        ____________

                                                                                                        I _ quit

                                                                                                        Joined: Jan 13 09
                                                                                                        Posts: 44
                                                                                                        ID: 5136
                                                                                                        Credit: 88,562
                                                                                                        RAC: 0
                                                                                                        Message 4500 - Posted 23 Jan 2009 16:15:26 UTC - in response to Message 4499.



                                                                                                          Feet1st , this is awesome - debugging on an unprecedented level :) Nice to get an idea of what all this looks like from your point of view.

                                                                                                          These docking tasks are new and not mine - lemme track down the person submiting these and make sure the graphics app can deal with it.

                                                                                                          Mike


                                                                                                          I've seen a number of other reports of the screen saver/graphic just displaying as a black window. And I had always assumed people just weren't waiting long enough for the display to refresh. It does often take a long time, and I always assumed this was due to the allowed % of CPU time for the graphic. But, on the other hand, this wasn't an issue before about 2 or 3 months ago.

                                                                                                          I meant to point out that the screenshot shows the graphic thread has used 4:02 of CPU time, but the corresponding Ralph thread is shown with only 2:11 of CPU received so far. So the % CPU shown in the screenshot is only the last interval, but you can see from the totals that the number is roughly what it's been during the entire 4 minutes the task has been running.

                                                                                                          I was wondering if perhaps the graphic could always display at least it's grid lines, and text immediately, and perhaps a "protein information being retrieved... please wait" message in the frames. That way at least it would never just be "blank".



                                                                                                          Cool idea...I would love to see that as well.

                                                                                                          Snagletooth

                                                                                                          Joined: May 4 07
                                                                                                          Posts: 65
                                                                                                          ID: 3020
                                                                                                          Credit: 112,890
                                                                                                          RAC: 0
                                                                                                          Message 4501 - Posted 23 Jan 2009 19:42:23 UTC

                                                                                                            test_cc_1_8_nocst4_hb_t327__IGNORE_THE_REST_2FSWA_6_6888_1
                                                                                                            andtest_cc_1_8_nocst4_hb_t327__IGNORE_THE_REST_2F2EA_10_6888_1_1

                                                                                                            both ended with:

                                                                                                            ERROR: unknown model name: 2FRHA_10
                                                                                                            ERROR:: Exit from: src/protocols/abinitio/PairingStatistics.hh line: 170

                                                                                                            This time they ran a few minutes instead of a few seconds, claimed to be done instead of declaring a compute error and received a validate error instead of a client error.


                                                                                                            Snags

                                                                                                            mtyka
                                                                                                            Forum moderator
                                                                                                            Project developer
                                                                                                            Project scientist

                                                                                                            Joined: Mar 19 08
                                                                                                            Posts: 79
                                                                                                            ID: 4144
                                                                                                            Credit: 0
                                                                                                            RAC: 0
                                                                                                            Message 4502 - Posted 23 Jan 2009 20:00:14 UTC - in response to Message 4496.


                                                                                                              Why not grab the set and issue them to us with high replication counts ... then if they fail for everyone, that tells us one thing ... if they only fail for some and not others ... that tells us something else ...


                                                                                                              we already submit identical jobs to many many machines - and they only fail *sometimes*. Even when the same random number seed is used.



                                                                                                              To my mind, the cases that fail are the ones that you should be saving and using as your issue tasks for each round of testing. Not to keep making new tasks in all cases, but to use those tasks that have proven their ability to cause a problem.


                                                                                                              again .. we're already doing that. But the errors are sporadic in nature so submitting the same task might not throw an error "this time".

                                                                                                              I have noticed though that some of the problems seem to be highly machine specific. Like one user will always produce this one kind of an error and nithing else. weired. something strange about their setup ? No idea.


                                                                                                              mtyka
                                                                                                              Forum moderator
                                                                                                              Project developer
                                                                                                              Project scientist

                                                                                                              Joined: Mar 19 08
                                                                                                              Posts: 79
                                                                                                              ID: 4144
                                                                                                              Credit: 0
                                                                                                              RAC: 0
                                                                                                              Message 4503 - Posted 23 Jan 2009 20:08:57 UTC - in response to Message 4502.

                                                                                                                Thank you all for reporting the problems! The problem divides it self into three parts, firstly cornering the error, turning it into an erro message, reproducing it here and then finding the root of the problem.

                                                                                                                At any stage there are three types of problem:

                                                                                                                a) Ones that fail with an error message like this:


                                                                                                                ERROR: unknown model name: 1DK8A_1
                                                                                                                ERROR:: Exit from: d:\boinc_build\minirosetta_windows\mini\src\protocols/abinitio/PairingStatistics.hh line: 170
                                                                                                                called boinc_finish


                                                                                                                These ones we've basically identified. Why they occur is merely a matter of tracking down the route. These ones i have already passed on to the relevant programmers who are fixing htem as we speak.

                                                                                                                b) Segfaults with traces:

                                                                                                                - Callstack -
                                                                                                                ChildEBP RetAddr Args to Child
                                                                                                                0012d270 00426312 08000000 077bf590 00000010 08000000 minirosetta_1.50_windows_intelx!UnwindUpVec+0x0 (F:\SP\vctools\crt_bld\SELF_X86\crt\src\Intel\MEMCPY.ASM:312)
                                                                                                                0012d28c 0046d11a 08000000 00000010 077bf590 00000010 minirosetta_1.50_windows_intelx!memmove_s+0xc (f:\sp\vctools\crt_bld\self_x86\crt\src\memmove_s.c:58)
                                                                                                                0012d2a8 0046f352 077bf590 077bf5a0 08000000 021a3d3c minirosetta_1.50_windows_intelx!stdext::unchecked_copy<double *,double *>+0x2a (c:\program files (x86)\microsoft visual studio 8\vc\include\xutility:3409)
                                                                                                                0012d2c8 005b1d2e 021a3d40 3d0497dd 0012d500 02006520 minirosetta_1.50_windows_intelx!std::vector<double,std::allocator<double> >::operator=+0x0 (c:\program files (x86)\microsoft visual studio 8\vc\include\vector:565)
                                                                                                                0012d3cc 005b1ed1 3d0496dd 02029f88 00000000 02006520 minirosetta_1.50_windows_intelx!core::scoring::Ramachandran::init_rama_sampling_table+0x2e (d:\boinc_build\minirosetta_windows\mini\src\core\scoring\ramachandran.cc:200)
                                                                                                                0012d3ec 005398fb 3d0496fd 02090718 02006520 0012d420 minirosetta_1.50_windows_intelx!core::scoring::Ramachandran::Ramachandran+0x0 (d:\boinc_build\minirosetta_windows\mini\src\core\scoring\ramachandran.cc:72)
                                                                                                                0012d408 005eccf9 3d049111 02029f88 3fc99999 02090718 minirosetta_1.50_windows_intelx!core::scoring::ScoringManager::get_Ramachandran+0x21 (d:\boinc_build\minirosetta_windows\mini\src\core\scoring\scoringmanager.cc:339)
                                                                                                                0012d428 0053b7e1 3d049131 02012f48 00000001 7c00003d minirosetta_1.50_windows_intelx!core::scoring::methods::RamachandranEnergy::RamachandranEnergy+0x59 (d:\boinc_build\minirosetta_windows\mini\src\core\scoring\methods\ramachandranenergy.cc:42)
                                                                                                                0012d4b4 004d8a28 0012d4e4 0012d500 02090718 3d0491c9 minirosetta_1.50_windows_intelx!core::scoring::ScoringManager::energy_method+0x2b (d:\boinc_build\minirosetta_windows\mini\src\core\scoring\scoringmanager.cc:566)
                                                                                                                0012d4dc 004d9135 0012d500 0012d544 3d0491f5 0012da20 minirosetta_1.50_windows_intelx!core::scoring::ScoreFunction::set_weight+0x0 (d:\boinc_build\minirosetta_windows\mini\src\core\scoring\scorefunction.cc:1377)
                                                                                                                0012d708 004ceacb 0012d738 3d04921d 020c68e0 01321ef0 minirosetta_1.50_windows_intelx!core::scoring::ScoreFunction::apply_patch_from_file+0x0 (d:\boinc_build\minirosetta_windows\mini\src\core\scoring\scorefunction.cc:235)
                                                                                                                0012d9e0 004cf1b1 0012da20 0000001f 6e617473 64726164 minirosetta_1.50_windows_intelx!core::scoring::ScoreFunctionFactory::create_score_function+0x4e (d:\boinc_build\minirosetta_windows\mini\src\core\scoring\scorefunctionfactory.cc:72)
                                                                                                                0012da90 005755b2 0012dac4 3d049fa5 00000000 0131af60 minirosetta_1.50_windows_intelx!core::scoring::getScoreFunction+0x31 (d:\boinc_build\minirosetta_windows\mini\src\core\scoring\scorefunctionfactory.cc:168)
                                                                                                                0012daec 0051b6a9 0131af60 0012dc20 3d049e05 01304498 minirosetta_1.50_windows_intelx!core::pack::pack_missing_sidechains+0xa (d:\boinc_build\minirosetta_windows\mini\src\core\pack\pack_missing_sidechains.cc:83)
                                                                                                                0012ddf8 0051bc8a 0131af60 01304498 3d049b09 00000010 minirosetta_1.50_windows_intelx!core::io::pdb::FileData::build_pose_as_is+0xe (d:\boinc_build\minirosetta_windows\mini\src\core\io\pdb\file_data.cc:749)
                                                                                                                0012de74 004dd0f2 0131af60 01304498 3d049b8d 0012ee68 minirosetta_1.50_windows_intelx!core::io::pdb::FileData::build_pose+0x0 (d:\boinc_build\minirosetta_windows\mini\src\core\io\pdb\file_data.cc:116)
                                                                                                                0012e01c 004dd1ae 0131af60 01304498 0012e070 00000000 minirosetta_1.50_windows_intelx!core::io::pdb::pose_from_pdb+0x0 (d:\boinc_build\minirosetta_windows\mini\src\core\io\pdb\pose_io.cc:194)
                                                                                                                0012e034 00798370 0131af60 0012e070 00000000 3d04a549 minirosetta_1.50_windows_intelx!core::io::pdb::pose_from_pdb+0x0 (d:\boinc_build\minirosetta_windows\mini\src\core\io\pdb\pose_io.cc:210)
                                                                                                                0012e0f8 007a7282 3d04a401 0000000f 00000007 00000000 minirosetta_1.50_windows_intelx!protocols::abinitio::AbrelaxApplication::setup+0x5d (d:\boinc_build\minirosetta_windows\mini\src\protocols\abinitio\abrelaxapplication.cc:496)
                                                                                                                0012edb4 0041a0d8 3d04a8c5 00000a28 00000002 0012ffc0 minirosetta_1.50_windows_intelx!protocols::abinitio::AbrelaxApplication::run+0x0 (d:\boinc_build\minirosetta_windows\mini\src\protocols\abinitio\abrelaxapplication.cc:2176)
                                                                                                                0012ef14 0041a465 0000001f 0012ef2c 00152310 0012ef2c minirosetta_1.50_windows_intelx!main+0x0 (d:\boinc_build\minirosetta_windows\mini\src\apps\public\boinc\minirosetta.cc:117)
                                                                                                                0012ff28 0042d207 00400000 00000000 00152352 0000000a minirosetta_1.50_windows_intelx!WinMain+0x0 (d:\boinc_build\minirosetta_windows\mini\src\apps\public\boinc\minirosetta.cc:159)
                                                                                                                0012ffc0 7c817067 0032005c 0064005c 7ffd8000 c0000005 minirosetta_1.50_windows_intelx!__tmainCRTStartup+0x1c (f:\sp\vctools\crt_bld\self_x86\crt\src\crt0.c:324)
                                                                                                                0012fff0 00000000 0042d270 00000000 78746341 00000020 kernel32!_BaseProcessStart@4+0x0 (f:\sp\vctools\crt_bld\self_x86\crt\src\crt0.c:324)



                                                                                                                Segfault - bad. But at least i get a trace - i can thus look into the code and understand (or guess) at why the program could fail at that point. Then i can turn them into a) type errors.


                                                                                                                <core_client_version>6.2.19</core_client_version>
                                                                                                                <![CDATA[
                                                                                                                <message>
                                                                                                                There are no child processes to wait for. (0x80) - exit code 128 (0x80)
                                                                                                                </message>
                                                                                                                <stderr_txt>
                                                                                                                BOINC:: Initializing ... ok.
                                                                                                                [2009- 1-22 4:45: 1:] :: BOINC :: boinc_init()
                                                                                                                BOINC:: Setting up shared resources ... ok.
                                                                                                                BOINC:: Setting up semaphores ... ok.
                                                                                                                BOINC:: Updating status ... ok.
                                                                                                                BOINC:: Registering timer callback... ok.
                                                                                                                BOINC:: Worker initialized successfully.


                                                                                                                Unhandled Exception Detected...

                                                                                                                - Unhandled Exception Record -
                                                                                                                Reason: Access Violation (0xc0000005) at address 0x7C81BD02 read attempt to address 0x00000002

                                                                                                                Engaging BOINC Windows Runtime Debugger...



                                                                                                                Unhandled Exception Detected...

                                                                                                                - Unhandled Exception Record -
                                                                                                                Reason: Access Violation (0xc0000005) at address 0x7C81BD02 read attempt to address 0x00000002

                                                                                                                Engaging BOINC Windows Runtime Debugger...


                                                                                                                </stderr_txt>
                                                                                                                ]]>




                                                                                                                These are random segfault that occured somewhere. I can do virtually nothing from here. If i rerun the *exact same commandline* from here all runs fine. So these ones i can only tentatively bracket with stderr statemenets.




                                                                                                                Right now i am mainly trying to turn b) and c) into a). Once an error is in the form of a) its usually trivial to solve!

                                                                                                                Anyway,.. just incase you guys were interested in what i'm trying to do.


                                                                                                                oh i guess there are also errors that are *behavioral* errors. Those are the ones i cannot see from here but you guys have to tell me about. Stuff like the graphics or overrunning models or other strange behaviour.

                                                                                                                :-)

                                                                                                                Snagletooth

                                                                                                                Joined: May 4 07
                                                                                                                Posts: 65
                                                                                                                ID: 3020
                                                                                                                Credit: 112,890
                                                                                                                RAC: 0
                                                                                                                Message 4504 - Posted 23 Jan 2009 20:46:27 UTC

                                                                                                                  Last modified: 23 Jan 2009 20:47:39 UTC

                                                                                                                  Hey, Mike, thanks for all the info you've been giving us lately. It's interesting and certainly provides a bit of extra motivation to keep crunching and reporting what we see at our end.

                                                                                                                  Snags

                                                                                                                  franfranlatulipe94@hotmail.com

                                                                                                                  Joined: Dec 12 08
                                                                                                                  Posts: 1
                                                                                                                  ID: 5048
                                                                                                                  Credit: 447
                                                                                                                  RAC: 0
                                                                                                                  Message 4505 - Posted 23 Jan 2009 21:22:42 UTC

                                                                                                                    Hi,
                                                                                                                    I'm running RALPH with Ubuntu Jaunty up to date.
                                                                                                                    My BOINC version is 6.2.18.
                                                                                                                    When I click to "show graphic" (or suchs, I'm French), I see a small window, like the screenshot in XP in this Post, but all my memory (1.5 Gio) and all my swap (2.0 Gio) are full…
                                                                                                                    If I do not it (click on "sow graphics"), there isn't any visible problem.

                                                                                                                    Rosetta mini 1.52
                                                                                                                    http://francois.linkmauve.fr/boinc.png
                                                                                                                    The screenshot show when the window has been killed.
                                                                                                                    If there are questions, ask me…

                                                                                                                    Snagletooth

                                                                                                                    Joined: May 4 07
                                                                                                                    Posts: 65
                                                                                                                    ID: 3020
                                                                                                                    Credit: 112,890
                                                                                                                    RAC: 0
                                                                                                                    Message 4506 - Posted 23 Jan 2009 22:12:04 UTC - in response to Message 4505.

                                                                                                                      Hi,
                                                                                                                      I'm running RALPH with Ubuntu Jaunty up to date.
                                                                                                                      My BOINC version is 6.2.18.
                                                                                                                      When I click to "show graphic" (or suchs, I'm French), I see a small window, like the screenshot in XP in this Post, but all my memory (1.5 Gio) and all my swap (2.0 Gio) are full…
                                                                                                                      If I do not it (click on "sow graphics"), there isn't any visible problem.

                                                                                                                      Rosetta mini 1.52
                                                                                                                      http://francois.linkmauve.fr/boinc.png
                                                                                                                      The screenshot show when the window has been killed.
                                                                                                                      If there are questions, ask me…


                                                                                                                      I'm showing the same or very similar behavior on the same type of workunit:lr6_D_score12_rlbn_1gu3_IGNORE_THE_REST_NATIVE_6909_1 running Leopard on an Intel Mac. No cpu use but tons of memory and I don't even get the box so nothing to show in a screen capture. Activity Manager says the process has hung and I killed it without trouble from there.

                                                                                                                      Snags

                                                                                                                      tralala

                                                                                                                      Joined: Apr 12 06
                                                                                                                      Posts: 52
                                                                                                                      ID: 1266
                                                                                                                      Credit: 15,257
                                                                                                                      RAC: 0
                                                                                                                      Message 4507 - Posted 23 Jan 2009 22:12:53 UTC

                                                                                                                        Mtyka,

                                                                                                                        can you disable graphics for some WUs and see if the error rate goes down? Might help turning c)'s into b)'s.
                                                                                                                        ____________

                                                                                                                        Profile feet1st

                                                                                                                        Joined: Mar 7 06
                                                                                                                        Posts: 312
                                                                                                                        ID: 1028
                                                                                                                        Credit: 110,522
                                                                                                                        RAC: 0
                                                                                                                        Message 4508 - Posted 23 Jan 2009 23:45:17 UTC

                                                                                                                          FYI, in French "Gio" is "GB" (gigabytes)
                                                                                                                          ____________

                                                                                                                          Evan

                                                                                                                          Joined: Dec 23 07
                                                                                                                          Posts: 75
                                                                                                                          ID: 3893
                                                                                                                          Credit: 69,584
                                                                                                                          RAC: 0
                                                                                                                          Message 4509 - Posted 24 Jan 2009 0:17:01 UTC

                                                                                                                            Last modified: 24 Jan 2009 0:38:32 UTC

                                                                                                                            I'm showing the same or very similar behavior on the same type of workunit:lr6_D_score12_rlbn_1gu3_IGNORE_THE_REST_NATIVE_6909


                                                                                                                            I'm also getting problems on several work units starting: test_B_cc ....
                                                                                                                            I get either a hung empty screen or a runtime error.

                                                                                                                            edit The graphics problem doesn't affect progress of the work unit.

                                                                                                                            Aegis Maelstrom

                                                                                                                            Joined: Jan 19 09
                                                                                                                            Posts: 12
                                                                                                                            ID: 5159
                                                                                                                            Credit: 4,751
                                                                                                                            RAC: 0
                                                                                                                            Message 4510 - Posted 24 Jan 2009 0:17:28 UTC

                                                                                                                              Hi Mike, Hi Fellows!

                                                                                                                              I'm just running Mini 1.52, this task.

                                                                                                                              Actually, I've found that RALPH WU has started only because of a prompt alert: BOINC wanted to turn the screensaver on and I got the Windows massage about a runtime terror and sudden termination of 1.40 graphics viewer. The process has been killed by the system.

                                                                                                                              So far, the WU seems to be continuing, I'm on 3,84% (0:11:35) and running.

                                                                                                                              Path7

                                                                                                                              Joined: Feb 11 08
                                                                                                                              Posts: 56
                                                                                                                              ID: 4036
                                                                                                                              Credit: 4,974
                                                                                                                              RAC: 0
                                                                                                                              Message 4511 - Posted 24 Jan 2009 0:25:28 UTC

                                                                                                                                Last modified: 24 Jan 2009 0:32:34 UTC

                                                                                                                                Hello all,

                                                                                                                                While crunching:
                                                                                                                                lr6_D_score12_rlbn_1ynv_IGNORE_THE_REST_NATIVE_NOCON_6909_1_0 (minirosetta 1.52)
                                                                                                                                I hit the “Show graphic” button, Windows XP replied with an error message: Runtime error.
                                                                                                                                Short message rapport:
                                                                                                                                Hung application: minirosetta_graphics_1.40_windows_intelx86.exe, version: 0.0.0.0, hung module: hungapp, version: 0.0.0.0, hung at: 0x00000000.

                                                                                                                                Retrying “Show graphics”opened the graphics window, empty.
                                                                                                                                At 16 minutes processor time the processor time stood still, CPU usage = 0 %.
                                                                                                                                Stopped and started BOINC, the WU proceeded where it has stopped.

                                                                                                                                Again tried to hit the “Show graphics button”, and the above repeated.
                                                                                                                                Now the WU has started for the third time, and keeps on running. I don't dare to touch the “Show graphics” button again!

                                                                                                                                Windows XP SP3, single core AMD Sempron 3000+ 1.8 GHz, 1.5 Gb RAM, BOINC 5.10.45.

                                                                                                                                Path7.

                                                                                                                                Evan

                                                                                                                                Joined: Dec 23 07
                                                                                                                                Posts: 75
                                                                                                                                ID: 3893
                                                                                                                                Credit: 69,584
                                                                                                                                RAC: 0
                                                                                                                                Message 4512 - Posted 24 Jan 2009 0:27:20 UTC

                                                                                                                                  This one failed (1st time)
                                                                                                                                  1259966

                                                                                                                                  Unhandled Exception Record -
                                                                                                                                  Reason: Access Violation (0xc0000005) at address 0x0044EFF4 read attempt to address 0x01062000

                                                                                                                                  Engaging BOINC Windows Runtime Debugger...

                                                                                                                                  mtyka
                                                                                                                                  Forum moderator
                                                                                                                                  Project developer
                                                                                                                                  Project scientist

                                                                                                                                  Joined: Mar 19 08
                                                                                                                                  Posts: 79
                                                                                                                                  ID: 4144
                                                                                                                                  Credit: 0
                                                                                                                                  RAC: 0
                                                                                                                                  Message 4513 - Posted 24 Jan 2009 0:43:10 UTC - in response to Message 4512.

                                                                                                                                    Last modified: 24 Jan 2009 0:44:09 UTC

                                                                                                                                    This one failed (1st time)
                                                                                                                                    1259966

                                                                                                                                    Unhandled Exception Record -
                                                                                                                                    Reason: Access Violation (0xc0000005) at address 0x0044EFF4 read attempt to address 0x01062000

                                                                                                                                    Engaging BOINC Windows Runtime Debugger...


                                                                                                                                    Yeah guess what. I just found a bug in the BOINC API! . Holy crap. Basically as far as i can see there's a memory leak when it's trying to unzip files. Mostly all you see is the application dying kicking and screaming just after initialization. One RALPH user though produced a suspicious trace (thank you philip in hongkong!). I have to stress that this is the only job out of hundreds such failures that has returned with a trace.


                                                                                                                                    LENOVO-A05B19D0 (10.9.3.121) [16000]
                                                                                                                                    User ID philip-in-hongkong [2191]
                                                                                                                                    CPU time 0
                                                                                                                                    XML doc in

                                                                                                                                    <core_client_version>6.2.18</core_client_version>
                                                                                                                                    <![CDATA[
                                                                                                                                    <message>
                                                                                                                                    - exit code -1073741819 (0xc0000005)
                                                                                                                                    </message>
                                                                                                                                    <stderr_txt>
                                                                                                                                    BOINC:: Initializing ... ok.
                                                                                                                                    [2009- 1-22 17:43:47:] :: BOINC :: boinc_init()
                                                                                                                                    BOINC:: Setting up shared resources ... ok.
                                                                                                                                    BOINC:: Setting up semaphores ... ok.
                                                                                                                                    BOINC:: Updating status ... ok.
                                                                                                                                    BOINC:: Registering timer callback... ok.
                                                                                                                                    BOINC:: Worker initialized successfully.
                                                                                                                                    Initializing options.... ok
                                                                                                                                    Initializing random generators... ok
                                                                                                                                    Initialization complete.


                                                                                                                                    Unhandled Exception Detected...

                                                                                                                                    - Unhandled Exception Record -
                                                                                                                                    Reason: Access Violation (0xc0000005) at address 0x0044FA48 read attempt to
                                                                                                                                    - Callstack -
                                                                                                                                    ChildEBP RetAddr Args to Child
                                                                                                                                    0012ecbc 0044fbc9 0114f008 00000001 010f1ffc 00000004 minirosetta_1.51_windows_intelx!unzip+0x0 (d:\boinc_build\minirosetta_windows\mini\external\boinc\zip\unzip\unzip.c:943)
                                                                                                                                    0012ecd4 0044e8f0 00000004 010f1ff0 14f97ded 0000000f minirosetta_1.51_windows_intelx!unzip_main+0x0 (d:\boinc_build\minirosetta_windows\mini\external\boinc\zip\unzip\unzip.c:629)
                                                                                                                                    0012ed10 0044ea7d 00000000 00000000 0108df08 0000000f minirosetta_1.51_windows_intelx!boinc_zip+0x7 (d:\boinc_build\minirosetta_windows\mini\external\boinc\zip\boinc_zip.cpp:151)
                                                                                                                                    0012ed68 0041981a 00000000 0000000f 0108bd58 0012ffc0 minirosetta_1.51_windows_intelx!boinc_zip+0x32 (d:\boinc_build\minirosetta_windows\mini\external\boinc\zip\boinc_zip.cpp:73)
                                                                                                                                    0012ef14 0041a015 0000001f 0012ef2c 00152310 0012ef2c minirosetta_1.51_windows_intelx!main+0x55 (d:\boinc_build\minirosetta_windows\mini\src\apps\public\boinc\minirosetta.cc:85)
                                                                                                                                    0012ff28 0042cdf7 00400000 00000000 00152352 0000000a minirosetta_1.51_windows_intelx!WinMain+0x0 (d:\boinc_build\minirosetta_windows\mini\src\apps\public\boinc\minirosetta.cc:159)
                                                                                                                                    0012ffc0 7c816fd7 00000000 00000000 7ffd3000 c0000005 minirosetta_1.51_windows_intelx!__tmainCRTStartup+0x1c (f:\sp\vctools\crt_bld\self_x86\crt\src\crt0.c:324)
                                                                                                                                    0012fff0 00000000 0042ce60 00000000 78746341 00000020 kernel32!_BaseProcessStart@4+0x0 (f:\sp\vctools\crt_bld\self_x86\crt\src\crt0.c:324)

                                                                                                                                    *** Dump of thread ID 4252 (state: Waiting): ***



                                                                                                                                    It fails in the unzip code ! OMG.
                                                                                                                                    Been tinkering with the code, the bug probably stems froma single byte not being set to 0. This explains the sporadic nature - if the relevant byte happens to already be 0 then all is fine. I'll push out a version soon to see if the fix works. Its all stipulation at this point.

                                                                                                                                    ALso soory bout the graphics error, i increased the buffersizes and (duh!) forgot to also update the graphics app .. i'll do that ogether with 1.53.

                                                                                                                                    Mike

                                                                                                                                    Profile Paul D. Buck

                                                                                                                                    Joined: Jan 14 09
                                                                                                                                    Posts: 62
                                                                                                                                    ID: 5139
                                                                                                                                    Credit: 33,293
                                                                                                                                    RAC: 0
                                                                                                                                    Message 4514 - Posted 24 Jan 2009 0:49:24 UTC - in response to Message 4502.

                                                                                                                                      I have noticed though that some of the problems seem to be highly machine specific. Like one user will always produce this one kind of an error and nithing else. weired. something strange about their setup ? No idea.


                                                                                                                                      Well, then it is time to start asking that user ... :)

                                                                                                                                      I know that some will OC their machine to the edge of instability and fail to recognize the implications of that ...

                                                                                                                                      Other cases can be because they are running other applications / projects and that might be the cause ... like running GPU Grid may raise the temperature enough at some times to cause the instability, but not other times ... So, it may not be the problem but the specific hardware ...

                                                                                                                                      I remember once where I reported a problem where the device we were testing would cause a tape read error ... impossible the engineers said ... I showed them ... still impossible ... not really ... the compiler was putting bad instructions on the tape and under an error condition the tape could not be read ...

                                                                                                                                      Anyway ...

                                                                                                                                      Just trying to stimulate the brain cells with some brain storming ... I do appreciate the feedback though ... and I got another task and it should be back in a little bit ... and yes, fascinated in what we as a collective are trying to do ...

                                                                                                                                      Oh, and the same RND seed, unless the machines are of the same specific type with identical CPU and OS the RND generator may or may not issue the same sequence of random numbers. Especially if the core of the generator relies on the noise at the end of floating point numbers ... Virtual Prairie is having this exact problem with their work ...
                                                                                                                                      ____________

                                                                                                                                      Aegis Maelstrom

                                                                                                                                      Joined: Jan 19 09
                                                                                                                                      Posts: 12
                                                                                                                                      ID: 5159
                                                                                                                                      Credit: 4,751
                                                                                                                                      RAC: 0
                                                                                                                                      Message 4515 - Posted 24 Jan 2009 2:13:24 UTC

                                                                                                                                        Last modified: 24 Jan 2009 2:25:53 UTC

                                                                                                                                        O.K., I got the new 1.53 version - when only 1.52 finishes we will see how it behaves. :)

                                                                                                                                        EDIT: So far O.K., and the graphics is working...

                                                                                                                                        mtyka
                                                                                                                                        Forum moderator
                                                                                                                                        Project developer
                                                                                                                                        Project scientist

                                                                                                                                        Joined: Mar 19 08
                                                                                                                                        Posts: 79
                                                                                                                                        ID: 4144
                                                                                                                                        Credit: 0
                                                                                                                                        RAC: 0
                                                                                                                                        Message 4516 - Posted 24 Jan 2009 2:24:49 UTC

                                                                                                                                          1.53 is out. This includes a fix in API causing crashes with unzipping .zip files. I hope. Fingers crossed ;)

                                                                                                                                          Also the graphics are updated and should not freeze.

                                                                                                                                          What I'd like to know from you:

                                                                                                                                          - Do you ever get any long running tasks ? (longer then PrefRuntime + 4 hrs)
                                                                                                                                          - Do the checkpoints work and honor the user's setting ? (Feet1st ? )
                                                                                                                                          - Do you get any jobs that are stuck ?
                                                                                                                                          - Do the graphics behave properly again ?

                                                                                                                                          Thanks ya all.

                                                                                                                                          Enjoy the weekend ;)

                                                                                                                                          Mike

                                                                                                                                          Aegis Maelstrom

                                                                                                                                          Joined: Jan 19 09
                                                                                                                                          Posts: 12
                                                                                                                                          ID: 5159
                                                                                                                                          Credit: 4,751
                                                                                                                                          RAC: 0
                                                                                                                                          Message 4517 - Posted 24 Jan 2009 2:43:05 UTC - in response to Message 4516.

                                                                                                                                            Last modified: 24 Jan 2009 3:24:16 UTC

                                                                                                                                            Hi,

                                                                                                                                            currently I'm running sr213_t077_1_NMR_NESG_SAVE_ALL_OUT_6972_24_0.

                                                                                                                                            - Do the graphics behave properly again ?

                                                                                                                                            Yes, it is working - however there was a funny thing.
                                                                                                                                            This WU has a native protein and RMSD - but once the RMSD part of the graphics (showing left/right how good the RMSD is) was not working. It was just blank.
                                                                                                                                            The low energy part was O.K.

                                                                                                                                            After turning off and turning on one-two minutes after, everything was just fine.

                                                                                                                                            Regarding checkpointing - according to boinccmd.exe it does make checkpoints every 100+ - 200+ seconds. It is quite close to my preferences (120 sec).

                                                                                                                                            EDIT: I have made an experiment to check the checkpointing and I have turned off the client and afterwards turned it back on.

                                                                                                                                            The task resumed from over 26 minutes (and according to the boinccmd the checkpoint was 1591 seconds, so it would fit). However, the WU started from Model 1, from the step 0 and was moving forward.
                                                                                                                                            On the other hand, the WU snaply went into SmoothFragmentMover_GunnCost and it didn't look that bad, so I am puzzled: I am not sure if it really started from the beginning, or if the checkpointing worked but the step information is misguiding.
                                                                                                                                            I think it must be checked on some further step when the protein is really pretty and the difference is easy to tell - but now I must go sleep. :D


                                                                                                                                            EDIT 2: I have repeated my experiment and I am pretty sure the checkpointing is not working correctly.

                                                                                                                                            I've stopped the WU after 46 minutes (2524 seconds) when it was quite beautiful (minus three hundred something energy etc.) and restarted.

                                                                                                                                            Yet again it started from step 0 and then it started to behave in a strange manner. There was no really folded protein, only one straight chain - but the program was clearly trying to use methods proper in the last stage of the process (namely MoverBase+Minimization). In a minute or so the chain was bent in something like two points and it got "fractalized" by the last step of the procedure (when you have all these small strings moving to get the lowest possible energy).

                                                                                                                                            If you want, I can provide you with a snapshot (printscreen of the graphics).

                                                                                                                                            Everything ended after circa 200 seconds (for a couple of seconds there was "stage: unknown") and got reported.

                                                                                                                                            Have a nice lecture.

                                                                                                                                            Interesting thing: it reduced the granted credit; claimed was 7.06, granted: 4.68.

                                                                                                                                            Well, one must suffer for the science. =) And now let me get some sleep...


                                                                                                                                            P.S. The only thing that collided with the experiment was that in a second time I had to halt a regular Rosetta task (I think I've halted the Rosetta but this time after restarting BOINC the manager didn't remember it, huh). I don't know if it could interfere - if it did it would mean there is some lack of reseting used variables.

                                                                                                                                            If it were not for the experiment, I am pretty sure the task would run alright.

                                                                                                                                            Profile feet1st

                                                                                                                                            Joined: Mar 7 06
                                                                                                                                            Posts: 312
                                                                                                                                            ID: 1028
                                                                                                                                            Credit: 110,522
                                                                                                                                            RAC: 0
                                                                                                                                            Message 4518 - Posted 24 Jan 2009 2:53:05 UTC

                                                                                                                                              The checkpoints seem more inline with my settings as of 1.52.

                                                                                                                                              I'm a bit unclear as to what exactly to expect now. Sounds like if I set my preference to "write at most..." 600 seconds (10min) that I should expect the following...

                                                                                                                                              I'll see checkpoint debug messages in my messages tab, but it may be that no data was written to disk?

                                                                                                                                              And I'll see for a given task, within a given model, that it won't write more then every 10min?

                                                                                                                                              ...except if it reaches the end of a model, at which point it will write, regardless of how recently the last checkpoint was??

                                                                                                                                              So, hypothetical example:

                                                                                                                                              Time Event
                                                                                                                                              mm:ss
                                                                                                                                              00:00 Model start
                                                                                                                                              01:15 checkpoint (but not written)
                                                                                                                                              02:45 checkpoint (but not written)
                                                                                                                                              05:15 checkpoint (but not written)
                                                                                                                                              09:15 checkpoint (but not written)
                                                                                                                                              11:30 checkpoint, all of the above written to disk
                                                                                                                                              12:10 checkpoint (but not written)
                                                                                                                                              13:40 checkpoint (but not written)
                                                                                                                                              14:25 model completed, write to disk, even though only 3min since last write.

                                                                                                                                              I think the above is what I should expect. Now... I've got a P4 running with HT, so 2 virtual cores... should I expect the tasks to behave independantly? i.e. each on their own 10 min "write at most" timer? Or should I expect them to both buffer for 10min. unless they reach a model end? And if a model end is reached on one, will that cause the buffers for the other to be written as well?

                                                                                                                                              Anyone have any good ideas on how to track over time the number of checkpoint messages, as compared to the number of disk writes?
                                                                                                                                              ____________

                                                                                                                                              Profile feet1st

                                                                                                                                              Joined: Mar 7 06
                                                                                                                                              Posts: 312
                                                                                                                                              ID: 1028
                                                                                                                                              Credit: 110,522
                                                                                                                                              RAC: 0
                                                                                                                                              Message 4519 - Posted 24 Jan 2009 3:03:50 UTC

                                                                                                                                                Last modified: 24 Jan 2009 3:09:27 UTC

                                                                                                                                                Anyone have any good ideas on how to track over time the number of checkpoint messages, as compared to the number of disk writes?


                                                                                                                                                OH! Is that what all these new files are??
                                                                                                                                                chk_S_1LARA_1_00000009_ClassicAbinitio___lc_3.out
                                                                                                                                                chk_S_1LARA_1_00000009_ClassicAbinitio___lc_3.rng.state
                                                                                                                                                chk_S_1LARA_1_00000010_ClassicAbinitio__stage_1.out
                                                                                                                                                chk_S_1LARA_1_00000010_ClassicAbinitio__stage_1.rng.state

                                                                                                                                                In fact I see many files now in my slot directory. Seems like blocks of 4 all written with same timestamp. But names vary, probably depending on what each of the model took the checkpoint.

                                                                                                                                                Ok, yes my "boinc_checkpoint_count" file says 92, but I've got no where near that many files. And then even less blocks of time where files were written. Ya, I've only got 14 different timestamps. So, that sounds good.

                                                                                                                                                I've got another docking task that says it's on model 258 after 22hrs (my preference is 24hrs), and so it's checkpoint count says 257, and I don't have any of the other files. Perhaps it is not taking checkpoints? The default.out file on that one is 43MB so far! Man is *THAT* upload going to take a while!
                                                                                                                                                ____________

                                                                                                                                                Aegis Maelstrom

                                                                                                                                                Joined: Jan 19 09
                                                                                                                                                Posts: 12
                                                                                                                                                ID: 5159
                                                                                                                                                Credit: 4,751
                                                                                                                                                RAC: 0
                                                                                                                                                Message 4520 - Posted 24 Jan 2009 3:31:43 UTC

                                                                                                                                                  Feel1st, others: could you consider running a similar test to the one I have described above?

                                                                                                                                                  Or maybe you would find some further/better tests? I guess there is no better way to see if the checkpointing is really working.

                                                                                                                                                  And I am awfully sorry for my subprime English ;) - it's 4:30 a.m. here and I am rushing to my bed. :)

                                                                                                                                                  Profile Paul D. Buck

                                                                                                                                                  Joined: Jan 14 09
                                                                                                                                                  Posts: 62
                                                                                                                                                  ID: 5139
                                                                                                                                                  Credit: 33,293
                                                                                                                                                  RAC: 0
                                                                                                                                                  Message 4521 - Posted 24 Jan 2009 4:21:08 UTC

                                                                                                                                                    Last modified: 24 Jan 2009 5:14:33 UTC

                                                                                                                                                    Graphics ...

                                                                                                                                                    Well, for me on the Mac Pro (Intel) trying the graphics got the system start to initialize and then die... never got the graphics window up ... I don't have a Rosetta task in work at the moment, but I am pretty sure that 1.47 the graphics does work on OS-X ... does not for me with 1.52 (maybe I need 1.53? Well, I will see when the other tasks become available and I can run them and try the grpahics ...

                                                                                                                                                    Could it be becuase I am running GPU Grid on my other computers and the Mac is jealous?

                                                                                                                                                    {edit add}
                                                                                                                                                    It looks like the task that failed for graphics did complete...

                                                                                                                                                    But this task failed for unknown model name ...

                                                                                                                                                    Not sure why ... and I just caught it ...


                                                                                                                                                    Initialization complete.
                                                                                                                                                    Watchdog active.

                                                                                                                                                    ERROR: unknown model name: 1DK8A_1
                                                                                                                                                    ERROR:: Exit from: src/protocols/abinitio/PairingStatistics.hh line: 170
                                                                                                                                                    called boinc_finish


                                                                                                                                                    ____________

                                                                                                                                                    Profile feet1st

                                                                                                                                                    Joined: Mar 7 06
                                                                                                                                                    Posts: 312
                                                                                                                                                    ID: 1028
                                                                                                                                                    Credit: 110,522
                                                                                                                                                    RAC: 0
                                                                                                                                                    Message 4522 - Posted 24 Jan 2009 5:54:35 UTC

                                                                                                                                                      Last modified: 24 Jan 2009 5:56:02 UTC

                                                                                                                                                      Yep, I was afraid of that...

                                                                                                                                                      1/23/2009 11:12:31 PM|ralph@home|Computation for task 1BJ1.mrtmdock.pdb_simpledocking.xml_3_8_6906_1_0 finished
                                                                                                                                                      1/23/2009 11:12:31 PM|ralph@home|Output file 1BJ1.mrtmdock.pdb_simpledocking.xml_3_8_6906_1_0_0 for task 1BJ1.mrtmdock.pdb_simpledocking.xml_3_8_6906_1_0 exceeds size limit.
                                                                                                                                                      1/23/2009 11:12:31 PM|ralph@home|File size: 30524004.000000 bytes. Limit: 25000000.000000 bytes


                                                                                                                                                      1BJ1.mrtmdock.pdb_simpledocking.xml_3_8_6906_1_0

                                                                                                                                                      The file was too large, so it calls it a compute error. Looks like it doesn't actually send the file when it grows too large. Too bad. It was 24 seconds short of 24hrs when it "failed", apparently writing the last model to the file.

                                                                                                                                                      Oh well... THIS is why I chose to run 24hr tasks on Ralph. To reveal these things. On Rosetta I do it because I like a tidy task list, and to minimize hits to the server.
                                                                                                                                                      ____________

                                                                                                                                                      Profile Paul D. Buck

                                                                                                                                                      Joined: Jan 14 09
                                                                                                                                                      Posts: 62
                                                                                                                                                      ID: 5139
                                                                                                                                                      Credit: 33,293
                                                                                                                                                      RAC: 0
                                                                                                                                                      Message 4523 - Posted 24 Jan 2009 6:13:44 UTC

                                                                                                                                                        I just chedcked a 1.53 task and the graphics window came up and seemed to post updates ... too tired to look too hard at it (sorry), but it does look like what I saw with 1.52 is fixed in 1.53 (or was it the task?) ...
                                                                                                                                                        ____________

                                                                                                                                                        Profile Paul D. Buck

                                                                                                                                                        Joined: Jan 14 09
                                                                                                                                                        Posts: 62
                                                                                                                                                        ID: 5139
                                                                                                                                                        Credit: 33,293
                                                                                                                                                        RAC: 0
                                                                                                                                                        Message 4524 - Posted 24 Jan 2009 6:16:04 UTC - in response to Message 4522.

                                                                                                                                                          Yep, I was afraid of that...

                                                                                                                                                          1/23/2009 11:12:31 PM|ralph@home|Computation for task 1BJ1.mrtmdock.pdb_simpledocking.xml_3_8_6906_1_0 finished
                                                                                                                                                          1/23/2009 11:12:31 PM|ralph@home|Output file 1BJ1.mrtmdock.pdb_simpledocking.xml_3_8_6906_1_0_0 for task 1BJ1.mrtmdock.pdb_simpledocking.xml_3_8_6906_1_0 exceeds size limit.
                                                                                                                                                          1/23/2009 11:12:31 PM|ralph@home|File size: 30524004.000000 bytes. Limit: 25000000.000000 bytes


                                                                                                                                                          1BJ1.mrtmdock.pdb_simpledocking.xml_3_8_6906_1_0

                                                                                                                                                          The file was too large, so it calls it a compute error. Looks like it doesn't actually send the file when it grows too large. Too bad. It was 24 seconds short of 24hrs when it "failed", apparently writing the last model to the file.

                                                                                                                                                          Oh well... THIS is why I chose to run 24hr tasks on Ralph. To reveal these things. On Rosetta I do it because I like a tidy task list, and to minimize hits to the server.


                                                                                                                                                          Hmmm, shouldn't that be a test? When it writes to the file it should check to see if it is hearing the limit, and if so, stop ... I mean, 24 hours worth of work down the tubes other than we discovered the error ... (thank you) ...



                                                                                                                                                          ____________

                                                                                                                                                          Profile Ian_D

                                                                                                                                                          Joined: Feb 16 06
                                                                                                                                                          Posts: 16
                                                                                                                                                          ID: 321
                                                                                                                                                          Credit: 39,518
                                                                                                                                                          RAC: 0
                                                                                                                                                          Message 4525 - Posted 24 Jan 2009 9:40:03 UTC

                                                                                                                                                            version 1.53

                                                                                                                                                            http://ralph.bakerlab.org/result.php?resultid=1263355

                                                                                                                                                            <core_client_version>6.4.5</core_client_version>
                                                                                                                                                            <![CDATA[
                                                                                                                                                            <message>
                                                                                                                                                            - exit code -1073741819 (0xc0000005)
                                                                                                                                                            </message>
                                                                                                                                                            <stderr_txt>
                                                                                                                                                            BOINC:: Initializing ... ok.
                                                                                                                                                            [2009- 1-24 8:50:37:] :: BOINC :: boinc_init()
                                                                                                                                                            BOINC:: Setting up shared resources ... ok.
                                                                                                                                                            BOINC:: Setting up semaphores ... ok.
                                                                                                                                                            BOINC:: Updating status ... ok.
                                                                                                                                                            BOINC:: Registering timer callback... ok.
                                                                                                                                                            BOINC:: Worker initialized successfully.
                                                                                                                                                            Registering options..
                                                                                                                                                            Registered extra options.
                                                                                                                                                            Initializing core...
                                                                                                                                                            Initializing options.... ok
                                                                                                                                                            Loaded options.... ok
                                                                                                                                                            Processed options.... ok
                                                                                                                                                            Initializing random generators... ok
                                                                                                                                                            Initialization complete.
                                                                                                                                                            Setting WU description ...
                                                                                                                                                            Unpacking zip data: ../../projects/ralph.bakerlab.org/minirosetta_database_rev26003.zip
                                                                                                                                                            <unzip> <-oq> <../../projects/ralph.bakerlab.org/minirosetta_database_rev26003.zip> <-d> <./>
                                                                                                                                                            Firstarg=true; pp=-d
                                                                                                                                                            first check it's not -dexdir
                                                                                                                                                            firstarg: <./>
                                                                                                                                                            End of unzipping.
                                                                                                                                                            Unpacking WU data ...
                                                                                                                                                            Unpacking data: ../../projects/ralph.bakerlab.org/testC_cc2_1_8_mammoth_mix_cen_cst.foldcst_chunk_general_mammoth_cst.t290_.mtyka.boinc_files.zip
                                                                                                                                                            <unzip> <-oq> <../../projects/ralph.bakerlab.org/testC_cc2_1_8_mammoth_mix_cen_cst.foldcst_chunk_general_mammoth_cst.t290_.mtyka.boinc_files.zip>
                                                                                                                                                            End of unzipping.
                                                                                                                                                            Setting database description ...
                                                                                                                                                            Setting up checkpointing ...
                                                                                                                                                            Setting up folding (abrelax) ...
                                                                                                                                                            Beginning folding (abrelax) ...
                                                                                                                                                            BOINC:: Worker startup.
                                                                                                                                                            Starting watchdog...
                                                                                                                                                            Watchdog active.
                                                                                                                                                            Starting work on structure: _1CYNA_10_00001


                                                                                                                                                            Unhandled Exception Detected...

                                                                                                                                                            - Unhandled Exception Record -
                                                                                                                                                            Reason: Access Violation (0xc0000005) at address 0x00572136 read attempt to address 0xAB9812D1

                                                                                                                                                            Engaging BOINC Windows Runtime Debugger...



                                                                                                                                                            ********************


                                                                                                                                                            BOINC Windows Runtime Debugger Version 6.5.0


                                                                                                                                                            Dump Timestamp : 01/24/09 09:25:27
                                                                                                                                                            Install Directory : D:\Program Files\BOINC\
                                                                                                                                                            Data Directory : D:\ProgramData\BOINC
                                                                                                                                                            Project Symstore : http://boinc.bakerlab.org/rosetta/symstore
                                                                                                                                                            Loaded Library : D:\Program Files\BOINC\\dbghelp.dll
                                                                                                                                                            Loaded Library : D:\Program Files\BOINC\\symsrv.dll
                                                                                                                                                            Loaded Library : D:\Program Files\BOINC\\srcsrv.dll
                                                                                                                                                            LoadLibraryA( D:\Program Files\BOINC\\version.dll ): GetLastError = 126
                                                                                                                                                            Loaded Library : version.dll
                                                                                                                                                            Debugger Engine : 4.0.5.0
                                                                                                                                                            Symbol Search Path: D:\ProgramData\BOINC\slots\1;D:\ProgramData\BOINC\projects\ralph.bakerlab.org;srv*D:\ProgramData\BOINC\projects\ralph.bakerlab.orgsymbols*http://msdl.microsoft.com/download/symbols;srv*D:\ProgramData\BOINC\projects\ralph.bakerlab.orgsymbols*http://boinc.bakerlab.org/rosetta/symstore;srv*D:\ProgramData\BOINC\projects\ralph.bakerlab.orgsymbols*http://boinc.berkeley.edu/symstore



                                                                                                                                                            ____________

                                                                                                                                                            Aegis Maelstrom

                                                                                                                                                            Joined: Jan 19 09
                                                                                                                                                            Posts: 12
                                                                                                                                                            ID: 5159
                                                                                                                                                            Credit: 4,751
                                                                                                                                                            RAC: 0
                                                                                                                                                            Message 4526 - Posted 24 Jan 2009 10:17:07 UTC

                                                                                                                                                              ...and now something completely different.

                                                                                                                                                              Mike, do you need more information about actual behaviour of your predicting models? Just to fine tune the energy function, folding procedure etc.?

                                                                                                                                                              One thing we all now is that some models of particular WU take much more time than the others. It's clearly seen when we have "long running WUs", however it is quite often as well in WUs which have very short models. In these cases you need to actually watch it or have good logs to see, that, i.e. most of models run in 5 minutes, and one takes 15-20.

                                                                                                                                                              I've been having such a task right now: testC_cc_1_8_nocst4_hb_t288__IGNORE_THE_REST_2FNEA_2_6976_1_0.
                                                                                                                                                              For many minutes the accepted screen was showing a very simple chain split on two, longer and shorter, and the shorter was standing still, showing some quite folded structure. It took over 400,000 steps to get something semifolded and more complicated on a "accepted" screen and go further.
                                                                                                                                                              This model (12th) took much more time and as the forecast runtime of a model increased, it was the last decoy allowed by the scheduler.

                                                                                                                                                              All of these looked like the procedure needed much more time to hit something acceptable to start with.

                                                                                                                                                              I'm sure you know about this issue but I have seen such a behaviour a couple times before and just wanted to let you know it is quite common.

                                                                                                                                                              P.S. Besided that everything - except checkpointing, see my posts above - is working like a charm.

                                                                                                                                                              P.S.S. Great work Feel1st!

                                                                                                                                                              Path7

                                                                                                                                                              Joined: Feb 11 08
                                                                                                                                                              Posts: 56
                                                                                                                                                              ID: 4036
                                                                                                                                                              Credit: 4,974
                                                                                                                                                              RAC: 0
                                                                                                                                                              Message 4528 - Posted 24 Jan 2009 12:38:47 UTC - in response to Message 4516.

                                                                                                                                                                Hello mtyka,

                                                                                                                                                                - Do the graphics behave properly again ?

                                                                                                                                                                After having problems with the graphics running minrosetta 1.52,
                                                                                                                                                                the combination of minrosetta 1.53 & minrosetta graphics 1.53 seems to work smoothly again.
                                                                                                                                                                Thanks and a good weekend,
                                                                                                                                                                Path7.

                                                                                                                                                                ramostol

                                                                                                                                                                Joined: Mar 29 07
                                                                                                                                                                Posts: 24
                                                                                                                                                                ID: 2840
                                                                                                                                                                Credit: 31,121
                                                                                                                                                                RAC: 0
                                                                                                                                                                Message 4529 - Posted 24 Jan 2009 12:55:36 UTC

                                                                                                                                                                  After Rosetta 1.47 totally crashing on my MacBook on 27 Dec 2008 (after one week of faultless computing, within hours after connecting to the Rosetta server; still haven't recovered) I have waited for an opportunity to test what is happening on Ralph. At last I received two 1.53-tasks - however, they show exactly the same symptoms as observed using 1.47 on Rosetta:

                                                                                                                                                                  abinitio_norelax_homfrag_natfrag_129_B_1pxuA_SAVE_ALL_OUT_7037_1_0

                                                                                                                                                                  <![CDATA[
                                                                                                                                                                  <message>
                                                                                                                                                                  process exited with code 193 (0xc1, -63)
                                                                                                                                                                  </message>

                                                                                                                                                                  This one crashed after 30 sec.

                                                                                                                                                                  ----

                                                                                                                                                                  abinitio_norelax_homfrag_natfrag_129_B_1ctf__SAVE_ALL_OUT_7037_1_0

                                                                                                                                                                  <![CDATA[
                                                                                                                                                                  <message>
                                                                                                                                                                  process exited with code 193 (0xc1, -63)
                                                                                                                                                                  </message>
                                                                                                                                                                  <stderr_txt>

                                                                                                                                                                  This one is reported using Boinc 6.6.2 but computed with 6.6.1.

                                                                                                                                                                  It started out relatively normally. But after about one hour’s computing time, steadily increasing "To completion" time, progress stuck at 0.240%, I had to conclude that this wu would turn into an everlasting task. "Show graphics" did not respond, even "Quit Boinc" was greyed out. After rebooting to install Boinc 6.6.2, and logging on as administrative and non-administrative user the wu quit after restarting using 32 sec.


                                                                                                                                                                  I should add that I mostly use this computer as a non-administrator, which may have some bearing (on my PPC it influences my graphics display at least). However, I do not receive more units to check, so I can not test the behaviour when changing configurations.

                                                                                                                                                                  ----

                                                                                                                                                                  PS
                                                                                                                                                                  For the curious:
                                                                                                                                                                  This is the report from the first wu crashing on Dec 27 2008 on Rosetta (I haven't too much time analyzing such data....). The error message is a bit more extensive than the messages for the rest of the bunch:

                                                                                                                                                                  http://boinc.bakerlab.org/rosetta/result.php?resultid=216101560 (obsolete link)

                                                                                                                                                                  Task ID
                                                                                                                                                                  216101560
                                                                                                                                                                  Name
                                                                                                                                                                  1r9pA_BOINC_MPZN_vanilla_abrelax_5901_6648_0
                                                                                                                                                                  Workunit
                                                                                                                                                                  196949149
                                                                                                                                                                  Created
                                                                                                                                                                  21 Dec 2008 12:53:24 UTC
                                                                                                                                                                  Sent
                                                                                                                                                                  21 Dec 2008 13:16:26 UTC
                                                                                                                                                                  Received
                                                                                                                                                                  27 Dec 2008 14:15:44 UTC
                                                                                                                                                                  Server state
                                                                                                                                                                  Over
                                                                                                                                                                  Outcome
                                                                                                                                                                  Client error
                                                                                                                                                                  Client state
                                                                                                                                                                  Compute error
                                                                                                                                                                  Exit status
                                                                                                                                                                  193 (0xc1)
                                                                                                                                                                  Computer ID
                                                                                                                                                                  936507
                                                                                                                                                                  Report deadline
                                                                                                                                                                  31 Dec 2008 13:16:26 UTC
                                                                                                                                                                  CPU time
                                                                                                                                                                  20.36099
                                                                                                                                                                  stderr out
                                                                                                                                                                  <core_client_version>6.5.0</core_client_version>
                                                                                                                                                                  <![CDATA[
                                                                                                                                                                  <message>
                                                                                                                                                                  process exited with code 193 (0xc1, -63)
                                                                                                                                                                  </message>
                                                                                                                                                                  <stderr_txt>
                                                                                                                                                                  minirosetta_1.47_i686-apple-darwin(69428,0xa031b720) malloc: *** error for object 0x1748220: Non-aligned pointer being freed (2)
                                                                                                                                                                  *** set a breakpoint in malloc_error_break to debug
                                                                                                                                                                  minirosetta_1.47_i686-apple-darwin(69428,0xa031b720) malloc: *** error for object 0x1748220: incorrect checksum for freed object - object was probably modified after being freed.

                                                                                                                                                                  *** set a breakpoint in malloc_error_break to debug
                                                                                                                                                                  SIGBUS: bus error

                                                                                                                                                                  Crashed executable name: minirosetta_1.47_i686-apple-darwin
                                                                                                                                                                  built using BOINC library version 6.5.0
                                                                                                                                                                  Machine type Intel 80486 (32-bit executable)
                                                                                                                                                                  System version: Macintosh OS 10.5.6 build 9G55
                                                                                                                                                                  Sat Dec 27 14:45:29 2008

                                                                                                                                                                  Profile robertmiles

                                                                                                                                                                  Joined: Jan 13 09
                                                                                                                                                                  Posts: 80
                                                                                                                                                                  ID: 5137
                                                                                                                                                                  Credit: 246,177
                                                                                                                                                                  RAC: 24
                                                                                                                                                                  Message 4530 - Posted 24 Jan 2009 14:48:54 UTC - in response to Message 4502.


                                                                                                                                                                    Why not grab the set and issue them to us with high replication counts ... then if they fail for everyone, that tells us one thing ... if they only fail for some and not others ... that tells us something else ...


                                                                                                                                                                    we already submit identical jobs to many many machines - and they only fail *sometimes*. Even when the same random number seed is used.



                                                                                                                                                                    To my mind, the cases that fail are the ones that you should be saving and using as your issue tasks for each round of testing. Not to keep making new tasks in all cases, but to use those tasks that have proven their ability to cause a problem.


                                                                                                                                                                    again .. we're already doing that. But the errors are sporadic in nature so submitting the same task might not throw an error "this time".

                                                                                                                                                                    I have noticed though that some of the problems seem to be highly machine specific. Like one user will always produce this one kind of an error and nithing else. weired. something strange about their setup ? No idea.


                                                                                                                                                                    You might want to try classifying such failures by the specific type of CPU they ran on, and what operating system they ran under. At least one other BOINC project I'm participating in found at least one problem where the results came out one way on Intel CPUs, and a different way on AMD CPUs. And more than one have found problems where workunits come out OK if run under one operating system, but not if they're run under another operating system.

                                                                                                                                                                    Profile robertmiles

                                                                                                                                                                    Joined: Jan 13 09
                                                                                                                                                                    Posts: 80
                                                                                                                                                                    ID: 5137
                                                                                                                                                                    Credit: 246,177
                                                                                                                                                                    RAC: 24
                                                                                                                                                                    Message 4531 - Posted 24 Jan 2009 14:54:50 UTC - in response to Message 4488.

                                                                                                                                                                      Should the name of this thread be editted to include 1.51, or should a new thread be started for 1.51?


                                                                                                                                                                      Same for 1.53, the version used in the last workunit I got. Also, this could be reflected in the description of the latest version being tested on the home page.

                                                                                                                                                                      Profile robertmiles

                                                                                                                                                                      Joined: Jan 13 09
                                                                                                                                                                      Posts: 80
                                                                                                                                                                      ID: 5137
                                                                                                                                                                      Credit: 246,177
                                                                                                                                                                      RAC: 24
                                                                                                                                                                      Message 4532 - Posted 24 Jan 2009 15:25:44 UTC - in response to Message 4503.

                                                                                                                                                                        Last modified: 24 Jan 2009 16:12:26 UTC

                                                                                                                                                                        Thank you all for reporting the problems! The problem divides it self into three parts, firstly cornering the error, turning it into an erro message, reproducing it here and then finding the root of the problem.


                                                                                                                                                                        <core_client_version>6.2.19</core_client_version>
                                                                                                                                                                        <![CDATA[
                                                                                                                                                                        <message>
                                                                                                                                                                        There are no child processes to wait for. (0x80) - exit code 128 (0x80)
                                                                                                                                                                        </message>
                                                                                                                                                                        <stderr_txt>
                                                                                                                                                                        BOINC:: Initializing ... ok.
                                                                                                                                                                        [2009- 1-22 4:45: 1:] :: BOINC :: boinc_init()
                                                                                                                                                                        BOINC:: Setting up shared resources ... ok.
                                                                                                                                                                        BOINC:: Setting up semaphores ... ok.
                                                                                                                                                                        BOINC:: Updating status ... ok.
                                                                                                                                                                        BOINC:: Registering timer callback... ok.
                                                                                                                                                                        BOINC:: Worker initialized successfully.


                                                                                                                                                                        Unhandled Exception Detected...

                                                                                                                                                                        - Unhandled Exception Record -
                                                                                                                                                                        Reason: Access Violation (0xc0000005) at address 0x7C81BD02 read attempt to address 0x00000002

                                                                                                                                                                        Engaging BOINC Windows Runtime Debugger...



                                                                                                                                                                        Unhandled Exception Detected...

                                                                                                                                                                        - Unhandled Exception Record -
                                                                                                                                                                        Reason: Access Violation (0xc0000005) at address 0x7C81BD02 read attempt to address 0x00000002

                                                                                                                                                                        Engaging BOINC Windows Runtime Debugger...


                                                                                                                                                                        </stderr_txt>
                                                                                                                                                                        ]]>




                                                                                                                                                                        These are random segfault that occured somewhere. I can do virtually nothing from here. If i rerun the *exact same commandline* from here all runs fine. So these ones i can only tentatively bracket with stderr statemenets.


                                                                                                                                                                        I wonder if a significant number of these are problems with restarting after suspending the task, so if you run them on a machine that doesn't participate in enough other BOINC projects to even cause the suspension, the code with problems won't even be used?

                                                                                                                                                                        Also, have you checked if this problem is restricted to certain BOINC versions, but not certain others?

                                                                                                                                                                        Also, it may matter whether the Leave In Memory option is used, and thus whether the restart is from what is left in memory or from a checkpoint.

                                                                                                                                                                        On my machine, I found that minirosetta ran better after I enabled Leave In Memory, increased the upper limit on how much disk space BOINC could use, and increased the percentage of the available swap space BOINC could use. The new upper limits are high enough to suggest that BOINC divides the available disk space and swap space equally between BOINC projects before deciding how much to give to each workunit, instead of letting the more demanding BOINC projects have more than an equal share if they need it. This does have a side effect, though - I'll probably need to run about 10 workunits of any new minirosetta version to have enough results to guess whether it is better than 1.47, unless it's significantly worse instead.

                                                                                                                                                                        Profile robertmiles

                                                                                                                                                                        Joined: Jan 13 09
                                                                                                                                                                        Posts: 80
                                                                                                                                                                        ID: 5137
                                                                                                                                                                        Credit: 246,177
                                                                                                                                                                        RAC: 24
                                                                                                                                                                        Message 4533 - Posted 24 Jan 2009 16:41:02 UTC - in response to Message 4513.

                                                                                                                                                                          Last modified: 24 Jan 2009 16:53:46 UTC

                                                                                                                                                                          Yeah guess what. I just found a bug in the BOINC API! . Holy crap. Basically as far as i can see there's a memory leak when it's trying to unzip files. Mostly all you see is the application dying kicking and screaming just after initialization. One RALPH user though produced a suspicious trace (thank you philip in hongkong!). I have to stress that this is the only job out of hundreds such failures that has returned with a trace.

                                                                                                                                                                          It fails in the unzip code ! OMG.
                                                                                                                                                                          Been tinkering with the code, the bug probably stems froma single byte not being set to 0. This explains the sporadic nature - if the relevant byte happens to already be 0 then all is fine. I'll push out a version soon to see if the fix works. Its all stipulation at this point.

                                                                                                                                                                          Mike


                                                                                                                                                                          Is the place with the bad code part of BOINC, instead of part of minirosetta? If it is, I'd expect a number of variations of the fix to be needed, depending on which version of BOINC is used.

                                                                                                                                                                          Profile robertmiles

                                                                                                                                                                          Joined: Jan 13 09
                                                                                                                                                                          Posts: 80
                                                                                                                                                                          ID: 5137
                                                                                                                                                                          Credit: 246,177
                                                                                                                                                                          RAC: 24
                                                                                                                                                                          Message 4534 - Posted 24 Jan 2009 17:18:19 UTC

                                                                                                                                                                            Last modified: 24 Jan 2009 17:32:26 UTC

                                                                                                                                                                            Could some of your problems with RALPH@home be due to it looking like an alpha test project, thus driving away participants willing to participate in beta test projects but not alpha test projects?

                                                                                                                                                                            I looked down the list of top participants, and found that I entered this list within days after adding RALPH@home to my list of BOINC projects, and not even giving it a very high percentage of the available CPU time on my machine. It then rose rapidly in the list the next few days. Is this due to the current updates being sent only to machines of specific types, or due to a very small number of people actually currently trying to be active in RALPH@home (I'd guess around 500)?

                                                                                                                                                                            At least the results I'm getting from RALPH@home don't look significantly worse than I'm getting at Rosetta@home, so I'll stay active for now.

                                                                                                                                                                            I've seen some other BOINC projects offer a "test" option for running workunits that are between their beta tests and their final release, but need a higher volume of workunits completed to check for performance problems in any new sections of the code used. Are you interested in adding this option on Rosetta@home, to restrict some of the testing done on Rosetta@home to people willing to tolerate a high bug level, and thus help attract more participants that object to the high bug level currently seen on Rosetta@home? You could even enable this option for users who haven't tried to set it at all, not not for new participants, if you're worried about getting enough participants willing to use it.

                                                                                                                                                                            Profile feet1st

                                                                                                                                                                            Joined: Mar 7 06
                                                                                                                                                                            Posts: 312
                                                                                                                                                                            ID: 1028
                                                                                                                                                                            Credit: 110,522
                                                                                                                                                                            RAC: 0
                                                                                                                                                                            Message 4535 - Posted 24 Jan 2009 17:29:25 UTC - in response to Message 4524.

                                                                                                                                                                              Last modified: 24 Jan 2009 17:34:20 UTC


                                                                                                                                                                              Hmmm, shouldn't that be a test? When it writes to the file it should check to see if it is hearing the limit, and if so, stop ... I mean, 24 hours worth of work down the tubes other than we discovered the error ... (thank you) ...

                                                                                                                                                                              Paul, I think you are saying "shouldn't it have detected the problem much sooner?". It dawned on me later looking at how far over the file size was, that there must be much more to it then just the last model. It was actually wrapping up the entire task when it failed.

                                                                                                                                                                              There are file size limits in BOINC, and it does check them as the task runs (so far as I know anyway). But this one may not have been detected because I'll bet the last thing the task did was zip it all up to send in. So, there was no way to know the final size until the compression completed.

                                                                                                                                                                              But yes, seems to indicate that the task is producing too many models. And perhaps that there is some intermediate file size that could be further limited to assure the compressed size does not exceed the limit there. The other way around it would be to limit the number of models the task is allowed to produce. But this would mess up my runtime preference, and my future requests for work, because it would end significantly sooner then my 24hr preference.

                                                                                                                                                                              Actually, this would be another good factor for the watchdog to keep tabs on. (or the logic about whether to start the next model) If he sees a problem, then he could get the current work reported rather then losing it.
                                                                                                                                                                              ____________

                                                                                                                                                                              Profile Ian_D

                                                                                                                                                                              Joined: Feb 16 06
                                                                                                                                                                              Posts: 16
                                                                                                                                                                              ID: 321
                                                                                                                                                                              Credit: 39,518
                                                                                                                                                                              RAC: 0
                                                                                                                                                                              Message 4536 - Posted 24 Jan 2009 17:30:12 UTC

                                                                                                                                                                                Is this due to the current updates being sent only to machines of specific types, or due to a very small number of people actually currently trying to be active in RALPH@home (I'd guess around 500)?


                                                                                                                                                                                Here's some numbers

                                                                                                                                                                                BOINC Stats for RALPH

                                                                                                                                                                                642 active users out of 3073

                                                                                                                                                                                HTH


                                                                                                                                                                                ____________

                                                                                                                                                                                Profile Paul D. Buck

                                                                                                                                                                                Joined: Jan 14 09
                                                                                                                                                                                Posts: 62
                                                                                                                                                                                ID: 5139
                                                                                                                                                                                Credit: 33,293
                                                                                                                                                                                RAC: 0
                                                                                                                                                                                Message 4537 - Posted 24 Jan 2009 19:45:06 UTC - in response to Message 4535.

                                                                                                                                                                                  [quote]
                                                                                                                                                                                  Hmmm, shouldn't that be a test? When it writes to the file it should check to see if it is hearing the limit, and if so, stop ... I mean, 24 hours worth of work down the tubes other than we discovered the error ... (thank you) ...

                                                                                                                                                                                  Paul, I think you are saying "shouldn't it have detected the problem much sooner?". It dawned on me later looking at how far over the file size was, that there must be much more to it then just the last model. It was actually wrapping up the entire task when it failed./quote]

                                                                                                                                                                                  Yes that is what I was trying to say. Again, not being aware of the exact circumstances and what stage you were in at the time of the issue, it seems that the system should have been more graceful. If it was trying to wrap up the model at the end is it possible that it should have been able to make several upload files rather than just one?

                                                                                                                                                                                  There are file size limits in BOINC, and it does check them as the task runs (so far as I know anyway). But this one may not have been detected because I'll bet the last thing the task did was zip it all up to send in. So, there was no way to know the final size until the compression completed./quote]

                                                                                                                                                                                  True, but you can make estimates based on input sizes and normal compression ratios. And if the system errors, back off and try smaller sets ... again, you did try to press the system and found an issue that needs some thinking ... after all it is *MY* understanding that they want us to run the tasks for longer periods (my setting on Rosetta is 6 hours) to get more attempts per issued task ...

                                                                                                                                                                                  [quote]But yes, seems to indicate that the task is producing too many models. And perhaps that there is some intermediate file size that could be further limited to assure the compressed size does not exceed the limit there. The other way around it would be to limit the number of models the task is allowed to produce. But this would mess up my runtime preference, and my future requests for work, because it would end significantly sooner then my 24hr preference./quote]

                                                                                                                                                                                  Or they would package the end data differently. Again, you stressed the system in a way they likely did not expect.

                                                                                                                                                                                  [quote]Actually, this would be another good factor for the watchdog to keep tabs on. (or the logic about whether to start the next model) If he sees a problem, then he could get the current work reported rather then losing it.


                                                                                                                                                                                  Agreed.
                                                                                                                                                                                  ____________

                                                                                                                                                                                  mtyka
                                                                                                                                                                                  Forum moderator
                                                                                                                                                                                  Project developer
                                                                                                                                                                                  Project scientist

                                                                                                                                                                                  Joined: Mar 19 08
                                                                                                                                                                                  Posts: 79
                                                                                                                                                                                  ID: 4144
                                                                                                                                                                                  Credit: 0
                                                                                                                                                                                  RAC: 0
                                                                                                                                                                                  Message 4538 - Posted 24 Jan 2009 19:52:41 UTC - in response to Message 4535.


                                                                                                                                                                                    Hmmm, shouldn't that be a test? When it writes to the file it should check to see if it is hearing the limit, and if so, stop ... I mean, 24 hours worth of work down the tubes other than we discovered the error ... (thank you) ...

                                                                                                                                                                                    Paul, I think you are saying "shouldn't it have detected the problem much sooner?". It dawned on me later looking at how far over the file size was, that there must be much more to it then just the last model. It was actually wrapping up the entire task when it failed.

                                                                                                                                                                                    There are file size limits in BOINC, and it does check them as the task runs (so far as I know anyway). But this one may not have been detected because I'll bet the last thing the task did was zip it all up to send in. So, there was no way to know the final size until the compression completed.

                                                                                                                                                                                    But yes, seems to indicate that the task is producing too many models. And perhaps that there is some intermediate file size that could be further limited to assure the compressed size does not exceed the limit there. The other way around it would be to limit the number of models the task is allowed to produce. But this would mess up my runtime preference, and my future requests for work, because it would end significantly sooner then my 24hr preference.

                                                                                                                                                                                    Actually, this would be another good factor for the watchdog to keep tabs on. (or the logic about whether to start the next model) If he sees a problem, then he could get the current work reported rather then losing it.


                                                                                                                                                                                    currently there is no monitoring of that. I'll look into putting safety stop on that. I'm not even sure how to get that info from the boinc api.

                                                                                                                                                                                    Profile Paul D. Buck

                                                                                                                                                                                    Joined: Jan 14 09
                                                                                                                                                                                    Posts: 62
                                                                                                                                                                                    ID: 5139
                                                                                                                                                                                    Credit: 33,293
                                                                                                                                                                                    RAC: 0
                                                                                                                                                                                    Message 4539 - Posted 24 Jan 2009 20:05:05 UTC - in response to Message 4534.

                                                                                                                                                                                      Could some of your problems with RALPH@home be due to it looking like an alpha test project, thus driving away participants willing to participate in beta test projects but not alpha test projects?


                                                                                                                                                                                      This is common for ALL BOINC projects. If you start any BOINC project you will see that you will leap up the lists if you run on that project for only a couple weeks. At some point you start to hit the level where the dedicated are accumulating work... then your rise will level off and will rise and fall depending on how many computers you have and what speed they are ...

                                                                                                                                                                                      I tend to have more machines than most people that BOINC, less than the really dedicated and so I am now solidly in the middle of the pack of participants though still rising slowly (at one point I was number 411 in the world for all projects)

                                                                                                                                                                                      But at the end of the day, test projects, Alpha or Beta needs people that will not only do the work, but are willing to report errors, try things out and run the risk of doing work and never getting "paid" for the effort because of failures.
                                                                                                                                                                                      ____________

                                                                                                                                                                                      Profile Paul D. Buck

                                                                                                                                                                                      Joined: Jan 14 09
                                                                                                                                                                                      Posts: 62
                                                                                                                                                                                      ID: 5139
                                                                                                                                                                                      Credit: 33,293
                                                                                                                                                                                      RAC: 0
                                                                                                                                                                                      Message 4540 - Posted 24 Jan 2009 20:06:32 UTC - in response to Message 4538.

                                                                                                                                                                                        currently there is no monitoring of that. I'll look into putting safety stop on that. I'm not even sure how to get that info from the boinc api.


                                                                                                                                                                                        Perhaps begging? :)
                                                                                                                                                                                        ____________

                                                                                                                                                                                        mtyka
                                                                                                                                                                                        Forum moderator
                                                                                                                                                                                        Project developer
                                                                                                                                                                                        Project scientist

                                                                                                                                                                                        Joined: Mar 19 08
                                                                                                                                                                                        Posts: 79
                                                                                                                                                                                        ID: 4144
                                                                                                                                                                                        Credit: 0
                                                                                                                                                                                        RAC: 0
                                                                                                                                                                                        Message 4541 - Posted 24 Jan 2009 22:01:40 UTC - in response to Message 4529.

                                                                                                                                                                                          After Rosetta 1.47 totally crashing on my MacBook on 27 Dec 2008 (after one week of faultless computing, within hours after connecting to the Rosetta server; still haven't recovered) I have waited for an opportunity to test what is happening on Ralph. At last I received two 1.53-tasks - however, they show exactly the same symptoms as observed using 1.47 on Rosetta:

                                                                                                                                                                                          abinitio_norelax_homfrag_natfrag_129_B_1pxuA_SAVE_ALL_OUT_7037_1_0

                                                                                                                                                                                          <message>
                                                                                                                                                                                          process exited with code 193 (0xc1, -63)
                                                                                                                                                                                          </message>
                                                                                                                                                                                          <stderr_txt>
                                                                                                                                                                                          minirosetta_1.47_i686-apple-darwin(69428,0xa031b720) malloc: *** error for object 0x1748220: Non-aligned pointer being freed (2)
                                                                                                                                                                                          *** set a breakpoint in malloc_error_break to debug
                                                                                                                                                                                          minirosetta_1.47_i686-apple-darwin(69428,0xa031b720) malloc: *** error for object 0x1748220: incorrect checksum for freed object - object was probably modified after being freed.

                                                                                                                                                                                          *** set a breakpoint in malloc_error_break to debug
                                                                                                                                                                                          SIGBUS: bus error

                                                                                                                                                                                          Crashed executable name: minirosetta_1.47_i686-apple-darwin
                                                                                                                                                                                          built using BOINC library version 6.5.0
                                                                                                                                                                                          Machine type Intel 80486 (32-bit executable)
                                                                                                                                                                                          System version: Macintosh OS 10.5.6 build 9G55
                                                                                                                                                                                          Sat Dec 27 14:45:29 2008




                                                                                                                                                                                          Hi ramostol, thanks for joining ralph. You error is new, i've not seen it on any other machine yet, but at least now we have a chance to cathc it. Shame the trace is giving so little information.

                                                                                                                                                                                          Mike

                                                                                                                                                                                          I _ quit

                                                                                                                                                                                          Joined: Jan 13 09
                                                                                                                                                                                          Posts: 44
                                                                                                                                                                                          ID: 5136
                                                                                                                                                                                          Credit: 88,562
                                                                                                                                                                                          RAC: 0
                                                                                                                                                                                          Message 4542 - Posted 25 Jan 2009 0:13:33 UTC - in response to Message 4536.

                                                                                                                                                                                            Last modified: 25 Jan 2009 0:14:05 UTC

                                                                                                                                                                                            Is this due to the current updates being sent only to machines of specific types, or due to a very small number of people actually currently trying to be active in RALPH@home (I'd guess around 500)?


                                                                                                                                                                                            Here's some numbers

                                                                                                                                                                                            BOINC Stats for RALPH

                                                                                                                                                                                            642 active users out of 3073

                                                                                                                                                                                            HTH



                                                                                                                                                                                            I got caught behind a ton of work in my buffer for Ralph and then ran up a big debt load to rosie. I have shrunk my buffer size since then. When there is work available I should get it automatically and be active in testing again.

                                                                                                                                                                                            Profile Paul D. Buck

                                                                                                                                                                                            Joined: Jan 14 09
                                                                                                                                                                                            Posts: 62
                                                                                                                                                                                            ID: 5139
                                                                                                                                                                                            Credit: 33,293
                                                                                                                                                                                            RAC: 0
                                                                                                                                                                                            Message 4543 - Posted 25 Jan 2009 22:29:50 UTC

                                                                                                                                                                                              Please mister, can I have some more?

                                                                                                                                                                                              Can't test the version with no work to do ... :)
                                                                                                                                                                                              ____________

                                                                                                                                                                                              mtyka
                                                                                                                                                                                              Forum moderator
                                                                                                                                                                                              Project developer
                                                                                                                                                                                              Project scientist

                                                                                                                                                                                              Joined: Mar 19 08
                                                                                                                                                                                              Posts: 79
                                                                                                                                                                                              ID: 4144
                                                                                                                                                                                              Credit: 0
                                                                                                                                                                                              RAC: 0
                                                                                                                                                                                              Message 4544 - Posted 25 Jan 2009 22:47:37 UTC - in response to Message 4543.

                                                                                                                                                                                                Please mister, can I have some more?

                                                                                                                                                                                                Can't test the version with no work to do ... :)


                                                                                                                                                                                                No worries - there shall be more work soon. I'm preparring another update.

                                                                                                                                                                                                I _ quit

                                                                                                                                                                                                Joined: Jan 13 09
                                                                                                                                                                                                Posts: 44
                                                                                                                                                                                                ID: 5136
                                                                                                                                                                                                Credit: 88,562
                                                                                                                                                                                                RAC: 0
                                                                                                                                                                                                Message 4547 - Posted 26 Jan 2009 10:51:31 UTC - in response to Message 4544.

                                                                                                                                                                                                  Please mister, can I have some more?

                                                                                                                                                                                                  Can't test the version with no work to do ... :)


                                                                                                                                                                                                  No worries - there shall be more work soon. I'm preparring another update.


                                                                                                                                                                                                  I just got 25 new tasks after changing my allocation percentage.

                                                                                                                                                                                                  ramostol

                                                                                                                                                                                                  Joined: Mar 29 07
                                                                                                                                                                                                  Posts: 24
                                                                                                                                                                                                  ID: 2840
                                                                                                                                                                                                  Credit: 31,121
                                                                                                                                                                                                  RAC: 0
                                                                                                                                                                                                  Message 4548 - Posted 26 Jan 2009 11:27:23 UTC - in response to Message 4541.


                                                                                                                                                                                                    *** set a breakpoint in malloc_error_break to debug
                                                                                                                                                                                                    minirosetta_1.47_i686-apple-darwin(69428,0xa031b720) malloc: *** error for object 0x1748220: incorrect checksum for freed object - object was probably modified after being freed.




                                                                                                                                                                                                    Hi ramostol, thanks for joining ralph. You error is new, i've not seen it on any other machine yet, but at least now we have a chance to cathc it. Shame the trace is giving so little information.

                                                                                                                                                                                                    Mike


                                                                                                                                                                                                    Since the computer in question is connected to the internet only in weekends I cannot be of much use without lots of wus...

                                                                                                                                                                                                    But -- in connection with modified object -- this computer isn't doing much more that computing BOINC projects at the moment, and the sole configuration made by yours truly when connecting to the Rosetta server was changing the runtime to keep within limits. Since my first (of three) Rosetta crash the last months (that one using BOINC 6.3.14) made it necessary to manually delete all project files before reattaching to get up and running, I wonder if some errors are created outside of miniRosetta itself...

                                                                                                                                                                                                    Profile feet1st

                                                                                                                                                                                                    Joined: Mar 7 06
                                                                                                                                                                                                    Posts: 312
                                                                                                                                                                                                    ID: 1028
                                                                                                                                                                                                    Credit: 110,522
                                                                                                                                                                                                    RAC: 0
                                                                                                                                                                                                    Message 4561 - Posted 27 Jan 2009 0:11:55 UTC

                                                                                                                                                                                                      Last modified: 27 Jan 2009 0:16:14 UTC

                                                                                                                                                                                                      My apologies in advance for this long post, but I thought it the best method of report. I know you wanted feedback on the checkpointing. Specifically the checkpoint buffering that I understand should be included in v1.53.

                                                                                                                                                                                                      I have this v1.53 task, it's been running for about 11.5hrs on my 24hr preference. My BOINC client confirms my "write at most..." setting is 600 seconds. This task has generated 122 checkpoints so far (as per the boinc_checkpoint_count.txt file) and is now on model 106, and perhaps so many models is the root cause, but here is what I see in the slots file.

                                                                                                                                                                                                      I believe you indicated that at the end of a model, we would see a disk write, regardless of preference and any other buffered checkpoints. But, I seem to be seeing a large number of writes in a single minute, with several file numbers.

                                                                                                                                                                                                      So, showing you the directory seems the only way... (sorted by date/time, and colorized where heavy disk periods seem to be. Some just seem to have written a large number of files, others seem to have written multiple checkpoints [if that is what the numbers are], in a single minute)



                                                                                                                                                                                                      11/18/2008 12:24 PM 138 t327_.fasta
                                                                                                                                                                                                      11/18/2008 01:19 PM 666 T0327.all0.T20.aln.filt.2F2EA_2.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 667 T0327.all0.T20.aln.filt.2F2EA_4.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 628 T0327.all0.T20.aln.filt.2F2EA_5.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 672 T0327.all0.T20.aln.filt.2F2EA_6.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 594 T0327.all0.T20.aln.filt.2F2EA_7.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 668 T0327.all0.T20.aln.filt.2F2EA_8.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 672 T0327.all0.T20.aln.filt.2F2EA_9.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 628 T0327.all0.T20.aln.filt.2F2EA_12.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 669 T0327.all0.T20.aln.filt.2F2EA_11.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 602 T0327.all0.T20.aln.filt.2F2EA_10.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 700 T0327.all0.T20.aln.filt.1UB9A_2.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 601 T0327.all0.T20.aln.filt.2F2EA_15.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 671 T0327.all0.T20.aln.filt.2F2EA_14.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 676 T0327.all0.T20.aln.filt.2F2EA_16.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 646 T0327.all0.T20.aln.filt.1UB9A_5.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 696 T0327.all0.T20.aln.filt.1UB9A_6.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 670 T0327.all0.T20.aln.filt.1UB9A_10.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 697 T0327.all0.T20.aln.filt.1UB9A_8.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 694 T0327.all0.T20.aln.filt.1UB9A_13.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 636 T0327.all0.T20.aln.filt.1XMAA_10.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 636 T0327.all0.T20.aln.filt.1XMAA_13.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 675 T0327.all0.T20.aln.filt.1XMAA_11.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 682 T0327.all0.T20.aln.filt.1XMAA_14.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 659 T0327.all0.T20.aln.filt.1XMAA_15.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 683 T0327.all0.T20.aln.filt.1XMAA_16.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 645 T0327.all0.T20.aln.filt.1XMAA_17.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 595 T0327.all0.T20.aln.filt.1YYVA_3.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 674 T0327.all0.T20.aln.filt.1YYVA_6.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 637 T0327.all0.T20.aln.filt.1YYVA_5.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 671 T0327.all0.T20.aln.filt.1YYVA_8.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 612 T0327.all0.T20.aln.filt.1YYVA_7.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 678 T0327.all0.T20.aln.filt.1YYVA_10.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 612 T0327.all0.T20.aln.filt.1YYVA_9.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 677 T0327.all0.T20.aln.filt.1YYVA_14.mapping
                                                                                                                                                                                                      11/18/2008 01:19 PM 676 T0327.all0.T20.aln.filt.1YYVA_12.mapping
                                                                                                                                                                                                      11/18/2008 01:20 PM 637 T0327.all0.T20.aln.filt.2FSWA_1.mapping
                                                                                                                                                                                                      11/18/2008 01:20 PM 604 T0327.all0.T20.aln.filt.2FSWA_3.mapping
                                                                                                                                                                                                      11/18/2008 01:20 PM 653 T0327.all0.T20.aln.filt.2FSWA_2.mapping
                                                                                                                                                                                                      11/18/2008 01:20 PM 674 T0327.all0.T20.aln.filt.2FSWA_5.mapping
                                                                                                                                                                                                      11/18/2008 01:20 PM 691 T0327.all0.T20.aln.filt.2FSWA_4.mapping
                                                                                                                                                                                                      11/18/2008 01:20 PM 612 T0327.all0.T20.aln.filt.2FSWA_6.mapping
                                                                                                                                                                                                      11/18/2008 01:20 PM 671 T0327.all0.T20.aln.filt.2FSWA_7.mapping
                                                                                                                                                                                                      11/18/2008 01:20 PM 674 T0327.all0.T20.aln.filt.2FSWA_8.mapping
                                                                                                                                                                                                      11/18/2008 01:20 PM 678 T0327.all0.T20.aln.filt.2FSWA_9.mapping
                                                                                                                                                                                                      11/18/2008 01:20 PM 587 T0327.all0.T20.aln.filt.2FSWA_10.mapping
                                                                                                                                                                                                      11/18/2008 01:20 PM 670 T0327.all0.T20.aln.filt.2FSWA_11.mapping
                                                                                                                                                                                                      11/18/2008 01:20 PM 566 T0327.all0.T20.aln.filt.2FSWA_12.mapping
                                                                                                                                                                                                      11/18/2008 01:20 PM 601 T0327.all0.T20.aln.filt.2FSWA_13.mapping
                                                                                                                                                                                                      11/18/2008 01:20 PM 683 T0327.all0.T20.aln.filt.2ESHA_14.mapping
                                                                                                                                                                                                      11/18/2008 01:21 PM 663 T0327.all0.T20.aln.filt.2FRHA_3.mapping
                                                                                                                                                                                                      11/18/2008 01:21 PM 698 T0327.all0.T20.aln.filt.2FRHA_10.mapping
                                                                                                                                                                                                      11/18/2008 01:24 PM 629,605 boinc_aat327_03_05.200_v1_3.gz
                                                                                                                                                                                                      11/18/2008 01:25 PM 1,731,406 boinc_aat327_09_05.200_v1_3.gz
                                                                                                                                                                                                      11/18/2008 01:30 PM 135,465 1ub9.pdb
                                                                                                                                                                                                      11/18/2008 01:31 PM 141,309 1xma.pdb
                                                                                                                                                                                                      11/18/2008 01:31 PM 149,160 1yyv.pdb
                                                                                                                                                                                                      11/18/2008 01:32 PM 154,587 2esh.pdb
                                                                                                                                                                                                      11/18/2008 01:33 PM 180,600 2f2e.pdb
                                                                                                                                                                                                      11/18/2008 01:34 PM 175,317 2frh.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 141,402 2fsw.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 2F2EA_2.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 2F2EA_4.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 2F2EA_5.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 2F2EA_6.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 2F2EA_7.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 2F2EA_8.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 2F2EA_10.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 2F2EA_9.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 2F2EA_11.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 2F2EA_12.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 2F2EA_14.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 2F2EA_16.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 2F2EA_15.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 1UB9A_2.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 1UB9A_5.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 1UB9A_6.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 1UB9A_8.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 1UB9A_10.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 1UB9A_13.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 1XMAA_10.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 1XMAA_11.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 1XMAA_13.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 1XMAA_15.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 1XMAA_14.pdb
                                                                                                                                                                                                      11/18/2008 01:35 PM 39,881 1XMAA_16.pdb
                                                                                                                                                                                                      11/18/2008 01:36 PM 39,881 1XMAA_17.pdb
                                                                                                                                                                                                      11/18/2008 01:36 PM 39,881 1YYVA_3.pdb
                                                                                                                                                                                                      11/18/2008 01:36 PM 39,881 1YYVA_5.pdb
                                                                                                                                                                                                      11/18/2008 01:36 PM 39,881 1YYVA_6.pdb
                                                                                                                                                                                                      11/18/2008 01:36 PM 39,881 1YYVA_7.pdb
                                                                                                                                                                                                      11/18/2008 01:36 PM 39,881 1YYVA_8.pdb
                                                                                                                                                                                                      11/18/2008 01:36 PM 39,881 1YYVA_9.pdb
                                                                                                                                                                                                      11/18/2008 01:36 PM 39,881 1YYVA_10.pdb
                                                                                                                                                                                                      11/18/2008 01:36 PM 39,881 1YYVA_12.pdb
                                                                                                                                                                                                      11/18/2008 01:36 PM 39,881 1YYVA_14.pdb
                                                                                                                                                                                                      11/18/2008 01:37 PM 39,881 2FSWA_1.pdb
                                                                                                                                                                                                      11/18/2008 01:37 PM 39,881 2FSWA_2.pdb
                                                                                                                                                                                                      11/18/2008 01:37 PM 39,881 2FSWA_3.pdb
                                                                                                                                                                                                      11/18/2008 01:37 PM 39,881 2FSWA_4.pdb
                                                                                                                                                                                                      11/18/2008 01:37 PM 39,881 2FSWA_6.pdb
                                                                                                                                                                                                      11/18/2008 01:37 PM 39,881 2FSWA_5.pdb
                                                                                                                                                                                                      11/18/2008 01:37 PM 39,881 2FSWA_7.pdb
                                                                                                                                                                                                      11/18/2008 01:37 PM 39,881 2FSWA_8.pdb
                                                                                                                                                                                                      11/18/2008 01:37 PM 39,881 2FSWA_10.pdb
                                                                                                                                                                                                      11/18/2008 01:37 PM 39,881 2FSWA_9.pdb
                                                                                                                                                                                                      11/18/2008 01:37 PM 39,881 2FSWA_11.pdb
                                                                                                                                                                                                      11/18/2008 01:37 PM 39,881 2FSWA_13.pdb
                                                                                                                                                                                                      11/18/2008 01:37 PM 39,881 2ESHA_14.pdb
                                                                                                                                                                                                      11/18/2008 01:38 PM 39,881 2FRHA_3.pdb
                                                                                                                                                                                                      11/18/2008 01:38 PM 39,881 2FRHA_10.pdb
                                                                                                                                                                                                      11/20/2008 05:14 PM 94,689 native.pdb
                                                                                                                                                                                                      12/01/2008 01:11 PM 43 core_2F2EA_16_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 43 core_2F2EA_14_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 66 core_2F2EA_11_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 66 core_2ESHA_14_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 89 core_1YYVA_9_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 89 core_1YYVA_7_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 66 core_2FSWA_7_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 43 core_2FRHA_10_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 89 core_1YYVA_5_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 89 core_1YYVA_10_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 66 core_1XMAA_17_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 66 core_1XMAA_15_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 66 core_1XMAA_14_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 89 core_1XMAA_10_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 43 core_1UB9A_8_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 43 core_1UB9A_6_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 66 core_2F2EA_2_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 66 core_2FSWA_5_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 66 core_2F2EA_4_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 89 core_2FSWA_3_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 43 core_2FSWA_2_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 89 core_2F2EA_7_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 66 core_2FSWA_13_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 66 core_2F2EA_9_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 66 core_2FSWA_10_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 89 core_2FRHA_3_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 66 core_2FSWA_8_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 66 core_2F2EA_6_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:11 PM 89 core_1YYVA_6_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 89 core_2FSWA_6_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 43 core_2FSWA_4_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 66 core_2FSWA_1_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 43 core_2FSWA_11_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 66 core_2F2EA_8_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 66 core_2F2EA_5_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 43 core_2F2EA_15_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 66 core_2F2EA_12_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 89 core_2F2EA_10_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 89 core_1YYVA_8_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 89 core_1YYVA_3_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 89 core_1YYVA_14_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 89 core_1YYVA_12_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 66 core_1XMAA_16_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 89 core_1XMAA_13_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 66 core_1XMAA_11_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 43 core_1UB9A_5_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 43 core_1UB9A_2_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 43 core_1UB9A_13_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 66 core_2FSWA_9_noloop_loops.txt
                                                                                                                                                                                                      12/01/2008 01:12 PM 43 core_1UB9A_10_noloop_loops.txt
                                                                                                                                                                                                      01/14/2009 11:51 PM 3,618 good_template_config_file.txt
                                                                                                                                                                                                      01/14/2009 11:51 PM 0 1
                                                                                                                                                                                                      01/26/2009 12:44 AM 114 graphics_app
                                                                                                                                                                                                      01/26/2009 12:44 AM 105 minirosetta_1.53_windows_intelx86.exe
                                                                                                                                                                                                      01/26/2009 12:44 AM 81 Helvetica.txf
                                                                                                                                                                                                      01/26/2009 12:44 AM 4,165 init_data.xml
                                                                                                                                                                                                      01/26/2009 12:44 AM 136 test_cc_1_8_nocst4.foldcst_chunk_general.t327_.mtyka.boinc_files.zip
                                                                                                                                                                                                      01/26/2009 12:44 AM 101 minirosetta_database.zip
                                                                                                                                                                                                      01/26/2009 12:44 AM 130 default.out.gz
                                                                                                                                                                                                      01/26/2009 12:44 AM 132 test_cc_1_8_nocst4.foldcst_chunk_general.t327_.mtyka.flags.boinc
                                                                                                                                                                                                      01/26/2009 12:44 AM 0 boinc_lockfile
                                                                                                                                                                                                      01/26/2009 12:44 AM <DIR> minirosetta_database
                                                                                                                                                                                                      01/26/2009 12:44 AM 3 boinc_init_count.txt
                                                                                                                                                                                                      01/26/2009 12:59 AM 12,359 chk_S_1UB9A_2_00000001_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 12:59 AM 8,677 chk_S_1UB9A_2_00000001_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 01:09 AM 12,680 chk_S_1UB9A_2_00000002_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 01:09 AM 8,551 chk_S_1UB9A_2_00000002_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 01:09 AM 8,630 chk_S_1UB9A_2_00000002_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 01:09 AM 12,365 chk_S_1UB9A_2_00000002_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 01:18 AM 8,691 chk_S_1UB9A_2_00000003_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 01:18 AM 12,365 chk_S_1UB9A_2_00000003_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 01:29 AM 8,632 chk_S_1UB9A_2_00000004_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 01:29 AM 12,365 chk_S_1UB9A_2_00000004_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 01:38 AM 8,666 chk_S_1UB9A_2_00000005_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 01:38 AM 12,682 chk_S_1UB9A_2_00000005_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 01:38 AM 12,365 chk_S_1UB9A_2_00000005_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 01:38 AM 8,699 chk_S_1UB9A_2_00000005_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 02:47 AM 8,647 chk_S_1UB9A_2_00000006_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 02:47 AM 12,680 chk_S_1UB9A_2_00000006_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 02:47 AM 8,629 chk_S_1UB9A_2_00000006_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 02:47 AM 12,365 chk_S_1UB9A_2_00000006_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 03:59 AM 12,680 chk_S_1UB9A_2_00000007_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 03:59 AM 8,698 chk_S_1UB9A_2_00000007_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 03:59 AM 12,365 chk_S_1UB9A_2_00000007_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 03:59 AM 8,724 chk_S_1UB9A_2_00000007_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:12 AM 8,740 chk_S_1UB9A_2_00000008_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:12 AM 12,680 chk_S_1UB9A_2_00000008_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 04:12 AM 8,722 chk_S_1UB9A_2_00000008_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:12 AM 12,365 chk_S_1UB9A_2_00000008_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 04:21 AM 8,701 chk_S_1UB9A_2_00000009_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:21 AM 12,365 chk_S_1UB9A_2_00000009_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 04:34 AM 8,702 chk_S_1UB9A_2_00000010_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:34 AM 12,680 chk_S_1UB9A_2_00000010_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 04:34 AM 12,365 chk_S_1UB9A_2_00000010_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 04:34 AM 8,757 chk_S_1UB9A_2_00000010_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:43 AM 12,365 chk_S_1UB9A_2_00000011_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 04:43 AM 8,738 chk_S_1UB9A_2_00000011_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:51 AM 8,657 chk_S_1UB9A_2_00000012_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:51 AM 12,365 chk_S_1UB9A_2_00000012_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 05:00 AM 8,719 chk_S_1UB9A_2_00000013_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:00 AM 12,365 chk_S_1UB9A_2_00000013_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 22,784 chk_S_1UB9A_2_00000001_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,687 chk_S_1UB9A_2_00000001_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,759 chk_S_1UB9A_2_00000001_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 23,193 chk_S_1UB9A_2_00000001_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,759 chk_S_1UB9A_2_00000002_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 22,790 chk_S_1UB9A_2_00000002_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,676 chk_S_1UB9A_2_00000002_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 23,178 chk_S_1UB9A_2_00000002_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,676 chk_S_1UB9A_2_00000003_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 22,790 chk_S_1UB9A_2_00000003_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 23,185 chk_S_1UB9A_2_00000003_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,624 chk_S_1UB9A_2_00000003_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 22,790 chk_S_1UB9A_2_00000004_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,624 chk_S_1UB9A_2_00000004_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,632 chk_S_1UB9A_2_00000004_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 23,192 chk_S_1UB9A_2_00000004_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 22,790 chk_S_1UB9A_2_00000005_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,632 chk_S_1UB9A_2_00000005_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,696 chk_S_1UB9A_2_00000005_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 23,185 chk_S_1UB9A_2_00000005_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 22,790 chk_S_1UB9A_2_00000006_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,696 chk_S_1UB9A_2_00000006_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 23,192 chk_S_1UB9A_2_00000006_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,709 chk_S_1UB9A_2_00000006_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,709 chk_S_1UB9A_2_00000007_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 22,790 chk_S_1UB9A_2_00000007_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 23,185 chk_S_1UB9A_2_00000007_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,704 chk_S_1UB9A_2_00000007_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 22,790 chk_S_1UB9A_2_00000008_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,704 chk_S_1UB9A_2_00000008_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 23,178 chk_S_1UB9A_2_00000008_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,689 chk_S_1UB9A_2_00000008_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,689 chk_S_1UB9A_2_00000009_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 22,790 chk_S_1UB9A_2_00000009_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,722 chk_S_1UB9A_2_00000009_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 23,185 chk_S_1UB9A_2_00000009_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 22,790 chk_S_1UB9A_2_00000010_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,722 chk_S_1UB9A_2_00000010_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,767 chk_S_1UB9A_2_00000010_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 23,178 chk_S_1UB9A_2_00000010_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,767 chk_S_1UB9A_2_00000011_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 22,790 chk_S_1UB9A_2_00000011_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,703 chk_S_1UB9A_2_00000011_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 23,192 chk_S_1UB9A_2_00000011_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,703 chk_S_1UB9A_2_00000012_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 22,790 chk_S_1UB9A_2_00000012_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 23,192 chk_S_1UB9A_2_00000012_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,733 chk_S_1UB9A_2_00000012_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 22,790 chk_S_1UB9A_2_00000013_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,733 chk_S_1UB9A_2_00000013_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,689 chk_S_1UB9A_2_00000013_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 23,178 chk_S_1UB9A_2_00000013_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,689 chk_S_1UB9A_2_00000014_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 22,790 chk_S_1UB9A_2_00000014_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 05:08 AM 8,657 chk_S_1UB9A_2_00000014_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:08 AM 23,185 chk_S_1UB9A_2_00000014_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 05:12 AM 8,726 chk_S_1UB9A_2_00000014_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:12 AM 12,680 chk_S_1UB9A_2_00000014_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 05:12 AM 12,365 chk_S_1UB9A_2_00000014_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 05:12 AM 8,694 chk_S_1UB9A_2_00000014_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:32 AM 8,705 chk_S_1UB9A_2_00000015_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:32 AM 12,365 chk_S_1UB9A_2_00000015_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 05:40 AM 8,660 chk_S_1UB9A_2_00000016_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:40 AM 12,365 chk_S_1UB9A_2_00000016_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 05:50 AM 12,365 chk_S_1UB9A_2_00000017_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 05:50 AM 8,665 chk_S_1UB9A_2_00000017_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:58 AM 12,365 chk_S_1UB9A_2_00000018_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 05:58 AM 8,725 chk_S_1UB9A_2_00000018_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 07:24 AM 12,680 chk_S_1UB9A_2_00000020_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 07:24 AM 8,636 chk_S_1UB9A_2_00000020_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 07:24 AM 8,665 chk_S_1UB9A_2_00000020_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 07:24 AM 12,365 chk_S_1UB9A_2_00000020_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 07:38 AM 12,923 chk_S_1UB9A_2_00000022_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 07:38 AM 8,711 chk_S_1UB9A_2_00000022_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 07:38 AM 8,617 chk_S_1UB9A_2_00000022_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 07:38 AM 12,680 chk_S_1UB9A_2_00000022_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 07:38 AM 8,637 chk_S_1UB9A_2_00000022_ClassicAbinitio___lc_3.rng.state.gz
                                                                                                                                                                                                      01/26/2009 07:38 AM 12,365 chk_S_1UB9A_2_00000022_ClassicAbinitio___lc_3.out
                                                                                                                                                                                                      01/26/2009 07:47 AM 12,680 chk_S_1UB9A_2_00000023_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 07:47 AM 8,705 chk_S_1UB9A_2_00000023_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 07:47 AM 12,365 chk_S_1UB9A_2_00000023_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 07:47 AM 8,689 chk_S_1UB9A_2_00000023_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 07:52 AM 8,684 chk_S_1UB9A_2_00000024_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 07:52 AM 12,365 chk_S_1UB9A_2_00000024_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 08:03 AM 8,649 chk_S_1UB9A_2_00000026_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:03 AM 12,365 chk_S_1UB9A_2_00000026_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 08:07 AM 12,365 chk_S_1UB9A_2_00000027_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 08:07 AM 8,729 chk_S_1UB9A_2_00000027_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:12 AM 8,726 chk_S_1UB9A_2_00000028_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:12 AM 12,365 chk_S_1UB9A_2_00000028_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 08:17 AM 8,641 chk_S_1UB9A_2_00000029_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:17 AM 12,365 chk_S_1UB9A_2_00000029_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 08:24 AM 12,365 chk_S_1UB9A_2_00000030_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 08:24 AM 8,683 chk_S_1UB9A_2_00000030_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:30 AM 8,652 chk_S_1UB9A_2_00000031_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:30 AM 12,680 chk_S_1UB9A_2_00000031_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 08:30 AM 8,728 chk_S_1UB9A_2_00000031_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:30 AM 12,365 chk_S_1UB9A_2_00000031_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 08:39 AM 8,760 chk_S_1UB9A_2_00000033_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:39 AM 12,365 chk_S_1UB9A_2_00000033_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 08:44 AM 8,623 chk_S_1UB9A_2_00000034_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:44 AM 12,365 chk_S_1UB9A_2_00000034_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 08:49 AM 8,680 chk_S_1UB9A_2_00000035_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:49 AM 12,365 chk_S_1UB9A_2_00000035_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 08:56 AM 12,684 chk_S_1UB9A_2_00000036_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 08:56 AM 8,692 chk_S_1UB9A_2_00000036_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:56 AM 7,811 chk_S_1UB9A_2_00000036_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:56 AM 12,365 chk_S_1UB9A_2_00000036_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,754 chk_S_1UB9A_2_00000015_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 23,178 chk_S_1UB9A_2_00000015_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,754 chk_S_1UB9A_2_00000016_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000016_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,667 chk_S_1UB9A_2_00000016_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 23,192 chk_S_1UB9A_2_00000016_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000017_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,667 chk_S_1UB9A_2_00000017_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 23,178 chk_S_1UB9A_2_00000017_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,723 chk_S_1UB9A_2_00000017_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,723 chk_S_1UB9A_2_00000018_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000018_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,603 chk_S_1UB9A_2_00000018_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 23,185 chk_S_1UB9A_2_00000018_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,603 chk_S_1UB9A_2_00000019_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000019_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,676 chk_S_1UB9A_2_00000019_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 23,178 chk_S_1UB9A_2_00000019_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000020_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,676 chk_S_1UB9A_2_00000020_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 23,185 chk_S_1UB9A_2_00000020_Abrelax__fastrelax_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,683 chk_S_1UB9A_2_00000020_Abrelax__fastrelax_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,683 chk_S_1UB9A_2_00000021_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000021_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,681 chk_S_1UB9A_2_00000022_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000022_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,746 chk_S_1UB9A_2_00000023_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000023_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000024_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,675 chk_S_1UB9A_2_00000024_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,689 chk_S_1UB9A_2_00000025_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000025_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000026_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,623 chk_S_1UB9A_2_00000026_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,654 chk_S_1UB9A_2_00000027_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000027_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,711 chk_S_1UB9A_2_00000028_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000028_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,673 chk_S_1UB9A_2_00000029_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000029_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000030_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,633 chk_S_1UB9A_2_00000030_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000031_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,709 chk_S_1UB9A_2_00000031_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,749 chk_S_1UB9A_2_00000032_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000032_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000033_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,640 chk_S_1UB9A_2_00000033_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,702 chk_S_1UB9A_2_00000034_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000034_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,629 chk_S_1UB9A_2_00000035_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000035_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,615 chk_S_1UB9A_2_00000036_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000036_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 22,790 chk_S_1UB9A_2_00000037_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 08:57 AM 8,722 chk_S_1UB9A_2_00000037_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 09:01 AM 12,365 chk_S_1UB9A_2_00000037_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 09:01 AM 8,622 chk_S_1UB9A_2_00000037_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 09:20 AM 8,718 chk_S_1UB9A_2_00000038_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 09:20 AM 12,365 chk_S_1UB9A_2_00000038_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 09:27 AM 8,704 chk_S_1UB9A_2_00000039_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 09:27 AM 12,365 chk_S_1UB9A_2_00000039_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 09:33 AM 8,759 chk_S_1UB9A_2_00000040_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 09:33 AM 12,365 chk_S_1UB9A_2_00000040_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 09:38 AM 8,666 chk_S_1UB9A_2_00000041_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 09:38 AM 12,365 chk_S_1UB9A_2_00000041_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 09:44 AM 12,680 chk_S_1UB9A_2_00000042_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 09:44 AM 8,684 chk_S_1UB9A_2_00000042_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 09:44 AM 8,724 chk_S_1UB9A_2_00000042_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 09:44 AM 12,365 chk_S_1UB9A_2_00000042_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 09:49 AM 12,680 chk_S_1UB9A_2_00000043_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 09:49 AM 8,689 chk_S_1UB9A_2_00000043_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 09:49 AM 8,579 chk_S_1UB9A_2_00000043_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 09:49 AM 12,365 chk_S_1UB9A_2_00000043_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 09:54 AM 12,365 chk_S_1UB9A_2_00000044_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 09:54 AM 8,619 chk_S_1UB9A_2_00000044_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 10:02 AM 8,700 chk_S_1UB9A_2_00000045_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 10:02 AM 12,680 chk_S_1UB9A_2_00000045_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 10:02 AM 8,720 chk_S_1UB9A_2_00000045_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 10:02 AM 12,365 chk_S_1UB9A_2_00000045_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 10:12 AM 8,646 chk_S_1UB9A_2_00000046_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 10:12 AM 12,365 chk_S_1UB9A_2_00000046_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 10:26 AM 12,680 chk_S_1UB9A_2_00000048_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 10:26 AM 8,720 chk_S_1UB9A_2_00000048_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 10:26 AM 8,757 chk_S_1UB9A_2_00000048_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 10:26 AM 12,365 chk_S_1UB9A_2_00000048_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 10:32 AM 8,665 chk_S_1UB9A_2_00000049_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 10:32 AM 12,365 chk_S_1UB9A_2_00000049_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 11:01 AM 12,365 chk_S_1UB9A_2_00000051_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 11:01 AM 8,719 chk_S_1UB9A_2_00000051_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 11:06 AM 8,665 chk_S_1UB9A_2_00000052_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 11:06 AM 12,685 chk_S_1UB9A_2_00000052_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 11:11 AM 8,638 chk_S_1UB9A_2_00000053_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 11:11 AM 12,681 chk_S_1UB9A_2_00000053_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 11:11 AM 8,597 chk_S_1UB9A_2_00000053_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 11:11 AM 12,365 chk_S_1UB9A_2_00000053_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 11:19 AM 12,684 chk_S_1UB9A_2_00000054_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 11:19 AM 8,651 chk_S_1UB9A_2_00000054_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 11:19 AM 8,704 chk_S_1UB9A_2_00000054_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 11:19 AM 12,365 chk_S_1UB9A_2_00000054_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 11:23 AM 8,683 chk_S_1UB9A_2_00000055_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 11:23 AM 12,365 chk_S_1UB9A_2_00000055_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 11:31 AM 8,585 chk_S_1UB9A_2_00000056_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 11:31 AM 12,365 chk_S_1UB9A_2_00000056_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 11:37 AM 8,644 chk_S_1UB9A_2_00000057_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 11:37 AM 12,365 chk_S_1UB9A_2_00000057_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 11:42 AM 8,710 chk_S_1UB9A_2_00000058_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 11:42 AM 12,680 chk_S_1UB9A_2_00000058_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 11:42 AM 12,365 chk_S_1UB9A_2_00000058_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 11:42 AM 8,690 chk_S_1UB9A_2_00000058_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 11:49 AM 12,365 chk_S_1UB9A_2_00000059_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 11:49 AM 8,735 chk_S_1UB9A_2_00000059_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 11:54 AM 12,682 chk_S_1UB9A_2_00000060_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 11:54 AM 8,735 chk_S_1UB9A_2_00000060_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 11:54 AM 12,365 chk_S_1UB9A_2_00000060_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 11:54 AM 8,716 chk_S_1UB9A_2_00000060_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:11 PM 8,699 chk_S_1UB9A_2_00000061_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:11 PM 12,365 chk_S_1UB9A_2_00000061_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 12:33 PM 8,690 chk_S_1UB9A_2_00000064_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:33 PM 12,680 chk_S_1UB9A_2_00000064_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 12:33 PM 8,676 chk_S_1UB9A_2_00000064_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:33 PM 12,365 chk_S_1UB9A_2_00000064_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 12:38 PM 8,674 chk_S_1UB9A_2_00000065_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:38 PM 12,365 chk_S_1UB9A_2_00000065_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,612 chk_S_1UB9A_2_00000039_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000039_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,661 chk_S_1UB9A_2_00000040_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000040_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,713 chk_S_1UB9A_2_00000041_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000041_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,703 chk_S_1UB9A_2_00000042_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000042_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,631 chk_S_1UB9A_2_00000043_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000043_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,628 chk_S_1UB9A_2_00000044_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000044_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,698 chk_S_1UB9A_2_00000045_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000045_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000046_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,624 chk_S_1UB9A_2_00000046_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000047_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,674 chk_S_1UB9A_2_00000047_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,746 chk_S_1UB9A_2_00000048_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000048_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,642 chk_S_1UB9A_2_00000049_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000049_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,699 chk_S_1UB9A_2_00000050_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000050_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,715 chk_S_1UB9A_2_00000051_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000051_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,779 chk_S_1UB9A_2_00000052_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000052_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,699 chk_S_1UB9A_2_00000053_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000053_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000054_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,685 chk_S_1UB9A_2_00000054_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000055_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,630 chk_S_1UB9A_2_00000055_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000056_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,764 chk_S_1UB9A_2_00000056_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,640 chk_S_1UB9A_2_00000057_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000057_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,688 chk_S_1UB9A_2_00000058_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000058_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,660 chk_S_1UB9A_2_00000059_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000059_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,718 chk_S_1UB9A_2_00000060_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000060_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,636 chk_S_1UB9A_2_00000061_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000061_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000062_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,655 chk_S_1UB9A_2_00000062_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,708 chk_S_1UB9A_2_00000063_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000063_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000064_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,644 chk_S_1UB9A_2_00000064_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,742 chk_S_1UB9A_2_00000065_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000065_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 22,790 chk_S_1UB9A_2_00000066_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 12:39 PM 8,707 chk_S_1UB9A_2_00000066_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:43 PM 12,365 chk_S_1UB9A_2_00000066_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 12:43 PM 8,617 chk_S_1UB9A_2_00000066_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:49 PM 12,680 chk_S_1UB9A_2_00000067_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 12:49 PM 8,669 chk_S_1UB9A_2_00000067_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:54 PM 8,728 chk_S_1UB9A_2_00000068_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 12:54 PM 12,365 chk_S_1UB9A_2_00000068_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 01:06 PM 8,640 chk_S_1UB9A_2_00000070_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 01:06 PM 12,365 chk_S_1UB9A_2_00000070_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 01:13 PM 12,910 chk_S_1UB9A_2_00000071_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 01:13 PM 8,664 chk_S_1UB9A_2_00000071_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 01:13 PM 12,667 chk_S_1UB9A_2_00000071_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 01:13 PM 8,671 chk_S_1UB9A_2_00000071_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 01:18 PM 12,911 chk_S_1UB9A_2_00000072_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 01:18 PM 8,563 chk_S_1UB9A_2_00000072_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 01:24 PM 12,365 chk_S_1UB9A_2_00000073_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 01:24 PM 8,634 chk_S_1UB9A_2_00000073_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 01:46 PM 8,680 chk_S_1UB9A_2_00000074_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 01:46 PM 12,365 chk_S_1UB9A_2_00000074_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 01:55 PM 12,925 chk_S_1UB9A_2_00000075_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 01:55 PM 8,745 chk_S_1UB9A_2_00000075_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 01:55 PM 12,680 chk_S_1UB9A_2_00000075_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 01:55 PM 8,620 chk_S_1UB9A_2_00000075_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 01:55 PM 8,699 chk_S_1UB9A_2_00000075_ClassicAbinitio___lc_3.rng.state.gz
                                                                                                                                                                                                      01/26/2009 01:55 PM 12,365 chk_S_1UB9A_2_00000075_ClassicAbinitio___lc_3.out
                                                                                                                                                                                                      01/26/2009 02:04 PM 8,708 chk_S_1UB9A_2_00000076_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 02:04 PM 12,685 chk_S_1UB9A_2_00000076_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 02:04 PM 12,365 chk_S_1UB9A_2_00000076_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 02:04 PM 8,676 chk_S_1UB9A_2_00000076_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 02:12 PM 12,687 chk_S_1UB9A_2_00000077_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 02:12 PM 8,717 chk_S_1UB9A_2_00000077_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 02:12 PM 8,686 chk_S_1UB9A_2_00000077_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 02:12 PM 12,365 chk_S_1UB9A_2_00000077_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 02:17 PM 12,365 chk_S_1UB9A_2_00000078_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 02:17 PM 8,684 chk_S_1UB9A_2_00000078_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 02:25 PM 8,666 chk_S_1UB9A_2_00000079_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 02:25 PM 12,365 chk_S_1UB9A_2_00000079_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 02:31 PM 12,365 chk_S_1UB9A_2_00000080_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 02:31 PM 8,595 chk_S_1UB9A_2_00000080_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 02:36 PM 12,365 chk_S_1UB9A_2_00000081_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 02:36 PM 8,681 chk_S_1UB9A_2_00000081_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 02:49 PM 8,640 chk_S_1UB9A_2_00000082_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 02:49 PM 12,909 chk_S_1UB9A_2_00000082_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 02:49 PM 12,681 chk_S_1UB9A_2_00000082_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 02:49 PM 8,708 chk_S_1UB9A_2_00000082_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 02:49 PM 8,660 chk_S_1UB9A_2_00000082_ClassicAbinitio___lc_3.rng.state.gz
                                                                                                                                                                                                      01/26/2009 02:49 PM 12,365 chk_S_1UB9A_2_00000082_ClassicAbinitio___lc_3.out
                                                                                                                                                                                                      01/26/2009 02:59 PM 12,909 chk_S_1UB9A_2_00000083_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 02:59 PM 8,654 chk_S_1UB9A_2_00000083_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 03:04 PM 12,680 chk_S_1UB9A_2_00000084_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 03:04 PM 8,647 chk_S_1UB9A_2_00000084_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 03:04 PM 8,685 chk_S_1UB9A_2_00000084_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 03:04 PM 12,365 chk_S_1UB9A_2_00000084_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 03:09 PM 12,365 chk_S_1UB9A_2_00000085_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 03:09 PM 8,680 chk_S_1UB9A_2_00000085_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 03:15 PM 8,709 chk_S_1UB9A_2_00000086_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 03:15 PM 12,679 chk_S_1UB9A_2_00000086_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 03:15 PM 8,737 chk_S_1UB9A_2_00000086_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 03:15 PM 12,365 chk_S_1UB9A_2_00000086_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 03:20 PM 8,699 chk_S_1UB9A_2_00000087_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 03:20 PM 12,913 chk_S_1UB9A_2_00000087_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 03:25 PM 8,667 chk_S_1UB9A_2_00000088_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 03:25 PM 12,365 chk_S_1UB9A_2_00000088_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 03:30 PM 8,683 chk_S_1UB9A_2_00000089_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 03:30 PM 12,365 chk_S_1UB9A_2_00000089_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 03:36 PM 8,707 chk_S_1UB9A_2_00000090_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 03:36 PM 12,365 chk_S_1UB9A_2_00000090_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 03:42 PM 8,750 chk_S_1UB9A_2_00000091_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 03:42 PM 12,681 chk_S_1UB9A_2_00000091_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 03:42 PM 12,365 chk_S_1UB9A_2_00000091_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 03:42 PM 8,668 chk_S_1UB9A_2_00000091_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 03:47 PM 8,737 chk_S_1UB9A_2_00000092_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 03:47 PM 12,664 chk_S_1UB9A_2_00000092_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 03:52 PM 8,605 chk_S_1UB9A_2_00000093_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 03:52 PM 12,365 chk_S_1UB9A_2_00000093_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 04:24 PM 8,644 chk_S_1UB9A_2_00000094_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:24 PM 12,683 chk_S_1UB9A_2_00000094_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 04:24 PM 12,365 chk_S_1UB9A_2_00000094_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 04:24 PM 8,725 chk_S_1UB9A_2_00000094_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000068_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,632 chk_S_1UB9A_2_00000068_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,752 chk_S_1UB9A_2_00000069_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000069_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,665 chk_S_1UB9A_2_00000070_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000070_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,704 chk_S_1UB9A_2_00000071_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000071_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000072_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,701 chk_S_1UB9A_2_00000072_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,700 chk_S_1UB9A_2_00000073_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000073_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,649 chk_S_1UB9A_2_00000074_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000074_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,632 chk_S_1UB9A_2_00000075_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000075_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000076_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,644 chk_S_1UB9A_2_00000076_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,678 chk_S_1UB9A_2_00000077_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000077_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,679 chk_S_1UB9A_2_00000078_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000078_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,772 chk_S_1UB9A_2_00000079_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000079_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000080_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,654 chk_S_1UB9A_2_00000080_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000081_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,646 chk_S_1UB9A_2_00000081_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,691 chk_S_1UB9A_2_00000082_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000082_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,600 chk_S_1UB9A_2_00000083_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000083_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,711 chk_S_1UB9A_2_00000084_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000084_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000085_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,603 chk_S_1UB9A_2_00000085_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,652 chk_S_1UB9A_2_00000086_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000086_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,671 chk_S_1UB9A_2_00000087_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000087_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,680 chk_S_1UB9A_2_00000088_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000088_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,702 chk_S_1UB9A_2_00000089_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000089_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,635 chk_S_1UB9A_2_00000090_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000090_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,590 chk_S_1UB9A_2_00000091_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000091_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,716 chk_S_1UB9A_2_00000092_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000092_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000093_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,746 chk_S_1UB9A_2_00000093_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,721 chk_S_1UB9A_2_00000094_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000094_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 22,790 chk_S_1UB9A_2_00000095_Abrelax__rg_state_fa.out
                                                                                                                                                                                                      01/26/2009 04:26 PM 8,648 chk_S_1UB9A_2_00000095_Abrelax__rg_state_fa.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:30 PM 8,701 chk_S_1UB9A_2_00000095_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:30 PM 12,365 chk_S_1UB9A_2_00000095_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 04:38 PM 8,630 chk_S_1UB9A_2_00000096_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:38 PM 12,680 chk_S_1UB9A_2_00000096_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 04:38 PM 12,365 chk_S_1UB9A_2_00000096_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 04:38 PM 7,780 chk_S_1UB9A_2_00000096_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:51 PM 12,680 chk_S_1UB9A_2_00000098_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 04:51 PM 8,701 chk_S_1UB9A_2_00000098_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:51 PM 12,365 chk_S_1UB9A_2_00000098_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 04:51 PM 8,685 chk_S_1UB9A_2_00000098_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:56 PM 8,696 chk_S_1UB9A_2_00000099_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 04:56 PM 12,365 chk_S_1UB9A_2_00000099_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 05:01 PM 8,717 chk_S_1UB9A_2_00000100_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:01 PM 12,365 chk_S_1UB9A_2_00000100_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 05:09 PM 8,720 chk_S_1UB9A_2_00000101_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:09 PM 12,664 chk_S_1UB9A_2_00000101_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 05:36 PM 12,680 chk_S_1UB9A_2_00000102_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 05:36 PM 8,689 chk_S_1UB9A_2_00000102_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:36 PM 8,696 chk_S_1UB9A_2_00000102_ClassicAbinitio___lc_3.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:36 PM 12,365 chk_S_1UB9A_2_00000102_ClassicAbinitio___lc_3.out
                                                                                                                                                                                                      01/26/2009 05:43 PM 81,786 score.fsc
                                                                                                                                                                                                      01/26/2009 05:43 PM 2,225,351 default.out
                                                                                                                                                                                                      01/26/2009 05:43 PM 8,688 rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:43 PM 3 boinc_checkpoint_count.txt
                                                                                                                                                                                                      01/26/2009 05:43 PM 10,423 jumps_pre.log
                                                                                                                                                                                                      01/26/2009 05:43 PM 5,969 stderr.txt
                                                                                                                                                                                                      01/26/2009 05:47 PM 658 stderrgfx.txt
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,639 chk_S_1UB9A_2_00000104_ClassicAbinitio___lc_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM 12,680 chk_S_1UB9A_2_00000104_ClassicAbinitio___lc_1.out
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,660 chk_S_1UB9A_2_00000104_ClassicAbinitio___lc_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM 12,365 chk_S_1UB9A_2_00000104_ClassicAbinitio___lc_2.out
                                                                                                                                                                                                      01/26/2009 05:48 PM 13,158 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_1.out
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,676 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,826 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM 13,234 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_2.out
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,635 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM 13,888 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_1.out
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,658 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM 13,888 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_2.out
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,657 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_3.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM 13,994 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_3.out
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,699 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_4.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM 13,994 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_4.out
                                                                                                                                                                                                      01/26/2009 05:48 PM 13,994 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_5.out
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,688 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_5.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM 13,994 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_6.out
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,659 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_6.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM 14,102 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_7.out
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,632 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_7.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,655 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_8.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM 14,102 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_8.out
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,591 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_9.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM 14,210 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_9.out
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,674 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_10.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM 14,289 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage_3_iter1_10.out
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,715 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage4_kk_1.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM 13,938 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage4_kk_1.out
                                                                                                                                                                                                      01/26/2009 05:48 PM 13,938 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage4_kk_2.out
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,619 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage4_kk_2.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM 8,640 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage4_kk_3.rng.state.gz
                                                                                                                                                                                                      01/26/2009 05:48 PM <DIR> ..
                                                                                                                                                                                                      01/26/2009 05:48 PM 13,938 chk_S_1UB9A_2_00000105_ClassicAbinitio__stage4_kk_3.out
                                                                                                                                                                                                      01/26/2009 05:48 PM <DIR> .
                                                                                                                                                                                                      01/26/2009 05:48 PM 10,423 jumps.log
                                                                                                                                                                                                      01/26/2009 05:49 PM 60,626 stdout.txt


                                                                                                                                                                                                      Perhaps you could offer advice on how to look for any checkpointing problems. It seems it is either writing checkpoints more then it should... sometimes. Or, that it's got some models that complete in less then a minute and others that run for some time.

                                                                                                                                                                                                      If you could tell us more about the naming used on these files, I guess that would clear up some of my doubts on what I'm looking at, and for.
                                                                                                                                                                                                      ____________

                                                                                                                                                                                                      mtyka
                                                                                                                                                                                                      Forum moderator
                                                                                                                                                                                                      Project developer
                                                                                                                                                                                                      Project scientist

                                                                                                                                                                                                      Joined: Mar 19 08
                                                                                                                                                                                                      Posts: 79
                                                                                                                                                                                                      ID: 4144
                                                                                                                                                                                                      Credit: 0
                                                                                                                                                                                                      RAC: 0
                                                                                                                                                                                                      Message 4575 - Posted 27 Jan 2009 22:37:28 UTC - in response to Message 4561.

                                                                                                                                                                                                        Feet1st:

                                                                                                                                                                                                        Looking at this, the checkpointing is working fine. As i said, rosetta will now buffer checkpoints up to a limit and them *dump all of them at the same time*.
                                                                                                                                                                                                        So those large blocks of files is when it dumped all those checkpoints in one go.
                                                                                                                                                                                                        Sadly we cannot just write the last checkpoint, we need all intermediate checkpoints to corerctly restore the state of the program.
                                                                                                                                                                                                        (rather then spreading them over time, which would keep the disc spinnng).

                                                                                                                                                                                                        HOWEVER: YOu;ve found another problem: minirosetta is not deleting all the checkpoints after its
                                                                                                                                                                                                        done with them !!! Theoretically it should delete checkpoints after each decoy.

                                                                                                                                                                                                        I'll look into that.

                                                                                                                                                                                                        Cheers, Mike

                                                                                                                                                                                                        Profile feet1st

                                                                                                                                                                                                        Joined: Mar 7 06
                                                                                                                                                                                                        Posts: 312
                                                                                                                                                                                                        ID: 1028
                                                                                                                                                                                                        Credit: 110,522
                                                                                                                                                                                                        RAC: 0
                                                                                                                                                                                                        Message 4576 - Posted 27 Jan 2009 22:49:27 UTC

                                                                                                                                                                                                          I was starting to think that must be what was occuring. And also was thinking that once the model was completed, that these temp files should be deletable. So, great minds think alike I guess... and so do we!

                                                                                                                                                                                                          Glad the info. came of some good use.
                                                                                                                                                                                                          ____________

                                                                                                                                                                                                          mtyka
                                                                                                                                                                                                          Forum moderator
                                                                                                                                                                                                          Project developer
                                                                                                                                                                                                          Project scientist

                                                                                                                                                                                                          Joined: Mar 19 08
                                                                                                                                                                                                          Posts: 79
                                                                                                                                                                                                          ID: 4144
                                                                                                                                                                                                          Credit: 0
                                                                                                                                                                                                          RAC: 0
                                                                                                                                                                                                          Message 4578 - Posted 27 Jan 2009 23:29:15 UTC - in response to Message 4576.

                                                                                                                                                                                                            I was starting to think that must be what was occuring. And also was thinking that once the model was completed, that these temp files should be deletable. So, great minds think alike I guess... and so do we!

                                                                                                                                                                                                            Glad the info. came of some good use.


                                                                                                                                                                                                            yeah the non-deleteing i think is due to a logic flaw when it gets to the end in the dcoy. Its not suuuuuper urgent, i'll include a fix with the next update. All it does is accumulate a few more structures until the WU ends. But frankly that's rarely goingot be more than 5-10 MB.

                                                                                                                                                                                                            Mike

                                                                                                                                                                                                            Profile feet1st

                                                                                                                                                                                                            Joined: Mar 7 06
                                                                                                                                                                                                            Posts: 312
                                                                                                                                                                                                            ID: 1028
                                                                                                                                                                                                            Credit: 110,522
                                                                                                                                                                                                            RAC: 0
                                                                                                                                                                                                            Message 4579 - Posted 27 Jan 2009 23:44:49 UTC

                                                                                                                                                                                                              Its not suuuuuper urgent...


                                                                                                                                                                                                              Agreed. The only ill effect should be slightly higher disk space usage. Even then, it should still be within the normal range.
                                                                                                                                                                                                              ____________

                                                                                                                                                                                                              Message boards : RALPH@home bug list : minirosetta v1.48-1.51 bug thread


                                                                                                                                                                                                              Home | Join | About | Participants | Community | Statistics

                                                                                                                                                                                                              Copyright © 2017 University of Washington

                                                                                                                                                                                                              Last Modified: 20 Nov 2008 19:41:56 UTC
                                                                                                                                                                                                              Back to top ^