Higher quorum

Message boards : Feedback : Higher quorum

To post messages, you must log in.

AuthorMessage
Transient

Send message
Joined: 7 Apr 12
Posts: 4
Credit: 34,062
RAC: 0
Message 5522 - Posted: 10 Apr 2012, 20:32:05 UTC

Since we're testing new batches of WU's and new applications here, does it make sense to institute a quorum, where multiple tasks have to validate aginst each other? Would that not make for a better test?
ID: 5522 · Report as offensive    Reply Quote
Rocco Moretti
Volunteer moderator
Project developer
Project scientist

Send message
Joined: 18 May 10
Posts: 11
Credit: 30,188
RAC: 0
Message 5524 - Posted: 12 Apr 2012, 19:05:30 UTC - in response to Message 5522.  

Rosetta (and Ralph) uses a protocol which is stochastic, rather than deterministic. This means that unlike many other projects, there isn't a single "canonical" result for each workunit - the same workunit will produce slightly different results when run on different machines, or even re-run by the same machine. Therefore, there is little value in trying to compare two different runs of a workunit against each other.

The primary goal with the testing on Ralph is to check that there aren't any subtle bugs with the job or the application which would result in jobs terminating with errors on certain platforms or certain small percentages of cases.
ID: 5524 · Report as offensive    Reply Quote
Transient

Send message
Joined: 7 Apr 12
Posts: 4
Credit: 34,062
RAC: 0
Message 5525 - Posted: 13 Apr 2012, 4:01:53 UTC

I thought tasks using the same starting point (same random seed) would produce the same result, provided the machines return the same number of models. But that's not true?
ID: 5525 · Report as offensive    Reply Quote
Rocco Moretti
Volunteer moderator
Project developer
Project scientist

Send message
Joined: 18 May 10
Posts: 11
Credit: 30,188
RAC: 0
Message 5526 - Posted: 13 Apr 2012, 19:33:27 UTC - in response to Message 5525.  

I thought tasks using the same starting point (same random seed) would produce the same result, provided the machines return the same number of models.


Theoretically, yes, the same seed should produce the same results. Unfortunately, because of slight differences in processors and timing, etc., that's not always the case. Because of the way protocols run, slight numerical differences can change the trajectory significantly (i.e. the butterfly effect).

Because we're not really interested in producing a "canonical" result for each workunit (it's more important to produce a valid distribution across the entire job), and the main thing Ralph is intended to catch is things which cause the application to error out, it's not really worth trying compare the same workunit between different machines.
ID: 5526 · Report as offensive    Reply Quote
Transient

Send message
Joined: 7 Apr 12
Posts: 4
Credit: 34,062
RAC: 0
Message 5527 - Posted: 14 Apr 2012, 7:48:30 UTC

I understand an actual quorum is not of real interest, but I thought the mechanism would allow for better testing of applications and WU batches.

It was just an idea :)
ID: 5527 · Report as offensive    Reply Quote

Message boards : Feedback : Higher quorum



©2024 University of Washington
http://www.bakerlab.org