Message boards : Feedback : Higher quorum
Author | Message |
---|---|
Transient Send message Joined: 7 Apr 12 Posts: 4 Credit: 34,062 RAC: 0 |
Since we're testing new batches of WU's and new applications here, does it make sense to institute a quorum, where multiple tasks have to validate aginst each other? Would that not make for a better test? |
Rocco Moretti Volunteer moderator Project developer Project scientist Send message Joined: 18 May 10 Posts: 11 Credit: 30,188 RAC: 0 |
Rosetta (and Ralph) uses a protocol which is stochastic, rather than deterministic. This means that unlike many other projects, there isn't a single "canonical" result for each workunit - the same workunit will produce slightly different results when run on different machines, or even re-run by the same machine. Therefore, there is little value in trying to compare two different runs of a workunit against each other. The primary goal with the testing on Ralph is to check that there aren't any subtle bugs with the job or the application which would result in jobs terminating with errors on certain platforms or certain small percentages of cases. |
Transient Send message Joined: 7 Apr 12 Posts: 4 Credit: 34,062 RAC: 0 |
I thought tasks using the same starting point (same random seed) would produce the same result, provided the machines return the same number of models. But that's not true? |
Rocco Moretti Volunteer moderator Project developer Project scientist Send message Joined: 18 May 10 Posts: 11 Credit: 30,188 RAC: 0 |
I thought tasks using the same starting point (same random seed) would produce the same result, provided the machines return the same number of models. Theoretically, yes, the same seed should produce the same results. Unfortunately, because of slight differences in processors and timing, etc., that's not always the case. Because of the way protocols run, slight numerical differences can change the trajectory significantly (i.e. the butterfly effect). Because we're not really interested in producing a "canonical" result for each workunit (it's more important to produce a valid distribution across the entire job), and the main thing Ralph is intended to catch is things which cause the application to error out, it's not really worth trying compare the same workunit between different machines. |
Transient Send message Joined: 7 Apr 12 Posts: 4 Credit: 34,062 RAC: 0 |
I understand an actual quorum is not of real interest, but I thought the mechanism would allow for better testing of applications and WU batches. It was just an idea :) |
Message boards :
Feedback :
Higher quorum
©2024 University of Washington
http://www.bakerlab.org