Since we're testing new batches of WU's and new applications here, does it make sense to institute a quorum, where multiple tasks have to validate aginst each other? Would that not make for a better test?

Rosetta (and Ralph) uses a protocol which is stochastic, rather than deterministic. This means that unlike many other projects, there isn't a single "canonical" result for each workunit - the same workunit will produce slightly different results when run on different machines, or even re-run by the same machine. Therefore, there is little value in trying to compare two different runs of a workunit against each other.

The primary goal with the testing on Ralph is to check that there aren't any subtle bugs with the job or the application which would result in jobs terminating with errors on certain platforms or certain small percentages of cases.

I thought tasks using the same starting point (same random seed) would produce the same result, provided the machines return the same number of models. But that's not true?

I thought tasks using the same starting point (same random seed) would produce the same result, provided the machines return the same number of models.

Theoretically, yes, the same seed should produce the same results. Unfortunately, because of slight differences in processors and timing, etc., that's not always the case. Because of the way protocols run, slight numerical differences can change the trajectory significantly (i.e. the butterfly effect).

Because we're not really interested in producing a "canonical" result for each workunit (it's more important to produce a valid distribution across the entire job), and the main thing Ralph is intended to catch is things which cause the application to error out, it's not really worth trying compare the same workunit between different machines.

I understand an actual quorum is not of real interest, but I thought the mechanism would allow for better testing of applications and WU batches.

It was just an idea :)