Fair is completely the right word as I stated it as a goal as in "the goal is to be fair."

Then, bear with me for a moment. I will type a long explanation of what I think would be far superior.

---

Fair is completely the right word as I stated it as a goal as in "the goal is to be fair."

Then, bear with me for a moment. I will type a long explanation of what I think would be far superior.

Problem : The credit system is easy to manipulate because benchmarks can be altered, and tags can be altered as simply as opening a text editor and entering in your own scores.

Question : Assuming you could remove the ability to alter scores from the public's hand, would this make the scoring system stronger?

Answer : Assuming that this is possible, the current scoring system would be much more fair than assigning an arbitrary average scoring system. However - since scores can be manipulated, you thought that only giving credit on a per/model basis was the only way to keep scores 'fair', as it is the only way to assign proof to work. You can't cheat if you don't have results for your work, and you shouldn't have results if you cheat. (but we all know you can, if anyone remembers SETI Classic)

Solution : The Rosetta core application should do it's own benchmarking, and assign encrypted scoring values inside each workunit. BOINC level scoring systems could be bypassed on a per project basis, or per workunit basis even if you wanted to. The validator could be changed to decrypt these values on the fly, and everyone is happy. Although now that I think about it, you don't -have- to give this responsibility to the validator, you could require some background communication from Rosetta core whenever it needs to send in these values. It could send them in to the scheduler if you prefer.

Why do you feel you need an entirely new system to fix one problem? What are the other problems with the current scoring system? So far I have heard only one, and that is that it can be manipulated by EU/GP

---

This is a great proposal. If such a system gets developed, my vote would be to use it. That doesn't mean we should abandon our current efforts, though.

---

Thanks for the last message Aaron. . I think it sums up what we agree on. The ideal would be something unhackable in Boinc iteslf. . but that wouldn't be easy due to the open source nature of the code and it would require everyone to install a new client to run Rosetta.

A benchmark within the Rosetta itself would be great as well. . I'm not sure how feasible that is either. My understanding is the code branches a lot, many many function calls. Is it possible to count every calculation? The benchmark would have to be time independent. . otherwise a slow machine working for an hour would want the same score as a fast machine-hour.

Back to counting calculations. . does an integer calculation could the same as a floating point? How about a double float? How much overhead is there in keeping track of billions of operations. Those C++'s would add up pretty quickly :)

You could base a benchmark off how long it takes to crunch a 'golden' work unit. . but that one protein might use completely different parts of Rosetta than HIV research. . and having to create new 'golden' benchmark units, normalize them to each other, and package them in each new type of wu is a lot of overhead (for scientists and cpus).

Just throwing those ideas out there. . there may be ways of doing them I hadn't though of.

The lowest hanging fruit seems to be the proposed new system. It eliminates the advantages of the optimized client. People will still claim more credit than they deserve, but those will average with standard client claims, and everyone receives the same score. The only overhead is averaging the wu claims for each wu type. . and it will work with all versions of boinc and not require recoding of rosetta.

---

This is a great proposal. If such a system gets developed, my vote would be to use it. That doesn't mean we should abandon our current efforts, though.

Well, it's disheartening that you feel that you have no other option but to change what I (and many of the creators of BOINC) felt was a much better system back to the old system which we felt wasn't as fair, just to make things more fair. Do you see why this would be hard to swallow?

The goal certainly should be to make things more fair, but I just don't feel that you should cut off everyone's feet simply because some unscrupulous people invented rollerskates. There wasn't anything wrong with feet until rollerskates came along - Just make it so there aren't any rollerskates.

The Rosetta core application is all in -your- hands, so I guess I just don't understand why you couldn't program it to perform as you see fit?

I know that it must be easier to simply change a few settings with the scoring algorythms, but believe me the current system works, it's merely got a backdoor that needs locking.

---

I know that it must be easier to simply change a few settings with the scoring algorythms, but believe me the current system works, it's merely got a backdoor that needs locking.

That's the problem, how do you lock it? Boinc and Rosetta are seperate things. Rosetta has nothing to due with credit calculation other than returning the time it took to complete the WU. Boinc then takes that, does some math based on your benchmarks (or edited benchmarks), and returns the results to the server. Rosetta has to trust the credit claims since the benchmarking part of Boinc is out of its control.

The only way to close the backdoor is to have a new version of Boinc developed and require everyone upgrade to have their credits counted. . . Or get into the business of determining if each seemingly large score is real or inflated.

---

prior to the currently in-testing credit system, I thought that flops counting was the ideal, with the benchmark as either a library to be included in the BOINC code during compilation (don't know if it'd be possible to create a moduel that isn't 'tamperable'), or a standard benchmark to be included in the project apps. Infact I remember my very first DC post over at UD said the very same thing back in 99 or 2000! I haven't seen anything that says that this is an accurate system though- as Ethan says, what value to you assign to FPU operations over Integer operations etc. There is some arbitration even in that system isn't there? (I might be wrong - my understaning of programming is pretty minimal beyond VB...)

---

I know that it must be easier to simply change a few settings with the scoring algorythms, but believe me the current system works, it's merely got a backdoor that needs locking.

That's the problem, how do you lock it? Boinc and Rosetta are seperate things. Rosetta has nothing to due with credit calculation other than returning the time it took to complete the WU. Boinc then takes that, does some math based on your benchmarks (or edited benchmarks), and returns the results to the server. Rosetta has to trust the credit claims since the benchmarking part of Boinc is out of its control.

The only way to close the backdoor is to have a new version of Boinc developed and require everyone upgrade to have their credits counted. . . Or get into the business of determining if each seemingly large score is real or inflated.

Didn't you read my post? I could have swore you did by your previous reply, but now I think you must have missed it.. In it, I said quite clearly to put this in the hands of the Rosetta core application. Go back and read the last 5-10 posts, it'll explain it in it's entirety.

You don't need BOINC to assign scores. It only does this as a courtesy to facilitate projects. Someone needs to get with David and/or Rom, and ask how you can incorporate benchmarking into the projects core app.

---

Solution : The Rosetta core application should do it's own benchmarking, and assign encrypted scoring values inside each workunit. BOINC level scoring systems could be bypassed on a per project basis, or per workunit basis even if you wanted to. The validator could be changed to decrypt these values on the fly, and everyone is happy. Although now that I think about it, you don't -have- to give this responsibility to the validator, you could require some background communication from Rosetta core whenever it needs to send in these values. It could send them in to the scheduler if you prefer.

Wow. Good stuff. Where were you for the last 3 months... ?

Solution : The Rosetta core application should do it's own benchmarking, and assign encrypted scoring values inside each workunit. BOINC level scoring systems could be bypassed on a per project basis, or per workunit basis even if you wanted to. The validator could be changed to decrypt these values on the fly, and everyone is happy. Although now that I think about it, you don't -have- to give this responsibility to the validator, you could require some background communication from Rosetta core whenever it needs to send in these values. It could send them in to the scheduler if you prefer.

Wow. Good stuff. Where were you for the last 3 months... ?

I'm very sorry.. I have been busy at work. Unfortunately, I'm supposed to be working right now, but my passion for this issue is distracting me. :(

---

Let's try to stay focused in this discussion.

We have already incorporated a simple benchmark in rosetta. But since we don't use redundancy people can still be malicious, so we decided not to go that route.

---

Aaron, even the creators of Boinc (people of seti) couldn't close "the door", they implemented the FPOPS sytem to do just that. For some reason I have yet to understand, Fpops isn't useable here, so they're trying what they can. If the creators of boinc can't fix the existing benchmark system without an overhaul, what chance does the Rosetta staff have??? Oh, and the Benchmark system does have ONE decent flaw, it doesn't give linux users a comparable benchmark to it's windows counterpart (my lord, I'll be joining XS soon. LOL)

Let's try to stay focused in this discussion.
We have already incorporated a simple benchmark in rosetta. But since we don't use redundancy people can still be malicious, so we decided not to go that route.

How can you be malicious (easily) if the core application has it's own encrypted method of 'benchmarking', the workunits have their own encrypted tags, and the results are also containing encrypted tags? I'm assuming that there are parts of the Rosetta Core application that are not available to the GP as source..

If you were to use a code (such as 512bit RSA? 1024-bit RSA enough?) that is (realistically) unhackable, then why do you think people could still alter their benchmarks if you control every step of the process?

---

Hello,

Mmclasro you have a good point and a funny one, which is quite bewildering at the same time, and understandable untill Linux really takes off in a couple years. Seeing as Xp has been around long enough for Widows users to need a "Safe", "Secure" O.S..The differece in Bench,s for Linux isnot comparable.Assides from that, there are many
valid points within this Thread.Rosetta is "Great" Science, therfore running
Ralph is just as Paramount.I am Happy to be a part of something, that "could"
save 1 Human life, if nothing else, Keep crunching.
"Happy Crunching Boincateers"

Sincerely

Sluger

---

Let's try to stay focused in this discussion.
We have already incorporated a simple benchmark in rosetta. But since we don't use redundancy people can still be malicious, so we decided not to go that route.

How can you be malicious (easily) if the core application has it's own encrypted method of 'benchmarking', the workunits have their own encrypted tags, and the results are also containing encrypted tags? I'm assuming that there are parts of the Rosetta Core application that are not available to the GP as source..

If you were to use a code (such as 512bit RSA? 1024-bit RSA enough?) that is (realistically) unhackable, then why do you think people could still alter their benchmarks if you control every step of the process?

I agree, if we add encryption to rosetta, we could use the internal benchmark. It's something to consider.

---

How can you be malicious (easily) if the core application has it's own encrypted method of 'benchmarking', the workunits have their own encrypted tags, and the results are also containing encrypted tags? I'm assuming that there are parts of the Rosetta Core application that are not available to the GP as source..

If you were to use a code (such as 512bit RSA? 1024-bit RSA enough?) that is (realistically) unhackable, then why do you think people could still alter their benchmarks if you control every step of the process?

Would that produce the same benchmark regardless of OS?

---

Would that produce the same benchmark regardless of OS?

I would assume so, yes. However, to get that result it may be only slightly more complicated.

For instance, the linux crowd may get one version of the benchmark, while the windows crowd would obviously get another.

Depending on how complicated the benchmark is, it could be only as difficult as tagging the compiler for your specific OS, or rewriting it as you see fit.

Either way - I only really see 3 maybe 4 OS versions being used currently, it surely couldn't be that difficult.

---

Aaron, certain have to agree that encryption is possible, because Rosetta is not open source. But to simply make an effort to encrypt the BOINC figures still apparently penalizes Linux, and allows for other means of modifying the BOINC benchmarks... and so I believe that is why David Kim suggests continuing to pursue the current credit system... because it provides the level playing field (all P4 2.4Ghz systems, baring any other disparities in capabilities should score the same, regardless of Linux or Windows or "optimized" or stock), and then you can later add the encryption, which provides an unmodified playing field. So the two aren't mutually exclusive.

Since Rosetta is but an application within BOINC... they aren't really in control of some of the code you are suggesting be changed. But encryption for the part they do control, the key portions of the .out file, is certainly possible. And, as I've expressed before, I believe it would be helpful. Although I'll point out that if you encrypt the whole file, then it adds to the mystic and the cynics which come along from time to time and assert that the whole project is a fraud, or that it is a virus that's stealing your data or whatever. So, just encrypt a tiny portion of the file, so folks can SEE what's being sent and gain further interest and knowledge about the science being done here.

---

But to simply make an effort to encrypt the BOINC figures still apparently penalizes Linux, and allows for other means of modifying the BOINC benchmarks...

K.. I'll bite. How? and be specific, because what you are saying is that simply because the code is encrypted it punishes the linux crowd. forgive me if your statement lost me there. As for the second half of your statement, you're obviously not getting what I'm saying, I'm not talking about BOINC benchmarks.

Since Rosetta is but an application within BOINC... they aren't really in control of some of the code you are suggesting be changed. But encryption for the part they do control, the key portions of the .out file, is certainly possible.

C'mon, use your imagination here. What difference does that make? Is rosetta forbidden to make it's own files to send back to the validator / scheduler / some other server? I think not. People, if you want to take the benchmarking out of BOINC's hands, we have to start thinking outside the box. The box being - Currently we let the allmighty BOINC handle everything except wiping our butts. YOU DON'T HAVE TO LET BOINC DO ANYTHING! Period. How many times, and in how many different languages do I have to say it - BOINC only handles these functions as a courtesy for projects. Yes, it makes things easier, but we're not talking about something that can be done -within- BOINC are we?

And, as I've expressed before, I believe it would be helpful. Although I'll point out that if you encrypt the whole file, then it adds to the mystic and the cynics which come along from time to time and assert that the whole project is a fraud, or that it is a virus that's stealing your data or whatever. So, just encrypt a tiny portion of the file, so folks can SEE what's being sent and gain further interest and knowledge about the science being done here.

That's great if it can be done, but if it makes things too difficult, I wouldn't make it that high of a priority. Reasoning:

Those people are either...

1. Delusional.
2. Not downloading the project in EITHER case.
3. Not educated enough to know how to open up the program and know what it's doing ANYWAY.
4. Running the project, but like to be forum trolls.

In case of #1 - Non-Issue
In case of #2 - Complete Non-Issue
In case of #3 - Non-Issue
and in case of #4 - I'm sorry, I don't respond to trolls.

Assuming they aren't in one of the 4 listed scenarios above, What are you going to lose - 1, maybe 2 conspiracy freaks? Yeah.. Those people seem like real contributors to a DC project.


One thing I would like to add, is that one thing I remember David saying is that he lays awake at night thinking about how to get more members. I can't remember where I heard that or where he said it. However - If you change the credit system, you are GOING to upset people, no matter how much more logical useful, or fair the system that you're moving to seems. This is why I think it would be best to only fine-tune the existing system, rather than switch to one of a different type. Otherwise, are you prepared to disgruntle your participants? If the switch to BOINC's crediting system (remember trying to explain the cobblestone?) was any indication, I can guarantee you - It will disgruntle some of them. I know David definately doesn't want to be laying awake at night thinking about why the project lost ~5000 users.

---

...

I agree, if we add encryption to rosetta, we could use the internal benchmark. It's something to consider.

You'd also need to encrypt the in-memory benchmark results as well as the upload and download, also obfuscate the actual benchmark code so that people can't follow what it's doing in a debugger and modify the values on the fly. (Remember if people can use a debugger, an out-of-process routine can do the same once the memory locations are known).

The recent debugging code (map files etc) would prove extremely useful to someone trying to do this, but aren't necessary.

The new system does the job, and should do it well after the bugs are ironed out, so why mess around with insecure ways of doing the same thing?

---