New crediting system

Message boards : Current tests : New crediting system

To post messages, you must log in.

Previous · 1 . . . 8 · 9 · 10 · 11 · 12 · 13 · Next

AuthorMessage
Aaron Finney

Send message
Joined: 16 Feb 06
Posts: 56
Credit: 1,457
RAC: 0
Message 2152 - Posted: 16 Aug 2006, 22:25:11 UTC - in response to Message 2151.  


Fair is completely the right word as I stated it as a goal as in "the goal is to be fair."


Then, bear with me for a moment. I will type a long explanation of what I think would be far superior.
ID: 2152 · Report as offensive    Reply Quote
Aaron Finney

Send message
Joined: 16 Feb 06
Posts: 56
Credit: 1,457
RAC: 0
Message 2153 - Posted: 16 Aug 2006, 22:34:03 UTC - in response to Message 2152.  
Last modified: 16 Aug 2006, 22:47:18 UTC


Fair is completely the right word as I stated it as a goal as in "the goal is to be fair."


Then, bear with me for a moment. I will type a long explanation of what I think would be far superior.


Problem : The credit system is easy to manipulate because benchmarks can be altered, and tags can be altered as simply as opening a text editor and entering in your own scores.

Question : Assuming you could remove the ability to alter scores from the public's hand, would this make the scoring system stronger?

Answer : Assuming that this is possible, the current scoring system would be much more fair than assigning an arbitrary average scoring system. However - since scores can be manipulated, you thought that only giving credit on a per/model basis was the only way to keep scores 'fair', as it is the only way to assign proof to work. You can't cheat if you don't have results for your work, and you shouldn't have results if you cheat. (but we all know you can, if anyone remembers SETI Classic)

Solution : The Rosetta core application should do it's own benchmarking, and assign encrypted scoring values inside each workunit. BOINC level scoring systems could be bypassed on a per project basis, or per workunit basis even if you wanted to. The validator could be changed to decrypt these values on the fly, and everyone is happy. Although now that I think about it, you don't -have- to give this responsibility to the validator, you could require some background communication from Rosetta core whenever it needs to send in these values. It could send them in to the scheduler if you prefer.

Why do you feel you need an entirely new system to fix one problem? What are the other problems with the current scoring system? So far I have heard only one, and that is that it can be manipulated by EU/GP
ID: 2153 · Report as offensive    Reply Quote
Profile dekim
Volunteer moderator
Project administrator
Project developer
Project scientist

Send message
Joined: 20 Jan 06
Posts: 250
Credit: 543,579
RAC: 0
Message 2154 - Posted: 16 Aug 2006, 22:47:42 UTC

This is a great proposal. If such a system gets developed, my vote would be to use it. That doesn't mean we should abandon our current efforts, though.
ID: 2154 · Report as offensive    Reply Quote
Ethan

Send message
Joined: 11 Feb 06
Posts: 18
Credit: 25,579
RAC: 0
Message 2155 - Posted: 16 Aug 2006, 22:55:22 UTC - in response to Message 2153.  

Thanks for the last message Aaron. . I think it sums up what we agree on. The ideal would be something unhackable in Boinc iteslf. . but that wouldn't be easy due to the open source nature of the code and it would require everyone to install a new client to run Rosetta.

A benchmark within the Rosetta itself would be great as well. . I'm not sure how feasible that is either. My understanding is the code branches a lot, many many function calls. Is it possible to count every calculation? The benchmark would have to be time independent. . otherwise a slow machine working for an hour would want the same score as a fast machine-hour.

Back to counting calculations. . does an integer calculation could the same as a floating point? How about a double float? How much overhead is there in keeping track of billions of operations. Those C++'s would add up pretty quickly :)

You could base a benchmark off how long it takes to crunch a 'golden' work unit. . but that one protein might use completely different parts of Rosetta than HIV research. . and having to create new 'golden' benchmark units, normalize them to each other, and package them in each new type of wu is a lot of overhead (for scientists and cpus).

Just throwing those ideas out there. . there may be ways of doing them I hadn't though of.

The lowest hanging fruit seems to be the proposed new system. It eliminates the advantages of the optimized client. People will still claim more credit than they deserve, but those will average with standard client claims, and everyone receives the same score. The only overhead is averaging the wu claims for each wu type. . and it will work with all versions of boinc and not require recoding of rosetta.

ID: 2155 · Report as offensive    Reply Quote
Aaron Finney

Send message
Joined: 16 Feb 06
Posts: 56
Credit: 1,457
RAC: 0
Message 2156 - Posted: 16 Aug 2006, 22:57:41 UTC - in response to Message 2154.  

This is a great proposal. If such a system gets developed, my vote would be to use it. That doesn't mean we should abandon our current efforts, though.



Well, it's disheartening that you feel that you have no other option but to change what I (and many of the creators of BOINC) felt was a much better system back to the old system which we felt wasn't as fair, just to make things more fair. Do you see why this would be hard to swallow?

The goal certainly should be to make things more fair, but I just don't feel that you should cut off everyone's feet simply because some unscrupulous people invented rollerskates. There wasn't anything wrong with feet until rollerskates came along - Just make it so there aren't any rollerskates.

The Rosetta core application is all in -your- hands, so I guess I just don't understand why you couldn't program it to perform as you see fit?

I know that it must be easier to simply change a few settings with the scoring algorythms, but believe me the current system works, it's merely got a backdoor that needs locking.
ID: 2156 · Report as offensive    Reply Quote
Ethan

Send message
Joined: 11 Feb 06
Posts: 18
Credit: 25,579
RAC: 0
Message 2157 - Posted: 16 Aug 2006, 23:04:19 UTC - in response to Message 2156.  


I know that it must be easier to simply change a few settings with the scoring algorythms, but believe me the current system works, it's merely got a backdoor that needs locking.


That's the problem, how do you lock it? Boinc and Rosetta are seperate things. Rosetta has nothing to due with credit calculation other than returning the time it took to complete the WU. Boinc then takes that, does some math based on your benchmarks (or edited benchmarks), and returns the results to the server. Rosetta has to trust the credit claims since the benchmarking part of Boinc is out of its control.

The only way to close the backdoor is to have a new version of Boinc developed and require everyone upgrade to have their credits counted. . . Or get into the business of determining if each seemingly large score is real or inflated.

ID: 2157 · Report as offensive    Reply Quote
dcdc

Send message
Joined: 15 Aug 06
Posts: 26
Credit: 89,834
RAC: 0
Message 2158 - Posted: 16 Aug 2006, 23:11:34 UTC - in response to Message 2156.  

prior to the currently in-testing credit system, I thought that flops counting was the ideal, with the benchmark as either a library to be included in the BOINC code during compilation (don't know if it'd be possible to create a moduel that isn't 'tamperable'), or a standard benchmark to be included in the project apps. Infact I remember my very first DC post over at UD said the very same thing back in 99 or 2000! I haven't seen anything that says that this is an accurate system though- as Ethan says, what value to you assign to FPU operations over Integer operations etc. There is some arbitration even in that system isn't there? (I might be wrong - my understaning of programming is pretty minimal beyond VB...)
ID: 2158 · Report as offensive    Reply Quote
Aaron Finney

Send message
Joined: 16 Feb 06
Posts: 56
Credit: 1,457
RAC: 0
Message 2159 - Posted: 16 Aug 2006, 23:12:32 UTC - in response to Message 2157.  
Last modified: 16 Aug 2006, 23:19:03 UTC


I know that it must be easier to simply change a few settings with the scoring algorythms, but believe me the current system works, it's merely got a backdoor that needs locking.


That's the problem, how do you lock it? Boinc and Rosetta are seperate things. Rosetta has nothing to due with credit calculation other than returning the time it took to complete the WU. Boinc then takes that, does some math based on your benchmarks (or edited benchmarks), and returns the results to the server. Rosetta has to trust the credit claims since the benchmarking part of Boinc is out of its control.

The only way to close the backdoor is to have a new version of Boinc developed and require everyone upgrade to have their credits counted. . . Or get into the business of determining if each seemingly large score is real or inflated.


Didn't you read my post? I could have swore you did by your previous reply, but now I think you must have missed it.. In it, I said quite clearly to put this in the hands of the Rosetta core application. Go back and read the last 5-10 posts, it'll explain it in it's entirety.

You don't need BOINC to assign scores. It only does this as a courtesy to facilitate projects. Someone needs to get with David and/or Rom, and ask how you can incorporate benchmarking into the projects core app.
ID: 2159 · Report as offensive    Reply Quote
Profile Jack Shaftoe

Send message
Joined: 8 Aug 06
Posts: 13
Credit: 105,163
RAC: 0
Message 2160 - Posted: 16 Aug 2006, 23:32:39 UTC - in response to Message 2153.  

Solution : The Rosetta core application should do it's own benchmarking, and assign encrypted scoring values inside each workunit. BOINC level scoring systems could be bypassed on a per project basis, or per workunit basis even if you wanted to. The validator could be changed to decrypt these values on the fly, and everyone is happy. Although now that I think about it, you don't -have- to give this responsibility to the validator, you could require some background communication from Rosetta core whenever it needs to send in these values. It could send them in to the scheduler if you prefer.


Wow. Good stuff. Where were you for the last 3 months... ?
ID: 2160 · Report as offensive    Reply Quote
Aaron Finney

Send message
Joined: 16 Feb 06
Posts: 56
Credit: 1,457
RAC: 0
Message 2161 - Posted: 16 Aug 2006, 23:36:58 UTC - in response to Message 2160.  
Last modified: 16 Aug 2006, 23:37:51 UTC

Solution : The Rosetta core application should do it's own benchmarking, and assign encrypted scoring values inside each workunit. BOINC level scoring systems could be bypassed on a per project basis, or per workunit basis even if you wanted to. The validator could be changed to decrypt these values on the fly, and everyone is happy. Although now that I think about it, you don't -have- to give this responsibility to the validator, you could require some background communication from Rosetta core whenever it needs to send in these values. It could send them in to the scheduler if you prefer.


Wow. Good stuff. Where were you for the last 3 months... ?


I'm very sorry.. I have been busy at work. Unfortunately, I'm supposed to be working right now, but my passion for this issue is distracting me. :(
ID: 2161 · Report as offensive    Reply Quote
Profile dekim
Volunteer moderator
Project administrator
Project developer
Project scientist

Send message
Joined: 20 Jan 06
Posts: 250
Credit: 543,579
RAC: 0
Message 2162 - Posted: 16 Aug 2006, 23:38:43 UTC

Let's try to stay focused in this discussion.

We have already incorporated a simple benchmark in rosetta. But since we don't use redundancy people can still be malicious, so we decided not to go that route.
ID: 2162 · Report as offensive    Reply Quote
Profile Astro

Send message
Joined: 16 Feb 06
Posts: 141
Credit: 32,977
RAC: 0
Message 2163 - Posted: 16 Aug 2006, 23:39:18 UTC
Last modified: 16 Aug 2006, 23:39:58 UTC

Aaron, even the creators of Boinc (people of seti) couldn't close "the door", they implemented the FPOPS sytem to do just that. For some reason I have yet to understand, Fpops isn't useable here, so they're trying what they can. If the creators of boinc can't fix the existing benchmark system without an overhaul, what chance does the Rosetta staff have??? Oh, and the Benchmark system does have ONE decent flaw, it doesn't give linux users a comparable benchmark to it's windows counterpart (my lord, I'll be joining XS soon. LOL)
ID: 2163 · Report as offensive    Reply Quote
Aaron Finney

Send message
Joined: 16 Feb 06
Posts: 56
Credit: 1,457
RAC: 0
Message 2164 - Posted: 16 Aug 2006, 23:48:22 UTC - in response to Message 2162.  
Last modified: 16 Aug 2006, 23:52:50 UTC

Let's try to stay focused in this discussion.
We have already incorporated a simple benchmark in rosetta. But since we don't use redundancy people can still be malicious, so we decided not to go that route.


How can you be malicious (easily) if the core application has it's own encrypted method of 'benchmarking', the workunits have their own encrypted tags, and the results are also containing encrypted tags? I'm assuming that there are parts of the Rosetta Core application that are not available to the GP as source..

If you were to use a code (such as 512bit RSA? 1024-bit RSA enough?) that is (realistically) unhackable, then why do you think people could still alter their benchmarks if you control every step of the process?
ID: 2164 · Report as offensive    Reply Quote
Profile [B^S] Doug Worrall
Avatar

Send message
Joined: 16 Feb 06
Posts: 10
Credit: 1,515
RAC: 0
Message 2165 - Posted: 18 Aug 2006, 19:00:10 UTC



Hello,

Mmclasro you have a good point and a funny one, which is quite bewildering at the same time, and understandable untill Linux really takes off in a couple years. Seeing as Xp has been around long enough for Widows users to need a "Safe", "Secure" O.S..The differece in Bench,s for Linux isnot comparable.Assides from that, there are many
valid points within this Thread.Rosetta is "Great" Science, therfore running
Ralph is just as Paramount.I am Happy to be a part of something, that "could"
save 1 Human life, if nothing else, Keep crunching.
"Happy Crunching Boincateers"

Sincerely

Sluger
ID: 2165 · Report as offensive    Reply Quote
Profile dekim
Volunteer moderator
Project administrator
Project developer
Project scientist

Send message
Joined: 20 Jan 06
Posts: 250
Credit: 543,579
RAC: 0
Message 2170 - Posted: 19 Aug 2006, 0:31:35 UTC - in response to Message 2164.  

Let's try to stay focused in this discussion.
We have already incorporated a simple benchmark in rosetta. But since we don't use redundancy people can still be malicious, so we decided not to go that route.


How can you be malicious (easily) if the core application has it's own encrypted method of 'benchmarking', the workunits have their own encrypted tags, and the results are also containing encrypted tags? I'm assuming that there are parts of the Rosetta Core application that are not available to the GP as source..

If you were to use a code (such as 512bit RSA? 1024-bit RSA enough?) that is (realistically) unhackable, then why do you think people could still alter their benchmarks if you control every step of the process?


I agree, if we add encryption to rosetta, we could use the internal benchmark. It's something to consider.
ID: 2170 · Report as offensive    Reply Quote
Profile Trog Dog
Avatar

Send message
Joined: 8 Aug 06
Posts: 38
Credit: 41,996
RAC: 0
Message 2172 - Posted: 19 Aug 2006, 3:19:19 UTC - in response to Message 2164.  



How can you be malicious (easily) if the core application has it's own encrypted method of 'benchmarking', the workunits have their own encrypted tags, and the results are also containing encrypted tags? I'm assuming that there are parts of the Rosetta Core application that are not available to the GP as source..

If you were to use a code (such as 512bit RSA? 1024-bit RSA enough?) that is (realistically) unhackable, then why do you think people could still alter their benchmarks if you control every step of the process?


Would that produce the same benchmark regardless of OS?
ID: 2172 · Report as offensive    Reply Quote
Aaron Finney

Send message
Joined: 16 Feb 06
Posts: 56
Credit: 1,457
RAC: 0
Message 2173 - Posted: 19 Aug 2006, 3:59:17 UTC - in response to Message 2172.  



Would that produce the same benchmark regardless of OS?


I would assume so, yes. However, to get that result it may be only slightly more complicated.

For instance, the linux crowd may get one version of the benchmark, while the windows crowd would obviously get another.

Depending on how complicated the benchmark is, it could be only as difficult as tagging the compiler for your specific OS, or rewriting it as you see fit.

Either way - I only really see 3 maybe 4 OS versions being used currently, it surely couldn't be that difficult.
ID: 2173 · Report as offensive    Reply Quote
Profile feet1st

Send message
Joined: 7 Mar 06
Posts: 313
Credit: 116,623
RAC: 0
Message 2174 - Posted: 19 Aug 2006, 4:47:28 UTC

Aaron, certain have to agree that encryption is possible, because Rosetta is not open source. But to simply make an effort to encrypt the BOINC figures still apparently penalizes Linux, and allows for other means of modifying the BOINC benchmarks... and so I believe that is why David Kim suggests continuing to pursue the current credit system... because it provides the level playing field (all P4 2.4Ghz systems, baring any other disparities in capabilities should score the same, regardless of Linux or Windows or "optimized" or stock), and then you can later add the encryption, which provides an unmodified playing field. So the two aren't mutually exclusive.

Since Rosetta is but an application within BOINC... they aren't really in control of some of the code you are suggesting be changed. But encryption for the part they do control, the key portions of the .out file, is certainly possible. And, as I've expressed before, I believe it would be helpful. Although I'll point out that if you encrypt the whole file, then it adds to the mystic and the cynics which come along from time to time and assert that the whole project is a fraud, or that it is a virus that's stealing your data or whatever. So, just encrypt a tiny portion of the file, so folks can SEE what's being sent and gain further interest and knowledge about the science being done here.
ID: 2174 · Report as offensive    Reply Quote
Aaron Finney

Send message
Joined: 16 Feb 06
Posts: 56
Credit: 1,457
RAC: 0
Message 2176 - Posted: 19 Aug 2006, 6:55:11 UTC - in response to Message 2174.  
Last modified: 19 Aug 2006, 7:04:19 UTC

But to simply make an effort to encrypt the BOINC figures still apparently penalizes Linux, and allows for other means of modifying the BOINC benchmarks...


K.. I'll bite. How? and be specific, because what you are saying is that simply because the code is encrypted it punishes the linux crowd. forgive me if your statement lost me there. As for the second half of your statement, you're obviously not getting what I'm saying, I'm not talking about BOINC benchmarks.


Since Rosetta is but an application within BOINC... they aren't really in control of some of the code you are suggesting be changed. But encryption for the part they do control, the key portions of the .out file, is certainly possible.


C'mon, use your imagination here. What difference does that make? Is rosetta forbidden to make it's own files to send back to the validator / scheduler / some other server? I think not. People, if you want to take the benchmarking out of BOINC's hands, we have to start thinking outside the box. The box being - Currently we let the allmighty BOINC handle everything except wiping our butts. YOU DON'T HAVE TO LET BOINC DO ANYTHING! Period. How many times, and in how many different languages do I have to say it - BOINC only handles these functions as a courtesy for projects. Yes, it makes things easier, but we're not talking about something that can be done -within- BOINC are we?



And, as I've expressed before, I believe it would be helpful. Although I'll point out that if you encrypt the whole file, then it adds to the mystic and the cynics which come along from time to time and assert that the whole project is a fraud, or that it is a virus that's stealing your data or whatever. So, just encrypt a tiny portion of the file, so folks can SEE what's being sent and gain further interest and knowledge about the science being done here.


That's great if it can be done, but if it makes things too difficult, I wouldn't make it that high of a priority. Reasoning:

Those people are either...

1. Delusional.
2. Not downloading the project in EITHER case.
3. Not educated enough to know how to open up the program and know what it's doing ANYWAY.
4. Running the project, but like to be forum trolls.

In case of #1 - Non-Issue
In case of #2 - Complete Non-Issue
In case of #3 - Non-Issue
and in case of #4 - I'm sorry, I don't respond to trolls.

Assuming they aren't in one of the 4 listed scenarios above, What are you going to lose - 1, maybe 2 conspiracy freaks? Yeah.. Those people seem like real contributors to a DC project.


One thing I would like to add, is that one thing I remember David saying is that he lays awake at night thinking about how to get more members. I can't remember where I heard that or where he said it. However - If you change the credit system, you are GOING to upset people, no matter how much more logical useful, or fair the system that you're moving to seems. This is why I think it would be best to only fine-tune the existing system, rather than switch to one of a different type. Otherwise, are you prepared to disgruntle your participants? If the switch to BOINC's crediting system (remember trying to explain the cobblestone?) was any indication, I can guarantee you - It will disgruntle some of them. I know David definately doesn't want to be laying awake at night thinking about why the project lost ~5000 users.
ID: 2176 · Report as offensive    Reply Quote
MikeMarsUK

Send message
Joined: 8 Aug 06
Posts: 5
Credit: 0
RAC: 0
Message 2178 - Posted: 19 Aug 2006, 7:40:35 UTC - in response to Message 2170.  
Last modified: 19 Aug 2006, 7:40:08 UTC

...

I agree, if we add encryption to rosetta, we could use the internal benchmark. It's something to consider.


You'd also need to encrypt the in-memory benchmark results as well as the upload and download, also obfuscate the actual benchmark code so that people can't follow what it's doing in a debugger and modify the values on the fly. (Remember if people can use a debugger, an out-of-process routine can do the same once the memory locations are known).

The recent debugging code (map files etc) would prove extremely useful to someone trying to do this, but aren't necessary.

The new system does the job, and should do it well after the bugs are ironed out, so why mess around with insecure ways of doing the same thing?
ID: 2178 · Report as offensive    Reply Quote
Previous · 1 . . . 8 · 9 · 10 · 11 · 12 · 13 · Next

Message boards : Current tests : New crediting system



©2024 University of Washington
http://www.bakerlab.org