Posts by Ethan
|
1)
Message boards :
Current tests :
New crediting system
(Message 2157)
Posted 16 Aug 2006 by Ethan Post:
That's the problem, how do you lock it? Boinc and Rosetta are seperate things. Rosetta has nothing to due with credit calculation other than returning the time it took to complete the WU. Boinc then takes that, does some math based on your benchmarks (or edited benchmarks), and returns the results to the server. Rosetta has to trust the credit claims since the benchmarking part of Boinc is out of its control. The only way to close the backdoor is to have a new version of Boinc developed and require everyone upgrade to have their credits counted. . . Or get into the business of determining if each seemingly large score is real or inflated. |
|
2)
Message boards :
Current tests :
New crediting system
(Message 2155)
Posted 16 Aug 2006 by Ethan Post: Thanks for the last message Aaron. . I think it sums up what we agree on. The ideal would be something unhackable in Boinc iteslf. . but that wouldn't be easy due to the open source nature of the code and it would require everyone to install a new client to run Rosetta. A benchmark within the Rosetta itself would be great as well. . I'm not sure how feasible that is either. My understanding is the code branches a lot, many many function calls. Is it possible to count every calculation? The benchmark would have to be time independent. . otherwise a slow machine working for an hour would want the same score as a fast machine-hour. Back to counting calculations. . does an integer calculation could the same as a floating point? How about a double float? How much overhead is there in keeping track of billions of operations. Those C++'s would add up pretty quickly :) You could base a benchmark off how long it takes to crunch a 'golden' work unit. . but that one protein might use completely different parts of Rosetta than HIV research. . and having to create new 'golden' benchmark units, normalize them to each other, and package them in each new type of wu is a lot of overhead (for scientists and cpus). Just throwing those ideas out there. . there may be ways of doing them I hadn't though of. The lowest hanging fruit seems to be the proposed new system. It eliminates the advantages of the optimized client. People will still claim more credit than they deserve, but those will average with standard client claims, and everyone receives the same score. The only overhead is averaging the wu claims for each wu type. . and it will work with all versions of boinc and not require recoding of rosetta. |
|
3)
Message boards :
Current tests :
New crediting system
(Message 2149)
Posted 16 Aug 2006 by Ethan Post: No offense, but the logic for this change astounds me. The current system itself is not flawed in it's fairness to all - on the contrary, it is the most fair. We astound each other :) So if a PGA golfer finished a round and reported his score as 17 (you keep your own score in golf), the rest of the players have to accept it even though obviously false? Everyone who uses optimized clients are raising their credit claims to levels not based on the properties of their hardware. It's not a fair system when you have to use modified software (or subtract 60 from your golf score) to stay competitive. There is not less accuracy with the new system, there is more. . by definition of averaging out work unit claims. It's like a quorum of 100 (or 1000, however many are used to create the average). I agree that the uber solution would be to have the Boinc folks release a scoring system that's hidden in compiled code. . but that's not feasible when the code is open source and out of Rosetta's hands. Fair is the wrong word, but I can see why you use it. It's much more political than saying 'The credit system will now be harder to manipulate by malicious users, at the expense of accuracy.' I still fail to see how the new system is less accurate. Senario 1, everyone has optimized clients and claim whatever they feel like. Result = no accuracy whatsoever. Senario 2, half the people use optimized clients, the other half use standard ones. Result = a more accurate scoring system and level playing field. |
|
4)
Message boards :
Current tests :
New crediting system
(Message 2143)
Posted 16 Aug 2006 by Ethan Post: First, a question - I must say, DeKim (david?) I really, honestly think that you are going the wrong direction here. I understand that you want to change the existing credit system, and because of that it is safe to infer that you felt the existing system wasn't working. Why? The current system places credits in the hands of individual particpants. . you essentially keep your own score. Want more credits? Just claim more credits by making your benchmarks higher than possible (some computers claim 15+ Gflops per cpu. . even at 4ghz and running two floating point calculations a cycle, that's 8 Gflops. . and cpu's don't get near theoretical). Don't get too greedy or you'll get zerod out. It's like speeding, if you go the same speed as everyone else in the left lane, even if 10 over, you're not likely to get in trouble. . . if you're going 40 over in the median, you'll get busted. Not a very good system imo. The new system is taking the law of averages to make the playing field more fair. It's taking the average credit claim from hundreds of machines and applying the same score to each. |
|
5)
Message boards :
Current tests :
New crediting system
(Message 2136)
Posted 16 Aug 2006 by Ethan Post: mmciastro, maybe I'm reading your graph wrong (thanks for posting it btw). . . but it looks very consistant on a credit/hour basis for a given computer. Is this the case? And if so, how do you feel about the credits/hour for the various machines (is machine 2 really about 2.5 faster than machine 1)? |
|
6)
Message boards :
Current tests :
New crediting system
(Message 2132)
Posted 16 Aug 2006 by Ethan Post: It's pretty easy to pick out extremes. The majority of wu's will be in the same ballpark, and if grossly off, they could probably be recalculated. The differences pointed out below are still only half as large as the difference between standard and optimized credits (if not more, I just checked rosetta, an identical computer to mine gets 3000 credits a day, I get 600 using the standard client). . so that's an improvement rather than a step back. |
|
7)
Message boards :
Current tests :
New crediting system
(Message 2078)
Posted 16 Aug 2006 by Ethan Post:
Optimized boinc clients don't make rosetta run any faster. Check out task manager (or top if you're linux). The app using your cpu time is rosetta, not boinc. Compiling boinc to give you a million credits an hour has no affect on how fast Rosetta runs. |
|
8)
Message boards :
Current tests :
New crediting system
(Message 2076)
Posted 16 Aug 2006 by Ethan Post:
I'm not suggesting the science claims are false, there hasn't been a case of that happening I'm aware of. I'm only suggesting the credit claim needs to be re-evaluated. It's not a matter of optimized or standard client scores. . it's a matter of work done for the project. If you have the most uber quad core overclocked liquid nitrogen powered rig dedicated 100 percent to Rosetta, you should be at the top of the leader board. . . you shouldn't be there just because you could edit a text file to add a couple zeros to your flops count. |
|
9)
Message boards :
Current tests :
New crediting system
(Message 2072)
Posted 16 Aug 2006 by Ethan Post: Star Trek references aside :) The governement makes you accountable for false claims on past returns. . why should people be given passes on clients that granted more credits than the standard one? Because they were the first to use it they should be given amnesty? What do they have against a level scoring system? If they are number one in the current credit system, they should still be number one when the actual work is calculated. . right? -E |
|
10)
Message boards :
Current tests :
New crediting system
(Message 2068)
Posted 16 Aug 2006 by Ethan Post: Given Zombie67's point (which is valid in my comparison in terms of investments, thanks). . . I think it's fare to reduce the credits of those who 'tricked' the system into accpeting an unreasonable value to start with. 9 out of 10 identical worksations reported a similar wu time, should the 10th be given 5x the credits just because the software was set different than the rest? Where on Rosetta's homepage did it say to download a different client to receive more credits? Users who read the Associated Press article who had no idea of other projects. . they should be punished because they didn't research things they had never heard of? How does an apples to apples comparison hurt anyone? If you did the work, you get the credit. . if you did the work faster than the other guy, you get more credit. I'm not sure how this hurts anyone. |
|
11)
Message boards :
Current tests :
New crediting system
(Message 2066)
Posted 16 Aug 2006 by Ethan Post: Given Zombie67's point (which is valid in my comparison in terms of investments, thanks). . . I think it's fair to reduce the credits of those who 'convinced' the system into accpeting an unreasonable value to start with. 9 out of 10 identical worksations reported a similar wu time, should the 10th be given 5x the credits for no reason? How does it hurt anyone to go apples to apples back to Feb? |
|
12)
Message boards :
Current tests :
New crediting system
(Message 2061)
Posted 16 Aug 2006 by Ethan Post: At the time people joined, a credit system was promised. You cannot pull that away from people retroactively. It's deceitful. You want to chage it going forward? Fine. Change the rules, tell people, and let them make their own decision to stay or go. Ok. . If you owned a bank and users who used Firefox got 5 percent interest, but users who used Opera only got 3 percent. . . Users of Opera would find out they were being short changed and expect the same return on their investment. Why should Opera people be denied their credits from back in the day? The only reason I can think of is so people who 'enhanced' their credits can keep that advantage. Since credits are an important metric, wouldn't everyone want them to be as neutral as possible? How is it less neutral to start them back in Feb rather than Aug? Especially if the old credit count is still visible on the Rosetta webpage. |
|
13)
Message boards :
Current tests :
New crediting system
(Message 2059)
Posted 16 Aug 2006 by Ethan Post: delete me. . must have been a duplicate post |
|
14)
Message boards :
Current tests :
New crediting system
(Message 2058)
Posted 16 Aug 2006 by Ethan Post: Totally against backdating the changes. Not nessesary nor needed - Time to fix the problems and just go forward from here. Why does it matter if old credits are re-evaluated? I think everyone agrees that credits are a benchmark for how much work a given user/computer does for the project. Now that a much more accurate mechanism exists to grant credit, why is it a negaive to retroactively re-score the user base? Should peple who intentially modified their credit 'wage' be granted a higher score than people who read a newspaper article and joined? I'm not for or against starting fresh or going back go Feb. . I think whatever results in the most computers crunching R@H is the correct outcome. . but proposing a scoring system based on user results when they are known to be fiddle-able, from users whose computes are hidden. . it just doesn't sound right. -Ethan |
|
15)
Message boards :
Current tests :
New crediting system
(Message 1936)
Posted 8 Aug 2006 by Ethan Post: Does granting credits on how long each computer was running the application separate how much they worked? I mean, does the server give the same amount of credits both to the machine which has Pentium and to that which has Conroe, if they run applications for the same length while Conroe can crunch more? Once ralph determines how many credits to grant for each simulation produced, then it only depends on how fast your computer is to get that many credits. A brand new 3ghz core processor will get many more credits per day than a p3-1ghz since it will be able to process more simulations in the same amount of time. |
|
16)
Message boards :
Current tests :
New crediting system
(Message 1933)
Posted 8 Aug 2006 by Ethan Post:
Hi Tony, It partially does, yes. I'm more curious that since both the whetstone/drystone times can be faked, as well as apparently runtime in the Trux client. . . would it be better to put a cpu time capture in the Rosetta code itself. Since Rosetta@home code is compiled in the lab, results shouldn't be fake-able. I don't know if it's the way to go, but it should get rid of any appearance of people getting a higher score due to 3rd party boinc clients. Oh, and thanks for the hello, I've been around awhile - ID: 2 :) |
|
17)
Message boards :
Current tests :
New crediting system
(Message 1931)
Posted 8 Aug 2006 by Ethan Post:
Is that a Boinc client? What if the time was kept within the Rosetta code (which is compiled and can't be manipulated as far as I know)? |
|
18)
Message boards :
Current tests :
New crediting system
(Message 1927)
Posted 8 Aug 2006 by Ethan Post: Why not just use Ralph to determine the average crunch time for each simulation for a given WU? I don't believe there's a way to manipulate the amount of time it takes to process, then that average time can be compared to a 'golden' ratio of credits/cpu hour for an average machine. The ratio would have to be revisited every couple months since computers will get faster over time, but this way, the credit system is completely bypassed (and its inherent problems). -E |
©2024 University of Washington
http://www.bakerlab.org