| Author | Message |
|---|
|
|
|
Please post comments and suggestions regarding the new crediting system here.
____________
|
|
|
|
|
Please post comments and suggestions regarding the new crediting system here.
Are there any jobs to run? I keep getting \"communication deferred\", and no jobs downloaded.
TIA
____________
Dublin, CA
SETI.USA - Stats - My stuff - BOINC IRC chat
 |
|
|
|
|
Please post comments and suggestions regarding the new crediting system here.
Are there any jobs to run? I keep getting \"communication deferred\", and no jobs downloaded.
TIA
Oh sure.... As soon as I posted this, the jobs downloaded.
Nevermind. =;^)
____________
Dublin, CA
SETI.USA - Stats - My stuff - BOINC IRC chat
|
|
|
|
|
|
Is there some basis used to come up with this initial estimate of 2 cobblestones/model, or was it a \"swag\" (scientific wild a** guess) LOL??
Are you now collecting data about the Avg decoys/WU per hour and comparing it to current values issued (with standard boinc client)?
Have you tried to identify a \"Golden\" machine (avg machine) and base credit issuance quantities based on that?
What weight does Rosetta place on credit parity across all projects?
Is there something we could do?
tony |
|
|
|
|
Please post comments and suggestions regarding the new crediting system here.
Are there any jobs to run? I keep getting \"communication deferred\", and no jobs downloaded.
TIA
Oh sure.... As soon as I posted this, the jobs downloaded.
Nevermind. =;^)
Okay, *now* I am getting the message \"No work from project\". This both on my intel mac, and my amd x2 linux machine, if that makes any difference.
____________
Dublin, CA
SETI.USA - Stats - My stuff - BOINC IRC chat
|
|
|
|
|
|
...and adding to mcciastro\'s questions: Will the two credits per structure be for one particular WU type or accross to board, independently of how long the strucutures take to complete ? It would be nice to let us know what you actually have in mind. Thanks, -H.
____________
|
|
|
|
|
Is there some basis used to come up with this initial estimate of 2 cobblestones/model, or was it a \"swag\" (scientific wild a** guess) LOL??
Are you now collecting data about the Avg decoys/WU per hour and comparing it to current values issued (with standard boinc client)?
Have you tried to identify a \"Golden\" machine (avg machine) and base credit issuance quantities based on that?
What weight does Rosetta place on credit parity across all projects?
Is there something we could do?
tony
I took it from a result from one of my computers as just a very rough value to test with. The version that will eventually run on Rosetta@home will have work unit specific credit per model values that are determined from test runs on Ralph. It will be a requirement for lab members to not only test new work units on Ralph but to also determine the average credit per model value from their test runs for production runs. The credits should remain somewhat consistent with other projects since the average values will be based on the standard boinc crediting scheme. If things look okay on Ralph, Rosetta@home will use the credit per model crediting method while Ralph will switch back to the standard method.
____________
|
|
|
|
|
I took it from a result from one of my computers as just a very rough value to test with. The version that will eventually run on Rosetta@home will have work unit specific credit per model values that are determined from test runs on Ralph. It will be a requirement for lab members to not only test new work units on Ralph but to also determine the average credit per model value from their test runs for production runs. The credits should remain somewhat consistent with other projects since the average values will be based on the standard boinc crediting scheme. If things look okay on Ralph, Rosetta@home will use the credit per model crediting method while Ralph will switch back to the standard method.
OK, great - in fact this seems to be very much along the lines I was thinking myself... -H. :-)
____________
|
|
|
|
|
|
How do you want to determine credit/model from Ralph runs? As an average of claimed credit based on X runs? As a median? Keep in mind that the reported specs and benchmarks on RALPH can be as manipulated for some hosts as on Rosetta. Some use special BOINC versions which claim about 3x the credit the standard client claims (including myself).
Possibly an average of claimed credit/model with 10 or more results is reliable enough to be used for Rosetta. If you take lesser results you have a higher impact of distorted credit reports on some WU and thus a WU which will give more credit and other which will give lesser credit (which is not good since people start cherrypicking \"good\" WU by aborting \"bad\" WU)
____________
|
|
|
|
|
|
tralala - the aim is to avoid and abandom always and ever ill-numbered benchmarks.
(or at least I hope and pray).
You can simply choose CPU type, divide it with CPU frequency - or golden machine, as tony suggested. I know it\'s not perfert, RAM speed takes place etc.
But you should fit in acceptable +- 10%, not something like 500% with benchmarks.
Just try to maintain inter-project parity, that will do...
____________
|
|
|
|
|
tralala - the aim is to avoid and abandom always and ever ill-numbered benchmarks.
(or at least I hope and pray).
You can simply choose CPU type, divide it with CPU frequency - or golden machine, as tony suggested. I know it\'s not perfert, RAM speed takes place etc.
But you should fit in acceptable +- 10%, not something like 500% with benchmarks.
Just try to maintain inter-project parity, that will do...
Well so far they revealed no details how they plan to establish the credit/model ratio. As I see it, there are two approaches. What you describe would require a bunch of trustable computers, preferably locally available, with different processors and OSs. 10-20 various machines would be probably sufficient for that (one golden machine will not work, since it will prefer one CPU-type over others and one OS over others). That is not a task for Ralph I\'d say since computer on RALPH are not trustable.
The second approach is different and just relies on the force of numbers where anything will balance out with more sampling. This would mean sending out WU on Ralph to \"untrustable\" hosts but with a lot of results, so that on average you will have also correct values.
The bottom line is, it is a tricky task which requires some testing and tuning until it works flawlessly. I hope they will discuss it with us before they use it over at Rosetta to avoid fixing it while it is being used at Rosetta.
____________
|
|
|
|
|
|
Okay here is one proposal:
Send each type of WU to at least 60 hosts. Discard the results with the lowest and highest 10% claimed credit and take the arithmetic average of the remaining hosts as fixed credit/model for production runs on Rosetta.
____________
|
|
|
|
|
Okay here is one proposal:
Send each type of WU to at least 60 hosts. Discard the results with the lowest and highest 10% claimed credit and take the arithmetic average of the remaining hosts as fixed credit/model for production runs on Rosetta.
But doesn\'t this way cut fair highest and lowest results at high rate? There may be some people who runs R@H on Pentium(I, II) or Kentsfield now. Especially a few users use genuine Kentsfield as testing already.
If the way would be applied, how about distributing WUs to same/similar spec\'s machine? This way could help the server to distinguish original claimed results from ones modified. Using coefficients figured of machine\'s spec, etc. I fear, however, this might be issue of client frameworks.
If thinking realistically, changing \"quorum\" from 1 to 3 is the easiest and more effective way applied overall. changing it to 5 isn\'t good:(
____________
|
|
|
|
|
|
Why not just use Ralph to determine the average crunch time for each simulation for a given WU? I don\'t believe there\'s a way to manipulate the amount of time it takes to process, then that average time can be compared to a \'golden\' ratio of credits/cpu hour for an average machine. The ratio would have to be revisited every couple months since computers will get faster over time, but this way, the credit system is completely bypassed (and its inherent problems).
-E
____________
|
|
|
|
|
|
no work from project :(
____________
|
|
|
|
|
Why not just use Ralph to determine the average crunch time for each simulation for a given WU? I don\'t believe there\'s a way to manipulate the amount of time it takes to process, then that average time can be compared to a \'golden\' ratio of credits/cpu hour for an average machine. The ratio would have to be revisited every couple months since computers will get faster over time, but this way, the credit system is completely bypassed (and its inherent problems).
-E
The time can be manipulated by truxes client. 5.3.12tx36 |
|
|
|
|
Okay here is one proposal:
Send each type of WU to at least 60 hosts. Discard the results with the lowest and highest 10% claimed credit and take the arithmetic average of the remaining hosts as fixed credit/model for production runs on Rosetta.
But doesn\'t this way cut fair highest and lowest results at high rate? There may be some people who runs R@H on Pentium(I, II) or Kentsfield now. Especially a few users use genuine Kentsfield as testing already.
If the way would be applied, how about distributing WUs to same/similar spec\'s machine? This way could help the server to distinguish original claimed results from ones modified. Using coefficients figured of machine\'s spec, etc. I fear, however, this might be issue of client frameworks.
If thinking realistically, changing \"quorum\" from 1 to 3 is the easiest and more effective way applied overall. changing it to 5 isn\'t good:(
I don\'t think that idea would unfairly cut any honest machine. It may not be the best way though, since it would tend to give higher numbers than expected if there are many cheating hosts. Maybe it would work better to throw out 10% of the highest standard deviations. In that case all the results thrown out could be high if there is a concentration of cheaters, but come from both ends if all standard clients get the same task.
Any attempt to change the quorum would not work since the number of structures is user configurable.
____________
BOINC WIKI
 
BOINCing since 2002/12/8 |
|
|
|
|
The time can be manipulated by truxes client. 5.3.12tx36
Is that a Boinc client? What if the time was kept within the Rosetta code (which is compiled and can\'t be manipulated as far as I know)?
____________
|
|
|
|
|
The time can be manipulated by truxes client. 5.3.12tx36
Is that a Boinc client? What if the time was kept within the Rosetta code (which is compiled and can\'t be manipulated as far as I know)?
Ethan, welcome to ralph by the way.
Trux 5.3.12tx36 is an optimized Boinc Core Client. The claimed credit formula currently is (whetstone+Dhrystone) * Cpu time (in seconds)/172800. Most optimized Boinc Core clients change the Benchmarks, but Truxs alters the time reported and benchmarks to get more credit.
does this answer your question?
tony
here is a Result ID from a trux client:
stderr out <core_client_version>5.3.12.tx36</core_client_version>
<real_cpu_time>2503</real_cpu_time>
<corrected_cpu_time>3930</corrected_cpu_time>
<corrected_Mfpops>11126.2</corrected_Mfpops>
see how it\'s \"corrected\" the time and benchmark? |
|
|
|
|
Ethan, welcome to ralph by the way.
Hi Tony,
It partially does, yes. I\'m more curious that since both the whetstone/drystone times can be faked, as well as apparently runtime in the Trux client. . . would it be better to put a cpu time capture in the Rosetta code itself. Since Rosetta@home code is compiled in the lab, results shouldn\'t be fake-able.
I don\'t know if it\'s the way to go, but it should get rid of any appearance of people getting a higher score due to 3rd party boinc clients.
Oh, and thanks for the hello, I\'ve been around awhile - ID: 2 :)
____________
|
|
|
|
|
. . . would it be better to put a cpu time capture in the Rosetta code itself. Since Rosetta@home code is compiled in the lab, results shouldn\'t be fake-able.
Not sure how the \"time capture\" works... wish I could use that for my self :) but once Rosetta determines the time, it should \"encrypt\" (not \"encode\") the number of CPU seconds in a way that only the Rosetta host can authenticate. And it should be encrypted along with something fairly random such as the complete WU name and the expiry date/time. That way you can\'t just replace a 3000 second WU with the time data from a 10000 second WU and have it pass authentication.
____________
|
|
|
|
|
|
Does granting credits on how long each computer was running the application separate how much they worked? I mean, does the server give the same amount of credits both to the machine which has Pentium and to that which has Conroe, if they run applications for the same length while Conroe can crunch more?
____________
|
|
|
|
|
Does granting credits on how long each computer was running the application separate how much they worked? I mean, does the server give the same amount of credits both to the machine which has Pentium and to that which has Conroe, if they run applications for the same length while Conroe can crunch more?
Once ralph determines how many credits to grant for each simulation produced, then it only depends on how fast your computer is to get that many credits. A brand new 3ghz core processor will get many more credits per day than a p3-1ghz since it will be able to process more simulations in the same amount of time.
____________
|
|
|
|
|
|
One thing one should keep in mind is that nothing which is reported from the clientside is trustable, not the benchmarks, not the cpu-type and speed - nothing! Anything can be edited and faked. So every scenario which relies on the accuracy of the data from (single) clients should be discarded. However on average there are more accurate and valid information than faked ones. So any mechanism which takes averages from big numbers should in fact work quite well - even the claimed credits which depend on the reported benchmarks. If the sampling is big enough (>100) no discarding of odd results might be necessary but in any case I like the idea of throwing out 10% of the highest standard deviation.
Ethan
Your idea of measuring the average time per model and compare it with a golden computer assumes that the computer pools on Rosetta and Ralph are similar, which is probably not true. I assume on Ralph there are on average faster computer than on Rosetta. This makes the idea imho unpractical.
____________
|
|
|
|
|
but in any case I like the idea of throwing out 10% of the highest standard deviation.
Could you tell me why do you think the value of the deviation should be 10%, the constant?
____________
|
|
|
|
|
but in any case I like the idea of throwing out 10% of the highest standard deviation.
Could you tell me why do you think the value of the deviation should be 10%, the constant?
Just a rough number thrown out. The actual cut-off would depend on how many copies are sent out and what percentage of misclaiming hosts we have on RALPLH.
____________
BOINC WIKI
BOINCing since 2002/12/8 |
|
|
|
|
but in any case I like the idea of throwing out 10% of the highest standard deviation.
Could you tell me why do you think the value of the deviation should be 10%, the constant?
Just a rough number thrown out. The actual cut-off would depend on how many copies are sent out and what percentage of misclaiming hosts we have on RALPLH.
Thanks.
Has the formula which determines the value submitted already or not yet?
edit: the issue is argued also on the Rosetta\'s thread. I think it should be combined with this thread.
____________
|
|
|
|
|
|
If inter-project parity is one of the objectives, I\'m not seeing it yet.
I started tracking credit when Seti devs asked for help with that, during the flame wars around their newest application. At that time I observed that my machine was generally collecting 9-10 credits per cpu hour on most projects. (Windows XP, standard BOINC client, no optimized apps except Einstein\'s \"official\" beta app). This # was not significantly affected by considering or ignoring the larger quotas on most projects I run.
My time setting for Ralph is 2 hours, and I was getting 20 credits or so per wu. Based on only the 3 wus credited so far (sorry not more, I\'ve been gone for 3 weeks & Ralph\'s workstream is fairly thin), 2 have wildly higher amounts (58 & 60 credits) and 1 much lower (6 credits). Actually I see the \"claimed credit\" is showing the older calculation so you can see the difference. Here\'s my results list.
Edit: a 4th wu came in with credit (24) just a bit higher than previous-normal.
I know you\'re just starting to experiment with this, but it seemed worth a comment. (Though I\'m cynically guessing that \"extra\" credit wouldn\'t cause the uproar Seti\'s perceived \"lesser\" credit caused...)
____________
 |
|
|
|
|
If inter-project parity is one of the objectives, I\'m not seeing it yet.
Inter-project parity is not an objective on RALPH and thus it may never happen on RALPH.
A credit of 2 per model is being given no matter how slow or fast the WU in order to test the awarding of credit.
As far as I can tell, the test is working fine and credit is being awarded as intended.
|
|
|
|
|
If inter-project parity is one of the objectives, I\'m not seeing it yet.
Inter-project parity is not an objective on RALPH and thus it may never happen on RALPH.
A credit of 2 per model is being given no matter how slow or fast the WU in order to test the awarding of credit.
As far as I can tell, the test is working fine and credit is being awarded as intended.
I\'m not asking about RAlph, I\'m asking about this scheme which is intended to be used, eventually, on Rosetta -- where Dekim\'s post earlier in this thread implies that parity remains an objective, at least \"somewhat\". I understand that the current per-model value is for test purposes only, and I see that credit is being awarded as described, just wondering about the path from here to project parity in production mode.
If I understood his description, it sounds like a lot of work for the lab members to have to run each wu, and come up with and record a credit-awarding value into it before it can be released to production.
The project team has a great track record so the above must not be a problem, but it will be interesting to see this work out over time. |
|
|
|
|
Ethan
Your idea of measuring the average time per model and compare it with a golden computer assumes that the computer pools on Rosetta and Ralph are similar, which is probably not true. I assume on Ralph there are on average faster computer than on Rosetta. This makes the idea imho unpractical. Not being Ethan but anyway...
No, it doesn\'t. Damn credit award can be postponed until each model gets cralibrated (in term of credit) on Ralph.
Or, taking on your idea, credit award can be estimated once a 100 hosts(or so) returns each results type (no need for ralph estimation)
Each model, as I understand, is not constant in terms of computng demands.
If it would be so...or better make it so - we can use calibration units.
The downside of this is that credit can\'t be graned immediately (not a bad one) and there will be more results rending.
____________
|
|
|
|
|
|
I think it\'s dangerous having a fixed global credit award for models produced. The variability of models per time is significant from one WU to the next. This would encourage cherry picking high model WU\'s and favor machines that are handed more of that typ of work. (Or at least run the risk of seeing an increase of aborted work on WU\'s that are found to produce a few number of models.) Examples from the same machine:
WU: 1qysA_BOINC_ABRELAX_SAVE_ALL_OUT__1108_42_0
Ran for 7021 seconds
Produced 5 models
Awarded 10 credits
WU: 1l2yA_BOINC_ABRELAX_SAVE_ALL_OUT__1108_42_0
Ran for 7195 seconds
Produced 62 models
Awarded 124 credits
It would be far better to take some scaling into account for each WU. Perhaps Ralph could assist in providing this figure to carry over into Rosetta?
EDIT: Just saw the post that this is indeed the plan, which should provide a very equitable credit system for the project. I\'m eager to see some of this testing take place.
____________
|
|
|
|
|
|
I have finished a WU, apparently with the new credit system. Here are the results:
Win XP Pro sp2
Pentium 4 3.0 HT
RAM 1Gb
BOINC 5.5.0 ;-)
WU: 1qysA_BOINC_ABRELAX_SAVE_ALL_OUT_BARCODE__1109_99
CPU Time: 21127 seconds
Models: 17
Claim credits: 98.1
Granted credits: 34.00
Average credits: 5.80 per hour
____________
|
|
|
|
|
|
http://ralph.bakerlab.org/result.php?resultid=238822
One of mine:
BOINC Alpha 5.5.9
Result ID: 238822
Name: 1enh__BOINC_ABRELAX_SAVE_ALL_OUT_flatss__1111_596_0
CPU Time: 3301sec = around 55min
Claimed credit: 4.88809081678063
Granted credit: 7(models)x2 = 14
Average credit: 12.83/h
____________
|
|
|
|
|
|
Here\'s a project comparison using what LITTLE ralph data I have. The other projects haven\'t been brought up to date yet and I have many more data points to add to them, so I wouldn\'t use this as anything more than a general idea of where you\'re at. I\'ve set other projects to NNW until I can get a decent sample quantity. I highlighted the \"ralph-new\" values in red. Since Rosetta and OLD Ralph used the same credit formula, you can compare the new ralph to the old ralph and even to Rosetta on each puter. Note: all the numbers here are from stock boinc core clients.
 |
|
|
|
|
|
Is ralph planning on sending more work? My results so far have been all over the map, and don\'t have enough data to draw any conclusions one way or the other.
tony |
|
|
|
|
|
There seems to be a lot of various misunderstandings floating around in this thread. I am no more \"in the know\" then any other participant, but I wanted to try and spell out my understanding of the new system and why it is reasonable, and address some of the concerns people have express below. I hope this help set a stake in the sand, and at least give a frame of reference. If I\'m off base then project team just has to identify where, rather than write up the whole of the details, so hopefully it leads to a clearer understanding.
Firstly, David Kim\'s comment:
The credits should remain somewhat consistent with other projects since the average values will be based on the standard boinc crediting scheme.
1) It was stated in passing. Kind of a reassurance that we\'re not trying to change the whole credit VALUE here. That the new system should remain in line with the old for the most part. It wasn\'t stated as an objective or a target to achieve. Since BOINC credits are SUPPOSED to represent the TFLOPS of the project, ya kinda have ta stick to that. It doesn\'t endorse any flawed system (read more below). It simply falls in line with the 100,000 credits = 1 TFLOPS system that BOINC uses.
2) YES, the fact that some machines are faster then others is fully accounted for. The slower machine will take longer to complete each \"model\". Models are not \"work units\". The number of models generated for a work unit varies by the user\'s time preference, and computer resources. This is why a credit system based on number of models crunched makes the most sense.
3) Yes, time to crunch a single model of a 400 amino acid long protein is much higher then the time to crunch a single model of a 55 amino acid long protein. This is fully recognized. No cherry picking is possible, because the credit granted per model for the first will be significantly higher then the credit granted per model for the second.
4) The idea (as I understand it), is to send a few of a given protein WUs out to Ralph and get a feel for how long each model takes to crunch for THAT WU. This credit value will then be used on Rosetta to award credits. So, from the moment the WU is released on Rosetta, the credit value is defined, fair, and requires no quarum or average or waiting for credit granting. All that work, or weeding or standard deviation and reversion to mean was done ahead of time on Ralph.
5) What\'s the deal with the 2 credits?? As I read it, David Kim found that for the WU he\'s released on Ralph, that the old system would have awared about 2 credits per model. The actual number doesn\'t matter. The point is that they are testing the idea of having a fixed value per model for THAT specific WU determined ahead of time. Because this is how Rosetta will be running. It will be handed a set of WUs to send out, along with the credit to award per model crunched.
6) You can\'t trust the client PCs... correct. Even with the new system it will be possible to falsify your results. The only value we\'re trusting from the client is the number of models they crunched on that protein... and if they report 50 models crunched, then obviously they\'ve got to report the results of the 50 crunched models... this COULD be falsified. No doubt! (unless you encrypt all of the output data so that only the Rosetta servers can authenticate it) So, you see, we\'re not counting on the client accurately reporting number of seconds crunching, or Ghz of it\'s CPU or FLOPS measured. It levels the playing field about how much actual work your machine did to help the project.
7) Why use Ralph to assess the credit value per model? Why not just use a \"golden computer\" in the lab?? ...because that golden computer has a fixed architecture with no variation. Technology is changing all the time. Say a new system comes out with dual math co-processors, and it\'s capable of doing roughly twice the actual Rosetta work of it\'s predecessor, at only a 30% higher clock speed on the base CPU... how do you determine credit for that new machine? You can\'t just take clock speed, because it\'s only 30% faster. You can\'t just take floating point speed, because there are many other factors in performance. By using Ralph, a wide variety of machines and operating systems can be tested for each given work unit. This gives the fairest perspective on how difficult it is to crunch this new protein, and how much harder it is then the last protein that was studied. It\'s kinda the opposite of the golden machine. And that is better, because the field is always going to be changing. This approach automatically callibrates itself as system capabilities evolve.
I hope I\'m helping here. I\'ve been hoping someone from the project team would have spelled all of this out for us, but I think they are probably still zonked from CASP7 :)
Again, the above is just my understanding of things. Not to be taken as gospel by any means. But if you have specific concerns about the above approach, I hope this gives us all a frame of reference for discussion. And I\'ve numbered paragraphs in hopes of NOT seeing all of this text quoted in responses.
____________
|
|
|
|
|
|
Thanks for your brief explanation:) Now I\'ve understood the system, I\'m for it.
____________
|
|
|
|
|
|
That seems a good solution.
However, the credit compaired to other projects will still be to high. Now it\'s impossible to compair Rosetta with other projects, because the claimed credit is always rewarded.
Changing the \"quorum\" from 1 to 3 on RALPH would solve that too.
____________
|
|
|
|
|
|
The 2 points / model on this WU is not even close.
http://ralph.bakerlab.org/result.php?resultid=240301
Lets se if it evens out :)
Anders n
____________
|
|
|
|
|
|
The 2 credits per model was just a stab in the dark. First trial. But the idea is that Ralph credits are disposable. So, run the WUs on Ralph, establish an appropriate credit per model. Then roll out to Rosetta using that established credit value per model.
The other idea is to use Ralph as a testing ground. And so they have to test all of the server code that they\'ve changed that\'s keeping track of credits claimed (on the old system) with the new credits. And so to run Ralph in the mannar the most similar to how Rosetta will be run, SOME arbitrary value had to be established before sending out WUs. Because this is how Rosetta will run.
Perhaps, with the experience on Ralph it is determined that a more reasonable value is 5.53 credits per model, or whatever. But they need to practice running under an approach where the number of credits per model is established up front, in order to help assure there are no glitches when rolling out to Rosetta.
____________
|
|
|
|
|
|
does anyone object to rolling this out to Rosetta@home?
____________
|
|
|
|
|
|
That seems OK to me. And with this you know exactly what you have done (as Seti Classic did).
The principle seems OK but... I have not followed all the discussions and there is something I don\'t understand very well.
I take an example:
- I have one PC which runs Rosetta with a \'Target CPU run time\' of 10 hours. The current running WU runs for 9 hours and have 143 models. So it will be something like 318 credits for 10 hours.
- I have another PC with a \'Target CPU run time\' of one day. The current WU runs for 2.5 hours and have now 5 models. That\'s something like 96 credits for 24 hours.
Isn\'t that strange?
____________
|
|
|
|
|
|
Not \"objecting\", but I\'d be prepared for more uproar given what I\'ve seen, especially since there\'s already been some heat on the boards there.
If this credit-per-model scheme can\'t end up yielding some consistent amount of credit per cpu hour on a given machine, and preferably something close to that machine\'s results on other proejcts, there will be confusion at best and likely more conflict around credit, rough project parity etc.
From my results so far it\'s nowhere near consistent, as mmciastro said they\'re all over the map. Though if you\'re rolling this out with the figure-out-the-right-credit-award-first mechanism already in place, that will be quite interesting to see.
Best of luck with it (& I\'m not going anywhere, regardless) |
|
|
|
|
|
Whats wrong with flops counting?? If its been answered before sorry not reading all the forums trying to find a answer
____________
Join us in Chat (see the forum) Click the Sig
Join UBT |
|
|
|
|
does anyone object to rolling this out to Rosetta@home?
Before you do that, you need to document what you\'ve got. Hopefully I\'ve made that process easier below. Did I get it pretty much right?
Once you document it, then folks can comment from an informed perspective with feedback. From there, Rosetta\'s next. Document it there, invite comment, address concerns, and then roll it out.
Right now, even the Ralph participants don\'t seem to have a grasp of where you\'re headed with the credit system, and they\'ve seen it at work. So, more description is needed.
...or just explain, \"we\'re going to keep credits the same for now... but look at these new numbers we\'re working on\". If that\'s the case.
____________
|
|
|
|
|
|
thierry
- I have one PC which runs Rosetta with a \'Target CPU run time\' of 10 hours. The current running WU runs for 9 hours and have 143 models. So it will be something like 318 credits for 10 hours.
- I have another PC with a \'Target CPU run time\' of one day. The current WU runs for 2.5 hours and have now 5 models. That\'s something like 96 credits for 24 hours.
Isn\'t that strange?
Exactly, this is what I\'m saying about the 2 credits per model was just an example. And the WU you are running on the first PC is going to yield less credits per model then the WU you are running on the second (assuming your two PCs are comparable speed). So, the idea would be to prototype these two WUs on Ralph first... get a grasp for how trivial models for the first are, and how difficult models for the second are... establish an appropriate credit value for each, which makes them equally attractive and does not afford any cherry picking, and in the end, you should yield same credits per hour with different WUs on different machines (that have the same relative speed).
____________
|
|
|
|
|
|
Thanks Feet1st. Got it. In fact the first is slower compares to the second (P4 3.0 HT vs Pentium D 2.8)
Just a question: what are you in the real life? Teacher in some subjects? Because your explanations are always so clear and understandable, it\'s a pleasure to read you.
|
|
|
|
|
|
It\'s my opinion that I don\'t have enough data to say one way or the other. We don\'t know what you know about what happened behind the scenes to determine the number. I\'m apologizing upfront for the following pic, but I had no other way of linking the two together into one pic, as both are needed to complete the picture and to be able to refer between them. What I\'m seeing is models/time all over the map. Perhaps it you upped the number of Ralph WUs distributed per day to more than the two/three I\'m getting now, that I could form an opinion. Perhaps with enough data the quantity of low number and high numbers will offset themselves to mean that the AVG granted credit over a large time period will be closer to Cross Project equality. At one point all I had were 18 cr/hour wus, then I got one that only came in at 2/hr and that offset themselves to be nearly what I would have gotten with just a standard boinc client.
Below are how it shapes up vs. other projects. It also shows every new Ralph WU I have on record. Remember Einstien and Seti are still working on theirs. Which way that goes is unknown other than Seti plans to up the \"load store adjustment\" (fpops multiplier) from 3.35 to 3.51 in the Seti App 5.17 (now in beta). If you look at the few results/computer you\'ll see what I mean by \"all over the place\".
I just think I need more time.
tony
 |
|
|
|
|
|
The new crediting system is pretty simple.
First, we determine how much credit to grant per model for each work unit by running tests on Ralph. We will use the average credit per model from the Ralph tests for production runs on Rosetta@home.
These values will be work unit specific so work units that take longer to generate structures will have higher values.
I have explained this below in a previous post. The only difference is that I decided to keep the old crediting system so users can compare their work with either system. If you take a look at the top participants page you will see two new columns, \"Recent average work credit\" and \"Total work credit\". These are the new credit per model based values. I\'ve been using 2 credits per model on Ralph JUST FOR TESTING. Rosetta@home will use work unit specific values determined from the Ralph test runs.
I took it from a result from one of my computers as just a very rough value to test with. The version that will eventually run on Rosetta@home will have work unit specific credit per model values that are determined from test runs on Ralph. It will be a requirement for lab members to not only test new work units on Ralph but to also determine the average credit per model value from their test runs for production runs. The credits should remain somewhat consistent with other projects since the average values will be based on the standard boinc crediting scheme. If things look okay on Ralph, Rosetta@home will use the credit per model crediting method while Ralph will switch back to the standard method.
____________
|
|
|
|
|
|
Sounds good :-)
____________
|
|
|
|
|
|
sounds good if the influence from over and underclaiming hosts is not too big to have a real impact on the credits per model :)
how will errored WUs be dealt with? do they report how many models they did before they crashed? |
|
|
|
|
The new crediting system is pretty simple. It finally dawned upon me that people didn\'t understand David Kim\'s original explanation of the new credit system that I reposted over at Rosetta (perhaps that was a mistake). Thanks to Feet1st and David Kim for the \'extended\' explanations ! :-) does anyone object to rolling this out to Rosetta@home? So what\'s the idea which of the two credit systems, the old or the new one, should be exported to the stats sites and be listed on the account pages (e.g. this one) ??
____________
|
|
|
|
|
|
thierry ...what are you in the real life? Teacher in some subjects? Because your explanations are always so clear and understandable, it\'s a pleasure to read you.
Well thank you! You just made my week! I\'m a \"computer programmer\". But I\'m in a job where what I\'m really doing is teaching people how to write code and perform diagnostics on their code, and optimize for performance, select the best option for their situation, etc. ...so ya, I\'m a \"teacher\" too.
____________
|
|
|
|
|
sounds good if the influence from over and underclaiming hosts is not too big to have a real impact on the credits per model :)
how will errored WUs be dealt with? do they report how many models they did before they crashed?
we can apply a correction factor to account for the over/under claiming hosts or as someone on the boards suggested, remove the top and bottom X percent.
Errored work units will not be granted credit with the new crediting method. They will continue to be with the existing method though. If it becomes a serious issue, we can try to come up with a reasonable solution.
____________
|
|
|
|
|
we can apply a correction factor to account for the over/under claiming hosts or as someone on the boards suggested, remove the top and bottom X percent. Assuming that you use the median instead of the mean you are already removing the top and bottom 50%, so removing any additional percentages from the distribution will not have any effect on the median. ;-)
____________
|
|
|
|
|
So what\'s the idea which of the two credit systems, the old or the new one, should be exported to the stats sites and be listed on the account pages (e.g. this one) ??
That\'s a tough one. I\'m inclined to keep using the current credit stats to remain consistent. But the leader lists/stats on our web site will be ordered by the new credit values by default as in
http://ralph.bakerlab.org/top_users.php.
These are all great questions/comments, by the way. Thanks! keep up the feedback. it really helps.
feet1st, your explanations are great and right on. I\'ll work on documentation when we come up with a final plan.
____________
|
|
|
|
|
does anyone object to rolling this out to Rosetta@home?
Oh yes I strongly object. Please don\'t hasten! First let\'s discuss the details of how to determine the credit/model. How many WU/Results you plan to receive before determing the credit/model? Then let\'s determine some real credit/model here on RALPH and then reissue the WU here with the new credit factor and check if anyone is pleased. Then we should discuss questions like removing the top and bottom X percent, \"adjusting\" the credit in order to match with other projects. Then make an announcement on the HOMEPAGE of Rosetta that you plan to switch and link to RALPH if anybody wants to see what is coming. Then after a week make the switch.
That\'s a slow approach but I think it does much less harm to Rosetta and the atmosphere in the message boards to quarrel over the new credit system BEFORE it gets deployed than after that.
____________
|
|
|
|
|
does anyone object to rolling this out to Rosetta@home?
I just crunched a few WU\'s for beta and I do object -
I have several WU that did 5 decoys, a couple WU that did 6 decoys, most WU has 3 or fewer. WU\'s crunched in about the same time frame.
If we were to look at cross project equalization of credits - Rosetta would be sitting at or very near the bottom. Granting the fewest credit per CPU hour out there. This could in fact hurt the project as many crunchers will seek out the project that grants higher credits per hour. Or is at least more consistant on granting credits per CPU hour.
I have some old PIII that sometimes will crunch for 2-3 hours with only returning a single decoy - Given this credit system, those machines would do about 1 credit per hour, even seti enhanced is not that low on these guys, as I am seeing about 15 an hour from them with SETI, and 18 and hour with them at E@H. This credit system would make these machines nearly worthless IMO for returning any benifit in running them.
____________
|
|
|
|
|
|
Everyone keep in mind that the current standard boinc crediting system will still be used.
Also, minor modifications to the credit/model values will not make that much of a difference in the long run. The important thing to know is that given any credit/model value, users will be on a level playing field. I think we can all agree that this is the major drive/motivation for coming up with a new method. Making sure it closely matches the BOIINC credit values is not as important since we will still use the old system along with the new.
____________
|
|
|
|
|
does anyone object to rolling this out to Rosetta@home?
I just crunched a few WU\'s for beta and I do object -
I have several WU that did 5 decoys, a couple WU that did 6 decoys, most WU has 3 or fewer. WU\'s crunched in about the same time frame.
If we were to look at cross project equalization of credits - Rosetta would be sitting at or very near the bottom. Granting the fewest credit per CPU hour out there. This could in fact hurt the project as many crunchers will seek out the project that grants higher credits per hour. Or is at least more consistant on granting credits per CPU hour.
I have some old PIII that sometimes will crunch for 2-3 hours with only returning a single decoy - this credit system would make these machines nearly worthless IMO for returning any benifit in running them.
2 credits/model is being used right now on Ralph FOR TESTING ONLY. For Rosetta@home, the value will be more true to the actual cpu time used per model since the value will be determined from the Ralph test runs for each work unit.
____________
|
|
|
|
|
|
Would extending the \"Time to Completion\" from my current 4 hours to 6 or 8 hours, generate more decoys and hence more credit? If so we can extend the time to do the WU models. Does Ralph (and Rosetta) require results as quickly as possible and therefore 4 hours is better for the project?
Two credits for a couple of hours work is not very exciting if only 1 decoy is generated.
By the way I am unable to check if your new credit system actually does credit anything as I am unable to send results back to Ralph at the moment, getting \"No Schedulers Responded\" for over a day now, it uploads the finished WU but not the WU results.
____________
 |
|
|
|
|
|
How about declaring a 2 week test period, when you deploy the new credit system to rosetta? After that period the new credits will by erased if there are serious flaws. So everybody can have a look and start complain :).
I would announce that all comments on the new system made within the first week will not be read and deleted. So everybody is forced to take a break of one week before he can complain. Hopefully that helps to get more comments of quality. |
|
|
|
|
we can apply a correction factor to account for the over/under claiming hosts or as someone on the boards suggested, remove the top and bottom X percent. Assuming that you use the median instead of the mean you are already removing the top and bottom 50%, so removing any additional percentages from the distribution will not have any effect on the median. ;-)
Perhaps you can \'sell\' the new credit system as WU/structure-specific \'Ralph quorum\' applied to Rosetta (reminds me of the discussion on \'pseudo-redundancy\' we had back in the old days ;-).
____________
|
|
|
|
|
Everyone keep in mind that the current standard boinc crediting system will still be used.
Also, minor modifications to the credit/model values will not make that much of a difference in the long run. The important thing to know is that given any credit/model value, users will be on a level playing field. I think we can all agree that this is the major drive/motivation for coming up with a new method. Making sure it closely matches the BOIINC credit values is not as important since we will still use the old system along with the new.
Why keep the old system in place?
____________
  |
|
|
|
|
Would extending the \"Time to Completion\" from my current 4 hours to 6 or 8 hours, generate more decoys and hence more credit?
YES.
Two credits for a couple of hours work is not very exciting if only 1 decoy is generated.
You\'re missing the point. If you are crunching a WU which takes that long to do a single model... then the credit awarded per model of that WU will be larger. And the overall idea is that regardless of which WU you happen to draw from the server, your credits per hour of crunching should be very consistent.
Many of you seem to be caught up on this 2 credit per model EXAMPLE. If they could have granted \"X\" rather then \"2\", I think it would have been more clear.
Let me outline how the workflow would be done. I think that might be where people are getting confused.
1) Calibration:
Run a mixture of WUs of various sizes on Ralph and come up with a benchmark. Take the time to crunch one model for each of 100 proteins on a given machine and compare machines and processor speeds and how many hours of crunching it took to do 1 model for each of the 100 proteins. Now, do some statistics work and try to normalize that in to FLOPs. Because BOINC credits are FLOPS based. Now divide that number of credits back across the 100 different proteins in proportion to the relative amount of time that protein\'s model took to crunch.
Example, say we crunch this 100 model benchmark in 24 hours on a PC that is capable of earning 100 credits per day under the old system. If we concur that the 100 credits is in line with the number of FLOPS that machine can do in a day, then we\'d take that 100 credits and divide it amongst each protein. One of the proteins took 1 hour to complete, those are worth a little more then 4 credits each model. 4 other proteins took 15 minutes each. Those would each be worth 1 credit per model.
This 100 model benchmark would be a frame of reference going forward. Only done the one time.
2) Release a new WU to study, on Ralph:
OK, so now we have a new protein to study, or a new approach at studying them (which we hope will take less compute time to resolve!) and we send out some WUs on Ralph. We look at the time various types of hosts report back, and we size up this new WU against the WUs in the benchmark we previously established. We basically need to define how difficult it is to crunch a model with this new protein or using this new approach.
Average out the results from Ralph, calculate credit per model for this WU. This would be done every time a new type of WU is going to be sent out.
3) Release new WU to study, on Rosetta:
\"We\'ve found this new protein to be most challenging... it\'s models are awarded 5.31 credits each... but they take about 82 minutes to complete\". And so eveyone on Rosetta knows exactly how much credit per model a given WU is worth... and this is fair, because you get credits for how many models you complete. i.e. you get paid (in credits) for each piece of production you return.
An apples to apples comparison:
Some people can pick apples faster then others. But the value to the orchard owner is measured in bushels, not hours. Pickers are paid PER BUSHEL. Now, hand everyone a different sized basket... wait for the complaints \"but Johnny\'s basket is smaller then mine... of COURSE he\'s going to be able to fill it more times!\" But if we pay based on BUSHELS, not baskets, then it\'s still fair to everyone; right? Fare to the farmer, fare to the pickers. And efficient pickers are paid more per day then inefficient pickers.
Well, during the calibration stage, \"bushels\" are defined, and the value of filling one with picked apples is determined by the farmer. During the Ralph release of a new \"basket\", we determine how it compares to a \"bushel\", is it larger? Smaller? or the same? Then during Rosetta release, we\'re all handed this new sized basket and asked to pick apples to the best of our unique abilities, and told this is what filling THIS basket is worth in credits. And visually, everyone can SEE that this new basket is smaller then a bushel, and so the credits for filling it are less... or visa versa. And, in the end, whether you REPORT your picking speed as 75 bushels an hour, or not, you are credited for the number of filled baskets under your trees at the end of the day.
____________
|
|
|
|
|
Why keep the old system in place?
In part so everyone can see they\'re still being credited fairly. Once folks get used to the new system and it is calibrated to BOINC flops and credit values, I presume the \"claimed credits\" information would eventually be phased out.
The problem people seem to be having is that on Ralph, the arbitrary 2 credits per model number, and the comparision of the credits claim under the old system and the credits awarded under the new is making it too easy to see a disparity and raise question about the new approach. That 2 credits per model number wasn\'t a \"calibrated\" value, and hence it wasn\'t adequate for the difficulty of the protein that was released. The \"TEST\" was of the mechanics of tallying and presenting the user data, and running with some predetermined credit value per model established.
I haven\'t heard any problems with the mechanics of the new system, and I believe that\'s why David Kim posed the question about proceeding to Rosetta when he did. Because his purpose for running on Ralph has proven the system works... the only thing I think people are still confused by is how many credits should be awarded. And the main reason we\'re confused by it is simply because we\'ve not seen the full cycle completed. So, to some extent, by releasing on Rosetta, he\'d be making it more clear to us what the new system is. Because then we could see the 5.4 credits per model award or whatever the calibrated number should be. And we\'ll be able to see that in the end, the number of credits for 4 hours of crunching is pretty much what we had under the old system (for the \"average\" WINDOWS user... Sounds like Linux folks will finally see a well-deserved boost... I don\'t want to get in to it here. But I believe for MOST people, the credit claimed will be inline with credit awarded).
____________
|
|
|
|
|
|
So, from Dekims description, we aren\'t seeing what it will be like at Rosetta. It satifies part of my curiosity to hear that different wus will be given credit at a different rate. This would/might make the adjustments to the \"all over the map\" issue and possibly even things out. I say if you feel confident, then \"Let her roll\".
I was under the impression it was going to be 2cr/model and that\'s what he wanted to release to Rosetta. I was not understanding this part of the process. When I read \"start with issuing 2 cr/model in ralph, then make adjustments\". I thought they meant change it to something other than 2, but apply it to all wus, without regard to crunch times or model/hour. |
|
|
|
|
|
@feet1st, thanks for your response. In relation to 2 credits for a couple of hours work, I was not \"missing the point\" as you stated. I quite understand what they are trying and I know it is a test to see if it works, which I will go along with. I was refering to a previous post by Kevint, who has an older P111 machine which he believes will only get about 1 credit an hour if the 2 credit per decoy is used. if by extending the WU time then that may solve his problem by generating more decoys and hence more credit.
I am still having trouble uploading completed WU\'s to Ralph at the moment so I have yet to have any of the new credit awarded to me yet, to see how it compares.
____________
|
|
|
|
|
|
Conan\'s not the only one have problems uploading. See this and this. Hope this gets the message out that there\'s something wrong with the servers even though the status pages states they\'re ok...
____________
 |
|
|
|
|
which he believes will only get about 1 credit an hour if the 2 credit per decoy is used. if by extending the WU time then that may solve his problem by generating more decoys and hence more credit.
I am still having trouble uploading completed WU\'s to Ralph at the moment so I have yet to have any of the new credit awarded to me yet, to see how it compares.
I understand that this is only a test and that the production WU\'s may be different. My response was based upon a comment made about anyone having problems releasing this to the production side. As is, yes I do have issues with it, if the credit per decoy remains the same in production.
However, even upping the WU time would not solve this issue, if the machine can do a decoy every 1 hour, that is what it can do, increasing the WU crunch time would only grant more credits per WU, not more credit per hour.
However - to oblige I will test and crunch some 6 or 8 hour WU\'s on the older machines to see if there is a difference. I would like a 24 hour session as is, 2 hour WU\'s first for a baseline, then I will change it to a 8 hour WU and let that run for 24 hours to verify the credit per hour.
____________
|
|
|
|
|
|
Hello,
I noticed that two sorts of value, \"Recent average work credit\" and \"Total work credit\", have been added to the list of my computers. Could anyone explain what they are?
Thanks,
____________
|
|
|
|
|
|
http://ralph.bakerlab.org/forum_thread.php?id=233#1973 |
|
|
|
|
|
ty |
|
|
|
|
Everyone keep in mind that the current standard boinc crediting system will still be used.
Also, minor modifications to the credit/model values will not make that much of a difference in the long run. The important thing to know is that given any credit/model value, users will be on a level playing field. I think we can all agree that this is the major drive/motivation for coming up with a new method. Making sure it closely matches the BOIINC credit values is not as important since we will still use the old system along with the new.
I think it is a smart idea to introduce the new crediting system to Rosetta first as an alternative. In the long run however you need to decide on one system, which can only be the credit/model system, since it will be much fairer. The exact transition can be determined later and indeed the credit/model scheme allows quick and smooth adjusting in any case. Nevertheless I suggest introducing this on Rosetta only after it has been seen in effect here on RALPH with real data not the fixed test-ration 2credits/model. If this is too complicated then roll it out on Rosetta but I strongly support the idea of feet1st of putting together a documentation, which should be placed prominently (News-Entry and Sticky in message board).
____________
|
|
|
|
|
1) Calibration:
...
2) Release a new WU to study, on Ralph:
...
Feet1st, I don\'t think what you say under 1) and 2) is correct - I liked your original explanation (id 1954) much better. The beauty of the new system is that it does NOT require any calibration or server-side benchmark to be performed. Also, it is not necessary to compare each new WU to benchmark WUs as you seem to imply under 2). On Ralph, one simply determines the median (or some other kind of average) of the old style credit that the Ralph participants claim per crunched structure. No other calibration/benchmark/comparison is required ! A good way to get this concept across might be to think of this as a BOINC style quorum of the (classical BOINC) credit claimed by the Ralph participants per returned structure. Just an idea - of course you are the one who is good at explaining...
I absolutely love the \'apples to apples comparison\'. ;-)
____________
|
|
|
|
|
...
I think it is a smart idea to introduce the new crediting system to Rosetta first as an alternative. In the long run however you need to decide on one system, which can only be the credit/model system, since it will be much fairer. The exact transition can be determined later and indeed the credit/model scheme allows quick and smooth adjusting in any case.
...
I would mostly agree with this, but the problem with having two sets of statistics is that it doesn\'t stop the \'cheating\' flame wars, and could even possibly make it worse. If it wasn\'t for that, I\'d quite like to have the two to compare.
Is there any chance of calculating the historical \'work credit\' based on average claimed credit per work unit? (i.e., for previous models crunched over the years).
If Rosetta publishes the user.xml stats etc, I think it should be the work credits stats which are published, but if this is to be done, it would be best to grant work credit for all decoys that the participant has processed. My guess is that Rosetta does have the data to do this (although perhaps not the time).
____________
|
|
|
|
|
|
I am eager to see the fair credit system on Rosetta, but first we need to make sure that the current server problems on RALPH are not somehow being caused by a bug in the new credit system.
It\'s not possible to have two official stat systems. The stats that are exported to all the third party stat sites are the official stats. To attract crunchers who want to compete with fair stats, these exported stats must be the fair stats.
|
|
|
|
|
|
Okay, I hope I fixed the recent bugs. Let me know if there are still issues with uploading and the team info. I had to update the cgi program with the new database code.
FOR TESTING: I also made the credit/model value for each work unit determined from the most recent average so the work credits should be more accurate for different work units (particularly as more results are returned) rather than the 2 credit/model value. This way, we can get an idea of how the credit granting will be for different sized work units. This is just for testing though. For R@h, we will use the average value from the Ralph runs so everyone will get the same credit/model for a given work unit rather than a value that may change a bit initially.
____________
|
|
|
|
|
For R@h, we will use the average value from the Ralph runs so everyone will get the same credit/model for a given work unit rather than a value that may change a bit initially. I propose not only using Ralph-values for Rosetta, but to include the values received so far from Rosetta (per WU-type ) too. You\'ll need less Ralph units of a new WU-type to start on Rosetta, because the credit per WU granted from Ralph-estimations is only a starting point, that will be calibrated further with the returned Rosetta-WUs.
Norbert
____________
|
|
|
|
|
|
hello dekim,
I confirm you that the bugs about the \"team info\" ans \"Resource share \" are fixed.
I prefer the new system of points which seems to be more fair than the system to give 2 credits/model value.
(Sorry for my bad English)
____________
|
|
|
|
|
|
So far only one result from my host:
http://ralph.bakerlab.org/result.php?resultid=242756
In this case it seems granted credit is a bit high. I suppose a host with my specs (Athlon 64@2.4 GHZ = Athlon 64 3800+) would get around 14-24 credits but I got even >25. This would even top Einstein which grants currently the most credit/hour.
____________
|
|
|
|
|
|
Looking only at the new credit, not the 2cr/model stats.
I have set my puters to run ralph with priority so I can get more samples. With one result in on 4 of my five puters, they all are granted from 33% to %300 more than the claimed credit (based upon stock boinc app/clients). I\'ll have more later. |
|
|
|
|
So far only one result from my host:
http://ralph.bakerlab.org/result.php?resultid=242756
In this case it seems granted credit is a bit high. I suppose a host with my specs (Athlon 64@2.4 GHZ = Athlon 64 3800+) would get around 14-24 credits but I got even >25. This would even top Einstein which grants currently the most credit/hour.
Not so bad - as long as credit is granted equally across the board. Should Rosetta, grant more credit per hour than other projects it would have the tendancy to draw in more crunchers.
Anything from 19-25 credits per hour per core(based on Pent D 920 ) is about correct for correct cross project equality.
I have several Pent D 920\'s and running non-optimized I get an average of 23 credit per hour per core on SETI, QMC, SIMAP, Currently Rosetta is a bit higher than that averaging about 28, E@H is around 30 but have not crunched that for a couple of weeks to test.
Currently there are talks at E@H for project optimization apps that will would increase the current credit per hour about 20%-30%(faster WU\'s more credit per hour). And SETI 5.17 is suppose to incorporate a higher muliplier to match E@H, and further optimization on SETI would allow for even a greater credit per hour.
I know that in prelimiary testing of highly optimized SETI 5.15 apps I was seeing 1700-1800 credit a day on a Pent D 930. On Rosetta that same machine gets around 1200-1300.
My fear is that should Rosetta not at least be equal to these other projects - there will be an exodus to these other higher yielding projects and the Rosetta Project as a whole would suffer as \"credit whores\" gravitate to the projects that yield the most bang for the buck.
____________
|
|
|
|
|
|
The claimed credit is based on the standard boinc method so actually for this example the credit/model method gave a lower amount of credit, 35 vs 25. It would be helpful if users indicate whether they are using optimized clients.
____________
|
|
|
|
|
|
The result listed by tralala is from 5.5.0 for windows and since Boinc never made a 5.5.0 for windows, it must be optimized/third party. The result he called out \"http://ralph.bakerlab.org/result.php?resultid=242756\" is from an optimized/third party boinc client. |
|
|
|
|
Why keep the old system in place?
In part so everyone can see they\'re still being credited fairly. Once folks get used to the new system and it is calibrated to BOINC flops and credit values, I presume the \"claimed credits\" information would eventually be phased out.
G\'day feet1st
You say you presume that the old system will eventually be abandoned, but so far there is nothing from the project devs to confirm this.
Maybe I\'m reading things wrong, but the official message as I read it is the old system will be kept in place and both will be reported,but the old system will continue to be exported.
____________
|
|
|
|
|
Currently there are talks at E@H for project optimization apps that will would increase the current credit per hour about 20%-30%(faster WU\'s more credit per hour). And SETI 5.17 is suppose to incorporate a higher muliplier to match E@H, and further optimization on SETI would allow for even a greater credit per hour.
G\'day Kevint
The new optimised Einstein apps are in beta testing at the moment and lead to significant speedups, once released as the official app the credits will be adjusted.
The adjustment will be to bring it back into line with the other projects.
____________
|
|
|
|
|
|
Thats good to hear; otherwise it\'d be a silly arms race with scientists that should be more mature than that.
Anyway, this whole issue is astounding. I think the new method seems to be the closest R@H can get to Folding@Home\'s method of granting credit, and F@H is as fair as you can get.
What I see is a knee-jerk reaction to no longer being able to cheat (tralla; anyone using an \'optimized\' client is attempting to get higher points than the masses and justifications of lame benchmarks is merely an excuse since everyone has lame benchmarks -- except the cheaters with optimized clients. Dictionary could tell anyone as much!) and get sky-high points. Everything else is not understanding the credit system. What else could be had against a system that promises fair credit distribution?
Only fault with the plan seems to be thinking that the majority of the RALPH pool will be more \'pure\' than the R@H pool of users.. if tralla is an example, maybe not the case? If F@H\'s system can be used directly here somehow I think it\'d be the best possible solution instead of trusting the masses. As long as BOINC is open-source or can be reverse engineered, there will be corrupted clients, and as long as there are corrupted clients, there will be people complaining (and rightfully so).
And regarding what seemed to be a misconception about the \'golden machine\' rule in general.. Why would you need 60 machines to establish a baseline? Yes, a P4 3.0ghz will behave differently than a A64 X2 4800 or an Core 2 Duo 6700. The other two will probably get more work done in a unit of time even at the same clock speeds (depending on optimizations). Therefore, they\'ll pound out more WU or at least WU worth more points. Thats the point, being credited based on scientific output, not CPU time or even artificial measurements of theoretical FLOPS preferably. Some CPU\'s get more work done than others, obviously! That type of system can\'t be tampered with in terms of rewarding points for science. I don\'t visit the main F@H forums often, but I do the top leading teams (HardOCP & those aussies) and virtually never has there been dissent over credit claims. I frequent the [H] board daily, and actually cant recall it ever being an issue. There really can\'t be; its set server side based on the baseline, and that is that. If one wants to \'cheat\' and get more points than one can overclock to the limits of stability, but beyond that, no real cheating.
Anyway, looking forward to some RALPH WU. Looks like this will be an important project, and someone indicated RALPH gets light WU streams. Finishing up 2 CPDN models on an X2 but want to help R@H in the mean time since I hadn\'t in a while so this will be perfect.
|
|
|
|
|
|
Here\'s the latest. Hope it\'s visible as it was shrunk a great deal. All results are from boinc 5.5.11 (standard). All results in green are from the 2cr/model run, results in black are from this latest run.
 |
|
|
|
|
Here\'s the latest. Hope it\'s visible as it was shrunk a great deal. All results are from boinc 5.5.11 (standard).
Why so many statistics with this 2 cr/model? Ralph is testing the software, the credits are nonsense now. They tell us nothing, because Rosetta will have different fixed credits/model for each WU-type. What might be interesting is, if the times/model for a given WU-type are consistent enough to go to a fixed credit model.
Norbert
____________
|
|
|
|
|
FOR TESTING: I also made the credit/model value for each work unit determined from the most recent average so the work credits should be more accurate for different work units (particularly as more results are returned) rather than the 2 credit/model value. This way, we can get an idea of how the credit granting will be for different sized work units. This is just for testing though. For R@h, we will use the average value from the Ralph runs so everyone will get the same credit/model for a given work unit rather than a value that may change a bit initially. I clicked through a number of recent results pages and it seems that the current credit/model averages are significantly higher than what the old crediting system would have assigned for Windows/standard client computers. I wonder why this is the case. Are you perhaps using the mean instead of the median to do the credits/model averaging and are thus affected by outliers on the high side ? I really think you should use the median which will make your averages largely independent of any low and high outliers and automatically \'select\' the (Windows/standard client) majority population as averages. Another option would be a weighted mean (weighting factors 1/value, mean = n/Sum_1..n(1/v_i)) which de-emphasizes the high values. Here is an example to demonstrate the effect ot the different averaging methods: For the 10 values 2 3 5 5 5 5 5 10 15 20 the mean is 7.5, the median is exactly 5 and the weighted average would be 4.88. Would be interesting to hear your thinking on that.
____________
|
|
|
|
|
G\'day feet1st
You say you presume that the old system will eventually be abandoned, but so far there is nothing from the project devs to confirm this.
Maybe I\'m reading things wrong, but the official message as I read it is the old system will be kept in place and both will be reported,but the old system will continue to be exported.
You are correct, I\'m reading in to their statement that they are \"changing the credit system\". From that I infer that eventually the new system, whether in it\'s initial form, or after some revisions, will be used for credit reporting. And that the dual credit system is a temporary measure as we all become familiar with the new system.
____________
|
|
|
|
|
|
Hoelder1in 1) Calibration:
...
2) Release a new WU to study, on Ralph:
...
Feet1st, I don\'t think what you say under 1) and 2) is correct...
I absolutely love the \'apples to apples comparison\'. ;-)
I had a fairly lengthy reply all written up when my satilite internet connection dropped :(
Basically, I agree that all of #1 I had described is specuation on my part... suffice it to say there is a black box #1, which is some means of calibration (i.e. defining how large \"bushels\" are) and once that is in place, then the process described in #2 and #3 can proceed. I see your point about using the BOINC reported credits and the averages David Kim described... I\'d still like to see a detailed explaination of the plan, and how a slow & fast machine on Ralph will impact the credit award per model that is determined.
The apples to apples comparison is based on personal experience :) My first gainful employment, age 12 I believe, was as an apple picker. Some days you\'d be assigned a row of trees which grow larger apples (i.e. less hand motions to fill a bushel), and other days trees that grow smaller apples (if your hands are large enough, sometimes you can grab more then one apple per grab). For some people the small apples take longer to fill a bushel, for others they could fill faster. Depends on your hands, experience, how far you can reach from the ladder without climbing down and moving it, etc. etc. ...but the farmer knew when he was assigning you tougher trees, and made a point of rotating around so everyone had to share the task.
____________
|
|
|
|
|
FOR TESTING: I also made the credit/model value for each work unit determined from the most recent average so the work credits should be more accurate for different work units (particularly as more results are returned) rather than the 2 credit/model value. This way, we can get an idea of how the credit granting will be for different sized work units. This is just for testing though. For R@h, we will use the average value from the Ralph runs so everyone will get the same credit/model for a given work unit rather than a value that may change a bit initially. I clicked through a number of recent results pages and it seems that the current credit/model averages are significantly higher than what the old crediting system would have assigned for Windows/standard client computers. I wonder why this is the case. Are you perhaps using the mean instead of the median to do the credits/model averaging and are thus affected by outliers on the high side ? I really think you should use the median which will make your averages largely independent of any low and high outliers and automatically \'select\' the (Windows/standard client) majority population as averages. Another option would be a weighted mean (weighting factors 1/value, mean = n/Sum_1..n(1/v_i)) which de-emphasizes the high values. Here is an example to demonstrate the effect ot the different averaging methods: For the 10 values 2 3 5 5 5 5 5 10 15 20 the mean is 7.5, the median is exactly 5 and the weighted average would be 4.88. Would be interesting to hear your thinking on that.
Currently, I am just using the quotient of the claimed_credit and model totals for each work unit batch which get saved in a project specific table that we created. I wanted to avoid having to query the result table which would be necessary to get the median. I\'d also have to add a project specific column to the result table which would hold the model count and I\'m trying to avoid modifying the BOINC tables. I could use a correction factor if the descrepancies are significant enough. Can you point me to the results that you are talking about?
____________
|
|
|
|
|
Currently, I am just using the quotient of the claimed_credit and model totals for each work unit batch which get saved in a project specific table that we created. I wanted to avoid having to query the result table which would be necessary to get the median. I\'d also have to add a project specific column to the result table which would hold the model count and I\'m trying to avoid modifying the BOINC tables. I could use a correction factor if the descrepancies are significant enough. Can you point me to the results that you are talking about? OK, I understand, if you only have these two numbers available that\'s of course the only thing you can do. I simply looked at the granted to claimed credit ratios on some of the recent results pages that looked like they were using the standard client. Here are a few examples: 1, 2, 3 (first six).
____________
|
|
|
|
|
|
Hi everyone - I\'ve just registered for Ralph, although I\'ve not really had any problems with Rosetta in the past so i\'ve never connected any computers up to it. I could drop a 12hr/day P3 1GHz on it in a few weeks if that\'s any use?
I read the entire thread last night(!) and I think the new credit system is definitely along the right lines. I\'m not sure about using Ralph to calculate a WU mean score though if the Ralph app differs from the Rosetta one (doesn\'t it produce debug info etc?). What about when you release new versions onto Ralph that take more or less time than the current Rosetta release? It might be wise to hand pick some known dependable machines for this task, whether on Ralph or Rosetta, and use those to calibrate the credits per decoy, rather than using the whole of Ralph as a test bed. They would just have to have a consistent hardware config (and preferably on 24/7 for a fast turn-around time), and you\'d need to monitor/be informed if this changed. Would that take a lot of work? I guess you\'d need a table of \'golden machines\' that would be sent the jobs first, then the credit per decoy could be calculated before general release. Of course the \'golden\' machines would be given whatever credit they request, so they\'d have to be from trusted sources, but there are plenty of those available.
I don\'t mind submitting a machine that will have a constant hardware config - i\'m going to be putting a backup server in the loft at some point so it\'d be fine to run that.
Is there any chance of calculating the historical \'work credit\' based on average claimed credit per work unit? (i.e., for previous models crunched over the years).
If Rosetta publishes the user.xml stats etc, I think it should be the work credits stats which are published, but if this is to be done, it would be best to grant work credit for all decoys that the participant has processed. My guess is that Rosetta does have the data to do this (although perhaps not the time).
I\'ve requested this over on the Rosetta forums as I\'d like to have a look at this too (although this is probably a much better place to ask!).
It seems to me it would probably be a fairly simple task to backdate the new credit model fairly accurately for all previous jobs, using computers which are known to have the same hardware/software config from the start, then calculating a credits/hour figure for each computer and then using the number of decoys/time taken to calculate the number of credits due for each type of WU decoy.
I know there is a risk that some people will use this as fuel for more flaming, but I think we\'ll be able to get a good idea of whether the credits can be backdated reasonably accurately from this. Could a table showing the WU name, number of decoys produced, time taken and the benchmark score be released? I know it\'s gonna be a big file!
Danny
____________
|
|
|
|
|
Currently, I am just using the quotient of the claimed_credit and model totals for each work unit batch which get saved in a project specific table that we created. I wanted to avoid having to query the result table which would be necessary to get the median. I\'d also have to add a project specific column to the result table which would hold the model count and I\'m trying to avoid modifying the BOINC tables. I could use a correction factor if the descrepancies are significant enough. Can you point me to the results that you are talking about? OK, I understand, if you only have these two numbers available that\'s of course the only thing you can do. I simply looked at the granted to claimed credit ratios on some of the recent results pages that looked like they were using the standard client. Here are a few examples: 1, 2, 3 (first six).
So you already did apply a correction factor, didn\'t you ? :-) I just randomly picked 13 standard client results that were sent out after midnight (UTC) and calculated the granted to claimed credit ratios. I get a mean of 1.16 with standard dev. 0.35 and an error of the mean of 0.10. So within the sampling error of this small sample your correction factor seems to be dead on. Presumably, you calculated it from a larger sample, so it probably is even more accurate. I guess the correction factor would have to be reviewed occasionally to correct for changes in the composition of the Ralph participants (Linux/Windows, standard/optimized client).
____________
|
|
|
|
|
G\'day feet1st
You say you presume that the old system will eventually be abandoned, but so far there is nothing from the project devs to confirm this.
Maybe I\'m reading things wrong, but the official message as I read it is the old system will be kept in place and both will be reported,but the old system will continue to be exported.
You are correct, I\'m reading in to their statement that they are \"changing the credit system\". From that I infer that eventually the new system, whether in it\'s initial form, or after some revisions, will be used for credit reporting. And that the dual credit system is a temporary measure as we all become familiar with the new system.
Cheers feet1st
It would be nice to get some official confirmation that this is the case.
____________
|
|
|
|
|
It would be nice to get some official confirmation that this is the case.
Agreed. David Kim mentioned preparing a more detailed description, and from the discussion here, he\'s got a pretty good idea what people on Rosetta will need clearly described in order to understand the new system. So, I\'ll wait patiently.
____________
|
|
|
|
|
|
dcdc:
We could calculate credits from February of this year using the new method but not from the start of the project because we have already cleaned up old result files. It wouldn\'t be worth the work required in my opinion. I like your idea of hand picking a representative set of hosts whose configurations we can trust but again, it would take a bit of work and organization. Good point about ralph and R@h app differences. If there is a significant change in the app that will skew the credit/model values, we can separate it as another application. Or we can always have a separate app for changes.
Hoelder1in:
Very observant. I did apply a simple correction factor and I\'m glad you see it seems to be working. And, yes, we will have to update it now and then to account for changes in the host pool.
feet1st and Trog Dog:
We should first see what the response to the new method is and whether users would prefer it over the old. Not everyone will be happy about the final decision. I prefer abandoning the old system but that\'s just my opinion. Users that prefer the old system cannot use as an argument that they get more credits for whatever reason with the old system. The goal to even the playing field.
____________
|
|
|
|
|
dcdc:
We could calculate credits from February of this year using the new method but not from the start of the project because we have already cleaned up old result files. It wouldn\'t be worth the work required in my opinion. I like your idea of hand picking a representative set of hosts whose configurations we can trust but again, it would take a bit of work and organization. Good point about ralph and R@h app differences. If there is a significant change in the app that will skew the credit/model values, we can separate it as another application. Or we can always have a separate app for changes.
I think the new credit system will be accepted much more willingly and quickly if the credits are backdated. If they could be calculated back to the start then that\'d be best but if they can only go back to Feb then fair enough.
If it\'s a lot of work to calculate then I\'m sure we could take some of the burden from you on that front if you\'re willing to send one of us the data.
Danny
____________
|
|
|
|
|
|
Could you do the credit calc from Feb, and then estimate the older credit based on the recent boinc credit vs. recent work credit ratio?
i.e., if the Feb-August boinc credit is 10% higher than the Feb-August work credit, then apply the same ratio to the pre-Feb boinc credit to get an estimated pre-Feb work credit?
The reason that I am trying to think of ways to get the old credit into the equation is so that the exported XML can reflect the new system ASAP. I also would like to move to the new system exclusively.
____________
|
|
|
|
|
|
I\'m extracting the archived data and will see what I can do. What do other users think about trying to backdate the credits using the new work based system?
____________
|
|
|
|
|
|
Totally against it. Changes are progressive, so what has been done is done, what is to be done will be done. The system is evolving as we work things out by trial and possibly error, therefore you have found some things aren\'t working as good as some people would wish, therefore you adapt the system and try that. What you have done in past is past, so lets move on and see how this new system goes, it may also not be the \'best\' system and will have to be changed again, what will you do then backdate all the credits again? You will lose a lot of people if you do that.
There is no need to backdate.
____________
|
|
|
|
|
I\'m extracting the archived data and will see what I can do. What do other users think about trying to backdate the credits using the new work based system?
If nothing else, it would be interesting to see what changes would be made. Could a mock run be done and show us side by side results?
____________
|
|
|
|
|
|
At Distributed Folding - we had the scoring system changed; and the scores were frozen. It was possible through 3rd party and the system itself to see overall scores (the final score from score system #1, and the current results of score system #2) and compare them to others. Personally, I thought it would have been fine to have continued on from the old scores; rather than starting out SS#2 with zero points. The system allowed either view to work..
As for rescoring the WUs we\'ve done back to Feb or earlier - it won\'t matter to me; but we should get feedback from the highly competitive teams on the top of the charts. To perform the work, and then show what the top 50 team scores would end up being using the new scoring technique would allow each team to vote whether they were willing to go with the current fair crediting system, or the old way.
____________
|
|
|
|
|
I\'m extracting the archived data and will see what I can do. What do other users think about trying to backdate the credits using the new work based system?
I would be very happy to see this happen - since it\'s obvious that the new system is going to be a much more level playing field we should show as many credits using this as possible.
Back date to Feb, include everything older than that as they were reported. That gets my vote. |
|
|
|
|
|
I don\'t care about the credits, just the science, but it seems to me that :
Currently, I am just using the quotient of the claimed_credit and model totals for each work unit batch which get saved in a project specific table that we created. I wanted to avoid having to query the result table which would be necessary to get the median. I\'d also have to add a project specific column to the result table which would hold the model count and I\'m trying to avoid modifying the BOINC tables. I could use a correction factor if the descrepancies are significant enough. Can you point me to the results that you are talking about?
If I understand that correctly, then I think you will have some trouble.
Linux machines receive less credit than Win machines, so depending on the amount of Linux machines you have crunching, Win credit will go down from what it is now.
So I would suggest extracting only the Win results for your calcultaions, that way Lin and Mac will be truly given credit they deserve, and Win machines won\'t be underscored.
Somebody else has probably thought of this and maybe even posted, but I\'m not going to read the entire thread :)
I\'m extracting the archived data and will see what I can do. What do other users think about trying to backdate the credits using the new work based system?
You will get a lot of flak and bad publicity (among BOINC\'ers) that you just don\'t need, let it go with \"Here is the new credit system, it\'s as fair as we could make it, so enjoy\".
____________
|
|
|
|
|
I\'m extracting the archived data and will see what I can do. What do other users think about trying to backdate the credits using the new work based system?
Totally against backdating the changes. Not nessesary nor needed - Time to fix the problems and just go forward from here.
Not a wise move in my mind - you risk loosing many crunchers should you decide to do this.
____________
|
|
|
|
|
Totally against backdating the changes. Not nessesary nor needed - Time to fix the problems and just go forward from here.
Not a wise move in my mind - you risk loosing many crunchers should you decide to do this.
Why does it matter if old credits are re-evaluated? I think everyone agrees that credits are a benchmark for how much work a given user/computer does for the project. Now that a much more accurate mechanism exists to grant credit, why is it a negaive to retroactively re-score the user base?
Should peple who intentially modified their credit \'wage\' be granted a higher score than people who read a newspaper article and joined? I\'m not for or against starting fresh or going back go Feb. . I think whatever results in the most computers crunching R@H is the correct outcome. . but proposing a scoring system based on user results when they are known to be fiddle-able, from users whose computes are hidden. . it just doesn\'t sound right.
-Ethan
____________
|
|
|
|
|
|
delete me. . must have been a duplicate post
____________
|
|
|
|
|
I\'m extracting the archived data and will see what I can do. What do other users think about trying to backdate the credits using the new work based system?
Horrible idea.
People joined this project based on the scoring method (other reasons too, of course). Think about it this way. The laws (in the US) today say that if you invest in a 401(k), you do so with the expectation that your money goes in exempt from income tax, but you will have to live with the limited investment options your employer offers. Now, 20 years later, they change the rules and tell you that they\'ve changed the rules, and 401(k)s are no longer valid. And in addition to having lost all that tax free advantage, you owe back taxes (money/credits lost).
And the real point here is that you have missed out on better potential profits had you just gone elsewhere to begin with.
At the time people joined, a credit system was promised. You cannot pull that away from people retroactively. It\'s deceitful. You want to chage it going forward? Fine. Change the rules, tell people, and let them make their own decision to stay or go.
____________
Dublin, CA
SETI.USA - Stats - My stuff - BOINC IRC chat
|
|
|
|
|
At the time people joined, a credit system was promised. You cannot pull that away from people retroactively. It\'s deceitful. You want to chage it going forward? Fine. Change the rules, tell people, and let them make their own decision to stay or go.
Ok. . If you owned a bank and users who used Firefox got 5 percent interest, but users who used Opera only got 3 percent. . . Users of Opera would find out they were being short changed and expect the same return on their investment. Why should Opera people be denied their credits from back in the day? The only reason I can think of is so people who \'enhanced\' their credits can keep that advantage. Since credits are an important metric, wouldn\'t everyone want them to be as neutral as possible? How is it less neutral to start them back in Feb rather than Aug? Especially if the old credit count is still visible on the Rosetta webpage.
____________
|
|
|
|
|
People joined this project based on the scoring method
Probably some will be same as you, but keep in mind others crunch this project just for science development. For them it doesn\'t matter whether the way in which their work will be evaluated again is changed. |
|
|
|
|
At the time people joined, a credit system was promised. You cannot pull that away from people retroactively. It\'s deceitful. You want to chage it going forward? Fine. Change the rules, tell people, and let them make their own decision to stay or go.
Ok. . If you owned a bank and users who used Firefox got 5 percent interest, but users who used Opera only got 3 percent. . . Users of Opera would find out they were being short changed and expect the same return on their investment. Why should Opera people be denied their credits from back in the day? The only reason I can think of is so people who \'enhanced\' their credits can keep that advantage. Since credits are an important metric, wouldn\'t everyone want them to be as neutral as possible? How is it less neutral to start them back in Feb rather than Aug? Especially if the old credit count is still visible on the Rosetta webpage.
Disagree. Everyone joined the bank under the same rules. Some people chose different investment methods. Was it open and honest? Yes. Nothing was hidden. Nothing anyone did was against the rules. Everyone had the same investment opportunities. And more importantly, everyone had right to not join in the first place if they didn\'t like the rules.
You can\'t change the laws/rules retroactively....ever. You can\'t tell me it is okay to drive 55 on the road today, change the limit to 45 tomorrow, and then give me a ticket for what I did while it was still legal.
____________
Dublin, CA
SETI.USA - Stats - My stuff - BOINC IRC chat
|
|
|
|
|
People joined this project based on the scoring method
Probably some will be same as you, but keep in mind others crunch this project just for science development. For them it doesn\'t matter whether the way in which their work will be evaluated again is changed.
Obviously, which is why I said, \"People joined this project based on the scoring method (other reasons too, of course).\" In fact, that was the whole point of the parenthetical portion.
But we\'re explicitly talking about the credit portion of the motivation here, with this whole thread. That\'s the pertinent part.
____________
Dublin, CA
SETI.USA - Stats - My stuff - BOINC IRC chat
|
|
|
|
|
|
I thought there were joining for science.
David,
Do whatever you want, as far credits are fair for everyone it\'s OK for me. There are +/- 76000 users registered at Rosetta. We only hear here a few who want keep their credits at any price. We never hear the 76000 minus a few.
|
|
|
|
|
|
Given Zombie67\'s point (which is valid in my comparison in terms of investments, thanks). . . I think it\'s fair to reduce the credits of those who \'convinced\' the system into accpeting an unreasonable value to start with. 9 out of 10 identical worksations reported a similar wu time, should the 10th be given 5x the credits for no reason? How does it hurt anyone to go apples to apples back to Feb?
____________
|
|
|
|
|
Given Zombie67\'s point (which is valid in my comparison in terms of investments, thanks). . . I think it\'s fare to reduce the credits of those who \'tricked\' the system into accpeting an unreasonable value to start with. 9 out of 10 identical worksations reported a similar wu time, should the 10th be given 5x the credits just because the software was set different than the rest?
If not everyone is taking equal advantage of the rules, and the powers that be want it to be less \"customizable\" , then change the rules to prevent it going forward. That\'s what we\'re doing here, right? Perhaps we\'re late in implementing chanes to those rules. That\'s our own fault, not those who took advantage of it. Punishing people for breaking no rules is unfair at best.
Worse terms come to mind.
____________
Dublin, CA
SETI.USA - Stats - My stuff - BOINC IRC chat
|
|
|
|
|
Given Zombie67\'s point (which is valid in my comparison in terms of investments, thanks). . . I think it\'s fare to reduce the credits of those who \'tricked\' the system into accpeting an unreasonable value to start with. 9 out of 10 identical worksations reported a similar wu time, should the 10th be given 5x the credits just because the software was set different than the rest?
If not everyone is taking equal advantage of the rules, and the powers that be want it to be less \"customizable\" , then change the rules to prevent it going forward. That\'s what we\'re doing here, right? Perhaps we\'re late in implementing chanes to those rules. That\'s our own fault, not those who took advantage of it. Punishing people for breaking no rules is unfair at best.
Worse terms come to mind.
Where on Rosetta\'s homepage did it say to download a different client to receive more credits? Users who read the Associated Press article who had no idea of other projects. . they should be punished because they didn\'t research things they had never heard of?
How does an apples to apples comparison hurt anyone? If you did the work, you get the credit. . if you did the work faster than the other guy, you get more credit. I\'m not sure how this hurts anyone.
____________
|
|
|
|
|
If not everyone is taking equal advantage of the rules, and the powers that be want it to be less \"customizable\" , then change the rules to prevent it going forward. That\'s what we\'re doing here, right? Perhaps we\'re late in implementing chanes to those rules. That\'s our own fault, not those who took advantage of it. Punishing people for breaking no rules is unfair at best. However it cannot become the reason that prevents us from not applying the rules and leaving them as they\'ve been, right? Moreover many participants are agree with that rules will be changed.
|
|
|
|
|
Where on Rosetta\'s homepage did it say to download a different client to receive more credits? Users who read the Associated Press article who had no idea of other projects. . they should be punished because they didn\'t research things they had never heard of?
Just like in the real world (especially in the investment world), you must research before you play. The more you know, the better you will do. Know the game in which you play, before you join. This is universal. Even a brief scan of the message boards would clue in a newbie.
How does an apples to apples comparison hurt anyone? If you did the work, you get the credit. . if you did the work faster than the other guy, you get more credit. I\'m not sure how this hurts anyone.
I agree. So going forward, make it so (number one). Changing the rules backwards is wrong. Some people (had they known) would have never participated to begin with.
And worse of all, they have now lost out on accumulating credit on other projects. Rosetta@home is unable to make that right. The people will have been played by R@H, used and discarded like a spent condom. Nice.
I sincerely hope it doesn\'t work out that way, as I don\'t believe that is anyone\'s intention.
____________
Dublin, CA
SETI.USA - Stats - My stuff - BOINC IRC chat
|
|
|
|
|
However it cannot become the reason that prevents us from not applying the rules and leaving them as they\'ve been, right? Moreover many participants are agree with that rules will be changed.
I\'m not sure I understand your point. I think you are saying \"But it should be okay to change the rules going forward, right?\" If so, I agree 100%. In fact, I am a *huge* fan of the way SETI is doing it. And optomized clients mean exactly that...they crunch the SAME job in less time. More work = more credit = more science! Who can argue with that?
____________
Dublin, CA
SETI.USA - Stats - My stuff - BOINC IRC chat
|
|
|
|
|
|
Star Trek references aside :)
The governement makes you accountable for false claims on past returns. . why should people be given passes on clients that granted more credits than the standard one? Because they were the first to use it they should be given amnesty? What do they have against a level scoring system? If they are number one in the current credit system, they should still be number one when the actual work is calculated. . right?
-E
____________
|
|
|
|
|
Star Trek references aside :)
The governement makes you accountable for false claims on past returns. . why should people be given passes on clients that granted more credits than the standard one? Because they were the first to use it they should be given amnesty? What do they have against a level scoring system? If they are number one in the current credit system, they should still be number one when the actual work is calculated. . right?
-E
The claims aren\'t false and therefore are not illegal. We have *ALL* read the posts from the admins, over and over, that optomized clients are not illegal/baned/against the rules/whatever.
Change the rules....Good! But don\'t punish people for *not* breaking the rules.
____________
Dublin, CA
SETI.USA - Stats - My stuff - BOINC IRC chat
|
|
|
|
|
...
I\'m not sure I understand your point. I think you are saying \"But it should be okay to change the rules going forward, right?\" If so, I agree 100%. In fact, I am a *huge* fan of the way SETI is doing it. And optomized clients mean exactly that...they crunch the SAME job in less time. More work = more credit = more science! Who can argue with that?
I think your argument is based on a misunderstanding - An optimised client (i.e., a drop-in replacement manager) does no more science work than the original manager it replaced.
You\'d need to optimise the science application (which is what Rosetta has been doing since it started) in order to increase the science output. The main effect optimising a manager has is to increase the credit score (a few have other functions as well, such as managing farms of computers, or giving you additional statistics).
What is being discussed is firstly whether \'science = credit\' is a good idea going forward, and secondly whether it\'s a good idea going back, as well. My vote is \'yes\' for both.
____________
|
|
|
|
|
...
I\'m not sure I understand your point. I think you are saying \"But it should be okay to change the rules going forward, right?\" If so, I agree 100%. In fact, I am a *huge* fan of the way SETI is doing it. And optomized clients mean exactly that...they crunch the SAME job in less time. More work = more credit = more science! Who can argue with that?
I think your argument is based on a misunderstanding - An optimised client (i.e., a drop-in replacement manager) does no more science work than the original manager it replaced.
No. You are wrong. Witht with SETI (and Einstein too if I understand correctly) the optomized clients actually do the job faster because they are optomized for the machine architecture. I know this for a fact for SETI and Macs.
Edit: I have seen the time get smaller for the same job, using optomized mac client*. This is not replacing the boincmgr, but instead the acutal client.
* http://tbp.berkeley.edu/~alexkan/seti/
____________
Dublin, CA
SETI.USA - Stats - My stuff - BOINC IRC chat
|
|
|
|
|
The claims aren\'t false and therefore are not illegal. We have *ALL* read the posts from the admins, over and over, that optomized clients are not illegal/baned/against the rules/whatever.
Change the rules....Good! But don\'t punish people for *not* breaking the rules.
I\'m not suggesting the science claims are false, there hasn\'t been a case of that happening I\'m aware of. I\'m only suggesting the credit claim needs to be re-evaluated.
It\'s not a matter of optimized or standard client scores. . it\'s a matter of work done for the project. If you have the most uber quad core overclocked liquid nitrogen powered rig dedicated 100 percent to Rosetta, you should be at the top of the leader board. . . you shouldn\'t be there just because you could edit a text file to add a couple zeros to your flops count.
____________
|
|
|
|
|
...
I\'m not sure I understand your point. I think you are saying \"But it should be okay to change the rules going forward, right?\" If so, I agree 100%. In fact, I am a *huge* fan of the way SETI is doing it. And optomized clients mean exactly that...they crunch the SAME job in less time. More work = more credit = more science! Who can argue with that?
I think your argument is based on a misunderstanding - An optimised client (i.e., a drop-in replacement manager) does no more science work than the original manager it replaced.
(add: R@H serves no optimised science application, unlike S@H and SIMAP, for example. Be careful not to confuse the client and the science app.)
____________
|
|
|
|
|
No. You are wrong. Witht with SETI (and Einstein too if I understand correctly) the optomized clients actually do the job faster because they are optomized for the machine architecture. I know this for a fact for SETI and Macs.
Optimized boinc clients don\'t make rosetta run any faster. Check out task manager (or top if you\'re linux). The app using your cpu time is rosetta, not boinc. Compiling boinc to give you a million credits an hour has no affect on how fast Rosetta runs.
____________
|
|
|
|
|
No. You are wrong. Witht with SETI (and Einstein too if I understand correctly) the optomized clients actually do the job faster because they are optomized for the machine architecture. I know this for a fact for SETI and Macs.
It is you that are wrong. What really compute is the science application. The client is just a manager.
|
|
|
|
|
The claims aren\'t false and therefore are not illegal. We have *ALL* read the posts from the admins, over and over, that optomized clients are not illegal/baned/against the rules/whatever.
Change the rules....Good! But don\'t punish people for *not* breaking the rules.
I\'m not suggesting the science claims are false, there hasn\'t been a case of that happening I\'m aware of. I\'m only suggesting the credit claim needs to be re-evaluated.
It\'s not a matter of optimized or standard client scores. . it\'s a matter of work done for the project. If you have the most uber quad core overclocked liquid nitrogen powered rig dedicated 100 percent to Rosetta, you should be at the top of the leader board. . . you shouldn\'t be there just because you could edit a text file to add a couple zeros to your flops count.
You are suggesting to change the rules, after people have already invested their machines\' time based on rules at the time the joined. How will you pay them back for that time?
____________
Dublin, CA
SETI.USA - Stats - My stuff - BOINC IRC chat
|
|
|
|
|
No. You are wrong. Witht with SETI (and Einstein too if I understand correctly) the optomized clients actually do the job faster because they are optomized for the machine architecture. I know this for a fact for SETI and Macs.
It is you that are wrong. What really compute is the science application. The client is just a manager.
Good lord. \"What we have here is a failure to commuicate.\" We have a terminology fubar. I mean that with SETI, they have optomized science applications (in your terminology). These \"applications\" are machine specific. They actually improve the efficiency of the machines.
Clear as mud?
____________
Dublin, CA
SETI.USA - Stats - My stuff - BOINC IRC chat
|
|
|
|
|
I mean that with SETI, they have optomized science applications (in your terminology). These \"applications\" are machine specific. They actually improve the efficiency of the machines. http://ralph.bakerlab.org/forum_reply.php?thread=233&post=2081#2074
the term \"science application\" is widely used
eg http://en.wikipedia.org/wiki/BOINC
|
|
|
|
|
I mean that with SETI, they have optomized science applications (in your terminology). These \"applications\" are machine specific. They actually improve the efficiency of the machines. http://ralph.bakerlab.org/forum_reply.php?thread=233&post=2081#2074
the term \"science application\" is widely used
eg http://en.wikipedia.org/wiki/BOINC
Right. My fault for not using the right terminology.
So we\'re on the same page now?
Does this mean you agree or disagree with my points?
____________
Dublin, CA
SETI.USA - Stats - My stuff - BOINC IRC chat
|
|
|
|
|
Right. My fault for not using the right terminology. So we\'re on the same page now? Does this mean you agree or disagree with my points?
Basically I\'m agree with Ethan\'s viewpoint.
|
|
|
|
|
|
While using the new system to re-score back to Feb or earlier will reduce the scores of those that used an optimized client, it\'ll be reduced the same degree as any other user of the optimized client based on the amount of time the optimized client was used.
On the other hand, keep in mind the poor Linux folks who are being penalized under the standard boinc benchmark (compared to Windows); and some of the Intel P4s that are getting rather poor scores on the standard boinc benchmark when compared to AMD Athlon cpus.
So the change will help out lots of people that weren\'t using an AMD cpu on Windows.
____________
|
|
|
|
|
Right. My fault for not using the right terminology. So we\'re on the same page now? Does this mean you agree or disagree with my points?
Basically I\'m agree with Ethan\'s viewpoint.
What does that mean in plain english? Could you be more vague?
My point in this part of the discussion (that you objected to) was:
\"In fact, I am a *huge* fan of the way SETI is doing it. And optomized clients mean exactly that...they crunch the SAME job in less time. More work = more credit = more science! Who can argue with that?\"
So replace OC with SA to correct my language isssue. Do you still object?
____________
Dublin, CA
SETI.USA - Stats - My stuff - BOINC IRC chat
|
|
|
|
|
I\'m extracting the archived data and will see what I can do. What do other users think about trying to backdate the credits using the new work based system?
I\'m afraid there will be quite an uproar if you correct the credit for past work. It would be the fair thing to do but in the end it will result in bitter discussions and a loss of hosts. I support transition from the old to the new system as soon as it is accepted in the boards to level the playing field but wouldn\'t touch the past.
P.S.: Yes I\'m using the cheating Boinc 5.5.0 but no I don\'t care for credits. I use it now for scientific reasons to check whether the new credit system will be immune against overclaiming hosts on RALPH.
____________
|
|
|
|
|
Right. My fault for not using the right terminology. So we\'re on the same page now? Does this mean you agree or disagree with my points?
Basically I\'m agree with Ethan\'s viewpoint.
What does that mean in plain english? Could you be more vague?
My point in this part of the discussion (that you objected to) was:
\"In fact, I am a *huge* fan of the way SETI is doing it. And optomized clients mean exactly that...they crunch the SAME job in less time. More work = more credit = more science! Who can argue with that?\"
So replace OC with SA to correct my language isssue. Do you still object?
Yes an optimised science application does more work in a given time (else it\'s badly optimised ;-)
BUT what has that got to do with what we are talking about here.
We are talking about people who either compile their own Boinc \'core client\' aka \'client\' or use a pre-compiled version that scores differently to the client given by boinc on their download page. (note: Everyone is free to do this, and the boinc site even points to some of them and gives instructions how to do this).
This is what the whole cofuffle is about. One one person score (claims credit for) for 12hrs work on a computer is often wildely different to what someone else may calim for that exact same computer if they use a different core client (akak crunch3rs, boinc studio, Mac superbench, truxoft or their own compiled not using \'boinc developers\' settings)
Note, no mork science work is done just the credit someone has claimed.
You are talking about somethng completely different and irrelevant to this problem.
Probably just the usual boinc naming convention mix up :-)
Me, I see no reason to waste time retrospectively adjusting the credits.
(and yes mine are a mix of Crunch3rs, Trux, boincStudio, official and develoment. I don\'t even know what most are running at the moment though they\'ll probably be 5.4.9 / crunc3rs 5.5.0 / dev5.5.6/9 when I was trying to get the benchmarks differences.)
The only thing you do is please a minority of people here that are trying to help fix this, but you will piss off a whole bunch of people. NOT something you want to do given that is the majority of the enthusiasts and the top teams!
They have technically done nothing wrong.
(also do you keep adjusting if you find this new method does not work out)
Just a quick question, we are not going to have this split system (work credit and credit) at Rosetta?
I hope not as it just complicates matters. Just make sure it works over here at Ralph first with actually real measurements for a while through a variety of task.
Though be sure to set aside any other work to fix and adjust it as results come back in from Rosetta. There will be people with calclators checking them all (and good on them for making the checks!)
I\'d say if youcan get a +-10% spread your doing very well and I\'d leave it at that.
____________
|
|
|
|
|
and some of the Intel P4s that are getting rather poor scores on the standard boinc benchmark when compared to AMD Athlon cpus.
Linux I don\'t know about, but keep in mind mhz for mhz, a P4 will bite the dust every time in comparison to an A64. Its architecture is just fundamentally inferior.
If boinc benchmarks are significantly different than the difference in other real, commonly accepted benchmarks (like those done at Anandtech, Bit-Tech, and HardOCP) then thats one thing. Otherwise, thats what one gets for buying a P4. Core 2 Duo, likewise, should womp Athlon 64 X2\'s. But considering an A64 3200+ isn\'t truly necessarily the equivalent of a P4 3.2ghz, as long as P4s generally trail roughly equivalent A64 chips then the benchmarks are actually right-on.
I can see why some of you guys see going back and updating the scores as a bad idea, stealing credits, etc. As far as you guys knew, and apparently with the admins blessing, cheating credits-wise was legal.
On the other hand, there\'s a ton of people who see the cheating going on, look at the rules and see no rule against it, and.. flame wars like this start up.
At the very least moving forward, I\'m glad the system is changing. Going backwards, got to say it doesn\'t matter much to me. To those that really, really want scores to be revised back to Feb, consider this. Cheaters RAC will drop to sane levels, right? Some will even become upset that they can no longer cheat and their RAC will slowly fade to 0. Given not too much time at all, they\'ll fall to a place in their ranks closer to where they deserve anyway.
If it takes too much effort to revise the old scores on part of the folks running the program I hope it just gets left as it is. Time will sort out the rest at the very least.
____________
 |
|
|
|
|
|
First off, I think it needs to be accepted that some people will stop crunching Rosetta because of this credit change, no matter how it\'s done. However, it is also the case that many people aren\'t currently crunching because of the credit system, and many aren\'t pushing what they can because of the lack of fair competition. Of course the drop-off will be quick and the gains will be gradual. However, if the situation isn\'t rectified then it will always be a contentious issue and I believe the project will suffer in the long run, so I\'m sure it\'s the right thing to do for the project.
I agree entirely with Ethan, and my position on the whole \'optimised\' boinc manager situation is that it isn\'t in breach of the rules, but I consider it poor sportsmanship. Taking it to its logical extreme, if I add a magic Cell board to my computer that boosts my computer\'s SSE performance by 50x, Rosetta won\'t benefit in the slightest, but I\'ll get 50x more credits if using an optimised client. The credits aren\'t alligned to the work done.
I\'d like that to be an irellevant point on this topic, but I\'m afraid its going to be the central theme.
I do, however, appreciate what Zombie67 has said about people joining with one set of crediting rules, and then these being changed retro-actively.
However, I can\'t sway myself from thinking that we have the opportunity to make the credit system fairly accurate based entirely on work done. I\'m fairly sure everyone agrees credit should be based on work done. We have the opportunity to back date this to make the credit system consistent and fair throughout. If you\'ve crunched more than me, you\'ll have more credit than me. If not, you won\'t. We know that\'s not the case at the moment, and I don\'t see how anyone can logically be against that.
The right thing to do for the project is to keep as much crunching power as possible long term.
Just a quick question, we are not going to have this split system (work credit and credit) at Rosetta?
I hope not as it just complicates matters. Just make sure it works over here at Ralph first with actually real measurements for a while through a variety of task.
I agree - I don\'t think the stats sites would appreciate running two credit systems and it would just add to confusion.
____________
|
|
|
|
|
|
In any case it is good to break the whole process into seperate steps:
1. New credit system as an alternative
2. Abandoning the old credit system
3. Correcting past credits according to the new system
Currently we are at step 1 which is already announced on the Rosetta-Homepage. I sincerely hope until tomorrow there will be a page online which explains the new credit system as feet1st did. In fact they can just copy and past it but they _really_ should do this in order to avoid dozens of posts with the same questions in the message boards.
Step 2 and 3 can be discussed and planned once the first step works as intended.
____________
|
|
|
|
|
|
We have the opportunity to back date this to make the credit system consistent and fair throughout. If you\'ve crunched more than me, you\'ll have more credit than me.
dcdc
New member
Joined: Aug 15, 2006
Posts: 3
ID: 1699
Credit: 0
RAC: 0
---
I can see why it wouldn\'t matter if the Credits are Back Dated to somebody that has 0 Credit 0 RAC & at the moment doesn\'t even have a Computer attached to the Project ... 0_o
It doesn\'t matter to me one way or the other if the Credits are Reset because it\'s what I\'ve come to expect from the BOINC Projects, thanks for the help now here\'s your slap in the face.
With data base loses, bad application releases, WU\'s Erring out for no apparent reason not just at this Project but across all the Projects I\'ve lost an enormous amount of Credits already so whats another 50,000 to 100,000 or more lost Credits.
I\'ll be the first to admit to using optimized clients that inflate the credits, but the Project said nor did nothing to curtail that practice. Now like a few people have said already the rules are going to be changed & lets back date it.
Who stands to gain the most from this Back Dating, the people with 0 Credit & 0 RAC I would suspect ... As a side NOTE when the Ralph Project started it was made clear the Credits were not important, In fact the Mods even made that clear a few times, now all of a sudden apparently the Credits are important ... Go Figure
|
|
|
|
|
|
Unless I\'m mistaken this new credit system has developed in three stages. The 2 credits/model stage, some calculated credit method stage used until roughly 14 August, and now some calculated credit method with a correction factor after that one. If you want to compare the 2 CR/Model stage my chart is here for the same puters as below. All are from standard boinc clients. From what I see, this new system needs to go through yet another stage prior to release, as my credit is still \"all over the map\", although is is closer to cross project parity than previous stages. I\'ve changed the colors between stages two and three for easier viewing.
 |
|
|
|
|
|
Poorboy, dcdc has many attached computers to Rosetta as of a month ago. Although, I agree that anyone with zero credit has little at stake in this decision, and that always makes it easier.
 |
|
|
|
|
|
Mine are all over the place to Tony, I get 2 Credits for 1 WU & 10 for another ... I also know dcdc has Computers attached to the Rosetta Project ... |
|
|
|
|
|
(Deleting post, misunderstood) |
|
|
|
|
I can\'t sway myself from thinking that we have the opportunity to make the credit system fairly accurate based entirely on work done. I\'m fairly sure everyone agrees credit should be based on work done. We have the opportunity to back date this to make the credit system consistent and fair throughout. If you\'ve crunched more than me, you\'ll have more credit than me. If not, you won\'t. We know that\'s not the case at the moment, and I don\'t see how anyone can logically be against that.
Precisely, very well said.
|
|
|
|
|
We have the opportunity to back date this to make the credit system consistent and fair throughout. If you\'ve crunched more than me, you\'ll have more credit than me.
dcdc
New member
Joined: Aug 15, 2006
Posts: 3
ID: 1699
Credit: 0
RAC: 0
---
I can see why it wouldn\'t matter if the Credits are Back Dated to somebody that has 0 Credit 0 RAC & at the moment doesn\'t even have a Computer attached to the Project ... 0_o
...
Who stands to gain the most from this Back Dating, the people with 0 Credit & 0 RAC I would suspect ... As a side NOTE when the Ralph Project started it was made clear the Credits were not important, In fact the Mods even made that clear a few times, now all of a sudden apparently the Credits are important ... Go Figure
I have no Ralph credits but we\'re discussing Rosetta.
[edit]i think there\'s a misunderstanding here - we\'re (or at least I\'m) not taking about back-dating Ralph credits - this is about Rosetta credits[/edit]
With data base loses, bad application releases, WU\'s Erring out for no apparent reason not just at this Project but across all the Projects I\'ve lost an enormous amount of Credits already so whats another 50,000 to 100,000 or more lost Credits.
I\'ll be the first to admit to using optimized clients that inflate the credits, but the Project said nor did nothing to curtail that practice. Now like a few people have said already the rules are going to be changed & lets back date it.
I know it\'s annoying to loose credits for work done, and I don\'t know about any of the other projects, but I don\'t know of any lost any data on Rosetta. Also, I know it didn\'t happen from the start, but you should have been credited for any WU errors for the last four months or so.
If you\'re being given credit for the work done I wouldn\'t consider that \'losing\' credits though. You\'d get the right credits for the amount of work completed.
____________
|
|
|
|
|
|
I have no Ralph credits but we\'re discussing Rosetta.
-----
From what I have gathered at the Rosetta Project there will be no Back Dating of the Credits & all the testing that is being done here with the WU\'s & Credit Revision is to be eventually for the Rosetta Project, so if theres going to be no Credit Back Dating there where the most blatant Cheating was going on then why should it be done here ...
|
|
|
|
|
From what I have gathered at the Rosetta Project there will be no Back Dating of the Credits & all the testing that is being done here with the WU\'s & Credit Revision is to be eventually for the Rosetta Project, so if theres going to be no Credit Back Dating there where the most blatant Cheating was going on then why should it be done here ...
It is being proposed here that ROSETTA credits get back dated to Feb and reflect work done. If you go back a couple posts you\'ll see this (he\'s talking about Rosetta):
dekim
Forum moderator
Project administrator
Project developer
Project scientist
I\'m extracting the archived data and will see what I can do. What do other users think about trying to backdate the credits using the new work based system?
|
|
|
|
|
Unless I\'m mistaken this new credit system has developed in three stages. The 2 credits/model stage, some calculated credit method stage used until roughly 14 August, and now some calculated credit method with a correction factor after that one. If you want to compare the 2 CR/Model stage my chart is here for the same puters as below. All are from standard boinc clients. From what I see, this new system needs to go through yet another stage prior to release, as my credit is still \"all over the map\", although is is closer to cross project parity than previous stages. I\'ve changed the colors between stages two and three for easier viewing.
Nice work, but could you explain the points which separated 2nd and 3rd stage in detail more?
It is being proposed here that ROSETTA credits get back dated to Feb and reflect work done. Just to make sure, does it mean that credits won\'t be deleted at all, and eveluate all of the credits granted with the new method and award again?
Thanks for reading,
____________
|
|
|
|
|
|
I\'m confused too, I thought they were discussing changing work done at Rosetta back to February to the \"work done\" system, not here. Maybe we need some clarification??? I don\'t know if that\'s eliminating the old sytem, or simply augmenting rosetta with the addition of \"work done\" credits and the displays that go with it, so that\'s there\'d be two credit systems in play one showing the old rosetta credits and one using the new \"work done\" system. I just don\'t know. |
|
|
|
|
|
I\'m extracting the archived data and will see what I can do. What do other users think about trying to backdate the credits using the new work based system?
----
It is my misunderstanding then if dekim\'s post was meant to be for the Rosetta Project, he wasn\'t specific about which Project he was referring to & I just took it as the Ralph Project ... As I haven\'t processed that much over @ the Rosetta Project for awhile I can\'t wait for the Fire Storm to hit the Project when & if it\'s done, I don\'t think the Dev\'s know what their letting themselves get into if thats implemented ... ;) |
|
|
|
|
Nice work, but could you explain the points which separated 2nd and 3rd stage in detail more?
Thanks for reading,
I don\'t understand your question. I used black to represent stage two, and deep red to show results credited under stage three (if it exists). There appears to be a different credit system which came into play on/about August 14/15, though I\'m not sure. If there is a stage three than it appears to be closer to project parity in terms of granted credit/hour, but values from my limited sample shows them all over the map still.
It\'s possible that with what they have, this is the best they can do. I just don\'t know. If it\'s the best then so be it, but if there\'s room for further improvement, then let\'s try out stage 4. lol For example, I know from my cross project analysis that for users of standard boinc clients, that granted credit is nearly the same (plus/minus 2 granted credits/hour) for a given system. To achieve project parity with results that vary, I would think as returned results which exceed \"claimed credit\" should be offset by the same number of results which are under the \"claimed credit\" value so that when they\'re averaged out come close to \"claimed credit\". If claimed credit for a machine is 10 credits/hour, then results should look like the following for one hours work:
10
10
7
13
5
15
9
11
so that they average out to 10. What I\'m seeing now is that they average out to something higher than \"claimed credit\". Of course my sample has a small number of points and they should have a better \"big picture\" than I see.
|
|
|
|
|
As I haven\'t processed that much over @ the Rosetta Project for awhile I can\'t wait for the Fire Storm to hit the Project when & if it\'s done, I don\'t think the Dev\'s know what their letting themselves get into if thats implemented ... ;)
I think you\'re probably right. The most vocal group might be the winners on this, but as FC says, this might be the best result for the project if fewer leave.
I\'d like to see what the effect on the data would be though as I expect it\'d bring the top teams a bit closer together which\'d be great for competition!
____________
|
|
|
|
|
feet1st and Trog Dog:
We should first see what the response to the new method is and whether users would prefer it over the old. Not everyone will be happy about the final decision. I prefer abandoning the old system but that\'s just my opinion. Users that prefer the old system cannot use as an argument that they get more credits for whatever reason with the old system. The goal to even the playing field.
Cheers dekim
Thank you for your comments.
I have 15 boxes, hardly the most awe inspiring collection, but I\'m constantly adding more and working on updating them, and they will all attach to Rosetta with a high resource share with the move to the new credit system.
____________
|
|
|
|
|
|
But some people use the optimized client to compensate some low credits due to this or that. So a new credit system and a credit recalulation from February shouldn\'t change their results that much. They would be happy to have finally the real credits for the work done......
____________
|
|
|
|
|
I agree entirely with Ethan, and my position on the whole \'optimised\' boinc manager situation is that it isn\'t in breach of the rules, but I consider it poor sportsmanship.
I agree 100%.
With regard to the use of optimised clients as the adage goes \"It\'s not cricket\" (Just like the underarm bowl delivered by the Aussies against the Kiwis back Feb 1, 1981 - a day no Aussie can be proud of)
____________
|
|
|
|
|
Where on Rosetta\'s homepage did it say to download a different client to receive more credits? Users who read the Associated Press article who had no idea of other projects. . they should be punished because they didn\'t research things they had never heard of?
Just like in the real world (especially in the investment world), you must research before you play. The more you know, the better you will do. Know the game in which you play, before you join. This is universal. Even a brief scan of the message boards would clue in a newbie.
How does an apples to apples comparison hurt anyone? If you did the work, you get the credit. . if you did the work faster than the other guy, you get more credit. I\'m not sure how this hurts anyone.
I agree. So going forward, make it so (number one). Changing the rules backwards is wrong. Some people (had they known) would have never participated to begin with.
And worse of all, they have now lost out on accumulating credit on other projects. Rosetta@home is unable to make that right. The people will have been played by R@H, used and discarded like a spent condom. Nice.
I sincerely hope it doesn\'t work out that way, as I don\'t believe that is anyone\'s intention.
I agree with your idea\'s of not back dating credits. It is wrong, no reason to \"fine\" or remove credits from past crunching. Just move forward. If rules are changed then these should not be grandfatherd backwards. I love the 401K analogy. Why take away what was given freely in the past. That is distastfull and could hurt the science as there would be an exoudus from the project, and bad press for the project. I for one would not join a project if it had been known in the past to remove credits because of some new fangled credit granting system.
It has not been done with other projects, when SETI changed the way it granted credits - they did not remove credits that had been granted, neither has any other DC project. I feel it would prove destructive to the project.
____________
|
|
|
|
|
Star Trek references aside :)
The governement makes you accountable for false claims on past returns. . why should people be given passes on clients that granted more credits than the standard one? Because they were the first to use it they should be given amnesty? What do they have against a level scoring system? If they are number one in the current credit system, they should still be number one when the actual work is calculated. . right?
-E
Ethan, these are not false claims, in this case these claims are allowed under the current credit system. The IRS could not go back and make current laws retroactive. Neither should Rosetta.
If caught cheating the credits have been adjusted, and removed and in some cases removed from the project. And they have the right to do that, but to go back at this time and \"adjust\" the credits for everyone is simply bad managment.
____________
|
|
|
|
|
|
I think we should make the changes to the credit system and back date them if that is acceptable to all the other BOINC projects.
edit: I deleted my original post because it was in response to someone who doesn\'t want to get it and I needed to fill the space.
How is this change going to be reconciled with all the other projects? I thought Berkeley were working on that.
|
|
|
|
|
|
I agree with mmciastro that the credit system in its current shape is not yet ready for deployment. :-( I get quite different granted credits/hour which will be a source of endless dispute and lead to cherrypicking of \"credit-rich\" WUs on Rosetta from those who aim for most credits. I wonder why the granted credits for different WU vary so much. How big is your sampling dekim? Perhaps there is too much inequality in the distribution of different WU on RALPH and too much uncertainty about the reported results that it does not work. It seems some WU were processed only by overclaiming clients while others were only processed by underclaiming clients. That needs to be corrected before rolling it out on Rosetta, either by restricting the estimation to trustable hosts or by ensuring the same host pool for each WU and the same factor each hosts contributes to the average credit/model. The more I think about it the more I think the easiest way would be to select different trustable hosts and use their values.
Why inventing a new credit system to level the playing field, which does not level it?
____________
|
|
|
|
|
|
I don\'t have a problem with a credit scheme that grants 10 credits/hour for one WU and 5 for another, as long as over time they average out to cross project parity. After all, everyone would be getting a similar mix and that would be fair. Now, If it was known that \"Xyz\" WU got you more credits/hour than \"Abc\" Wus, then I can guarantee a small population of the community WILL abort the \"Abc\" WUs until they get a full cache of \"Xyz\" WUs. |
|
|
|
|
I don\'t have a problem with a credit scheme that grants 10 credits/hour for one WU and 5 for another, as long as over time they average out to cross project parity. After all, everyone would be getting a similar mix and that would be fair. Now, If it was known that \"Xyz\" WU got you more credits/hour than \"Abc\" Wus, then I can guarantee a small population of the community WILL abort the \"Abc\" WUs until they get a full cache of \"Xyz\" WUs. tralala & mmciastro, due to the random nature of the Rosetta algorithm each model takes a somewhat different time to complete, even for models of the same WU type. I think this is what causes the +/-30% or so variability in credits/hour that you are observing. Once you take averages of 10 crunched WUs or more you will get pretty stable average credits/hour values. In fact, looking at the numbers I have seen so far, Windows/standard client users should see about a 10% increase of their RAC or credits/hour, compared to the old system. Seems to be acceptable to me. Cherry-picking will not be possible because you will only know how many models you completed _after_ you crunched your WU.
____________
|
|
|
|
|
I agree with your idea\'s of not back dating credits. It is wrong, no reason to \"fine\" or remove credits from past crunching.
How about rewarding people who have done more work than they have received credit for? That\'s \"wrong\" too?
The question is, do we punish the people who have done more work and received less credit (don\'t back-date) or do we punish the people who have claimed more credit than the amount of work they have done (back-date)? The right thing to do is back date it. There\'s no getting around it. |
|
|
|
|
|
As I understand it, a fixed credit ratio shall be granted for completed decoys. And this ratio shall be determined from the type of work unit. If this is so, and it is nearing roll out time, then these ratios must have been already determined, or are being determined. If this is so, could those ratios be put in place here to allow us to experience the full flavor of the new system?
____________
 |
|
|
|
|
|
I think as long as overall it will average out to within a reasonable margin after running a few WUs then that\'s good enough for anyone. Do we need a reasonably large scale test of this or can the testing be done on Rosetta concurrent with the current system?
Cherry-picking will not be possible because you will only know how many models you completed _after_ you crunched your WU.
Cherry-picking will be possible unfortunatley, but it would take a bit of work to set up and I\'m not going to say how ;) It will also be identifiable!
____________
|
|
|
|
|
I can\'t sway myself from thinking that we have the opportunity to make the credit system fairly accurate based entirely on work done. I\'m fairly sure everyone agrees credit should be based on work done. We have the opportunity to back date this to make the credit system consistent and fair throughout. If you\'ve crunched more than me, you\'ll have more credit than me. If not, you won\'t. We know that\'s not the case at the moment, and I don\'t see how anyone can logically be against that.
Precisely, very well said.
Ditto, very well said!
HERE IS THE CURRENT PLAN FOR ROSETTA@HOME
FIRST NOTE: The old system will still be in place and the exported stats will still be from the old system for now. We are just going to include new columns on the leader board lists for the new work based credits. This way people can compare the two, and there will finally be a ranking based on actual work.
For the work based columns, I will determine the actual work done from our archives which dates back to about February. Credit before February will be from the old system. So your total credit will include claimed credit before February plus your work based credit.
The hope is to eventually phase out the old system.
____________
|
|
|
|
|
|
With what I\'ve seen so far in one day with up to a 168% difference from lowest to highest credits for one single computer it\'s nowhere near ready to roll out on Rosetta.
You\'re moving to a cherry picking heaven at the moment I would guess.
Wouldn\'t be hard for some of the larger teams (or bord individual) to create a program grab the stats, see what the initial credits claimed are for that type and tell the team.
Could even build it into the boinc client and get it all done automatically.
____________
|
|
|
|
|
|
Please show me results you are basing this on. With regards to cherry picking, we can easily prevent it if it becomes a problem, which I do not think it will.
____________
|
|
|
|
|
|
With just 4 results returned I have a range between 12 and 19 credits/hour, which is a difference of 60%. As Fluffy has pointed out this WILL result in cherrypicking and it WILL result in constant complaining on the boards. It would average out if people weren\'t so mad at getting more credits than others. As fluffy also pointed out it is easy for teams and individuals to almost automate this.
I have one constructive and easy proposal though . ;-) Limit the number of WU sent to a specific host to one at a time with a one hour interval between sending work. This would result in a more even distribution of WU since no host can grab 20 WU of the same kind and inflate the average credit/model of such a WU with over- or underclaiming on all those results. I described it here how it is done and here is the BOINC user guide entry on that.
edit: \"Good result\" (get more of those WU) and \"bad result\" (abort those WU)
edit1: fluffy\'s results differ even more: Result 1, Result 2
____________
|
|
|
|
|
|
these are very small numbers. it should even out. there will always be differences even with the same work unit because of the random nature of the runs.
____________
|
|
|
|
|
I have one constructive and easy proposal though . ;-) Limit the number of WU sent to a specific host to one at a time with a one hour interval between sending work.
That would also limit people that don\'t give a rip about credits from loading a normal 2 or 3 day cache with work. Some (especially dial up users) only connect occaisionally, and if they can\'t get WUs during that time, they will be idle for the next 2 or 3 days.
____________
|
|
|
|
|
these are very small numbers. it should even out. there will always be differences even with the same work unit because of the random nature of the runs.
You call 70% small numbers? That are definitely not small numbers! 10-20% I would call acceptable (not worth the cherry picking).
If you can easily prevent cherry picking why not do it?
____________
|
|
|
|
|
I have one constructive and easy proposal though . ;-) Limit the number of WU sent to a specific host to one at a time with a one hour interval between sending work.
That would also limit people that don\'t give a rip about credits from loading a normal 2 or 3 day cache with work. Some (especially dial up users) only connect occaisionally, and if they can\'t get WUs during that time, they will be idle for the next 2 or 3 days.
This is intended for RALPH only where loading a cache is not senseful and doing other projects inbetween is recommended (especially Rosetta).
____________
|
|
|
|
|
With what I\'ve seen so far in one day with up to a 168% difference from lowest to highest credits for one single computer it\'s nowhere near ready to roll out on Rosetta.
You\'re moving to a cherry picking heaven at the moment I would guess.
Wouldn\'t be hard for some of the larger teams (or bord individual) to create a program grab the stats, see what the initial credits claimed are for that type and tell the team. Fluffy, dcdc, tralala, I think you are mistaken: Cherry-picking is NOT possible !
The variability you are seeing in the credits is not between different WU types but because of the different completion times of the models _within_ the WUs. Even if, say, the first model of a WU takes a long time to complete this doesn\'t tell you anything about how long the following models will take. This is a completely random process. So terminating WUs that start with a \'slow\' model won\'t help you either.
Fluffy, instead of 168% difference you could also say +/-45% difference with respect to the average. Example: average=10, 10-45%=5.5, 10+45%=14.5, 164% difference between lowest and highest. I think this is acceptable, considering that most values will be much closer to the average.
____________
|
|
|
|
|
these are very small numbers. it should even out. there will always be differences even with the same work unit because of the random nature of the runs.
You call 70% small numbers? That are definitely not small numbers! 10-20% I would call acceptable (not worth the cherry picking).
If you can easily prevent cherry picking why not do it?
4 results - that is a small number.
____________
|
|
|
|
|
|
It\'s pretty easy to pick out extremes. The majority of wu\'s will be in the same ballpark, and if grossly off, they could probably be recalculated.
The differences pointed out below are still only half as large as the difference between standard and optimized credits (if not more, I just checked rosetta, an identical computer to mine gets 3000 credits a day, I get 600 using the standard client). . so that\'s an improvement rather than a step back.
____________
|
|
|
|
|
these are very small numbers. it should even out. there will always be differences even with the same work unit because of the random nature of the runs.
You call 70% small numbers? That are definitely not small numbers! 10-20% I would call acceptable (not worth the cherry picking).
If you can easily prevent cherry picking why not do it?
4 results - that is a small number.
Well, you\'ve been warned. ;-) If you are not convinced it\'s a problem try it out. :-)
I\'m not sure what you meant with \"we can easily prevent cherry picking\". How would you do that? The only way to prevent cherry picking is to grant very similar credits/hour for each WU and if this is indeed easily achieved why not do it?
____________
|
|
|
|
|
|
We can limit the number of wu you can abort. We can change the work unit distribution to be more homogenous. We can update the credit/model values. We can penalize people trying to cherry pick. It seems easy enough to me if it becomes a problem. just some ideas off the top of my head.
____________
|
|
|
|
|
|
Dekim is right, our quantity of individual samples is small. Given that this system is supposed to be moved to Rosetta in a day, we are left to bring up possiblilities of what might happen. Things that the project might have not thought about. They possess the \"real numbers\", in terms of the big picture as they hold the entire DB. I say \"cherry picking\" might be possible, but my limited amount of data doesn\'t support that this is possible.
now, if I collected more data, would a pattern appear which might indicate I\'d get more credit per hour running one wu type over another??? If you look at the results in bold, and IF it became apparant that that type WU consistently returned more per hour, then I could delete all that weren\'t that type.
we have to try to find things to improve your final product before you release it. We are left supposing, guessing, speculating in an effort to help you.
Note: this pic is the same as my earlier one. It\'s just had the WU names appended to it. |
|
|
|
|
|
mmciastro, maybe I\'m reading your graph wrong (thanks for posting it btw). . . but it looks very consistant on a credit/hour basis for a given computer.
Is this the case? And if so, how do you feel about the credits/hour for the various machines (is machine 2 really about 2.5 faster than machine 1)?
____________
|
|
|
|
|
With what I\'ve seen so far in one day with up to a 168% difference from lowest to highest credits for one single computer it\'s nowhere near ready to roll out on Rosetta.
You\'re moving to a cherry picking heaven at the moment I would guess.
Wouldn\'t be hard for some of the larger teams (or bord individual) to create a program grab the stats, see what the initial credits claimed are for that type and tell the team. Fluffy, dcdc, tralala, I think you are mistaken: Cherry-picking is NOT possible !
The variability you are seeing in the credits is not between different WU types but because of the different completion times of the models _within_ the WUs. Even if, say, the first model of a WU takes a long time to complete this doesn\'t tell you anything about how long the following models will take. This is a completely random process. So terminating WUs that start with a \'slow\' model won\'t help you either.
Fluffy, instead of 168% difference you could also say +/-45% difference with respect to the average. Example: average=10, 10-45%=5.5, 10+45%=14.5, 164% difference between lowest and highest. I think this is acceptable, considering that most values will be much closer to the average.
I know I could say that and I certainly wouldn\'t say a 45% difference was acceptible, yes it\'s better than the x3 optimised usualy give over the standard.
But the way I currently see it is that you may as well just count the number of models you creat instead of assigning a crdit value to it.
It\'ll save an lot of bother ;-)
All in all if you are going to go down this route I would have thought that you put everything into pending credit, then apply the awarding of credit till you have a statistically sound credit awarding per job type. Then adjust for time taken (using an internal timing procedure, not the boinc core client... This alters for actual work done not assumed work done)
Pending credit happens on any project that uses a quorum anyway so that\'s not really problem and it would only happen for the first \'however many you think necessary x00\'s of retuned jobs before it could start graning instantly again. Preferably from trusted clients you know of on Rosetta. Not here on Ralph. that free\'s up ralph to actually impove the client without getting in the way of credit. That should make it quicker as you\'ll get though far more task in a day than you would in a week in rosetta (I should hope).
P.S
How do you actually intend to get the \'credit per model\' across all the platform. The only way I can see it is to build a client (if you really still want to use Ralph for the benchmarking) is to send out the ralph-client with an internal benchmark to the testers, that would bypass having to worry about the boinc client used.
____________
|
|
|
|
|
mmciastro, maybe I\'m reading your graph wrong (thanks for posting it btw). . . but it looks very consistant on a credit/hour basis for a given computer.
Is this the case? And if so, how do you feel about the credits/hour for the various machines (is machine 2 really about 2.5 faster than machine 1)?
I think your reading it wrong, the credit/hr (C/H) should be consistent as it\'s the boinc benchmarks credit per hour.
The G/H is the new method
EDIT:
Machine one is a Celeron 500, two is a Pentium 4 1.8GHz so probably not fa off.
http://i65.photobucket.com/albums/h228/mmciastro/Ralphnewprojectcomparison5.jpg
mmciastro, culd you set them to run at 3hr units. which I believe is the default here (and so I guess at Rosetta, I cannot check as it\'s under maintenace there) Since that\'s what the majority will be using, err, by defualt ;-)
____________
|
|
|
|
|
mmciastro, maybe I\'m reading your graph wrong (thanks for posting it btw). . . but it looks very consistant on a credit/hour basis for a given computer.
Is this the case? And if so, how do you feel about the credits/hour for the various machines (is machine 2 really about 2.5 faster than machine 1)?
The first four columns are the same as you would see if you looked at \"your results\" page, the last four are my own composition \"C/H\" is claimed credit/hour, G/H is granted credit/hour, Model is the number of models done, and ofcourse the last column is the name of the wu done. Claimed credit, and C/H are consistent as they are based on the benchmark credit sytem. Granted credit and G/H are what I\'m actually getting. It\'s Granted Credit or G/H that is all over the map, but as long as they average out to the same as other projects, then they have done a good job in \"cross project parity\". Currently my G/H is higher (on average) than I get from other projects (see earlier posted cross project comparison chart).
does this help?
tony |
|
|
|
|
mmciastro, culd you set them to run at 3hr units. which I believe is the default here (and so I guess at Rosetta, I cannot check as it\'s under maintenace there) Since that\'s what the majority will be using, err, by defualt ;-)
Done, though I don\'t know what it will do if they release this new system tomorrow. LOL |
|
|
|
|
|
First, a question - I must say, DeKim (david?) I really, honestly think that you are going the wrong direction here. I understand that you want to change the existing credit system, and because of that it is safe to infer that you felt the existing system wasn\'t working. Why?
Back in the beginning (pre-boinc), SETI@Home used a system that was not too unlike your proposed method. There also was no quorum for results, and the credit system was fairly basic - 1 credit per workunit.
Everyone thought this was fine and fair, but in reality this system was more bugged and UNFAIR than you could imagine. Some workunits would take much much longer, and there really wasn\'t any way that you could determine an average run time for every workunit without completely processing all of them (which defeats the purpose of having the DC project) ALTHOUGH, with enough care and thought, you could predict runtimes for 95%+ of the work.
When BOINC was created, it was a harsh change to move to a crediting system that was based more on the actual work done closer to the process thread level than to assign arbitrary values to each workunit. If you want to change the existing system - Work with David Anderson and Rom Walton, and see if you can iron out the wrinkles. The -ONLY- fair way of granting credit is to calculate actual work done using total FLOPS or some other completely scientific method. NOT the only seemingly accurate (yet still arbitrary) averaging system you have implemented here.
Please understand, I have been with BOINC since version 0.07, and SETI for years before that. The problems that exist now with the current credit system FAR OUTWEIGH the problems we had beforehand. I hope that you take this message to heart, and understand that what I feel you are doing is taking a step backwards in your attempt to be revolutionary. I also support you and this project in whatever changes you make; However, this doesn\'t mean that I am not more than mildly upset at the change.
I\'m sorry that I haven\'t spoken on the subject earlier, but work has ahold of me as of late. As we speak, I\'m at the world AIDS conference in Toronto, and could not wait to comment here until I returned home as I feel that strongly that you could be making a big mistake. CERTAINLY, (and in the very least) - I would convey that MUCH more testing is needed.
Credit is something that lives at the core of many of your constituents. Monopoly just isn\'t Monopoly without Boardwalk and Free Parking. Tread extremely carefully here if you do nothing else!
We can limit the number of wu you can abort. We can change the work unit distribution to be more homogenous. We can update the credit/model values. We can penalize people trying to cherry pick. It seems easy enough to me if it becomes a problem. just some ideas off the top of my head.
____________
 |
|
|
|
|
these are very small numbers. it should even out. there will always be differences even with the same work unit because of the random nature of the runs.
David, is there any way to further refine credit? I mean for example to measure the value of a compelte relax run as compared to a model that aborts it? Do the WU results return that level of detail as to how many full atom relaxes were aborted and at what point?
I\'ve mentioned the idea of encrypting the .out data previously, but later it dawned on me that this would tend to get in the way of fun things like displaying your completed predictions... I suppose in that case it still runs the .out file through Rosetta, so it could do the needed decyption. But anyway, if you could encrypt portions of the results file to enable simple authentication of the results being claimed, and yet leave as much of the human readable stuff as you can that would be a good compromise. I\'m hoping over time you can describe more of the contents of the .out file, so participants can better see the mechanics of how it all works and what results our clients are reporting back.
____________
|
|
|
|
|
First, a question - I must say, DeKim (david?) I really, honestly think that you are going the wrong direction here. I understand that you want to change the existing credit system, and because of that it is safe to infer that you felt the existing system wasn\'t working. Why?
The current system places credits in the hands of individual particpants. . you essentially keep your own score. Want more credits? Just claim more credits by making your benchmarks higher than possible (some computers claim 15+ Gflops per cpu. . even at 4ghz and running two floating point calculations a cycle, that\'s 8 Gflops. . and cpu\'s don\'t get near theoretical). Don\'t get too greedy or you\'ll get zerod out. It\'s like speeding, if you go the same speed as everyone else in the left lane, even if 10 over, you\'re not likely to get in trouble. . . if you\'re going 40 over in the median, you\'ll get busted. Not a very good system imo.
The new system is taking the law of averages to make the playing field more fair. It\'s taking the average credit claim from hundreds of machines and applying the same score to each.
____________
|
|
|
|
|
...I understand that you want to change the existing credit system, and because of that it is safe to infer that you felt the existing system wasn\'t working. Why?
The reasons why are numerous and sprawled throughout the Rosetta boards, including the infamous (and deleted) cheating thread.
I you were not aware of it... the current BOINC implementation allows a user to basically modify a simple file with notepad and claim their machine is 10x (or 1000x) faster then it really is. That\'s the basic premise of the need for change. And that\'s why many of the BOINC projects are changing in ways appropriate for each project\'s work.
In the end, the new system will still equate back to the FLOPS that BOINC proports to measure. But it will be much more difficult to modify your results and try to claim more credits.
____________
|
|
|
|
|
The current system places credits in the hands of individual particpants. . you essentially keep your own score. Want more credits? Just claim more credits by making your benchmarks higher than possible (some computers claim 15+ Gflops per cpu. . even at 4ghz and running two floating point calculations a cycle, that\'s 8 Gflops. . and cpu\'s don\'t get near theoretical). Don\'t get too greedy or you\'ll get zerod out. It\'s like speeding, if you go the same speed as everyone else in the left lane, even if 10 over, you\'re not likely to get in trouble. . . if you\'re going 40 over in the median, you\'ll get busted. Not a very good system imo.
The new system is taking the law of averages to make the playing field more fair. It\'s taking the average credit claim from hundreds of machines and applying the same score to each.
No offense, but the logic for this change astounds me. The current system itself is not flawed in it\'s fairness to all - on the contrary, it is the most fair. The problem is that the current system places too much responsibility for ones credit in the hands of the unknown. The only fair and appropriate way to fix it is to remove the power to control ones own credit from the CURRENT system.
What has been proposed here is to add back in the inaccurracies of the past at the cost of losing fairness to all.
Credit manipulating should never have been allowed by the public. These values should be encrypted and no access to them should be provided to the GP.
____________
|
|
|
|
|
...I understand that you want to change the existing credit system, and because of that it is safe to infer that you felt the existing system wasn\'t working. Why?
The reasons why are numerous and sprawled throughout the Rosetta boards, including the infamous (and deleted) cheating thread.
I asked for an answer, not a generalization. Even if they are numerous, then there is no better place to index them than here.
I you were not aware of it... the current BOINC implementation allows a user to basically modify a simple file with notepad and claim their machine is 10x (or 1000x) faster then it really is. That\'s the basic premise of the need for change. And that\'s why many of the BOINC projects are changing in ways appropriate for each project\'s work.
Then -THAT- would be the problem to fix. As I said, encrypt these values and do not allow the GP access to the tools needed to change them.
In the end, the new system will still equate back to the FLOPS that BOINC proports to measure. But it will be much more difficult to modify your results and try to claim more credits.
The new system will be a system of averages. Nothing more.
Averages are hardly accurate. Where are my Swiss and German friends of yesteryear? Where is Jens? Where is Riedel? The founders of BOINC would be uproarious about this change.
____________
|
|
|
|
|
First, a question - I must say, DeKim (david?) I really, honestly think that you are going the wrong direction here.
I am just adding more info for user\'s for now. If you want to know how much actual work you\'ve done compared to others, look at the new info once it\'s up. If not, just ignore it. My goal in response to users and just plain old logic is to offer a more fair credit system. -- David K
____________
|
|
|
|
|
First, a question - I must say, DeKim (david?) I really, honestly think that you are going the wrong direction here.
I am just adding more info for user\'s for now. If you want to know how much actual work you\'ve done compared to others, look at the new info once it\'s up. If not, just ignore it. My goal in response to users and just plain old logic is to offer a more fair credit system. -- David K
Fair is the wrong word, but I can see why you use it. It\'s much more political than saying \'The credit system will now be harder to manipulate by malicious users, at the expense of accuracy.\'
I understand the problem, but I think the solution is improper.
____________
|
|
|
|
|
No offense, but the logic for this change astounds me. The current system itself is not flawed in it\'s fairness to all - on the contrary, it is the most fair.
We astound each other :) So if a PGA golfer finished a round and reported his score as 17 (you keep your own score in golf), the rest of the players have to accept it even though obviously false? Everyone who uses optimized clients are raising their credit claims to levels not based on the properties of their hardware. It\'s not a fair system when you have to use modified software (or subtract 60 from your golf score) to stay competitive.
There is not less accuracy with the new system, there is more. . by definition of averaging out work unit claims. It\'s like a quorum of 100 (or 1000, however many are used to create the average).
I agree that the uber solution would be to have the Boinc folks release a scoring system that\'s hidden in compiled code. . but that\'s not feasible when the code is open source and out of Rosetta\'s hands.
Fair is the wrong word, but I can see why you use it. It\'s much more political than saying \'The credit system will now be harder to manipulate by malicious users, at the expense of accuracy.\'
I still fail to see how the new system is less accurate. Senario 1, everyone has optimized clients and claim whatever they feel like. Result = no accuracy whatsoever. Senario 2, half the people use optimized clients, the other half use standard ones. Result = a more accurate scoring system and level playing field.
____________
|
|
|
|
|
We astound each other :) So if a PGA golfer finished a round and reported his score as 17 (you keep your own score in golf), the rest of the players have to accept it even though obviously false? Everyone who uses optimized clients are raising their credit claims to levels not based on the properties of their hardware. It\'s not a fair system when you have to use modified software (or subtract 60 from your golf score) to stay competitive.
Just because some people have found an open door allowing them to sidestep the system -DOES NOT MEAN- that the system in it\'s design is not fair.
You do not reinvent the system simply because somebody found a back door. You close the door and put a lock on it.
____________
|
|
|
|
|
First, a question - I must say, DeKim (david?) I really, honestly think that you are going the wrong direction here.
I am just adding more info for user\'s for now. If you want to know how much actual work you\'ve done compared to others, look at the new info once it\'s up. If not, just ignore it. My goal in response to users and just plain old logic is to offer a more fair credit system. -- David K
Fair is the wrong word, but I can see why you use it. It\'s much more political than saying \'The credit system will now be harder to manipulate by malicious users, at the expense of accuracy.\'
I understand the problem, but I think the solution is improper.
Fair is completely the right word as I stated it as a goal as in \"the goal is to be fair.\"
____________
|
|
|
|
|
Fair is completely the right word as I stated it as a goal as in \"the goal is to be fair.\"
Then, bear with me for a moment. I will type a long explanation of what I think would be far superior.
____________
|
|
|
|
|
Fair is completely the right word as I stated it as a goal as in \"the goal is to be fair.\"
Then, bear with me for a moment. I will type a long explanation of what I think would be far superior.
Problem : The credit system is easy to manipulate because benchmarks can be altered, and tags can be altered as simply as opening a text editor and entering in your own scores.
Question : Assuming you could remove the ability to alter scores from the public\'s hand, would this make the scoring system stronger?
Answer : Assuming that this is possible, the current scoring system would be much more fair than assigning an arbitrary average scoring system. However - since scores can be manipulated, you thought that only giving credit on a per/model basis was the only way to keep scores \'fair\', as it is the only way to assign proof to work. You can\'t cheat if you don\'t have results for your work, and you shouldn\'t have results if you cheat. (but we all know you can, if anyone remembers SETI Classic)
Solution : The Rosetta core application should do it\'s own benchmarking, and assign encrypted scoring values inside each workunit. BOINC level scoring systems could be bypassed on a per project basis, or per workunit basis even if you wanted to. The validator could be changed to decrypt these values on the fly, and everyone is happy. Although now that I think about it, you don\'t -have- to give this responsibility to the validator, you could require some background communication from Rosetta core whenever it needs to send in these values. It could send them in to the scheduler if you prefer.
Why do you feel you need an entirely new system to fix one problem? What are the other problems with the current scoring system? So far I have heard only one, and that is that it can be manipulated by EU/GP
____________
|
|
|
|
|
|
This is a great proposal. If such a system gets developed, my vote would be to use it. That doesn\'t mean we should abandon our current efforts, though.
____________
|
|
|
|
|
|
Thanks for the last message Aaron. . I think it sums up what we agree on. The ideal would be something unhackable in Boinc iteslf. . but that wouldn\'t be easy due to the open source nature of the code and it would require everyone to install a new client to run Rosetta.
A benchmark within the Rosetta itself would be great as well. . I\'m not sure how feasible that is either. My understanding is the code branches a lot, many many function calls. Is it possible to count every calculation? The benchmark would have to be time independent. . otherwise a slow machine working for an hour would want the same score as a fast machine-hour.
Back to counting calculations. . does an integer calculation could the same as a floating point? How about a double float? How much overhead is there in keeping track of billions of operations. Those C++\'s would add up pretty quickly :)
You could base a benchmark off how long it takes to crunch a \'golden\' work unit. . but that one protein might use completely different parts of Rosetta than HIV research. . and having to create new \'golden\' benchmark units, normalize them to each other, and package them in each new type of wu is a lot of overhead (for scientists and cpus).
Just throwing those ideas out there. . there may be ways of doing them I hadn\'t though of.
The lowest hanging fruit seems to be the proposed new system. It eliminates the advantages of the optimized client. People will still claim more credit than they deserve, but those will average with standard client claims, and everyone receives the same score. The only overhead is averaging the wu claims for each wu type. . and it will work with all versions of boinc and not require recoding of rosetta.
____________
|
|
|
|
|
This is a great proposal. If such a system gets developed, my vote would be to use it. That doesn\'t mean we should abandon our current efforts, though.
Well, it\'s disheartening that you feel that you have no other option but to change what I (and many of the creators of BOINC) felt was a much better system back to the old system which we felt wasn\'t as fair, just to make things more fair. Do you see why this would be hard to swallow?
The goal certainly should be to make things more fair, but I just don\'t feel that you should cut off everyone\'s feet simply because some unscrupulous people invented rollerskates. There wasn\'t anything wrong with feet until rollerskates came along - Just make it so there aren\'t any rollerskates.
The Rosetta core application is all in -your- hands, so I guess I just don\'t understand why you couldn\'t program it to perform as you see fit?
I know that it must be easier to simply change a few settings with the scoring algorythms, but believe me the current system works, it\'s merely got a backdoor that needs locking.
____________
|
|
|
|
|
I know that it must be easier to simply change a few settings with the scoring algorythms, but believe me the current system works, it\'s merely got a backdoor that needs locking.
That\'s the problem, how do you lock it? Boinc and Rosetta are seperate things. Rosetta has nothing to due with credit calculation other than returning the time it took to complete the WU. Boinc then takes that, does some math based on your benchmarks (or edited benchmarks), and returns the results to the server. Rosetta has to trust the credit claims since the benchmarking part of Boinc is out of its control.
The only way to close the backdoor is to have a new version of Boinc developed and require everyone upgrade to have their credits counted. . . Or get into the business of determining if each seemingly large score is real or inflated.
____________
|
|
|
|
|
|
prior to the currently in-testing credit system, I thought that flops counting was the ideal, with the benchmark as either a library to be included in the BOINC code during compilation (don\'t know if it\'d be possible to create a moduel that isn\'t \'tamperable\'), or a standard benchmark to be included in the project apps. Infact I remember my very first DC post over at UD said the very same thing back in 99 or 2000! I haven\'t seen anything that says that this is an accurate system though- as Ethan says, what value to you assign to FPU operations over Integer operations etc. There is some arbitration even in that system isn\'t there? (I might be wrong - my understaning of programming is pretty minimal beyond VB...)
____________
|
|
|
|
|
I know that it must be easier to simply change a few settings with the scoring algorythms, but believe me the current system works, it\'s merely got a backdoor that needs locking.
That\'s the problem, how do you lock it? Boinc and Rosetta are seperate things. Rosetta has nothing to due with credit calculation other than returning the time it took to complete the WU. Boinc then takes that, does some math based on your benchmarks (or edited benchmarks), and returns the results to the server. Rosetta has to trust the credit claims since the benchmarking part of Boinc is out of its control.
The only way to close the backdoor is to have a new version of Boinc developed and require everyone upgrade to have their credits counted. . . Or get into the business of determining if each seemingly large score is real or inflated.
Didn\'t you read my post? I could have swore you did by your previous reply, but now I think you must have missed it.. In it, I said quite clearly to put this in the hands of the Rosetta core application. Go back and read the last 5-10 posts, it\'ll explain it in it\'s entirety.
You don\'t need BOINC to assign scores. It only does this as a courtesy to facilitate projects. Someone needs to get with David and/or Rom, and ask how you can incorporate benchmarking into the projects core app.
____________
|
|
|
|
|
Solution : The Rosetta core application should do it\'s own benchmarking, and assign encrypted scoring values inside each workunit. BOINC level scoring systems could be bypassed on a per project basis, or per workunit basis even if you wanted to. The validator could be changed to decrypt these values on the fly, and everyone is happy. Although now that I think about it, you don\'t -have- to give this responsibility to the validator, you could require some background communication from Rosetta core whenever it needs to send in these values. It could send them in to the scheduler if you prefer.
Wow. Good stuff. Where were you for the last 3 months... ?
|
|
|
|
|
Solution : The Rosetta core application should do it\'s own benchmarking, and assign encrypted scoring values inside each workunit. BOINC level scoring systems could be bypassed on a per project basis, or per workunit basis even if you wanted to. The validator could be changed to decrypt these values on the fly, and everyone is happy. Although now that I think about it, you don\'t -have- to give this responsibility to the validator, you could require some background communication from Rosetta core whenever it needs to send in these values. It could send them in to the scheduler if you prefer.
Wow. Good stuff. Where were you for the last 3 months... ?
I\'m very sorry.. I have been busy at work. Unfortunately, I\'m supposed to be working right now, but my passion for this issue is distracting me. :(
____________
|
|
|
|
|
|
Let\'s try to stay focused in this discussion.
We have already incorporated a simple benchmark in rosetta. But since we don\'t use redundancy people can still be malicious, so we decided not to go that route.
____________
|
|
|
|
|
|
Aaron, even the creators of Boinc (people of seti) couldn\'t close \"the door\", they implemented the FPOPS sytem to do just that. For some reason I have yet to understand, Fpops isn\'t useable here, so they\'re trying what they can. If the creators of boinc can\'t fix the existing benchmark system without an overhaul, what chance does the Rosetta staff have??? Oh, and the Benchmark system does have ONE decent flaw, it doesn\'t give linux users a comparable benchmark to it\'s windows counterpart (my lord, I\'ll be joining XS soon. LOL) |
|
|
|
|
Let\'s try to stay focused in this discussion.
We have already incorporated a simple benchmark in rosetta. But since we don\'t use redundancy people can still be malicious, so we decided not to go that route.
How can you be malicious (easily) if the core application has it\'s own encrypted method of \'benchmarking\', the workunits have their own encrypted tags, and the results are also containing encrypted tags? I\'m assuming that there are parts of the Rosetta Core application that are not available to the GP as source..
If you were to use a code (such as 512bit RSA? 1024-bit RSA enough?) that is (realistically) unhackable, then why do you think people could still alter their benchmarks if you control every step of the process?
____________
|
|
|
|
|
|
Hello,
Mmclasro you have a good point and a funny one, which is quite bewildering at the same time, and understandable untill Linux really takes off in a couple years. Seeing as Xp has been around long enough for Widows users to need a \"Safe\", \"Secure\" O.S..The differece in Bench,s for Linux isnot comparable.Assides from that, there are many
valid points within this Thread.Rosetta is \"Great\" Science, therfore running
Ralph is just as Paramount.I am Happy to be a part of something, that \"could\"
save 1 Human life, if nothing else, Keep crunching.
\"Happy Crunching Boincateers\"
Sincerely
Sluger
____________
|
|
|
|
|
Let\'s try to stay focused in this discussion.
We have already incorporated a simple benchmark in rosetta. But since we don\'t use redundancy people can still be malicious, so we decided not to go that route.
How can you be malicious (easily) if the core application has it\'s own encrypted method of \'benchmarking\', the workunits have their own encrypted tags, and the results are also containing encrypted tags? I\'m assuming that there are parts of the Rosetta Core application that are not available to the GP as source..
If you were to use a code (such as 512bit RSA? 1024-bit RSA enough?) that is (realistically) unhackable, then why do you think people could still alter their benchmarks if you control every step of the process?
I agree, if we add encryption to rosetta, we could use the internal benchmark. It\'s something to consider.
____________
|
|
|
|
|
How can you be malicious (easily) if the core application has it\'s own encrypted method of \'benchmarking\', the workunits have their own encrypted tags, and the results are also containing encrypted tags? I\'m assuming that there are parts of the Rosetta Core application that are not available to the GP as source..
If you were to use a code (such as 512bit RSA? 1024-bit RSA enough?) that is (realistically) unhackable, then why do you think people could still alter their benchmarks if you control every step of the process?
Would that produce the same benchmark regardless of OS?
____________
|
|
|
|
|
Would that produce the same benchmark regardless of OS?
I would assume so, yes. However, to get that result it may be only slightly more complicated.
For instance, the linux crowd may get one version of the benchmark, while the windows crowd would obviously get another.
Depending on how complicated the benchmark is, it could be only as difficult as tagging the compiler for your specific OS, or rewriting it as you see fit.
Either way - I only really see 3 maybe 4 OS versions being used currently, it surely couldn\'t be that difficult.
____________
|
|
|
|
|
|
Aaron, certain have to agree that encryption is possible, because Rosetta is not open source. But to simply make an effort to encrypt the BOINC figures still apparently penalizes Linux, and allows for other means of modifying the BOINC benchmarks... and so I believe that is why David Kim suggests continuing to pursue the current credit system... because it provides the level playing field (all P4 2.4Ghz systems, baring any other disparities in capabilities should score the same, regardless of Linux or Windows or \"optimized\" or stock), and then you can later add the encryption, which provides an unmodified playing field. So the two aren\'t mutually exclusive.
Since Rosetta is but an application within BOINC... they aren\'t really in control of some of the code you are suggesting be changed. But encryption for the part they do control, the key portions of the .out file, is certainly possible. And, as I\'ve expressed before, I believe it would be helpful. Although I\'ll point out that if you encrypt the whole file, then it adds to the mystic and the cynics which come along from time to time and assert that the whole project is a fraud, or that it is a virus that\'s stealing your data or whatever. So, just encrypt a tiny portion of the file, so folks can SEE what\'s being sent and gain further interest and knowledge about the science being done here.
____________
|
|
|
|
|
But to simply make an effort to encrypt the BOINC figures still apparently penalizes Linux, and allows for other means of modifying the BOINC benchmarks...
K.. I\'ll bite. How? and be specific, because what you are saying is that simply because the code is encrypted it punishes the linux crowd. forgive me if your statement lost me there. As for the second half of your statement, you\'re obviously not getting what I\'m saying, I\'m not talking about BOINC benchmarks.
Since Rosetta is but an application within BOINC... they aren\'t really in control of some of the code you are suggesting be changed. But encryption for the part they do control, the key portions of the .out file, is certainly possible.
C\'mon, use your imagination here. What difference does that make? Is rosetta forbidden to make it\'s own files to send back to the validator / scheduler / some other server? I think not. People, if you want to take the benchmarking out of BOINC\'s hands, we have to start thinking outside the box. The box being - Currently we let the allmighty BOINC handle everything except wiping our butts. YOU DON\'T HAVE TO LET BOINC DO ANYTHING! Period. How many times, and in how many different languages do I have to say it - BOINC only handles these functions as a courtesy for projects. Yes, it makes things easier, but we\'re not talking about something that can be done -within- BOINC are we?
And, as I\'ve expressed before, I believe it would be helpful. Although I\'ll point out that if you encrypt the whole file, then it adds to the mystic and the cynics which come along from time to time and assert that the whole project is a fraud, or that it is a virus that\'s stealing your data or whatever. So, just encrypt a tiny portion of the file, so folks can SEE what\'s being sent and gain further interest and knowledge about the science being done here.
That\'s great if it can be done, but if it makes things too difficult, I wouldn\'t make it that high of a priority. Reasoning:
Those people are either...
1. Delusional.
2. Not downloading the project in EITHER case.
3. Not educated enough to know how to open up the program and know what it\'s doing ANYWAY.
4. Running the project, but like to be forum trolls.
In case of #1 - Non-Issue
In case of #2 - Complete Non-Issue
In case of #3 - Non-Issue
and in case of #4 - I\'m sorry, I don\'t respond to trolls.
Assuming they aren\'t in one of the 4 listed scenarios above, What are you going to lose - 1, maybe 2 conspiracy freaks? Yeah.. Those people seem like real contributors to a DC project.
One thing I would like to add, is that one thing I remember David saying is that he lays awake at night thinking about how to get more members. I can\'t remember where I heard that or where he said it. However - If you change the credit system, you are GOING to upset people, no matter how much more logical useful, or fair the system that you\'re moving to seems. This is why I think it would be best to only fine-tune the existing system, rather than switch to one of a different type. Otherwise, are you prepared to disgruntle your participants? If the switch to BOINC\'s crediting system (remember trying to explain the cobblestone?) was any indication, I can guarantee you - It will disgruntle some of them. I know David definately doesn\'t want to be laying awake at night thinking about why the project lost ~5000 users.
____________
|
|
|
|
|
...
I agree, if we add encryption to rosetta, we could use the internal benchmark. It\'s something to consider.
You\'d also need to encrypt the in-memory benchmark results as well as the upload and download, also obfuscate the actual benchmark code so that people can\'t follow what it\'s doing in a debugger and modify the values on the fly. (Remember if people can use a debugger, an out-of-process routine can do the same once the memory locations are known).
The recent debugging code (map files etc) would prove extremely useful to someone trying to do this, but aren\'t necessary.
The new system does the job, and should do it well after the bugs are ironed out, so why mess around with insecure ways of doing the same thing?
____________
|
|
|
|
|
|
I\'ll throw something else in here. One advantage of the proposed new credit system over any that use benchmarks is credit-crunching alignment, which is the very thing we\'ve been requesting here. Benchmarks are rarely perfectly aligned with the work to be done. As one example, the size of CPU cache may play a large part in crunching ability, but it is rarely a factor that influences benchmarks as they tend to be small and quick.
By not using a benchmark, or rather using a whole decoy as a benchmark, the credits are exactly aligned with the credit system. The only issue remaining is that there is variation in time taken to produce decoys on any given machine, but no variation in credit awarded. If this averages accurately, then I believe it to be the best system. If someone tweaks their computer by, say increasing the FSB, which removes a CPU-RAM bottleneck, this is unlikely to be picked up in any benchmark, but will show up in this system.
The other option of course is to try and make the benchmark realistic. This is difficult, because of the size and variation in algorithms used within the WUs. The \'new credit system\' includes for this.
My final point regards the golden machines used. I\'ve posted somewhere - here or at Rosetta - that the makeup of the golden machines needs to reflect the wild population for the following reason:
If the golden machines are 10 AMDs running Windows, which are very good at 2 in 10 WUs (as they fit into a small cache), but take much longer with the remaining 8 (as they are slightly larger than the AMD cache) then the first 2 WUs will be assigned little credit, while the remaining 8 will be assigned more. In the wild, although a Pentium isn\'t affected by the difference between the WUs sizes due to its larger cache, it will still get the credits that reflect how the AMDs crunched them.
This may be a minor point with regard to CPU cache sizes (or it may not), but there are a great number of factors that could influence this, including whetstone performance, dhrystone performance, CPU cache size, FSB/HyperTransport speeds, RAM availability etc...
The place that matches the wild population of computers the most is, of course, the wild population! Thats my current take on a credit system.
I can see that a well aligned benchmark can be more accurate in the short term, which is good for those wanting to push their computers, but after a week or two of crunching I\'d be suprised if this system didn\'t prove more accurate.
cheers
Danny
____________
|
|
|
|
|
...
I agree, if we add encryption to rosetta, we could use the internal benchmark. It\'s something to consider.
You\'d also need to encrypt the in-memory benchmark results as well as the upload and download, also obfuscate the actual benchmark code so that people can\'t follow what it\'s doing in a debugger and modify the values on the fly. (Remember if people can use a debugger, an out-of-process routine can do the same once the memory locations are known).
The recent debugging code (map files etc) would prove extremely useful to someone trying to do this, but aren\'t necessary.
The new system does the job, and should do it well after the bugs are ironed out, so why mess around with insecure ways of doing the same thing?
I think it is way easier to fake the numbers of decoys generated than to fake the internal benchmark. So the currently tested system is not fake-proof either. I think though that this is not necessary since there will be only a very few people who really put in some \"criminal energy\" to achieve more credits.
I think we need to implement a new credit system soon. I voted for thorough testing on RALPH and not testing on Rosetta until it was really mature. However since we did test it on Rosetta togehter with the disastrous backdating discussion there is no way back and it should be solved soon, which means the new credit system must be implemented.
____________
|
|
|
|
|
I think it is way easier to fake the numbers of decoys generated than to fake the internal benchmark. So the currently tested system is not fake-proof either.
I\'d be very suprised if that were true, but I\'d like to hear DK\'s thoughts on the subject.
____________
|
|
|
|
|
I think it is way easier to fake the numbers of decoys generated than to fake the internal benchmark. So the currently tested system is not fake-proof either.
I\'d be very suprised if that were true, but I\'d like to hear DK\'s thoughts on the subject.
A user can fake the [edit: claimed credits, not the internal benchmark] by just editing a file just as a user can fake the numbers of decoys generated. The difference is that we can validate the decoys and catch cheaters. Unless we start encrypting data or use redundancy, the system is not going to be fake proof.
____________
|
|
|
|
|
Unless we start encrypting data ... the system is not going to be fake proof.
You sir, hold all the cards in regards to that.
____________
|
|
|
|
|
A user can fake the [edit: claimed credits, not the internal benchmark] by just editing a file just as a user can fake the numbers of decoys generated. The difference is that we can validate the decoys and catch cheaters. Unless we start encrypting data or use redundancy, the system is not going to be fake proof.
No, it\'s not.
I have my last Ralph WU in my cache, and when that\'s returned, I\'ll move my Ralph share to uFluids. So this is goodbye from me.
I don\'t expect a personal thank you for my contribution from David Baker, let alone a thank you from him on my teamboard.
Goodbye.
____________
"I'm trying to maintain a shred of dignity in this world." - Me
 |
|
|
|
|
A user can fake the [edit: claimed credits, not the internal benchmark] by just editing a file just as a user can fake the numbers of decoys generated. The difference is that we can validate the decoys and catch cheaters. Unless we start encrypting data or use redundancy, the system is not going to be fake proof.
No, it\'s not.
I have my last Ralph WU in my cache, and when that\'s returned, I\'ll move my Ralph share to uFluids. So this is goodbye from me.
I don\'t expect a personal thank you for my contribution from David Baker, let alone a thank you from him on my teamboard.
Goodbye.
Forgive me.. I know you\'re having a moment - but I\'m confused, and I need your help to understand things - You\'re leaving because we\'re trying to figure out how to prevent falsification of credit data?
____________
|
|
|
|
|
We can limit the number of wu you can abort. We can change the work unit distribution to be more homogenous. We can update the credit/model values. We can penalize people trying to cherry pick. It seems easy enough to me if it becomes a problem. just some ideas off the top of my head.
Well I hope you don\'t limit me for all the WU\'s I have been aborting recently in both Ralph and (mainly) Rosetta, due to WU\'s locking up and doing nothing (I have reported this a number of times but nothing has happened). Happens on Linux machines and needs rebooting to clear.
By the way if problems are occuring and WU\'s have to be aborted, as in my case due to WU lock ups, and Rosetta/Ralph admin limits what you can do with the WU\'s how can you keep working? And are we still going to get credit for errored work or units that fail for other reasons?
____________
|
|
|
|
|
Forgive me.. I know you\'re having a moment - but I\'m confused, and I need your help to understand things - You\'re leaving because we\'re trying to figure out how to prevent falsification of credit data?
I think Fuzzy is signing off from Ralph and Rosetta over David Bakers postings on the XS boards, and the disregard that this showed to all the Rosetta/Ralph participants.
____________
|
|
|
|
|
|
Hi,
I am not sure if this of value for the crediting system, but I have done a check of some 20 Rosetta WUs and have found out that the time to calculate one decoy is pretty exactly proportional to the number of amino acids in the protein to the power of 1,3.
My formula is (number of amino acids)^1.3*n(decoys) / time = const.(for a given machine)
It yields pretty good values that vary by an average of 2.3% around the median. I am still collecting numbers to compare, but the latest two samples I put in after adjusting the proportionality factor had 99,9 and 100,2 of the average \"work factor\" for my machine. And the length of proteins varies from 28 to 157 amino acids, which is a factor of nearly six in length.
If that also accounts for other machines it would be pretty easy to just calculate a number of given WUs of classified types several times to get a group of average results that can be applied to all other machines out there. Then all you would need to grant credits for any other WU is the number of amino acids in the protein and the method used for the calculation. You would have to repeat the calibration as soon as you change the method, but as long as the method is unchanged the credit is predictable and can thus be directly be derived from the number of decoys computed.
Of course, the new system seems to get into the swing pretty well. But maybe that observation might help to implement a system that relies less on precalculating credits on Ralph machines but on an empirical formula that can be derived from the number of amino acids in the template and the specific mathematical method used in the WU?
Regards,
Christoph
[edited for clarity] |
|
|
|
|
|
For anyone left that questions the need for some form of new credit system, I illustrait how easy it is to falsify the BOINC client credit claims. I think the ease with which it is achieved will be news to some.
One test for the new system would be for someone to publish similar step-by-step instructions on how to falsify results under the new system. At this point I believe it is still possible, but more difficult. And with a few more fairly minor changes (I\'ve suggested encryption of small amounts of the result data which will be required for your WU to be authenticated by the Rosetta server), this can be thwarted as well.
____________
|
|
|
|
|
I thought there were joining for science.
Blame Misfit!
____________
Join BOINC Synergy!
misfit@boincsynergy.com |
|
|
|
|
I thought there were joining for science.
Wish it were true more then it appears to be.
____________
|
|
|
|
|
I thought there were joining for science.
Wish it were true more then it appears to be.
Appearances can be deceiving.. My nephew handed me a blue plastic light saber, and picked up the red plastic light saber. I deflected his swings a few times, and then he pressed the red blade against the blue blade and I made a poor imitation of the crackling noise that is supposed to occur when the blades are in contact. The Irish Wolf hound in the house goes nuts and lunges at me for attacking my nephew, when in fact it\'s the little miscreant that\'s attacking and he\'s enjoying the activity.
Obviously not my favorite mutt since the attacks started; but he also freaks out and lets everyone know when diabetics in the house go into insulin shock. (His life has been saved due to his being an instant medical alert system..)
As for the credit system, here\'s hoping that the last few snags in the system get worked out, so we can get back to the fun of competitive crunching or reliable crunching - whichever of the two is the main justification for our person contributions here.
____________
|
|
|
|
|
|
argh.. timed out, and it still posted the first time. :)
____________
|
|
|
|
|
No, it\'s not.
I have my last Ralph WU in my cache, and when that\'s returned, I\'ll move my Ralph share to uFluids. So this is goodbye from me.
I don\'t expect a personal thank you for my contribution from David Baker, let alone a thank you from him on my teamboard.
Goodbye.
I left rosetta too. 28k credits a month I was giving to Rosetta.. now 0.
I tried vocally to argue against the credit system on the rosetta boards - BOY WAS THAT A MISTAKE.
The german mod deleted my account because I posted David\'s email and the # to wash U (both of which are on his webpage.) SORRY - DIDN\'T KNOW POSTING PUBLIC INFO YOU ARE GIVING FREELY AWAY ANYWAY WAS AGAINST YOUR RULES.
Don\'t expect me to reattach.
I have also removed my endorsement for this project from the International AIDS Conference. It\'s clear that your moderators have lost control of the project for you.
____________
|
|
|
|
|
I left rosetta too. 28k credits a month I was giving to Rosetta.. now 0.
I tried vocally to argue against the credit system on the rosetta boards - BOY WAS THAT A MISTAKE.
The german mod deleted my account because I posted David\'s email and the # to wash U (both of which are on his webpage.) SORRY - DIDN\'T KNOW POSTING PUBLIC INFO YOU ARE GIVING FREELY AWAY ANYWAY WAS AGAINST YOUR RULES.
Don\'t expect me to reattach.
I have also removed my endorsement for this project from the International AIDS Conference. It\'s clear that your moderators have lost control of the project for you.
I hope someone rectifies this soon, you should certainly not have an account deleted for posting public information. Yes hide the post by all means (if the mod was unsure and wanted to protect something until he could get hold of the Administrator/David Baker)
Bad, Bad ,Bad managment.
Especially as they\'re new mods, they certainly shouldn\'t be doing that!
____________
|
|
|
|
|
I hope someone rectifies this soon, you should certainly not have an account deleted for posting public information. Yes hide the post by all means (if the mod was unsure and wanted to protect something until he could get hold of the Administrator/David Baker)
Bad, Bad ,Bad managment.
Especially as they\'re new mods, they certainly shouldn\'t be doing that!
David Kim deleted the account of Aaron Finney which was absolutely necessary. He first deleted a few flaimbaits of him, than Aaron started to post the email and phone number of Dr. Baker together with a call to complain to David Baker by phone. This post was also removed and after he repeated it, his account was deleted. It was long after David Kim wanted to go to bed and in order to avoid more damage while he was at sleep the account was deleted. Had the ban feature be available over on Rosetta he might decided to just ban him, but the deletion was certainly justified.
Guys, please don\'t trust all the accusations which are thrown towards the project management at the moment.
____________
|
|
|
|
|
I hope someone rectifies this soon, you should certainly not have an account deleted for posting public information. Yes hide the post by all means (if the mod was unsure and wanted to protect something until he could get hold of the Administrator/David Baker)
Bad, Bad ,Bad managment.
Especially as they\'re new mods, they certainly shouldn\'t be doing that!
David Kim deleted the account of Aaron Finney which was absolutely necessary. He first deleted a few flaimbaits of him, than Aaron started to post the email and phone number of Dr. Baker together with a call to complain to David Baker by phone. This post was also removed and after he repeated it, his account was deleted. It was long after David Kim wanted to go to bed and in order to avoid more damage while he was at sleep the account was deleted. Had the ban feature be available over on Rosetta he might decided to just ban him, but the deletion was certainly justified.
Guys, please don\'t trust all the accusations which are thrown towards the project management at the moment.
Repeatedly posting is bad especially when you\'ve been told not to, it is rosetta board after all not ours. Though still I don\'t get the point of the ban, you can just sign back up.
I had also assumed they had implemented the ban feature (since they updated the boards (or was that only here).
____________
|
|
|
|
|
This post was also removed and after he repeated it, his account was deleted.
BULL*&%.
I did NOT repeat the post. NO warning, No NOTHING. If it posted the exact same message twice then it\'s -ONLY- because your server had a hiccup and double posted the damn thing. It is absolutely unconscionable that you would use DELETION for posting publicly available information you can grab from U Of W. Also - Maybe if your german moderator wasn\'t deleting EVERY SINGLE NEGATIVE MESSAGE about the credit system, It wouldn\'t have led to this.
Your moderator over there overstepped his bounds in the first place. We should be able to comment as we feel appropriate regarding systemwide credit changes. IT SEEMS FITTING THAT THE MOD WAS FROM GERMANY, DOESN\'T IT? The general idea of information control over there is appalling. It\'s a shame you\'ve let this fool ruin your \'public\' forum.
Forum is supposed to mean \'an assembly, meeting place, television program, etc., for the open discussion of questions of public interest.\'
NOT - We\'re going to silence anything that sounds negative so that a little less &$%% hits the wall after going through the fan.
Repeatedly posting is bad especially when you\'ve been told not to, it is rosetta board after all not ours. Though still I don\'t get the point of the ban, you can just sign back up.
I had also assumed they had implemented the ban feature (since they updated the boards (or was that only here).
Also.. Wasn\'t a ban. DELETION. Check my boincstats. You will see no Rosetta there. 90,000 for Rosetta, POOF, because I copied and pasted from www.washington.edu
F-ING BAD FORM FOR ROSETTA.
____________
|
|
|
|
|
|
Aaron may honestly believe that posting public information and calling on thousands of others to act on that information is within his every right... so let\'s encourage him to post his own phone number here. After all, it\'s public information that many who read these board would like to have!
____________
|
|
|
|
|
Aaron may honestly believe that posting public information and calling on thousands of others to act on that information is within his every right... so let\'s encourage him to post his own phone number here. After all, it\'s public information that many who read these board would like to have!
If I was running a project that included LINKS TO MY PHONE NUMBER, that project is not only entitled to that information, but is given express permission to use it.
Take your delusion somewhere else.
____________
|
|
|
|
|
|
When you over step the boundaries, you will have your account removed. I don\'t care if you have 1 million credits or 1 credit. Quite frankly I don\'t even know how many credits Aaron had and I don\'t care. I warned him and he reposted. There was no glitch in the system. If anyone posts personal information like a home phone number whether it is public information or not without permission and in bad intent we will delete your account. This type of stuff is way way way out of line particularly in the middle of the night when people are sleeping.
We have your account information backed up and may reinstate you if you show us that you will not be a burden to the project.
edit: If you have any issues with this, please feel free to email me.
____________
|
|
|
|
|
When you over step the boundaries, you will have your account removed. I don\'t care if you have 1 million credits or 1 credit. Quite frankly I don\'t even know how many credits Aaron had and I don\'t care. I warned him and he reposted. There was no glitch in the system. If anyone posts personal information like a home phone number whether it is public information or not without permission and in bad intent we will delete your account. This type of stuff is way way way out of line particularly in the middle of the night when people are sleeping.
We have your account information backed up and may reinstate you if you show us that you will not be a burden to the project.
edit: If you have any issues with this, please feel free to email me.
The \'Warning\' I received (an HOUR or so LATER) was in EMAIL. and if there was a double post (as I am assuming) then I hardly see how the few seconds (if that, probably have the same time stamp) between the two (surely identical) posts would be enough time to step away from the conversation, load up outlook express or whatever email client I use, log in, check my email, and then notice - \'HEY ! I got an email from project staff I better be a good little boy now. \' If I was just spamming the whole damn forum, I could understand, but so far, you claim only one repetition. In any case, I hardly call the use of E-Mail as a \'warning method\' appropriate for forum moderation.
That\'s akin to the National Weather Service sending out Tornado warnings via the US Post Office. And if you can\'t see the illogic in that argument, god help us all.
If I had received -ANY- response (rather than pure deletion of messages, threads, accounts) believe me, I would not have been posting.
We have your account information backed up and may reinstate you if you show us that you will not be a burden to the project.
Oddly.. I don\'t see how I can promise you that. Your moderators seem to think that -ANY- kind of negative post on the rosetta boards is \'burden\' enough to merit deletion. How am I supposed to contribute if I cannot share my opinions, both positive and negative?
I\'d like to tell you that I have no ill intentions towards this project, but I can\'t see how that would be very honest, as I don\'t think I\'m capable of blindly kissing someones butt day after day, no matter if I disagree or not. Apparrently those are the only threads/posts that don\'t get deleted around there.
If disagreeing and sharing my opinion about it on a forum is a \'burden\' to you, then apparrently I\'m incapable of promising you that, because that\'s not what a forum is supposed to be.
However, if you\'re asking me not to post DAB\'s number and email address (even though the University of Washington has it merely 4 or 5 clicks away from the page you are looking at right now) then fine.. I will restrain myself.
____________
|
|
|
|
|
|
Aaron,
I agree. It wasn\'t much warning. It was late and the post was so out of line that my only choice was to delete your account to prevent you from posting again. You are free to share your opinions on the forum but do not post personal information like home phone numbers without permission. We have been deleting a lot of posts due to the recent degradation of threads due to flaming and hostility towards others, etc. I\'m sorry you feel like you could not express your opinions.
____________
|
|
|
|
|
Aaron,
I agree. It wasn\'t much warning. It was late and the post was so out of line that my only choice was to delete your account to prevent you from posting again. You are free to share your opinions on the forum but do not post personal information like home phone numbers without permission. We have been deleting a lot of posts due to the recent degradation of threads due to flaming and hostility towards others, etc. I\'m sorry you feel like you could not express your opinions.
Error: No account found with email address #############@###.com
____________
|
|
|
|
|
|
I\'ll reinstate your R@h account tomorrow.
____________
|
|
|
|
|
I\'ll reinstate your R@h account tomorrow.
Error : No such user.
____________
|
|
|
|
|
I\'ll reinstate your R@h account tomorrow.
Error : No such user.
Error: No account found with email address #############@###.com
____________
|
|
|
