New crediting system

Message boards : Current tests : New crediting system

To post messages, you must log in.

Previous · 1 . . . 3 · 4 · 5 · 6 · 7 · 8 · 9 . . . 13 · Next

AuthorMessage
Profile dekim
Volunteer moderator
Project administrator
Project developer
Project scientist

Send message
Joined: 20 Jan 06
Posts: 250
Credit: 543,579
RAC: 0
Message 2048 - Posted: 15 Aug 2006, 17:47:09 UTC

dcdc:

We could calculate credits from February of this year using the new method but not from the start of the project because we have already cleaned up old result files. It wouldn't be worth the work required in my opinion. I like your idea of hand picking a representative set of hosts whose configurations we can trust but again, it would take a bit of work and organization. Good point about ralph and R@h app differences. If there is a significant change in the app that will skew the credit/model values, we can separate it as another application. Or we can always have a separate app for changes.

Hoelder1in:

Very observant. I did apply a simple correction factor and I'm glad you see it seems to be working. And, yes, we will have to update it now and then to account for changes in the host pool.

feet1st and Trog Dog:

We should first see what the response to the new method is and whether users would prefer it over the old. Not everyone will be happy about the final decision. I prefer abandoning the old system but that's just my opinion. Users that prefer the old system cannot use as an argument that they get more credits for whatever reason with the old system. The goal to even the playing field.

ID: 2048 · Report as offensive    Reply Quote
dcdc

Send message
Joined: 15 Aug 06
Posts: 27
Credit: 90,652
RAC: 0
Message 2049 - Posted: 15 Aug 2006, 21:23:12 UTC - in response to Message 2048.  

dcdc:

We could calculate credits from February of this year using the new method but not from the start of the project because we have already cleaned up old result files. It wouldn't be worth the work required in my opinion. I like your idea of hand picking a representative set of hosts whose configurations we can trust but again, it would take a bit of work and organization. Good point about ralph and R@h app differences. If there is a significant change in the app that will skew the credit/model values, we can separate it as another application. Or we can always have a separate app for changes.


I think the new credit system will be accepted much more willingly and quickly if the credits are backdated. If they could be calculated back to the start then that'd be best but if they can only go back to Feb then fair enough.

If it's a lot of work to calculate then I'm sure we could take some of the burden from you on that front if you're willing to send one of us the data.

Danny
ID: 2049 · Report as offensive    Reply Quote
MikeMarsUK

Send message
Joined: 8 Aug 06
Posts: 5
Credit: 0
RAC: 0
Message 2050 - Posted: 15 Aug 2006, 22:55:16 UTC

Could you do the credit calc from Feb, and then estimate the older credit based on the recent boinc credit vs. recent work credit ratio?

i.e., if the Feb-August boinc credit is 10% higher than the Feb-August work credit, then apply the same ratio to the pre-Feb boinc credit to get an estimated pre-Feb work credit?

The reason that I am trying to think of ways to get the old credit into the equation is so that the exported XML can reflect the new system ASAP. I also would like to move to the new system exclusively.
ID: 2050 · Report as offensive    Reply Quote
Profile dekim
Volunteer moderator
Project administrator
Project developer
Project scientist

Send message
Joined: 20 Jan 06
Posts: 250
Credit: 543,579
RAC: 0
Message 2051 - Posted: 15 Aug 2006, 23:35:22 UTC

I'm extracting the archived data and will see what I can do. What do other users think about trying to backdate the credits using the new work based system?
ID: 2051 · Report as offensive    Reply Quote
Profile Conan
Avatar

Send message
Joined: 16 Feb 06
Posts: 364
Credit: 1,368,421
RAC: 0
Message 2052 - Posted: 15 Aug 2006, 23:58:32 UTC

Totally against it. Changes are progressive, so what has been done is done, what is to be done will be done. The system is evolving as we work things out by trial and possibly error, therefore you have found some things aren't working as good as some people would wish, therefore you adapt the system and try that. What you have done in past is past, so lets move on and see how this new system goes, it may also not be the 'best' system and will have to be changed again, what will you do then backdate all the credits again? You will lose a lot of people if you do that.
There is no need to backdate.
ID: 2052 · Report as offensive    Reply Quote
Robert Everly

Send message
Joined: 16 Feb 06
Posts: 10
Credit: 2,333
RAC: 0
Message 2053 - Posted: 16 Aug 2006, 1:01:02 UTC - in response to Message 2051.  

I'm extracting the archived data and will see what I can do. What do other users think about trying to backdate the credits using the new work based system?


If nothing else, it would be interesting to see what changes would be made. Could a mock run be done and show us side by side results?

ID: 2053 · Report as offensive    Reply Quote
BennyRop

Send message
Joined: 11 Mar 06
Posts: 14
Credit: 674
RAC: 0
Message 2054 - Posted: 16 Aug 2006, 1:24:06 UTC

At Distributed Folding - we had the scoring system changed; and the scores were frozen. It was possible through 3rd party and the system itself to see overall scores (the final score from score system #1, and the current results of score system #2) and compare them to others. Personally, I thought it would have been fine to have continued on from the old scores; rather than starting out SS#2 with zero points. The system allowed either view to work..

As for rescoring the WUs we've done back to Feb or earlier - it won't matter to me; but we should get feedback from the highly competitive teams on the top of the charts. To perform the work, and then show what the top 50 team scores would end up being using the new scoring technique would allow each team to vote whether they were willing to go with the current fair crediting system, or the old way.
ID: 2054 · Report as offensive    Reply Quote
Profile Jack Shaftoe

Send message
Joined: 8 Aug 06
Posts: 13
Credit: 105,163
RAC: 0
Message 2055 - Posted: 16 Aug 2006, 2:36:00 UTC - in response to Message 2051.  
Last modified: 16 Aug 2006, 2:46:17 UTC

I'm extracting the archived data and will see what I can do. What do other users think about trying to backdate the credits using the new work based system?


I would be very happy to see this happen - since it's obvious that the new system is going to be a much more level playing field we should show as many credits using this as possible.

Back date to Feb, include everything older than that as they were reported. That gets my vote.
ID: 2055 · Report as offensive    Reply Quote
hugothehermit

Send message
Joined: 17 Feb 06
Posts: 17
Credit: 2,170
RAC: 0
Message 2056 - Posted: 16 Aug 2006, 2:38:28 UTC

I don't care about the credits, just the science, but it seems to me that :


Currently, I am just using the quotient of the claimed_credit and model totals for each work unit batch which get saved in a project specific table that we created. I wanted to avoid having to query the result table which would be necessary to get the median. I'd also have to add a project specific column to the result table which would hold the model count and I'm trying to avoid modifying the BOINC tables. I could use a correction factor if the descrepancies are significant enough. Can you point me to the results that you are talking about?


If I understand that correctly, then I think you will have some trouble.
Linux machines receive less credit than Win machines, so depending on the amount of Linux machines you have crunching, Win credit will go down from what it is now.

So I would suggest extracting only the Win results for your calcultaions, that way Lin and Mac will be truly given credit they deserve, and Win machines won't be underscored.

Somebody else has probably thought of this and maybe even posted, but I'm not going to read the entire thread :)

I'm extracting the archived data and will see what I can do. What do other users think about trying to backdate the credits using the new work based system?


You will get a lot of flak and bad publicity (among BOINC'ers) that you just don't need, let it go with "Here is the new credit system, it's as fair as we could make it, so enjoy".

ID: 2056 · Report as offensive    Reply Quote
kevint

Send message
Joined: 24 Feb 06
Posts: 8
Credit: 1,568,696
RAC: 0
Message 2057 - Posted: 16 Aug 2006, 4:34:28 UTC - in response to Message 2051.  

I'm extracting the archived data and will see what I can do. What do other users think about trying to backdate the credits using the new work based system?



Totally against backdating the changes. Not nessesary nor needed - Time to fix the problems and just go forward from here.
Not a wise move in my mind - you risk loosing many crunchers should you decide to do this.
ID: 2057 · Report as offensive    Reply Quote
Ethan

Send message
Joined: 11 Feb 06
Posts: 18
Credit: 25,579
RAC: 0
Message 2058 - Posted: 16 Aug 2006, 5:09:18 UTC - in response to Message 2057.  
Last modified: 16 Aug 2006, 5:16:23 UTC

Totally against backdating the changes. Not nessesary nor needed - Time to fix the problems and just go forward from here.
Not a wise move in my mind - you risk loosing many crunchers should you decide to do this.


Why does it matter if old credits are re-evaluated? I think everyone agrees that credits are a benchmark for how much work a given user/computer does for the project. Now that a much more accurate mechanism exists to grant credit, why is it a negaive to retroactively re-score the user base?

Should peple who intentially modified their credit 'wage' be granted a higher score than people who read a newspaper article and joined? I'm not for or against starting fresh or going back go Feb. . I think whatever results in the most computers crunching R@H is the correct outcome. . but proposing a scoring system based on user results when they are known to be fiddle-able, from users whose computes are hidden. . it just doesn't sound right.

-Ethan
ID: 2058 · Report as offensive    Reply Quote
Ethan

Send message
Joined: 11 Feb 06
Posts: 18
Credit: 25,579
RAC: 0
Message 2059 - Posted: 16 Aug 2006, 5:15:13 UTC - in response to Message 2058.  
Last modified: 16 Aug 2006, 5:17:47 UTC

delete me. . must have been a duplicate post
ID: 2059 · Report as offensive    Reply Quote
zombie67 [MM]
Avatar

Send message
Joined: 8 Aug 06
Posts: 75
Credit: 2,396,363
RAC: 6,299
Message 2060 - Posted: 16 Aug 2006, 5:17:35 UTC - in response to Message 2051.  

I'm extracting the archived data and will see what I can do. What do other users think about trying to backdate the credits using the new work based system?


Horrible idea.

People joined this project based on the scoring method (other reasons too, of course). Think about it this way. The laws (in the US) today say that if you invest in a 401(k), you do so with the expectation that your money goes in exempt from income tax, but you will have to live with the limited investment options your employer offers. Now, 20 years later, they change the rules and tell you that they've changed the rules, and 401(k)s are no longer valid. And in addition to having lost all that tax free advantage, you owe back taxes (money/credits lost).

And the real point here is that you have missed out on better potential profits had you just gone elsewhere to begin with.

At the time people joined, a credit system was promised. You cannot pull that away from people retroactively. It's deceitful. You want to chage it going forward? Fine. Change the rules, tell people, and let them make their own decision to stay or go.



Reno, NV
Team: SETI.USA
ID: 2060 · Report as offensive    Reply Quote
Ethan

Send message
Joined: 11 Feb 06
Posts: 18
Credit: 25,579
RAC: 0
Message 2061 - Posted: 16 Aug 2006, 5:26:15 UTC - in response to Message 2060.  
Last modified: 16 Aug 2006, 5:32:09 UTC

At the time people joined, a credit system was promised. You cannot pull that away from people retroactively. It's deceitful. You want to chage it going forward? Fine. Change the rules, tell people, and let them make their own decision to stay or go.


Ok. . If you owned a bank and users who used Firefox got 5 percent interest, but users who used Opera only got 3 percent. . . Users of Opera would find out they were being short changed and expect the same return on their investment. Why should Opera people be denied their credits from back in the day? The only reason I can think of is so people who 'enhanced' their credits can keep that advantage. Since credits are an important metric, wouldn't everyone want them to be as neutral as possible? How is it less neutral to start them back in Feb rather than Aug? Especially if the old credit count is still visible on the Rosetta webpage.
ID: 2061 · Report as offensive    Reply Quote
suguruhirahara

Send message
Joined: 5 Mar 06
Posts: 40
Credit: 11,320
RAC: 0
Message 2062 - Posted: 16 Aug 2006, 5:42:36 UTC - in response to Message 2058.  
Last modified: 16 Aug 2006, 5:43:48 UTC

People joined this project based on the scoring method

Probably some will be same as you, but keep in mind others crunch this project just for science development. For them it doesn't matter whether the way in which their work will be evaluated again is changed.
ID: 2062 · Report as offensive    Reply Quote
zombie67 [MM]
Avatar

Send message
Joined: 8 Aug 06
Posts: 75
Credit: 2,396,363
RAC: 6,299
Message 2063 - Posted: 16 Aug 2006, 5:51:50 UTC - in response to Message 2061.  

At the time people joined, a credit system was promised. You cannot pull that away from people retroactively. It's deceitful. You want to chage it going forward? Fine. Change the rules, tell people, and let them make their own decision to stay or go.


Ok. . If you owned a bank and users who used Firefox got 5 percent interest, but users who used Opera only got 3 percent. . . Users of Opera would find out they were being short changed and expect the same return on their investment. Why should Opera people be denied their credits from back in the day? The only reason I can think of is so people who 'enhanced' their credits can keep that advantage. Since credits are an important metric, wouldn't everyone want them to be as neutral as possible? How is it less neutral to start them back in Feb rather than Aug? Especially if the old credit count is still visible on the Rosetta webpage.


Disagree. Everyone joined the bank under the same rules. Some people chose different investment methods. Was it open and honest? Yes. Nothing was hidden. Nothing anyone did was against the rules. Everyone had the same investment opportunities. And more importantly, everyone had right to not join in the first place if they didn't like the rules.

You can't change the laws/rules retroactively....ever. You can't tell me it is okay to drive 55 on the road today, change the limit to 45 tomorrow, and then give me a ticket for what I did while it was still legal.





Reno, NV
Team: SETI.USA
ID: 2063 · Report as offensive    Reply Quote
zombie67 [MM]
Avatar

Send message
Joined: 8 Aug 06
Posts: 75
Credit: 2,396,363
RAC: 6,299
Message 2064 - Posted: 16 Aug 2006, 5:57:46 UTC - in response to Message 2062.  

People joined this project based on the scoring method

Probably some will be same as you, but keep in mind others crunch this project just for science development. For them it doesn't matter whether the way in which their work will be evaluated again is changed.


Obviously, which is why I said, "People joined this project based on the scoring method (other reasons too, of course)." In fact, that was the whole point of the parenthetical portion.

But we're explicitly talking about the credit portion of the motivation here, with this whole thread. That's the pertinent part.






Reno, NV
Team: SETI.USA
ID: 2064 · Report as offensive    Reply Quote
Profile [B^S] thierry@home
Avatar

Send message
Joined: 15 Feb 06
Posts: 20
Credit: 17,624
RAC: 0
Message 2065 - Posted: 16 Aug 2006, 6:04:31 UTC

I thought there were joining for science.

David,
Do whatever you want, as far credits are fair for everyone it's OK for me. There are +/- 76000 users registered at Rosetta. We only hear here a few who want keep their credits at any price. We never hear the 76000 minus a few.
ID: 2065 · Report as offensive    Reply Quote
Ethan

Send message
Joined: 11 Feb 06
Posts: 18
Credit: 25,579
RAC: 0
Message 2066 - Posted: 16 Aug 2006, 6:04:44 UTC - in response to Message 2064.  
Last modified: 16 Aug 2006, 6:23:06 UTC

Given Zombie67's point (which is valid in my comparison in terms of investments, thanks). . . I think it's fair to reduce the credits of those who 'convinced' the system into accpeting an unreasonable value to start with. 9 out of 10 identical worksations reported a similar wu time, should the 10th be given 5x the credits for no reason? How does it hurt anyone to go apples to apples back to Feb?
ID: 2066 · Report as offensive    Reply Quote
zombie67 [MM]
Avatar

Send message
Joined: 8 Aug 06
Posts: 75
Credit: 2,396,363
RAC: 6,299
Message 2067 - Posted: 16 Aug 2006, 6:22:46 UTC - in response to Message 2066.  

Given Zombie67's point (which is valid in my comparison in terms of investments, thanks). . . I think it's fare to reduce the credits of those who 'tricked' the system into accpeting an unreasonable value to start with. 9 out of 10 identical worksations reported a similar wu time, should the 10th be given 5x the credits just because the software was set different than the rest?


If not everyone is taking equal advantage of the rules, and the powers that be want it to be less "customizable" , then change the rules to prevent it going forward. That's what we're doing here, right? Perhaps we're late in implementing chanes to those rules. That's our own fault, not those who took advantage of it. Punishing people for breaking no rules is unfair at best.

Worse terms come to mind.

Reno, NV
Team: SETI.USA
ID: 2067 · Report as offensive    Reply Quote
Previous · 1 . . . 3 · 4 · 5 · 6 · 7 · 8 · 9 . . . 13 · Next

Message boards : Current tests : New crediting system



©2024 University of Washington
http://www.bakerlab.org