[Outdated] Rating correction and changes

tl;dr We’re looking to correct our rating and ranking system. On everyones profile you’ll find the estimated new rating and ranking for that user, it will be under their current rating and rank:

Please check out not only your own rating and ranking, but peer ratings and rankings (that is, players who you think you have a good feel for being equal, stronger, or weaker to that of your own rank), and let us know how things look.

Also, it’s not a good idea to adjust your rank now in an attempt to outsmart the upcoming adjustment. The adjustment system takes into account a lot of factors when determining a starting position, including any manual rank changes you’ve made, so the results are going to be notably less predictable than one might hope if manual changes are made in an attempt to land yourself at a particular spot after the adjustment.

If after the adjustment you find your rank is pretty far off, contact a mod, myself, or matburt, and we’ll get you squared away with something more appropriate. But please bring up any perceived problems now as opposed to just waiting until after the update, this way we can look into your case and see if there are any abnormal things biasing you one way or another.


The problem

The ranks at OGS have been severely depressed for a long time now. There are several causes for this, all of which play a factor, here are several that we’ve identified as being notable contributors to the problem, either directly or indirectly.

  1. The ELO/GoR ranking system is a zero sum system, this means if everyone starts off at 30k, the average rank will always be 30k. In both “the old” and “the new” OGS, while you could select and change your rank, it was very common for folks to just leave rank at 30k and “let the system figure it out”. With no substantial way to inject points into the system, this is the number one reason why our ranks are depressed.

  2. When people transfer in from other servers, if they pick a rank at all it seems more common than not to sandbag a little, as they don’t want to start off too hard. This problem is only compounded by the fact that OGS has a reputation for having depressed ranks.

  3. We use a ranking system starting at 30k. KGS and IGS use ranking systems from 20k 30k (but seems to bias heavily towards 20k+?) and 17k respectively, which makes transfers in from those servers harder to place correctly within our system, particularly at the lower kyu levels, which we believe increases the likelihood someone will either pick 30k as their starting point, or lowball their rank.

  4. There have been several instances where very active (and pretty good) players have, for one reason or another, timed out on dozens of games they had going on, which decimated their rank. Many of these players came back to the site and continued playing with their new ranking, which meant they would naturally drag down everyone else as a result. (While this isn’t a huge number of instances, the players that were drug down far enough for this to matter were prolific enough players that they did actually have an impact on the system as a whole.)


The solution

We are proposing the following adjustments and changes to attempt to find a happy medium for solving our ranking problem while still remaining very beginner friendly.

  1. We don’t want to take the approach of dropping everyone in the middle of the ranking system and let new players get slaughtered, so in order to solve the beginner start problem we will begin injecting points into the system by scaling back the K factor in the EGF GoR function for the losing player, dependent on their rating. In layman’s terms, beginners won’t lose as many points when they lose a game as they will gain when they win a game. Intuitively, this makes sense as beginners playing each other can increase in skill greatly even from a loss, so their ratings shouldn’t plummet from a single loss. We will also be capping the K factor at 122 for ranks less than 20k, as this seems to improve results in our case. Our proposed scaling looks like this:


    Reading the graph, it’s telling us that the number of points injected into the system will be substantial in the lower ranks, but demish fairly quickly as you approach the SDK’s and beyond.

  2. Players who timeout of a correspondence game will be flagged. Their ratings will be adjusted normally for the first timeout, however if they timeout of any more correspondence games while the flag is set we will no longer ding their rating, protecting them and the system from mass timeouts. Players with this flag set will not be able to join correspondence tournaments or ladders until they clear the flag by completing a ranked correspondence game over the course of at least two days with some kind player willing to give them another chance.

  3. We will be targeting a 20k+ rating system to be more compatible with KGS/IGS/EGF/AGA. Brand new players will begin at 25k, and with the help of the rating injection system should be pushed into the 20k+ “normal” pool at what we hope will be a fairly natural pace, and without dragging down the ranks because we will be adding in points as they go along to account for the fact that they are starting at 25k instead of the system average. We will also be restricting player’s ability to manually change their ranks below 20k after they’ve signed up, hopefully allowing us to maintain a safe zone for new players in the system, but naturally graduating them to the larger pool of 20k+ as they become ready.

  4. To boost the system up into the 20k base range we’ll be injecting a lot of points and recomputing everyones rating based on the ranked games they’ve played in the past that have not been correspondence timeouts.


The results

The following graphs show the distribution of players who have played more than 5 ranked, non correspondence timeout, games on OGS. We are starting with a heavily bias graph that looks like this:

After injecting some base points and recomputing players ratings based on the new rules (points injection, discounting corr. timeouts), the resulting distribution looks like this:

With an average rank increase looking like this:

Additionally, the ability to predict the outcome of a game based on relative skill increased from 69.47% to 71.91%. While this is not a significant change, our goal with this adjustment is to ensure that after the rating adjustment the ability for a player to find a well matched game based on rating should hopefully be better, but certainly no worse than what they can find presently. We believe this value tells us this will be the case more often than not.


Final notes

This is a big adjustment and will squish the ranks together a bit. There will likely be a bit of a correction period right after the adjustment, but we’re hoping to make this process as smooth as possible, and for it to result in a better system as soon as possible. Especially if you are in the bottom ranks, it may look like you are being propelled far beyond your skill, but so long as you end up near the same rank as your peers, all should be well. So please take a look at where things landed for you and your friends and let us know how it holds up, good or bad.

Also, please refrain from making suggestions to switch to alternative rating systems (glicko 1/2, aga, kgs, whr, etc…) in this thread.

21 Likes

This looks so interesting!

My current OGS rank is 24k, and probable new rank is 15k. This agrees with my current KGS rank. Incidentally, when I initially started on KGS, my rank started at 24k? and quickly moved through the ranks between that and 15k. so KGS does go below 20kyu, though there don’t seem to be many players in the 20’s there.

I took a look at the few players whose strength I have a feel for, and what I saw seems reasonable. There are a few players who always seem to beat me who are ranked lower than me. I don’t understand that, but the new numbers seem to be in the right ballpark.

1 Like

This post was flagged by the community and is temporarily hidden.

Yes, @anoek’s statement is wrong, but 20k and below (as you show in your case, @LinuxGooo) dissapear almost instantly.

I wonder if the ranks shouldn’t be skewed even stronger than is proposed. If you look at other ranking systems, they tend to have the most players per rank in the mid SDK range, with few players in the 20k range and then a gradual buildup to that point. Stronger than that, the ranks tend to taper off more quickly in the dan range. You can see a comparison of data from several rating systems here: http://senseis.xmp.net/?RatingHistogramComparisons

It seems odd that the user base for this site would skew so differently from the other systems.

skydyr@ Maybe the more accessible OGS site leads to having more beginners? After all, setting up KGS or IGS client is a bigger pain.

1 Like

Ah thanks for the rank correction about KGS starting at 30k, I updated the text. I was going off the senseis chart, and the fact that cgoban was raising an error for having 30k ranks when I tried to import one of our games awhile ago :slight_smile: (at least i thought it was 30k… maybe it was lower). Or maybe they start at 29k? Anyways, I’ve updated the text.

@skydyr I wondered about that too, but I also didn’t want to overcorrect, and it’s a bit hard to tell which 30k’s should be sdk’s and which 30ks should be 20ks when they’ve done nothing but play other “30ks” for the past long while :slight_smile: As it is we may have a few instances of over and under correction already, but my hope is that folks wont be too far off, and with this adjustment we’ll at least have broken up that 30k mass enough so that things can start to settle out into increasingly accurate ranks. Over time, if we believe our mass should have more SDKs than it does, my thought is we can increase the point injection spread a bit to push folks up to where things should be. Right now it keeps pushing people a non negligible amount up until about 10-9k, so we’ll see if that’s enough to form up a reliable and solid SDK range or not, but I’d rather low ball it a little until we get a better feel about how it affects things for a few months.

@saxmaam And that’s the other side of that coin too, I do believe we cater to beginners a lot more than kgs/igs, and we’d like to continue doing so :slight_smile:

2 Likes

I think OGS does attract more beginners than some other places. Also I think the graph shown above includes all players, but the histograms you’re talking about are only active players (last 6 months for KGS). So the OGS graph is showing the many people who played some and left. And that is going to include more beginners who played a few games and quit.

1 Like

Hands down, this update is a great update into the right direction of the OGS future. :blush:

And even if we have to wipe out some errors here and there, and maybe tune and fix some stuff, this is still great news. Thank you very much, anoek and matburt! :smiley:

2 Likes

Looks good to me. 9k is about where I was on KGS last time I played…

So … my profile says:

[quote]Rating 666
Rank 15k
Approx. new rating 1124
Approx. new ranking 10k[/quote]

How to handle this?

My last KGS rank was ~13k, my current EGF rank (based on two tournaments) is 13k … and IIRC here I was at 13k, too, until I lost a few correspondence games on time due to Real Life™ interference, and I thought, “well, bad for me, but time management is part of Go skills” …

Would it be best if I set my rank to 13k now?

·

< edit >

OK, I’ve changed it to 13k, and I dearly hope the system will not impose that 10k unto me :non-potable_water:

< /edit >

·

TIA, Tom

So I probably should have said this in the original post (and i’ll edit it to include it), but it’s probably best to not adjust your rank to try and outsmart the adjustment - the adjustment takes into account a lot of factors when determining a starting position, including any adjustments you’ve made, so the results are going to be a little less predictable than one might hope if you change things.

If after the adjustment you find your rank is pretty far off, contact a mod, myself, or matburt, and we’ll get you squared away with something more appropriate.

I’ll look into your case though and see if there is anything biasing you to be higher than you think you should be.

3 Likes

It says my new rank will be around 11k. That sounds right to me. The highest I ever got on KGS was 9k, and just barely. I’ve probably lost a few stones since then anyway. This is a great update!

I think that a 76% accuracy rate for predicting games from relative rank is pretty good, from what I understand of ranking systems. If you have the data, it would be interesting to see how that breaks down according to average rank of the players and by rank difference.

1 Like

@thouis FTR If we guess based off of rank the percentages change, we go from 71.73% up to 74.48%, the previous numbers were based off of rating. Might not be a 76% but at least it’s moving in the right direction. The hope is after this it’ll be easier for GoR to more easily take over and get those numbers looking even better.

1 Like

Maybe I missed it above, but when should we expect the rating corrections to go into effect? Very much looking forward to playing more often on OGS.

@cmkirkla we’ll do it soon… we wanted to give people a chance to see what we’ve done and take feedback on it.

2 Likes

Yep, it’s a function of how well things look this next week, but so long as there aren’t any major surprises we’re thinking of doing it not this weekend but the next (so maybe around July 27th or so)

2 Likes

I am delighted at this initiative to tackle a rather difficult problem. In my opinion, the server absolutely needs a reliable ranking system which roughly corresponds to other servers. When rank represents playing strength it becomes possible to play handicap games which is impossible currently. I always suspected that the system would not just “correct itself in time” and some kind of correction like this would be needed.

I applaud the devs heartily for taking on this challenge. It is the most important thing that can be done for players of live games. I haven’t played a ranked game or a handicap game for 6 months because the ranks were previously meaningless in representing playing strength. I have played unranked games against countless players with rank 20K+ who are in reality better players than myself. I look forward to seeing how this works out.

Yes I could easily have joined the herd and created myself a 25K account here. But I have played on 5 other servers and my rank was always about 5K so why should I settle for something different here ?

4 Likes

Congratburts and thanoeks to the devs, I think this is the number one problem identified by experienced players on the site, and it’ll make a huge difference.

The only concern I have is that our users will have nothing left to talk about in the chat rooms :laughing:

14 Likes